PIV authentication with API

I am working on a .Net Core Application with the requirement of using PIV to login. I followed the flow below.

  1. Application checks if user has session, does nothing if session does not exist.
  2. User clicks on button to login with PIV
  3. User is logged in with PIV
  4. Application checks if user has session, makes /authorize redirect because session exists.
  5. User winds up on the callback route with the authorization_code
  6. Make /token to get Access and Refresh Token

I am stuck at step 6. I tried to use the Code from /authorize to generate to call “/oauth2/default/v1/token” get access and refresh token without any luck. I’m getting 400 Status Code ( ‘Bad Request’) from response.
Any suggestion to get both access and refresh token?

Are you using the same authorization server when making the /authorize request as you do in the /token request?

Yes, we have only 1 default authorization server,

sorry, didn’t see your update! Do you get a specific error description when you see the 400 that will help us narrow this down?