I believe this is an either/or. The client can be configured to use AuthorizationMode.PrivateKey (in which case, the SDK will fetch the access token for you) OR AuthorizationMode.BearerToken (where you provide the raw Access Token JWT yourself).
If when you use BearerToken mode, nothing is returned, you likely want to check if that token was issued with the appropriate scopes AND ensure that it was issued to a Service app with appropriate permissions OR an admin user with appropriate permissions to read users.