Cannot get .well-known discovery data back when using custom domain

I am having trouble getting the okta custom urls to work properly
The setup of the custom domain in the okta ui (with me setting dns as per the values okta ui shows, and leaving okta to generate the tls cert) works.
But after I have set it up, attempts to get the well-known metadata fail when referencing the custom domain (but still succeed with the original dev domain)

with the dev domain, these all succeed
e.g. https://dev-19504381.okta.com/.well-known/openid-configuration
https://dev-19504381.okta.com/oauth2/default/.well-known/openid-configuration
any other authorization servers I setup other than “default” also succeed
and for the oauth ones above the urls returned refer to the custom domain (e.g. https://okta.aamyidokta.com/oauth2/default/v1/keys)

But attempt to get any discovery data on the custom domain
https://okta.aamyidokta.com/.well-known/openid-configuration
fails with
{“errorCode”:“E0000006”,“errorSummary”:“You do not have permission to perform the requested action”,“errorLink”:“E0000006”,“errorId”:“oaetL4j09V-Qn-QhctRrBIs9A”,“errorCauses”:}
and
https://okta.aamyidokta.com/oauth2/default/.well-known/openid-configuration
or using oauth-authorization-server instead of openid-configuration or using any of my other authorization servers I setup instead of default
fails with
{“errorCode”:“E0000015”,“errorSummary”:“You do not have permission to access the feature you are requesting”,“errorLink”:“E0000015”,“errorId”:“oae824tbbhOSaCxiLEVo0fjnw”,“errorCauses”:}
Also calling the authorize endpoint on dev domain succeeds, but on custom domain fails
calling the keys endpoint on custom domain also fails with E0000015

The fact that the custom domain is reaching an okta error message suggests the dns is pointing to okta - hopefully the right okta server!
As per instructions, the CNAME of okta dot aamyidokta dot com points to dev-19504381 dot customdomains dot okta dot com
and when I use nslookup or dig on my custom domain I cannot spot any issue, it seems to bounce through a few hops
dev-19504381 dot customdomains dot okta dot com → ok12-custom-crtrs dot okta dot com → ok12-custom-crtrs dot aga dot okta dot com
and ends up with the error above.
When I setup the custom domain okta dot aamyidokta dot com, I let okta generate the tls cert, and I believe the right tls cert is being used (as when I get the errors, the name on the TLS cert is consistent with the domain, and the issue datetime is consistent with when I setup the custom domain)
(sorry about use of “dot” above - my post is being rejected due to too many URLs so trying to express it as text)

So the way I see it, the DNS routing of my custom domain is getting to okta and okta is mapping it to my account, but then (when going via the custom domain) is refusing to process the request (e.g. serve discovery data)
In Okta I have setup my apps and auth server to use the custom url, in fact even when I look at the authorization servers in the Okta UI, it gives me a url that points to the custom domain - but GETting that url fails with E0000015 (as above)
which then resolves to the E0000015 error as above
So I know I must be “trying” to use an appropriate discovery URL since it is a URL that okta UI has given me, which then fails with E0000015

I have removed and resetup the custom domain in okta (then repointed authservers at the custom domain) but that didnt change the above behaviour.

If I change the id in the url of the authorization server (eg .well-known where you would have default, or another authorization server ID (e.g. change the default or aus5gdvz5ghGbBGqc5d7 to aus5gdvz5ghGbBGqc5d7BADBAD) it also fails with E0000015, so this leads me to wonder if okta is not happy that my configured authorization servers are actually on that custom domain (although their Issuer is set to the custom domain, and if I use the metadata URI that the okta ui gives me, it still fails with E0000015)

So my problem is that while I can get Okta working with the default dev domain (and have relying parties use it to authenticate users), I cannot get it to work with the custom domain I have setup.
if I tell my app to use the custom domain to retrieve metadata it fails right away, and if I tell my app to use the dev domain to get discovery metadata, that call works but then subsequent calls (e.g. getting the jwks data) fail as those are referenced using the custom domain which results in E0000015
Unfortunately my use case requires a custom domain under the end-customers control.

I have scoured the internet and have as far as I can tell, gone through every setting I can find in the okta UI to no avail over the last couple of days.
I dont suppose anyone has hit this issue themselves and knows a fix, or if something jumps out something obviously wrong with what I am trying to or maybe the dns configuration above, or if there are relevant configurations in Okta that might be relevant here - because at this point I am going in circles.
Regards,

Can you log into your org at all via the custom domain? Because when I try to access it I keep getting redirected. Looks to me like there’s an issue with your DNS entries.

You mentioned a CNAME entry, but do you also have the TXT entry that’s part of the set-up? For reference, I use Digital Ocean for one of my orgs, and these are the entries I have

When I go to the custom domain to login, firstly (I dont know if this is relevant, or if is not normal it confirms the dns problem) it redirects from the custom domain to https://okta.okta.com/oauth2/v1/… but does show a signin screen, but I cannot login (error “unable to sign in” - I guess because it hasnt gone to my tenant so doesnt know my user).

For the DNS, yes I did setup a TXT (and I thought that must be right because that is verified as part of the setup/okta managed TLS certificates).
The TXT entry Okta gave me started with _acme-challenge , followed by my custom domain name - and I notice yours is _oktaverification… (and other parts of okta docs show the TXT as _oktaverification). I am not sure if that points to the problem, although I have a theory that the difference is that I chose Okta manager TLS certs and maybe you have supplied your own TLS cert in the example above. (or is there meant to be both an _acme-challenge AND a _oktaverification TXT entry - the Okta UI didnt give me _oktaverification entry as far as I observerd).
If I do:
nslookup -q=TXT _acme-challenge.okta.aamyidokta.com
then it shows me the _acme-challenge value (long b64 string) I was given by Okta (if the way I am using nslookup to confirm this is wrong, let me know), If I do
nslookup -q=TXT okta.aamyidokta.com
then it doesnt find a txt record
nslookup -q=CNAME okta.aamyidokta.com
gives me
okta.aamyidokta.com canonical name = dev-19504381.customdomains.okta.com

If I look on my domain hosting provider I can see for the subdomain the CNAME and TXT records (which are shown by the nslookup output above)

So my theories currently are

  • is it normal for signin to okta on your custom domain to be routed to okta (dot) okta (dot) com - I guess not, and that it should keep you on your custom domain on the sign in screen
  • does the problem relate to my DNS, don’t suppose you could see from the nslookup anything that jumps out?
    ** I had ruled out TXT previously on the reasoning that I thought this was checked during domain setup in okta/TLS cert issuance and since that worked assumed it was OK.
    ** But I am still not 100% certain why my TXT is _acme-challenge and yours and others is _oktaverification, and whether I might need both
  • This sounds silly, but I am half wondering if there are restrictions for the allowable subdomain names and somehow my subdomain called okta is confusing the routing inside okta.

(I did put some info, but when I edited the post to add more info it complained about too many links then removed my original post - I will try to summarise it again), then follow this the edit.
Yes - I had a TXT, which was _acme-challenge (dot) okta, as well as a CNAME
my theory is that with octa managed certs you get an _acme-challenge instead of an _oktaverification (that you show above)
nslookup -type=TXT _acme-challenge.okta.aamyidokta.com
retuned the b64 string of the data okta gave me
I could also use nslookup to see the cname
everything “looked” correct when I use nslookup and dig

My okta (dot) aamyidocta (dot) com domain was being redirected to okta (dot) okta - like you observed

Edit: so I had a theory about the subdomain being called okta causing some kind of routing issue inside okta, I created a different domain - signin.aamyidokta.com and went through the same process as before.

when I go to signin.aamyidokta.com it now shows the okta signin page on that domain (doesnt redirect to okta.okta like it did before), BUT I cant login, and any attempt to get well-known metadata on that domain results in 404 and returning the signin page html. Not sure what to make of this.)
dig and nslookup show what i believe to be expected results, for the TXT _acme-challenge and the CNAME (pointing at my dev-(number),customdomains.okta.com)

So some change of behaviour (in that the custom domain no longer redirects somewhere else), but still not working in that I cant logon when using the custom domain, and (wheras before attempting to retrieve well-known metadata resulted in a json errorcode), now retrieving .well-known metadata results in a 404 and the login page being returned

I hope (if my theory that a subdomain called okta was probematic is correct), that now a subdomain called signin is not also problematic.