I’m investigating an issue in my application server logs, and I can see several entries related to abrupt connection interruptions before an Okta API response (an average of 10x/day). We’re using the Pekko HTTP library for performing API calls to Okta.
The stack trace follows the pattern below:
Okta request failed due to [The HTTP server closed the connection unexpectedly before delivering responses for 1 outstanding requests], retrying in [229 milliseconds]. Entity [None]
I’m looking for an explanation for this behavior and if it could be prevented from the client side. Any insights are appreciated. We’re currently implementing retries when we get such errors and they are usually effective.
Retries are your best bet, unless you’re ready to dig deep into the network layer.
It looks like there’s something on your server side / or http libary either not reusing pooled idle established connections or dropping connections after the SYN-ACK phase. Okta closes idle connections if ports don’t send packets for too long after the connection is established, as a preventative measure.
Your best bet is to test all egress points from your service to Okta and see if there’s any drop-off along the way during these issues.
Start by capturing sent network traces from your application for each egress point and investigate the idle time if you are able (I am not sure about pekko layer here - assuming it has some debug flags or higher idle timeout). Here’s a typical path to trace: Assumes you have any of these layers This path
If you confirm that the connection is idle and the library is suppose to use that and it is correctly reused but okta still closes it. Then it might be an okta server side issue with the handshake (Which rarely is never the case!!)
However, Okta does ever so often recommend retry mechanism for issues in network layer which fixes it without hitches.