Best practice for exporting system logs to long term storage

api

#1

Hi guys,

My company use OKTA for a range of applications, and for compliance reasons we have been tasked to backup the system logs on a daily basis. We have chosen to use the new System Log beta REST API to retrieve the logs, however we seem to have run into an issue whereby there appears to be no effective way to do this with the current API.

The system log documentation states:

You can export your log events to a separate system for analysis or compliance. To obtain the entire dataset, query from the appropriate point of time in the past.

And provides the following example:

“https://{yourOktaDomain}.com/api/v1/logs?since=2017-10-01T00:00:00.000Z”

Simple enough, except that that this generates an infinite number of rel=“next” pagnation links as specified in the design principles documentation:

When you first make an API call and get a cursor-paged list of objects, the end of the list will be the point at which you do not receive another next link value with the response. This holds true for all but two cases:

Events API: The next link always exists, since the Events API is like a stream of data with a cursor.

System Log API: The next link will always exist in polling queries in the System Log API. A polling query is defined as an ASCENDING query with an empty or absent until parameter. Like in the Events API, the polling query is a stream of data.

So to make a non polling query to get around this, we would have to use since & until parameters to split up the data in to manageable chunks in order for us to not hit the 60 API calls per minute constraint. No problem, except this can return duplicates or miss events entirely according to the System Log documentation:

Do not attempt to transfer data by manually paginating using since and until as this may lead to skipped or duplicated events. Instead, always follow the next links.

Our last thought was to manually craft queries using the after keyword, to make sure we don’t run into any issues with duplicates, but again this is not recommended in the documentation:

The after parameter is system generated for use in “next” links. Users should not attempt to craft requests using this value and rely on the system generated links instead.

With all of these opposing constraints, is collecting an unbroken chain of system logs by querying the System Log REST API on a daily basis currently achievable?

Thank you for any help in this matter!


#2

This sounds like a polling use case to me, and the next link should work. Daily, you will use the next link from the previous day until you get an empty array, and pause until you are ready to go again.

Maybe I’m missing something here. It doesn’t sound like you need the until parameter, so this should work.


#3

I’m using the Node lib to pull in logs for archiving, and it seemed to be working for a while, but have been getting HTTP 500 errors since last night. I noticed that all of the successful requests have been for dates within the last 6 months. Is there a problem with fetching data more than 6 months old?


#4

In regards to data retention:
https://developer.okta.com/docs/api/resources/system_log#data-retention

If you are trying to get getting that is outside of your retention window, it shouldn’t be 500’ing. Are you using an after / next link? Can you show me the request?


#5

Like I said, I’m using the Node API, which I presume to be doing everything correctly. It also doesn’t log anything, but here’s the exact error message:

Okta HTTP 500 E0000009 Sorry, there’s been an error. We aren’t sure what caused it, but we’ve logged this and will work to address it. Please try your request again.


#6

@jelder Can you post the Node code that’s failing? Also, if possible, using a tool like Fiddler or Charles to get some HTTP traces would help us track down the problem.


#7

What is the retention window officially? I’ve scanned the docs and can’t find anything explicitly defining this.

Charles won’t work since its HTTPS. I presume Okta uses an error tracker like Rollbar, Datadog, Honeybadger, New Relic, etc? The error message implies that debug data was collected on the server side.


#8

The reason that I didn’t want to put the actual retention on the message board is it might change over time.

https://support.okta.com/help/Documentation/Knowledge_Article/Okta-Data-Retention-Policy

Please reach out the developers@okta.com. People on this board don’t have access to individual org’s system log or our internal error tracking system.