Duplicate syslog events when using Okta SDK

Questions
1

I’m working on a small application to periodically pull the latest syslog events from Okta using the Java Okta SDK. The flow essentially goes something like this:

  1. Create com.okta.sdk.client.Client object and pass it into a new Runnable task
  2. Submit Runnable task to ScheduledExecutorService with 5 minute interval
  3. For each execution of task:
    1. Check to see if I have an existing sinceTime checkpoint.
      a. If I have a sinceTime checkpoint, use it to call Client.getLogs(null, sinceTime, null, null, null)
      b. If I don’t have a sinceTime checkpoint, use a default sinceTime to call Client.getLogs(null, defaultSinceTime, null, null, null)
    2. For each event, process and note the publishTime
    3. Store the last seen publishTime as my new sinceTime checkpoint

For each execution of the Runnable task after the first (essentially whenever I call Client.getLogs() using the publishTime of the last seen log event for the sinceTime parameter), that previously-seen event (sometimes multiple previously-seen events if they all had publishTime values in the same second) gets repeated in the new LogEventList.

Is there any way, using the Java Okta SDK, to avoid this repetition? Would I be able to enforce a strictly-greater-than-this-timestamp query by using the filter field instead of relying on sinceTime with its apparent greater-than-or-equal-to-this-timestamp behavior? Am I simply using the SDK incorrectly (should I be making a single Client.getLogs() call and then periodically attempting to get new events from the LogEventList stream?

Any insight would be appreciated. Thanks in advance!

2

I think your approach is valid, to the best of my knowledge it’s the only approach to get events based on their timestamp. Filter won’t be able to help you with timestamp (if I correctly understood your idea to use it for filtering based on time). Filter only supports subjects/objects/eventTypes Filtering Events

I also had some troubles with precise timestamp processing by Okta, where I’d present a sinceTime, but Okta would give me an event which happened before this time. Also events not being visible with log api immediately, only after a delay

3

Updating to share what I learned through experimentation after posting this question.

It appears the LogEventList object returned by the Okta SDK Client is intended to be a long-lived object. It implements the Iterator interface, but it is more than just a simple iterator. When you call hasNext(), if it doesn’t have anything cached it calls back to the Okta server to see if there are more. Thus, I’m updating my flow to be more like the following:

  1. Create com.okta.sdk.client.Client object and pass it into a new Runnable task
    a. In the Runnable constructor, if I have a sinceTime checkpoint, use it to call Client.getLogs(null, sinceTime, null, null, null)
    b. If I don’t have a checkpoint, call getLogs() with a default value
    c. Keep a reference to the LogEventList.iterator()
  2. Submit Runnable to a ScheduledExecutorService with a 5 minute interval
  3. For each execution of the task:
    a. Loop through available events using while ( logEventIterator.hasNext() )
    b. Process each event and note the publish time
    c. Store the last seen publish time as my new checkpoint (just in case I need to shut down and restart)

This method still results in me getting one or more duplicate records when I restart the process, but I no longer get duplicate records on every execution of my Runnable task. I have also found a way to dig into the Okta Java SDK and extract the after token that gets used on the REST API (and a way to use it to get a new Iterator), but I am a bit afraid of depending on that since it depends on the internal implementation of the SDK rather than on the “official” SDK APIs and I fear that a future update to the SDK could break my code.

Any chance the Okta devs would consider updating the SDK API to allow acces to (and use of) the after toke for checkpointing?