De-provisioning Groups with SCIM not working as expected

Questions SCIM
1

Originally posted in the Okta Help Center here.

I’m currently implementing SCIM provisioning on a SAML app and having issues de-provisioning groups.

TLDR:

If the members are rapidly removed from a Pushed Group prior to the group being deleted, Okta is issuing a bunch of GET/PUT requests on the target group as the group members are deleted (as expected), but never actually issues the SCIM DELETE request for the group itself group (unexpected).

It also leaves behind an APP_GROUP that I can’t delete in the Okta UI.

We originally noticed this behaviour when managing groups with Terraform, but have since been able to replicate it via the Okta console by manually removing group members and deleting the group in quick succession.

We have also replicated this behaviour with a third-party SCIM server, so we’re reasonably sure something is wrong on the Okta side.

Detail:

  • The App is an Okta SAML app with SCIM provisioning enabled
  • Push New Users, Push Profile Updates & Push Groups are all enabled
  • Groups are created via terraform with the name group-nr-x , where x is an index number
  • Groups are added to the App’s PushGroups via a rule (i.e. anything starting with group-nr- )
  • Group members are added via terraform provider okta_group_memberships resources
  • None of the group-nr-x groups are used to assign the app to users

When I apply a terraform change that deletes a group I see:

  1. Terraform explicitly deleting each member from the group via the REST API
  2. Each deleted membership is reflected in the Okta log by a pair of events showing a successful group.user_membership.remove and subsequent successful application.provision.group_push.push_memberships targeting the appropriate apps
  3. A flurry of SCIM GET/PUT requests on the downstream group. The PUT requests remove members from the downstream list, but they never seem to get all the way to an empty list. All of the downstream reconciliation succeeds, and the the internal state of the downstream application matches the Group resource supplied by the PUT request.
  4. The Group resource returned by the downstream SCIM service to both the PUT and subsequent GET requests is identical to the PUT request body until the next PUT (Note: the member list ordering may be different, but the list content is the same).
  5. I don’t see any SCIM PATCH requests at all.
  6. Once Terraform has issued requests to delete all of the group members, it explicitly deletes the target Okta group
  7. The target group deletion is reflected in the Okta log with successful application.provision.group_push.delete_appgroup, group.lifecycle.delete and application.provision.group_push.updated events
  8. No SCIM DELETE request is ever issued on the downstream app
  9. Because no DELETE request is ever issued, the downstream application is left in an invalid state.

A non-editable group with the same name as the target group is also left behind in the Okta UI. I cannot edit or delete it through the UI and have had to use the REST API to delete it. This group’s ID appears nowhere that I can see in the Okta log as an event target, and shows up as an APP_GROUP when queried through the Okta Groups REST API. NOTE THAT The ID of this leftover group is not the same as the AppGroup mentioned in any of the log events

As stated before, when I delete a group via the Okta console or via the REST API (i.e. with a DELETE group, without any of the membership preamble), this all works as expected.

Does anyone have any idea about what (if anything) I’m doing wrong here? Or any workarounds we can use to avoid this behaviour?

Thanks & regards,

Trent