Hello @asafga,
This is expected behavior, and it occurs because the same Group is used both for Group Push and Assignment, and the assignment takes precedence.
As a user is unassigned from the application, they are set to inactive in your application via the SCIM request and no subsequent calls are made for the user, as they are no longer a member, and will not be affected by any subsequent group operations.
As a best practice, or recommendation, try to separate Group Push and Assignment Groups, as much as possible to minimize if this is in any way impacting your application.