Just to verify, you do have deactivate users enabled for this SCIM integration in your SAML app?
deactivate users