You you also make sure you are NOT trying to assign the user to Everyone group as part of your Google IdP config?
Everyone