The app embed link of an Application in the Okta Admin Console takes you to the login pane of the okta-signin-widget. Is it possible to modify the app embed link, or the /sso/saml URL it redirects you to, perhaps by adding a query string parameter, to take you straight to the signup pane instead?
I’m at least not aware of any way to send someone directly to the signup page. They should instead hit the login page and, if configured in your org, the Sign Up link should appear and they can use that to register.
The only thing I can think of to automate that a bit more is to add custom code to your Okta hosted login page (only available for org’s with a custom domain!) to simulate a mouse click on that link when users access application Y, but I imagine that could lead to poor experience for users who already have an account