I am using following callback URL provided by rocket chat
https://test.auth.com:553/_oauth/testauth
Then I've set same URL in OpenID Connect > Clients > Rocket.Chat > Redirect Login URI's
Redirect Login URI is : https://test.auth.com:553/_oauth/testauth
I've reffered https://www.gluu.org/docs/gluu-server/4.1/integration/saas/rocketchat/
this documentation.
And in our case the gluu server and Rocket chat both are installed on same cloud server.
Please let me know if you need some more information.