Gluu throws error on restart

By: Nilay Bhima user 11 Oct 2020 at 11:22 p.m. CDT

9 Responses
Nilay Bhima gravatar
Hi, Whenever I restart the Gluu server, I get a 503 on the UI and the following is what I get in the logs: ==> /opt/gluu-server/opt/gluu/jetty/oxauth/logs/http_request_response.log <== ==> /opt/gluu-server/opt/gluu/jetty/oxauth/logs/oxauth.log <== at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837) [jetty-xml-9.4.26.v20200117.jar:9.4.26.v20200117] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_222] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_222] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_222] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_222] at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) [start.jar:9.4.26.v20200117] at org.eclipse.jetty.start.Main.start(Main.java:491) [start.jar:9.4.26.v20200117] at org.eclipse.jetty.start.Main.main(Main.java:77) [start.jar:9.4.26.v20200117] 2020-10-12 04:15:39,637 ERROR [main] [org.gluu.oxauth.model.config.ConfigurationFactory] (ConfigurationFactory.java:372) - Failed to load configuration from file: /etc/gluu/conf/oxauth-config.json 2020-10-12 04:15:39,637 ERROR [main] [org.gluu.oxauth.model.config.ConfigurationFactory] (ConfigurationFactory.java:188) - Failed to load configuration from LDAP. Please fix it!!!. ==> /opt/gluu-server/opt/gluu/jetty/oxauth/logs/oxauth_audit.log <== ==> /opt/gluu-server/opt/gluu/jetty/oxauth/logs/oxauth_persistence.log <== Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_222] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_222] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_222] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_222] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_222] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_222] at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:666) ~[?:1.8.0_222] at com.unboundid.util.ssl.SetEnabledProtocolsAndCipherSuitesSocket.connect(SetEnabledProtocolsAndCipherSuitesSocket.java:131) ~[unboundid-ldapsdk-4.0.14.jar:4.0.14] at com.unboundid.ldap.sdk.ConnectThread.run(ConnectThread.java:148) ~[unboundid-ldapsdk-4.0.14.jar:4.0.14]
Ubuntu 18.04
closed

Answers

By Mohit Mali staff 12 Oct 2020 at 12:31 a.m. CDT

Copy
Mohit Mali gravatar
Hi Nilay Bhima, Thank you for reaching out gluu support , its look like its failed to find the config file , tell me more about the installation is it fresh install or you upgrade it. Thanks and Regards Mohit Mali

By Mohib Zico staff 12 Oct 2020 at 12:58 a.m. CDT

Copy
Mohib Zico gravatar
Please try with latest 4.2.1. When LDAP doesn't start upon reboot, Gluu Server tries to 'search' for configuration file to load it's config; however there isn't any configuration file available in file system. All are in LDAP. So, if LDAP doesn't start... your server won't load.

By Nilay Bhima user 12 Oct 2020 at 1:01 a.m. CDT

Copy
Nilay Bhima gravatar
Hi Mohib, Thanks for that. How do I get around this issue? I tried 4.2.1 but I couldnt get the gluu UI started so I went back to 4.1. Kind regards, Nilay.

By Nilay Bhima user 12 Oct 2020 at 1:03 a.m. CDT

Copy
Nilay Bhima gravatar
I tried upgrading it by changing the LDAP server to point to my own redis and changing the oxIDPAuthentication to point to a load balancer. I am following the instructions on here: https://gluu.org/docs/gluu-server/4.2/installation-guide/cluster/ This 503 error happens to me quite often though during a restart of the server. So I don't think it is to do with the changes in the LDAP server. Thanks, Nilay.

By Mohib Zico staff 12 Oct 2020 at 1:09 a.m. CDT

Copy
Mohib Zico gravatar
>> I am following the instructions on here: https://gluu.org/docs/gluu-server/4.2/installation-guide/cluster/ Actually I would have a better tip for you when doing manual clustering. For automated clustering, you should use [Cluster manager](https://www.gluu.org/docs/cm/4.2/) or use kubernetes / docker version. For manual clustering, I'll share a new method's doc. Re-assigning ticket to me.

By Nilay Bhima user 12 Oct 2020 at 1:14 a.m. CDT

Copy
Nilay Bhima gravatar
Ok, thanks Mohib. I will wait for you to send the new doc. In the meantime, do you think we can fix this issue? I have already built the cluster. Its just this minor issue. Kind regards, Nilay.

By Mohib Zico staff 12 Oct 2020 at 1:41 a.m. CDT

Copy
Mohib Zico gravatar
>> In the meantime, do you think we can fix this issue? I have already built the cluster. Its just this minor issue. it's not minor and beyond community support. There are lot of points which need to check.

By Michael Schwartz Account Admin 12 Oct 2020 at 4:17 p.m. CDT

Copy
Michael Schwartz gravatar
My thoughts are that it's probably a timing issue. Also, it could be a resource issue... for example make sure that the Shib IDP starts after identity so they are not competing for CPU.

By Vadim Saratovtsev user 13 Oct 2020 at 10 a.m. CDT

Copy
Vadim Saratovtsev gravatar
Hello, jumping in to share some of my recommendations for you to try. Are you using a chroot or -nochroot installation? Especially for -nochroot 4.2.1 installation, it is recommended that you try and leverage /sbin/gluu-serverd script to start the Gluu server components. Format is 'gluu-serverd start/stop/status/restart/version'. You can modify the script to add delays between components. If your LDAP footprint is large or you sycnronize lots of data on start-up, introducing a delay to OpenDJ start may be good. You would create a new system file for gluu-startup and copy contents of either opendj or oxauth system startup file. Start this new gluu-startup system file after syslog. Remember to systemctl disable oxauth/opendj/identity/idp/casa/passport ; so they no longer start automatically. Check your HTTPD startup up to make sure there are no conflicts. Please note that Cluster Manager is not a part of the gluu-serverd startup script. If you need to use the individual system files for startup, it is good to play around with 'after' positions for each service. OpenDJ must go first and it starts after the syslog. Not after the network please. Thsi way if something goes wrong there is a chance of some logs being captured. For some cases, especially high security environments with FIPS and SELinux enabled you may want to start HTTPD after OpenDJ. OXAuth must go after that. The rest depends on the machine, OS, hardware, VM foot print, etc. Below is a most consistent sequence we have seen, hope it works for you, though remember, with system files we did not introduce delays..You still may need to test. Our sequence: Opendj Httpd Oxauth IDP (Shibboleth) Oxd Identity Casa Having HTTPD start before Oxauth is a must for the FIPS and SELinux enabled environments because it needs to allow/open the network connection for HTTPD. In other environments, you could have OpenDJ followed by Oxauth. This is something to play with, check for conflicts. OpenDJ always goes first. HTTPD is a special case. OXAUTH second or third depending on the HTTPD case. Sequencing for the rest is up to your test or you can try our recommended approach first.

Post an answer