By: Patrick Ethier user 07 Feb 2017 at 11:28 a.m. CST

10 Responses
Patrick Ethier gravatar
The Ubuntu "init" script gluu-server-3.0.0 doesn't seem to survive a start/stop/start cycle. Doing so, off the default install, results in a "Service Unavailable 503" page when recovering from a reboot cycle (or a start/stop/start cycle). It seems this is due to the openldap (solserver?) service not being fully "up" when the oxauth and identity jetty instances are starting.... To reproduce follow this process: 1. Follow install guide for stock Ubuntu VM (limits, etc.), make sure /etc/hosts entry contains IP/hostname to rule out DNS issues 2. Follow setup guide for .deb file via apt-get (see looks like there's some repository issue so I ended up downloading .deb from the repo and running dpkg -i instead) 3. sudo service gluu-server-3.0.0 start 4. sudo service gluu-server-3.0.0 login 5. cd /install/community-edition-setup/ 6. run setup.py 7. Log into https://<yourhost> with the admin creds supplied by setup.py to confirm everything is OK 8. At this point, exit out of the chroot/jail and issue a "sudo reboot" command... OR "sudo service gluu-server-3.0.0 stop" 9. Upon reboot, issue "sudo service gluu-server-3.0.0 start" 10. Browse to https://<yourhost>, 503 error ensues every time! ------ Temporary workaround ----- 11. sudo gluu-server-3.0.0 login 12. cd /etc/init.d 13. ./identity stop 14. ./oxauth stop 15. ./solserver stop 16. ./solserver start 17. ./oxauth start 18. ./identity start 19. Log onto https://<yourhost>, login prompt should have returned...

By Adrian Alves staff 07 Feb 2017 at 1:44 p.m. CST

Adrian Alves gravatar
Hello Patrick, Can you post: 1- repositories issue 2- the output error of /etc/init.d/gluu-server-3.0.0 start Thanks, Adrian.-

By Patrick Ethier user 07 Feb 2017 at 1:53 p.m. CST

Patrick Ethier gravatar
There is no wrong output for the gluu-server-3.0.0 start command, it executes normally, it just doesn't start the oxauth and identity service properly. It takes 5-10 minutes to set up the situation above, I've reproduced it with 4 different VMs. I'm looking at the github code, I'm trying to modify the init.d scripts to include the LSB definitions to see if it fixes the race conditions. If it does, I'll submit a pull request.

By Patrick Ethier user 07 Feb 2017 at 1:56 p.m. CST

Patrick Ethier gravatar
As for the output of apt-get ``` user@login:~$ sudo apt-get install gluu-server-3.0.0 Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be upgraded: gluu-server-3.0.0 1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 559 MB of archives. After this operation, 509 kB disk space will be freed. Get:1 https://repo.gluu.org/ubuntu xenial/main amd64 gluu-server-3.0.0 amd64 1-1~xenial+Ub16.04 [559 MB] Err:1 https://repo.gluu.org/ubuntu xenial/main amd64 gluu-server-3.0.0 amd64 1-1~xenial+Ub16.04 Writing more data than expected (558989312 > 558974742) E: Failed to fetch https://repo.gluu.org/ubuntu/pool/main/xenial/gluu-server-3.0.0_1-1~xenial+Ub16.04_amd64.deb Writing more data than expected (558989312 > 558974742) E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? ```

By Adrian Alves staff 07 Feb 2017 at 3:27 p.m. CST

Adrian Alves gravatar
can you update the metadata of your repo and try again, apt-get clean; apt-get update; apt-get install gluu-server-3.0.0

By Patrick Ethier user 07 Feb 2017 at 4:52 p.m. CST

Patrick Ethier gravatar
apt-get clean & apt-get update seems to have fixed the corrupt repository entry. Performing apt-get upgrade seems to have updated the package I installed from dpkg -i prior. (from version 1-1 to 1-2) I then did the following: ``` sudo service gluu-server-3.0.0 start sudo service gluu-server-3.0.0 login /install/community-edition-setup/setup.py -f /install/community-edition-setup/setup.properties.last exit sudo service gluu-server-3.0.0 stop sudo service gluu-server-3.0.0 start ``` This now gets me back to the 503 error (the identity service isn't running correctly) So I did the following: ``` sudo service gluu-server-3.0.0 login /etc/init.d/identity stop /etc/init.d/identity start ``` Now it's restored...

By Patrick Ethier user 07 Feb 2017 at 4:53 p.m. CST

Patrick Ethier gravatar
Note, stopping and restarting the gluu-server-service still poses the same problem. I need to manually start/stop the services as in the previous posts for things to start working again.

By Dimitri Gavrilov user 09 Feb 2017 at 4:36 a.m. CST

Dimitri Gavrilov gravatar
Hello Patrick, could you please try this fix: ``` cp /opt/jetty/bin/jetty.sh /opt/jetty/bin/jetty_id.sh sed -i 's/# Provides: jetty/# Provides: identity/' /opt/jetty/bin/jetty_id.sh cd /etc/init.d/ ln -sf /opt/jetty/bin/jetty_id.sh identity insserv identity ``` You can also add the dependency to the openldap if you add "slapd" to the "Required-Start" line before running insserv command.

By Patrick Ethier user 09 Feb 2017 at 7:14 a.m. CST

Patrick Ethier gravatar
insserv doesn't seem to be an available command under the jail/chroot. But with all the other steps (now have two different jetty service scripts) doesn't seem to have changed anything. from outside the jail/chroot: sudo service gluu-server-3.0.0 restart I still need to manually log into the chroot/jail and stop/start all three services.

By Dimitri Gavrilov user 09 Feb 2017 at 7:46 a.m. CST

Dimitri Gavrilov gravatar
Ubuntu is a debian based distro, so it should be very similar. insserv binary is provided by the insserv package, but maybe it is not installed / not supported by your Ubuntu version. So in this case maybe the following command will help: update-rc.d identity enable You can check if there is a identity startup in the /etc/rc3.d. I assume you have one for oxauth, but no for identity. The numbers in the script names are referring to the startup order. So you should have something like this to start oxauth and identity services after the openldap: ``` S03solserver -> ../init.d/solserver S04identity -> ../init.d/identity S04oxauth -> ../init.d/oxauth ```

By Patrick Ethier user 09 Feb 2017 at 8:26 a.m. CST

Patrick Ethier gravatar
note, breaking out the jetty.sh links for asimba and idp are also necessary... ``` cp /opt/jetty/bin/jetty.sh /opt/jetty/bin/jetty-asimba.sh cp /opt/jetty/bin/jetty.sh /opt/jetty/bin/jetty-idp.sh cp /opt/jetty/bin/jetty.sh /opt/jetty/bin/jetty-identity.sh update-rc.d identity disable update-rc.d oxauth disable update-rc.d idp disable update-rc.d asimba disable update-rc.d solserver disable unlink /etc/init.d/identity unlink /etc/init.d/idp unlink /etc/init.d/asimba ln -sf /opt/jetty/bin/jetty-identity.sh /etc/init.d/identity ln -sf /opt/jetty/bin/jetty-asimba.sh /etc/init.d/asimba ln -sf /opt/jetty/bin/jetty-idp.sh /etc/init.d/idp sed -i 's/# Provides: jetty/# Provides: identity/' /opt/jetty/bin/jetty_identity.sh sed -i 's/# Provides: jetty/# Provides: idp/' /opt/jetty/bin/jetty_idp.sh sed -i 's/# Provides: jetty/# Provides: asimba/' /opt/jetty/bin/jetty_asimba.sh sed -i 's/# Provides: jetty/# Provides: oxauth/' /opt/jetty/bin/jetty.sh ``` Then you need to add the following lines in the LSB section for each file as stated: ``` # /opt/jetty/binjetty.sh # LSB Tags ### BEGIN INIT INFO # Provides: oxauth # Required-Start: $local_fs $network # Required-Stop: $local_fs $network # X-Start-Before: identity # /opt/jetty/bin/jetty-identity.sh # LSB Tags ### BEGIN INIT INFO # Provides: oxauth # Required-Start: $local_fs $network # Required-Stop: $local_fs $network # X-Start-Before: idp asimba # /etc/init.d/solserver ### BEGIN INIT INFO # Provides: slapd # Required-Start: $network $syslog # X-Start-Before: oxauth ``` Then you issue the following commands: ``` update-rc.d solserver enable update-rc.d oxauth enable update-rc.d identity enable update-rc.d idp enable update-rc.d asimba enable ``` This seems to untangle the whole mess and it works now by starting/stopping the gluu-server-3.0.0 service outside the jail.