503 Service Unavailable

By: Vitalii Bendyna user 14 Jun 2017 at 10:59 a.m. CDT

11 Responses
Vitalii Bendyna gravatar
Hello! I'm receiving 503 error after service restart. What I have: ``` 1) Fresh AWS instance with Ubuntu 16.04 (t2.large: 2 cores, 8Gb RAM) 2) It has internal (10.10.10.100) and external IP addresses 3) It has DNS domain name (like gluu.somedomain.com) ``` How to reproduce: 1) Run this command: ``` apt-get update && apt-get upgrade -y ``` 2) add in the and of the file /etc/security/limits.conf: ``` soft nofile 65536 hard nofile 262144 ``` 3) Add the following line to /etc/pam.d/login: ``` session required pam_limits.so ``` 4) Value in /proc/sys/fs/file-max is greater then 65535 5) Run this commands: ``` echo "deb https://repo.gluu.org/ubuntu/ xenial main" > /etc/apt/sources.list.d/gluu-repo.list curl https://repo.gluu.org/ubuntu/gluu-apt.key | apt-key add - apt-get update apt-get install gluu-server-3.0.1 service gluu-server-3.0.1 start service gluu-server-3.0.1 login cd /install/community-edition-setup ./setup.py ``` 6) During installation: ``` Enter IP Address [10.10.10.100] : Enter hostname [localhost] : gluu.somedomain.com Enter your city or locality : Zurich Enter your state or province two letter code : ZH Enter two letter Country Code : CH Enter Organization Name : Gluu Enter email address for support at your organization : a@b.cc Enter maximum RAM for applications in MB [3072] : 6144 Optional: enter password for oxTrust and LDAP superuser [hlE3vzf0hMdD] : <only letters and digits, 10 symbols> Install oxAuth OAuth2 Authorization Server? [Yes] : Install oxTrust Admin UI? [Yes] : Install LDAP Server? [Yes] : Install Apache HTTPD Server [Yes] : Install Shibboleth SAML IDP? [No] : Yes Install Asimba SAML Proxy? [No] : Yes Install oxAuth RP? [No] : Yes Install Passport? [No] : Install JCE 1.8? [Yes] : You must accept the Oracle Binary Code License Agreement for the Java SE Platform Products to download this software. Accept License Agreement? [Yes] : ``` 7) Follow https://gluu.somedomain.com in the browser - all works 8) Restart service: ``` service gluu-server-3.0.1 stop service gluu-server-3.0.1 start ``` 9) In some while try to follow https://gluu.somedomain.com. You'll receive redirect to https://gluu.somedomain.com/identity/ and then 503 Service Unavailable This steps has been performed on 3 different instances (but on the same AMI: ami-a8d2d7ce - standard Ubuntu 16 image on AWS): t2.medium, t2.large and m4.xlarge Can someone help me?
OS Version : N/A
closed

Answers

By Mohib Zico staff 15 Jun 2017 at 8:42 a.m. CDT

Copy
Mohib Zico gravatar
>> 9) In some while try to follow https://gluu.somedomain.com. You'll receive redirect to https://gluu.somedomain.com/identity/ and then 503 Service Unavailable Can you please check if all services are running properly? Services means... oxauth, identity, httpd, idp and solserver.

By Vitalii Bendyna user 15 Jun 2017 at 8:50 a.m. CDT

Copy
Vitalii Bendyna gravatar
Definitely they're not working. In start service script missing commands that start them. After "service gluu.... start" I have to write this command: ``` for i in identity idp oxauth oxauth-rp; do service $i restart; done ``` Only in this case all will work.

By Mohib Zico staff 15 Jun 2017 at 8:52 a.m. CDT

Copy
Mohib Zico gravatar
My apologies for your trouble. This has fix in upcoming 3.0.2 release.

By Vitalii Bendyna user 15 Jun 2017 at 9:01 a.m. CDT

Copy
Vitalii Bendyna gravatar
Could I ask you when this release is going to be published? Your application is quite good, but now it's really buggy - on Ubuntu 16.04 somehow RAM memory that app using getting out of the limits, to restart application (service) we have to execute additional commands (for research we spent more than 5 hours), sometimes after installation on new clean system it even doesn't want to start... According to all this we're searching for a similar solution, but more stable. That's why I'm asking about new release.

By Mohib Zico staff 15 Jun 2017 at 9:05 a.m. CDT

Copy
Mohib Zico gravatar
>> Could I ask you when this release is going to be published? Next week, it's already in final QA. If you want I can share the link of that deb package with you. >> on Ubuntu 16.04 somehow RAM memory that app using getting out of the limits, Have you tried modifying memory allocation which we included in another of your ticket?

By Vitalii Bendyna user 15 Jun 2017 at 9:08 a.m. CDT

Copy
Vitalii Bendyna gravatar
> Have you tried modifying memory allocation which we included in another of your ticket? Yes, I tried to set limits, but that didn't help. > Next week, it's already in final QA. If you want I can share the link of that deb package with you. It would be really great if you'll share with me the link to new release - I would like to test it. Could you send me the link to my email address?

By Mohib Zico staff 15 Jun 2017 at 9:10 a.m. CDT

Copy
Mohib Zico gravatar
>> Yes, I tried to set limits, but that didn't help. I think I am gonna fire up a Amazon VM with your specifications and see what's up there. >> It would be really great if you'll share with me the link to new release - I would like to test it. Could you send me the link to my email address? [Here](https://repo.gluu.org/ubuntu/pool/main/xenial-devel/gluu-server-3.0.2_1-1~xenial+Ub16.04_amd64.deb) it is.

By Vitalii Bendyna user 15 Jun 2017 at 9:32 a.m. CDT

Copy
Vitalii Bendyna gravatar
Thank you. I'm testing new application right now

By Vitalii Bendyna user 15 Jun 2017 at 10:39 a.m. CDT

Copy
Vitalii Bendyna gravatar
You know, I'm not able to set up this package on Ubuntu 16. Look: ``` root@localhost:~# service gluu-server-3.0.2 status ● gluu-server-3.0.2.service - LSB: This shell script takes care of starting and stopping Loaded: loaded (/etc/init.d/gluu-server-3.0.2; bad; vendor preset: enabled) Active: active (running) since Thu 2017-06-15 14:55:46 UTC; 7min ago Docs: man:systemd-sysv-generator(8) Process: 1527 ExecStart=/etc/init.d/gluu-server-3.0.2 start (code=exited, status=0/SUCCESS) Tasks: 67 Memory: 8.7M CPU: 437ms CGroup: /system.slice/gluu-server-3.0.2.service ├─1582 /usr/sbin/rsyslogd ├─1601 /usr/sbin/apache2 -k start ├─1604 /usr/sbin/apache2 -k start ├─1605 /usr/sbin/apache2 -k start ├─1666 /usr/sbin/cron ├─1676 /usr/bin/dbus-daemon --system └─1682 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 Jun 15 14:55:42 gluu systemd[1]: Starting LSB: This shell script takes care of starting and stopping... Jun 15 14:55:42 gluu gluu-server-3.0.2[1527]: Starting Gluu server, please wait... Jun 15 14:55:46 localhost systemd[1]: Started LSB: This shell script takes care of starting and stopping. ``` ``` root@localhost:~# /opt/gluu-server-3.0.2/etc/init.d/identity -bash: /opt/gluu-server-3.0.2/etc/init.d/identity: No such file or directory ``` ``` root@localhost:~# ls -la /opt/gluu-server-3.0.2/etc/init.d/ total 180 drwxr-xr-x 2 root root 4096 Jun 15 14:55 . drwxr-xr-x 63 root root 4096 Jun 15 14:57 .. -rwxr-xr-x 1 root root 8087 Apr 5 2016 apache2 -rwxr-xr-x 1 root root 2210 Apr 5 2016 apache-htcacheclean -rwxr-xr-x 1 root root 1275 Jan 19 2016 bootmisc.sh -rwxr-xr-x 1 root root 3807 Jan 19 2016 checkfs.sh -rwxr-xr-x 1 root root 1098 Jan 19 2016 checkroot-bootclean.sh -rwxr-xr-x 1 root root 9353 Jan 19 2016 checkroot.sh -rwxr-xr-x 1 root root 3049 Apr 5 2016 cron -rwxr-xr-x 1 root root 2813 Dec 2 2015 dbus -rw-r--r-- 1 root root 663 Feb 3 17:31 .depend.boot -rw-r--r-- 1 root root 310 Feb 3 17:31 .depend.start -rw-r--r-- 1 root root 459 Feb 3 17:31 .depend.stop -rwxr-xr-x 1 root root 1336 Jan 19 2016 halt -rwxr-xr-x 1 root root 1423 Jan 19 2016 hostname.sh -rwxr-xr-x 1 root root 3809 Mar 12 2016 hwclock.sh -rwxr-xr-x 1 root root 1300 Jan 19 2016 killprocs -rwxr-xr-x 1 root root 3466 Mar 15 2016 memcached -rwxr-xr-x 1 root root 703 Jan 19 2016 mountall-bootclean.sh -rwxr-xr-x 1 root root 2301 Jan 19 2016 mountall.sh -rwxr-xr-x 1 root root 1461 Jan 19 2016 mountdevsubfs.sh -rwxr-xr-x 1 root root 1564 Jan 19 2016 mountkernfs.sh -rwxr-xr-x 1 root root 711 Jan 19 2016 mountnfs-bootclean.sh -rwxr-xr-x 1 root root 2456 Jan 19 2016 mountnfs.sh -rwxr-xr-x 1 root root 1581 Oct 16 2015 ondemand -rwxr-xr-x 1 root root 1192 Sep 6 2015 procps -rwxr-xr-x 1 root root 6366 Jan 19 2016 rc -rwxr-xr-x 1 root root 820 Jan 19 2016 rc.local -rwxr-xr-x 1 root root 117 Jan 19 2016 rcS -rw-r--r-- 1 root root 2427 Jan 19 2016 README -rwxr-xr-x 1 root root 661 Jan 19 2016 reboot -rwxr-xr-x 1 root root 4355 Jul 10 2014 rsync -rwxr-xr-x 1 root root 2796 Feb 3 2016 rsyslog -rwxr-xr-x 1 root root 3927 Jan 19 2016 sendsigs -rwxr-xr-x 1 root root 597 Jan 19 2016 single -rw-r--r-- 1 root root 1087 Jan 19 2016 skeleton -rwxr-xr-x 1 root root 2737 Jan 19 2016 umountfs -rwxr-xr-x 1 root root 2202 Jan 19 2016 umountnfs.sh -rwxr-xr-x 1 root root 1879 Jan 19 2016 umountroot -rwxr-xr-x 1 root root 3111 Jan 19 2016 urandom ``` There are no identity, oxauth, oxauth-rp, idp... Also: ``` root@localhost:~# service gluu-server-3.0.2 start root@localhost:~# service gluu-server-3.0.2 login gluu-server-3.0.2 is running... logging in... root@localhost:~# logout root@localhost:~# ps aux | grep java root 2902 0.0 0.0 12944 984 pts/0 S+ 15:10 0:00 grep --color=auto java root@localhost:~# ``` Wired thing, right? Then I've installed Oracle java 1.8 and reinstalled the app: ``` root@localhost:~# java -version java version "1.8.0_131" Java(TM) SE Runtime Environment (build 1.8.0_131-b11) Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode) ``` ``` root@localhost:~# ps aux | grep java 998 7277 10.5 8.9 4948724 733360 ? Sl 15:22 0:11 /opt/jre/bin/java -server -Xms256m -XX:MaxMetaspaceSize=512m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dcatalina.base=/opt/gluu/jetty/oxauth-rp -Djetty.logging.dir=/opt/gluu/jetty/oxauth-rp/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/oxauth-rp -Djava.io.tmpdir=/opt/jetty-9.3/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8085 jetty.state=/opt/gluu/jetty/oxauth-rp/oxauth-rp.state jetty-logging.xml jetty-started.xml start-log-file=/opt/gluu/jetty/oxauth-rp/logs/start.log 998 7346 22.0 15.0 5296320 1232376 ? Sl 15:22 0:22 /opt/jre/bin/java -server -Xms256m -XX:MaxMetaspaceSize=2048m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dpython.home=/opt/jython -Dcatalina.base=/opt/gluu/jetty/oxauth -Djetty.logging.dir=/opt/gluu/jetty/oxauth/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/oxauth -Djava.io.tmpdir=/opt/jetty-9.3/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8081 jetty.state=/opt/gluu/jetty/oxauth/oxauth.state jetty-logging.xml jetty-started.xml start-log-file=/opt/gluu/jetty/oxauth/logs/start.log 998 7446 18.6 13.1 5292416 1072432 ? Sl 15:22 0:16 /opt/jre/bin/java -server -Xms256m -XX:MaxMetaspaceSize=1024m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dcatalina.base=/opt/gluu/jetty/idp -Djetty.logging.dir=/opt/gluu/jetty/idp/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/idp -Djava.io.tmpdir=/opt/jetty-9.3/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8086 jetty.state=/opt/gluu/jetty/idp/idp.state jetty-logging.xml jetty-started.xml start-log-file=/opt/gluu/jetty/idp/logs/start.log 998 7533 20.3 12.5 5264552 1024276 ? Sl 15:22 0:16 /opt/jre/bin/java -server -Xms256m -XX:MaxMetaspaceSize=683m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dcatalina.base=/opt/gluu/jetty/asimba -Djetty.logging.dir=/opt/gluu/jetty/asimba/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/asimba -Djava.io.tmpdir=/opt/jetty-9.3/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8084 jetty.state=/opt/gluu/jetty/asimba/asimba.state jetty-logging.xml jetty-started.xml start-log-file=/opt/gluu/jetty/asimba/logs/start.log 998 7615 43.2 17.4 5424012 1423188 ? Sl 15:22 0:29 /opt/jre/bin/java -server -Xms256m -XX:MaxMetaspaceSize=1365m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dcatalina.base=/opt/gluu/jetty/identity -Dpython.home=/opt/jython -Dorg.eclipse.jetty.server.Request.maxFormContentSize=50000000 -Djetty.logging.dir=/opt/gluu/jetty/identity/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/identity -Djava.io.tmpdir=/opt/jetty-9.3/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8082 jetty.state=/opt/gluu/jetty/identity/identity.state jetty-logging.xml jetty-started.xml start-log-file=/opt/gluu/jetty/identity/logs/start.log ``` But application still doesn't work. Even after this command: ``` for i in identity idp oxauth oxauth-rp asimba; do service $i restart; done ``` And apache logs are clean... And netstat: ``` root@localhost:~# netstat -ltn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 127.0.0.1:1636 0.0.0.0:* LISTEN tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN tcp6 0 0 :::443 :::* LISTEN tcp6 0 0 :::80 :::* LISTEN tcp6 0 0 127.0.0.1:8081 :::* LISTEN tcp6 0 0 127.0.0.1:8082 :::* LISTEN tcp6 0 0 127.0.0.1:8084 :::* LISTEN tcp6 0 0 127.0.0.1:8085 :::* LISTEN tcp6 0 0 127.0.0.1:8086 :::* LISTEN tcp6 0 0 :::22 :::* LISTEN root@localhost:~# ```

By Mohib Zico staff 15 Jun 2017 at 10:41 a.m. CDT

Copy
Mohib Zico gravatar
Thanks, sharing your report with QA team.

By Mohib Zico staff 15 Jun 2017 at 11:30 a.m. CDT

Copy
Mohib Zico gravatar
Did a quick test, I didn't have any issue installing Gluu Server. Used Amazon AWS Ubuntu 16.04 and deb package I shared before. Attaching the whole output below. However, after reboot... 'identity' piece is not starting. Attaching another output here. Created github [issue](https://github.com/GluuFederation/community-edition-setup/issues/305) as well.
after_reboot_vm_ubuntu1604_gluu_server_302.txt ubuntu_testing_gluu_server_302.txt

Post an answer