By: Conrad Krinock user 28 Oct 2020 at 8:10 a.m. CDT

21 Responses
Conrad Krinock gravatar
This is a continuation of https://support.gluu.org/identity-management/8883/503-error/ which has been closed. ##Expected behavior Successfully connect to server console ##Actual Behavior 503 error returned to browser ## Details We successfully deployed the Gluu authentication server earlier this year as a prototype/proof-of concept and the installation was totally successful. After not running it for a few months, we are now encountering a 503 error. We have captured SUBSTANTIAL DETAILS regarding this tech support request; however your web site apparently does not allow attachments(?) Nevertheless we can provide the details upon you request. Conrad J Krinock CISSP, Certified Scrum Professional 775-450-1202

By Mohib Zico staff 29 Oct 2020 at 9:23 a.m. CDT

Mohib Zico gravatar
Please share logs in pastebin or something like that.

By Conrad Krinock user 29 Oct 2020 at 11:40 a.m. CDT

Conrad Krinock gravatar
I've extracted 18 log files in /opt/gluu-server/opt/gluu/jetty/identity/logs/. Let me know if there are other directories where relevant log files may exist. Unless I'm missing it, this tech support page does not have the ability to upload files. Therefore I'm going to send the 18 log files to Davin Cooke.

By Mohib Zico staff 29 Oct 2020 at 1 p.m. CDT

Mohib Zico gravatar
Thanks. Davin shared your email. Quick question... You are getting 503 error from oxTrust or oxAuth? ( One easy way to understand, let's look at the url which is throwing this 503 ).

By Conrad Krinock user 29 Oct 2020 at 3:32 p.m. CDT

Conrad Krinock gravatar
We're encountering the 503 error when we connect to the gluu a/s dashboard via a browser at https://test.gluu.org/ Does that answer your question? Also we have two gluu a/s servers running as poc/prototypes: - ubuntu on VMWare Player15 -- 6GB ram - ubuntu on physical server -- 10GB ram Both are gluu version 4.2.1. Both are encountering the 503 error Conrad J Krinock CISSP, Certified Scrum Professional 775-450-1202

By Mohib Zico staff 29 Oct 2020 at 10:33 p.m. CDT

Mohib Zico gravatar
Can you please share the screenshot of the error ( full screenshot with url ).

By Bob Sirwaitis Account Admin 05 Nov 2020 at 8:26 a.m. CST

Bob Sirwaitis gravatar
Zico - need followup to email sent 10/30 - Conrad sent screenshot as attachment

By Mohib Zico staff 05 Nov 2020 at 8:33 a.m. CST

Mohib Zico gravatar
Received. Conrad, Let's do one thing... - Restart your services in below way: - Become user `ldap`. `su - ldap` and `stop.ds/start.ds` opendj from `/opt/opendj/bin/` directory. - As root, restart oxauth service: `service oxauth stop/start` - As root, restart identity service: `service identity stop/start` See how things go.

By Conrad Krinock user 05 Nov 2020 at 3:29 p.m. CST

Conrad Krinock gravatar
Hello Mohit, Seem we don't have an ldap user on our ubuntu server: $ su $whoami root $ su ldap su: **user ldap does not exist** We also checked to /etc/passwd file and it does not contain a ldap user. We have two ubuntu servers: * vm running on vmware player * physical server So now we might be getting some traction. It's possible I did something wrong during the installation(?) (1) please send me your email address so I can ** provide my installation log**, or (2) IS THERE SOMEWAY FOR ME TO ATTACH FILES TO THIS TECH SUPPORT REQUEST?? When we click on the "Link URL" nothing happens. Conrad

By Michael Schwartz staff 05 Nov 2020 at 4:31 p.m. CST

Michael Schwartz gravatar
I think you are not in the chroot container.... did you do ``` # /sbin/gluu-serverd login `` Try that and then follow Zico's suggestion?

By Conrad Krinock user 06 Nov 2020 at 5:54 a.m. CST

Conrad Krinock gravatar
We tried > /sbin/gluu-serverd login ... but we apparently don't know the password for the chroot container. What is the procedure for changing this password?

By Mohib Zico staff 06 Nov 2020 at 6:01 a.m. CST

Mohib Zico gravatar
>> but we apparently don't know the password for the chroot container You need to have `root` access to perform installation and log into Gluu Server container as standard procedure.

By Conrad Krinock user 06 Nov 2020 at 11:18 a.m. CST

Conrad Krinock gravatar
We ran these commands: > su > /sbin/gluu-serverd login (connect to chroot container) > su ldap > cd /opt/opendj/bin/ > ./stop-ds > ./start-ds > exit (from ldap to root) > service oxauth stop > service oxauth start > service identity stop > service identity start On our physical ubuntu server : -- all commands appear to have run successfully, except for: > service identity start which results in this error: Job for identity.service failed because the control process exited with error code. See "systemctl status identity.service" and "journalctl -xe" for details. . . On our VMWARE/ubuntu server : -- all commands appear to have run successfully -- https://test.gluu.org/ still results in a 503 error

By Michael Schwartz staff 06 Nov 2020 at 11:49 a.m. CST

Michael Schwartz gravatar
A 503 error means that `/identity` did not start. 6GB is still pretty tight, especially if you installed all the services. Try: ``` # service identity status # service opendj status ``` Or you can also do : ``` # ps -ef | grep java ``` Which should give you an idea of what's running--you should see oxauth, identity (oxtrust), and opendj all running.

By Conrad Krinock user 06 Nov 2020 at 12:44 p.m. CST

Conrad Krinock gravatar
Does this shed any light on the subject? . . service identity status ● identity.service - Identity service Loaded: loaded (/lib/systemd/system/identity.service; enabled; vendor preset: enabled Active: failed (Result: exit-code) since Fri 2020-11-06 11:40:30 EST; 1h 59min ago Process: 183658 ExecStart=/opt/dist/scripts/identity start (code=exited, status=1/FAIL Nov 06 11:39:30 test.gluu.org identity[183658]: 2020-11-06 16:39:30.797:INFO::main: Cons Nov 06 11:39:43 test.gluu.org identity[183658]: . . . WARNING: An illegal reflective acc Nov 06 11:39:43 test.gluu.org identity[183658]: WARNING: Illegal reflective access by or Nov 06 11:39:43 test.gluu.org identity[183658]: WARNING: Please consider reporting this Nov 06 11:39:43 test.gluu.org identity[183658]: WARNING: Use --illegal-access=warn to en Nov 06 11:39:43 test.gluu.org identity[183658]: WARNING: All illegal access operations w Nov 06 11:40:30 test.gluu.org identity[183658]: . . . . . . . . . . . . FAILED Fri Nov Nov 06 11:40:30 test.gluu.org systemd[1]: identity.service: Control process exited, code Nov 06 11:40:30 test.gluu.org systemd[1]: identity.service: Failed with result 'exit-cod Nov 06 11:40:30 test.gluu.org systemd[1]: Failed to start Identity service. . . . service opendj status ● opendj.service - OpenDJ Directory Service Loaded: loaded (/etc/systemd/system/opendj.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2020-11-06 07:22:52 EST; 6h ago Process: 76 ExecStart=/opt/opendj/bin/start-ds (code=exited, status=0/SUCCESS) Main PID: 518 (code=exited, status=143) Nov 01 11:06:17 test.gluu.org start-ds[76]: [01/Nov/2020:16:06:16 +0000] category=BACKEN Nov 01 11:06:17 test.gluu.org start-ds[76]: [01/Nov/2020:16:06:17 +0000] category=BACKEN Nov 01 11:06:17 test.gluu.org start-ds[76]: [01/Nov/2020:16:06:17 +0000] category=EXTENS Nov 01 11:06:18 test.gluu.org start-ds[76]: [01/Nov/2020:16:06:18 +0000] category=CORE s Nov 01 11:06:18 test.gluu.org start-ds[76]: [01/Nov/2020:16:06:18 +0000] category=CORE s Nov 01 11:06:18 test.gluu.org start-ds[76]: [01/Nov/2020:16:06:18 +0000] category=PROTOC Nov 01 11:06:18 test.gluu.org start-ds[76]: [01/Nov/2020:16:06:18 +0000] category=PROTOC Nov 01 11:06:19 test.gluu.org systemd[1]: Started OpenDJ Directory Service. Nov 06 07:22:52 test.gluu.org systemd[1]: opendj.service: Main process exited, code=exit Nov 06 07:22:52 test.gluu.org systemd[1]: opendj.service: Failed with result 'exit-code'

By Mohib Zico staff 06 Nov 2020 at 9:37 p.m. CST

Mohib Zico gravatar
Hi Conrad, Can you please do one thing for me? - Reboot your VM. - Immediately after reboot, log into container and use command `ps ax | grep java`. Please post the list here.

By Conrad Krinock user 09 Nov 2020 at 1:43 p.m. CST

Conrad Krinock gravatar
OMG I decided to perform a full re-install (again). I over-wrote my entire hard-drive by re-installing ubunutu from the ISO file (again) and re-installed GLUU (again). I am now able to access the gluu dashboard at https://test.gluu.org. So like Halleluljah!!!!!! Let's keep this ticket open. Tomorrow I'm going to make sure I can connect using my gluu java spring client. Conrad

By Conrad Krinock user 10 Nov 2020 at 7:49 a.m. CST

Conrad Krinock gravatar
Hi Mike, As requested, here are the results of ps -ef | grep java I believe this is showing we've got 3 services running: - opendj, oxauth, and identiy Correct? 5602 ? Sl 6:28 /opt/jre/bin/java -server -Xms1g -Xmx2g -XX:+UseCompressedOops -Dorg.opends.server.scriptName=start-ds org.opends.server.core.DirectoryServer --configFile /opt/opendj/config/config.ldif 5769 ? Sl 7:46 /opt/jre/bin/java -server -Xms256m -Xmx1433m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/oxauth -Dlog.base=/opt/gluu/jetty/oxauth -Dpython.home=/opt/jython -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/oxauth -Djava.io.tmpdir=/opt/jetty-9.4/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8081 jetty.state=/opt/gluu/jetty/oxauth/oxauth.state jetty-started.xml 5899 ? Sl 8:37 /opt/jre/bin/java -server -Xms256m -Xmx1433m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/identity -Dlog.base=/opt/gluu/jetty/identity -Dpython.home=/opt/jython -Dorg.eclipse.jetty.server.Request.maxFormContentSize=50000000 -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/identity -Djava.io.tmpdir=/opt/jetty-9.4/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8082 jetty.state=/opt/gluu/jetty/identity/identity.state jetty-started.xml 52892 pts/1 S+ 0:00 grep --color=auto java

By Mohib Zico staff 10 Nov 2020 at 7:55 a.m. CST

Mohib Zico gravatar
>> I believe this is showing we've got 3 services running: Correct.

By Conrad Krinock user 10 Nov 2020 at 7:56 a.m. CST

Conrad Krinock gravatar
Hi Again Mike -- also as requested we ran service identity status and service opendj status. Here are the results: . . service identity status ● identity.service - Identity service Loaded: loaded (/lib/systemd/system/identity.service; enabled; vendor preset: Active: active (running) since Mon 2020-11-09 14:24:01 EST; 18h ago Process: 5871 ExecStart=/opt/dist/scripts/identity start (code=exited, status= Main PID: 5899 (java) CGroup: /system.slice/identity.service └─5899 /opt/jre/bin/java -server -Xms256m -Xmx1433m -XX:+DisableExpli Nov 09 14:23:28 test.gluu.org systemd[1]: Starting Identity service... Nov 09 14:23:29 test.gluu.org identity[5871]: Starting Jetty: 2020-11-09 19:23:2 Nov 09 14:23:29 test.gluu.org identity[5871]: 2020-11-09 19:23:29.951:INFO::main Nov 09 14:23:42 test.gluu.org identity[5871]: . . . WARNING: An illegal reflecti Nov 09 14:23:42 test.gluu.org identity[5871]: WARNING: Illegal reflective access Nov 09 14:23:42 test.gluu.org identity[5871]: WARNING: Please consider reporting Nov 09 14:23:42 test.gluu.org identity[5871]: WARNING: Use --illegal-access=warn Nov 09 14:23:42 test.gluu.org identity[5871]: WARNING: All illegal access operat Nov 09 14:24:01 test.gluu.org identity[5871]: . . . . OK Mon Nov 9 14:24:01 EST Nov 09 14:24:01 test.gluu.org systemd[1]: Started Identity service. . . . service opendj status ● opendj.service - OpenDJ Directory Service Loaded: loaded (/etc/systemd/system/opendj.service; enabled; vendor preset: e Active: active (running) since Mon 2020-11-09 14:23:03 EST; 18h ago Process: 5484 ExecStop=/opt/opendj/bin/stop-ds (code=exited, status=0/SUCCESS) Process: 5556 ExecStart=/opt/opendj/bin/start-ds (code=exited, status=0/SUCCES Main PID: 5602 (java) CGroup: /system.slice/opendj.service └─5602 /opt/jre/bin/java -server -Xms1g -Xmx2g -XX:+UseCompressedOops Nov 09 14:22:59 test.gluu.org start-ds[5556]: [09/Nov/2020:19:22:59 +0000] categ Nov 09 14:23:00 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:00 +0000] categ Nov 09 14:23:01 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:01 +0000] categ Nov 09 14:23:01 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:01 +0000] categ Nov 09 14:23:01 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:01 +0000] categ Nov 09 14:23:02 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:01 +0000] categ Nov 09 14:23:02 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:01 +0000] categ Nov 09 14:23:02 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:02 +0000] categ Nov 09 14:23:02 test.gluu.org start-ds[5556]: [09/Nov/2020:19:23:02 +0000] categ Nov 09 14:23:03 test.gluu.org systemd[1]: Started OpenDJ Directory Service.

By Michael Schwartz staff 10 Nov 2020 at 12:02 p.m. CST

Michael Schwartz gravatar
Should we close this issue?

By Conrad Krinock user 11 Nov 2020 at 8:45 a.m. CST

Conrad Krinock gravatar
Yes Mike we're closing this tech support request. We unfortunately did not find a root cause. Davin Cooke has assigned Larry Jackson to assist us to hopefully (1) find a root cause to the gluu-as-server 503 error, and also (2) revive our java spring client web app. Status: gluu-as-server still functioning even after ubuntu host reboot Thanks, Conrad