502 Bad Gateway Error (examples/single-host 3.1.4)

By: Corin Lawson user 20 Jan 2019 at 7:32 p.m. CST

3 Responses
Corin Lawson gravatar
Hi, I am attempting to spin up and evaluate the docker-compose project under examples/single-host in the GluuFederation/gluu-docker repo (I have cloned the repo and have 00d88e9 checked out). As this is for evaluation purposes I only want to run it locally without exposing it outside of my machine's network, so I decided to set the HOST_IP to be the IP of the docker0 network bridge and DOMAIN to be a local name with the same IP in my /etc/hosts file. Here's the complete output of `docker-compose logs` http://sprunge.us/9NSXdM http://sprunge.us/CiNRa7 (--no-color) I find the output quite hard to decipher, but from what I can tell, nginx is attempting to pass to the oxtrust backend which hasn't received the oxtrust service from consul, browsing consul services (using the consul web ui) only the consul service itself appears to be there. Looking at the oxtrust logs, it appears to start correctly (I see `oejsh.ContextHandler:main: Started`) but then periodically reports: ``` EntryPersistenceException: Failed to find entries with baseDN: ou=resetPasswordRequests ``` So, I'm guessing it can't talk to the ldap container but everything looks fine there: ``` ldap | [21/Jan/2019:00:29:11 +0000] category=CORE severity=NOTICE msgID=org.opends.messages.core.135 msg=The Directory Server has started successfully ldap | [21/Jan/2019:00:29:11 +0000] category=CORE severity=NOTICE msgID=org.opends.messages.core.139 msg=The Directory Server has sent an notification generated by class org.opends.server.core.DirectoryServer ( type org.opends.server.DirectoryServerStarted, ID org.opends.messages.core-135): The Directory Server has started successfully ``` I did notice this: ``` ldap | 2019-01-21 00:26:24,841 [WARNING] [wait-for-it] - HTTPConnectionPool(host='consul', port=8500): Max retries exceeded with url: /v1/kv/gluu/config/oxauth_openid_jwks_fn?stale=1 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd5b35ef190>: Failed to establish a new connection: [Errno 111] Connection refused',)) ``` And thought that that might be why the service isn't registered in consul, so I tried restarting the ldap container but it started normally with no change to consul: ``` ldap | 2019-01-21 01:13:07,933 [INFO] [wait-for-it] - Hi world, waiting for config backend to be ready before running /opt/scripts/entrypoint.sh ldap | 2019-01-21 01:13:07,936 [INFO] [wait-for-it] - Config backend is ready. ldap | 2019-01-21 01:13:07,936 [INFO] [wait-for-it] - Now executing the arguments passed to /opt/scripts/wait-for-it: /opt/scripts/entrypoint.sh ldap | [21/Jan/2019:01:13:09 +0000] category=CORE severity=NOTICE msgID=org.opends.messages.core.134 msg=Gluu-OpenDJ 3.0.1-gluu (build 20180801142102, revision number c5ad2e4846d8aeb501ffdfe5ae2dfd35136dfa68) starting up ldap | [21/Jan/2019:01:13:11 +0000] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.21 msg=Installation Directory: /opt/opendj ldap | [21/Jan/2019:01:13:11 +0000] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.23 msg=Instance Directory: /opt/opendj ldap | [21/Jan/2019:01:13:11 +0000] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.17 msg=JVM Information: 1.8.0_171-b11 by Oracle Corporation, 64-bit architecture, 7467958272 bytes heap size ldap | [21/Jan/2019:01:13:11 +0000] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.18 msg=JVM Host: 5eae93cabf2d, running Linux 4.16.2 amd64, 33599057920 bytes physical memory size, number of processors available 8 ldap | [21/Jan/2019:01:13:11 +0000] category=UTIL severity=NOTICE msgID=org.opends.messages.runtime.19 msg=JVM Arguments: "-Dorg.opends.server.scriptName=start-ds" ldap | [21/Jan/2019:01:13:11 +0000] category=PLUGGABLE severity=NOTICE msgID=org.opends.messages.backend.513 msg=The database backend site containing 2 entries has started ldap | [21/Jan/2019:01:13:12 +0000] category=PLUGGABLE severity=NOTICE msgID=org.opends.messages.backend.513 msg=The database backend userRoot containing 207 entries has started ldap | [21/Jan/2019:01:13:12 +0000] category=EXTENSIONS severity=NOTICE msgID=org.opends.messages.extension.221 msg=DIGEST-MD5 SASL mechanism using a server fully qualified domain name of: 172.23.0.5 ldap | [21/Jan/2019:01:13:12 +0000] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on Administration Connector 0.0.0.0 port 4444 ldap | [21/Jan/2019:01:13:12 +0000] category=PROTOCOL severity=NOTICE msgID=org.opends.messages.protocol.276 msg=Started listening for new connections on LDAPS Connection Handler 0.0.0.0 port 1636 ldap | [21/Jan/2019:01:13:12 +0000] category=CORE severity=NOTICE msgID=org.opends.messages.core.135 msg=The Directory Server has started successfully ldap | [21/Jan/2019:01:13:12 +0000] category=CORE severity=NOTICE msgID=org.opends.messages.core.139 msg=The Directory Server has sent an notification generated by class org.opends.server.core.DirectoryServer ( type org.opends.server.DirectoryServerStarted, ID org.opends.messages.core-135): The Directory Server has started successfully ``` I note that consul cannot even talk to itself: ``` $ DOMAIN=gauss.docker HOST_IP=172.17.0.1 docker-compose exec consul consul info Error querying agent: Get http://127.0.0.1:8500/v1/agent/self: dial tcp 127.0.0.1:8500: connect: connection refused ``` Is that normal? I'm a little lost, please advise. Additional info: ``` $ uname -a Linux gauss 4.16.2 #23 SMP Mon Apr 30 11:01:40 AEST 2018 x86_64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz GenuineIntel GNU/Linux $ docker-compose version docker-compose version 1.19.0, build 9e633ef docker-py version: 2.7.0 CPython version: 3.6.5 OpenSSL version: OpenSSL 1.0.2q 20 Nov 2018 $ docker version Client: Version: 18.06.1-ce API version: 1.38 Go version: go1.10.3 Git commit: e68fc7a Built: Tue Dec 11 13:38:18 2018 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Dec 11 13:37:43 2018 OS/Arch: linux/amd64 Experimental: false ```
Gentoo/Linux 4.16
closed

Answers

By Isman Firmansyah staff 25 Jan 2019 at 4:46 p.m. CST

Copy
Isman Firmansyah gravatar
``` EntryPersistenceException: Failed to find entries with baseDN: ou=resetPasswordRequests ``` This is known issue in 3.1.4 and it is fixed in upcoming 3.1.5 release. ``` $ DOMAIN=gauss.docker HOST_IP=172.17.0.1 docker-compose exec consul consul info Error querying agent: Get http://127.0.0.1:8500/v1/agent/self: dial tcp 127.0.0.1:8500: connect: connection refused ``` You are seeing the error because Consul is not listening to its loopback interface. Consul is bind to its `eth0` interface. Try this command instead: ``` docker-compose exec consul ifconfig eth0 # get the container IP docker-compose exec consul consul info -http-addr=<IP>:8500 ``` Unfortunately I can't reproduce this issue. I also use `docker0` IP as `HOST_IP` value, but everything worked as expected. The `502 Bad Gateway` you've seen probably caused by oxTrust hasn't started and nginx hasn't seen the service being up. It may takes few seconds before nginx re-configure itself.

By Corin Lawson user 25 Jan 2019 at 6:02 p.m. CST

Copy
Corin Lawson gravatar
Thanks for the reply and the tips, I won't be at my workstation for a few days but I will try again then. I checked out the 3.1.5 branch but had issues with the vault setup... I didn't report the issue since it still appeared to be an alpha version... but I will open a separate issue if I need to.

By Corin Lawson user 29 Jan 2019 at 5:48 p.m. CST

Copy
Corin Lawson gravatar
I have it running now, after a few `docker-compose down`, `rm -rf volumes` and rerunning the `run_all.sh` script and waiting for all the checks to pass in consul web interface. I didn't do anything special. I do note that the logs are very chatty and there's a few errors reported and I'm not sure if action is required for each error but overall it's a very impressive setup.

Post an answer