Docker single host oxtrust can not download ssl certificate

By: Marco Weiss user 30 Aug 2018 at 9:24 a.m. CDT

10 Responses
Marco Weiss gravatar
Dear community, i'm playing with the docker beta. But i cannot get it to fly. What i did as root on a test server. mkdir docker-gluu-server cd docker-gluu-server wget https://raw.githubusercontent.com/GluuFederation/gluu-docker/master/examples/single-host/run_all.sh wget https://raw.githubusercontent.com/GluuFederation/gluu-docker/master/examples/single-host/docker-compose.yml chmod +x run_all.sh ./run_all.sh [I] Creating new configuration, please input the following parameters Enter Domain: cr02.kesslernetworks.de Enter Country Code: DE Enter State: BY Enter City: Raisting Enter Email: support@kesslernetworks.de Enter Organization: kesslernetworks Enter Admin/LDAP Password: supersecurepassword Continue with the above settings? [Y/n]y Waiting for 10 minutes and as i see the oxtrust is throwing that error... am i missing something? ``` oxtrust | 2018-08-30 14:13:46,739 INFO [qtp1190524793-24] [org.gluu.oxtrust.service.filter.AuthorizationProcessingFilter] (AuthorizationProcessingFilter.java:78) - Path is protected, proceeding with authorization processing... oxtrust | 2018-08-30 14:13:46,743 INFO [qtp1190524793-24] [org.gluu.oxtrust.service.uma.PassportUmaProtectionService] (PassportUmaProtectionService.java:90) - UMA passport authentication is disabled oxtrust | 2018-08-30 14:13:49,867 ERROR [Thread-142] [org.gluu.oxtrust.ldap.service.StatusCheckerTimer] (StatusCheckerTimer.java:216) - Can not download ssl certificate oxtrust | java.net.NoRouteToHostException: Host is unreachable (Host unreachable) oxtrust | at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_151] oxtrust | at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_151] oxtrust | at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_151] oxtrust | at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_151] oxtrust | at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_151] oxtrust | at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_151] oxtrust | at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673) ~[?:1.8.0_151] oxtrust | at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:173) ~[?:1.8.0_151] oxtrust | at sun.net.NetworkClient.doConnect(NetworkClient.java:180) ~[?:1.8.0_151] oxtrust | at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[?:1.8.0_151] oxtrust | at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[?:1.8.0_151] oxtrust | at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:1.8.0_151] oxtrust | at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367) ~[?:1.8.0_151] oxtrust | at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191) ~[?:1.8.0_151] oxtrust | at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[?:1.8.0_151] oxtrust | at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[?:1.8.0_151] oxtrust | at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177) ~[?:1.8.0_151] oxtrust | at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162) ~[?:1.8.0_151] oxtrust | at org.gluu.oxtrust.ldap.service.StatusCheckerTimer.setCertificateExpiryAttributes(StatusCheckerTimer.java:204) [classes/:?] oxtrust | at org.gluu.oxtrust.ldap.service.StatusCheckerTimer.processInt(StatusCheckerTimer.java:162) [classes/:?] oxtrust | at org.gluu.oxtrust.ldap.service.StatusCheckerTimer.process(StatusCheckerTimer.java:129) [classes/:?] oxtrust | at org.gluu.oxtrust.ldap.service.StatusCheckerTimer$Proxy$_$$_WeldSubclass.process$$super(Unknown Source) [classes/:?] oxtrust | at sun.reflect.GeneratedMethodAccessor270.invoke(Unknown Source) ~[?:?] oxtrust | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151] oxtrust | at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151] oxtrust | at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl-3.0.1.Final.jar:3.0.1.Final] oxtrust | at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl-3.0.1.Final.jar:3.0.1.Final] oxtrust | at org.xdi.service.cdi.async.AsynchronousInterceptor$1.get(AsynchronousInterceptor.java:36) [oxcore-service-3.1.3.Final.jar:?] oxtrust | at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_151] oxtrust | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151] ```
Docker on Centos 18.06
closed

Answers

By Chris Blanton user 30 Aug 2018 at 10:24 a.m. CDT

Copy
Chris Blanton gravatar
Marco, Can your containers access this domain `cr02.kesslernetworks.de`? `oxTrust` is trying to talk to `oxAuth` on that domain to gather it's OpenID Connect information. In the `docker-compose.yaml` you can see it's added to the `/etc/hosts` file [here](https://github.com/GluuFederation/gluu-docker/blob/3.1.3/examples/single-host/docker-compose.yml#L93). The domain is gathered from the prompt and the IP Address is determined using (this function)[https://github.com/GluuFederation/gluu-docker/blob/3.1.3/examples/single-host/run_all.sh#L6]. There seems to be a communication issue here somewhere.

By Chris Blanton user 30 Aug 2018 at 12:55 p.m. CDT

Copy
Chris Blanton gravatar
Here's the pertinent part of the error log: ``` oxtrust | 2018-08-30 14:13:49,867 ERROR [Thread-142] [org.gluu.oxtrust.ldap.service.StatusCheckerTimer] (StatusCheckerTimer.java:216) - Can not download ssl certificate oxtrust | java.net.NoRouteToHostException: Host is unreachable (Host unreachable ```

By Marco Weiss user 04 Sep 2018 at 1:13 a.m. CDT

Copy
Marco Weiss gravatar
Hi Chris, it think yes, the container should be able to connect to that domain. I tried it by executing a ping on that domain inside every container and got a response. Also the entry in the /etc/hosts file is available. Can i test it using a curl on that certificate? Do you know the URL for that certificate? Could it be that the container on a single setup is not allowed to access the underlaying docker host port? Thank you! Marco

By Chris Blanton user 04 Sep 2018 at 10:44 a.m. CDT

Copy
Chris Blanton gravatar
Marco, Can you run `curl -kLv cr02.kesslernetworks.de/oxauth` from the oxTrust container to see if you can access the OpenID Connect well-known information from the oxAuth container? You can access it by running `docker exec -it oxtrust sh` and running `apk update && apk add curl` to get curl functionality.

By Marco Weiss user 14 Sep 2018 at 3:24 a.m. CDT

Copy
Marco Weiss gravatar
Hi Chris, sorry for my delay, i had a lot of to do on my plate for a upgrade yesterday. Now i tested what you suggested unfortunately it does not work. Here is what i did, again a fresh setup. ``` [root@cr02 docker-gluu-server]# ls docker-compose.yml run_all.sh [root@cr02 docker-gluu-server]# ./run_all.sh [I] Preparing cluster-wide configuration [W] Configuration not found in Consul [I] Creating new configuration, please input the following parameters Enter Domain: cr02.kesslernetworks.de Enter Country Code: DE Enter State: BY Enter City: Raisting Enter Email: support@kesslernetworks.de Enter Organization: kesslernetworks.de Enter Admin/LDAP Password: MySecurePasswordHere Continue with the above settings? [Y/n]y [I] Deploying containers [I] Generating configuration for the first time; this may take a moment Config backend is ready. Generating config. Saving config. Config saved to backend. Config saved to /opt/config-init/db/config.json. [root@cr02 docker-gluu-server]# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES fc1bb5c4a1a8 gluufederation/nginx:latest "/opt/scripts/wait-f…" 4 minutes ago Restarting (0) 6 seconds ago nginx 1391e9754ff5 gliderlabs/registrator "/bin/registrator -i…" 4 minutes ago Up 4 minutes registrator e4db7d103361 gluufederation/oxauth:latest "/opt/scripts/wait-f…" 4 minutes ago Up 4 minutes 8080/tcp oxauth 421144cb7d3a gluufederation/oxshibboleth:latest "/opt/scripts/wait-f…" 4 minutes ago Up 4 minutes 8080/tcp oxshibboleth 027755f5d33d redis:alpine "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 6379/tcp redis d54d1c29c55a gluufederation/oxtrust:latest "/opt/scripts/wait-f…" 4 minutes ago Up 4 minutes 8080/tcp oxtrust 511b1799b632 consul "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp consul a80f5acde19b gluufederation/opendj:latest "/opt/scripts/wait-f…" 4 minutes ago Up 4 minutes 1636/tcp, 4444/tcp, 8989/tcp ldap 5d7ab169c019 gluufederation/oxpassport:latest "/opt/scripts/wait-f…" 4 minutes ago Up 4 minutes 8090/tcp oxpassport [root@cr02 docker-gluu-server]# DOMAIN=cr02.kesslernetworks.de HOST_IP=172.20.30.11 docker-compose logs -f ``` Here are the Log output https://pastebin.com/Fexj7aH5 ``` [root@cr02 docker-gluu-server]# docker exec -it oxtrust sh / # apk update && apk add curl fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz v3.7.0-270-g50b32074fd [http://dl-cdn.alpinelinux.org/alpine/v3.7/main] v3.7.0-265-g734499ca93 [http://dl-cdn.alpinelinux.org/alpine/v3.7/community] OK: 9065 distinct packages available (1/3) Installing libssh2 (1.8.0-r2) (2/3) Installing libcurl (7.61.0-r0) (3/3) Installing curl (7.61.0-r0) Executing busybox-1.27.2-r7.trigger OK: 160 MiB in 74 packages / # curl -kLv cr02.kesslernetworks.de/oxauth * Trying 172.20.30.11... * TCP_NODELAY set * connect to 172.20.30.11 port 80 failed: Host is unreachable * Failed to connect to cr02.kesslernetworks.de port 80: Host is unreachable * Closing connection 0 curl: (7) Failed to connect to cr02.kesslernetworks.de port 80: Host is unreachable ```

By Chris Blanton user 14 Sep 2018 at 1:14 p.m. CDT

Copy
Chris Blanton gravatar
Ah so there are some connectivity issues. One thing that stands out to me: ``` [root@cr02 docker-gluu-server]# DOMAIN=cr02.kesslernetworks.de HOST_IP=172.20.30.11 docker-compose logs -f ``` Your HOST_IP looks like an internal IP and not the servers IP address.

By Chris Blanton user 14 Sep 2018 at 1:17 p.m. CDT

Copy
Chris Blanton gravatar
I'm also seeing this in the logs which is odd: ``` nginx | 2018/09/14 08:01:53 [emerg] 14#14: socket() [::]:80 failed (97: Address family not supported by protocol) nginx | nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol) ``` I see your Nginx container is restarting repeatedly as well. Your OS is listed as CentOS 18. Did you mean Ubuntu 18?

By Chris Blanton user 14 Sep 2018 at 1:30 p.m. CDT

Copy
Chris Blanton gravatar
After some research, my colleague, Isman, discovered that that last error is due to ipv6 not being enabled on a machine. [https://admin-ahead.com/forum/general-linux/address-family-not-supported-by-protocol-nginx-error/](https://admin-ahead.com/forum/general-linux/address-family-not-supported-by-protocol-nginx-error/). We are going to remove the `listen [::]:80 default_server ipv6only=on;` section for compatability purposes and push an update to the Docker images.

By Isman Firmansyah staff 14 Sep 2018 at 1:39 p.m. CDT

Copy
Isman Firmansyah gravatar
The latest image for `gluufederation/nginx:latest` is ready. Please pull the updates and re-deploy the nginx container.

By Marco Weiss user 20 Sep 2018 at 2:52 a.m. CDT

Copy
Marco Weiss gravatar
Hi Chris and Isman, thanks for fixing that. But it did not solve it at all. I now activated IPv6 to test if it will work then. I deleted all on the docker host, images, volumes, container... pruned the hole system. Cloned you repository and started the run_all.sh as the single example. Afte a while all service up but i'm getting connection errors again, but from other services. I now can call my url but it will end up in /identitiy/error with Oop something wrong happened page. Here is my output https://pastebin.com/Ui32Q0pe My local ip configuration looks like that. Is there something wrong? It looks like oxtrust's host unreachable error is the point where all got stuck. ``` TYPE="Ethernet" BOOTPROTO="none" DEFROUTE="yes" IPV4_FAILURE_FATAL="no" IPV6INIT="yes" IPV6_AUTOCONF="yes" IPV6_DEFROUTE="yes" IPV6_FAILURE_FATAL="no" NAME="eth0" UUID="b560be72-da92-4cb4-8fad-7106f1733831" DEVICE="eth0" ONBOOT="yes" MTU="64000" IPADDR="172.20.30.11" DNS1="172.20.30.1" GATEWAY="172.20.30.1" PREFIX="24" DOMAIN="kesslernetworks.de" PEERDNS="no" PEERROUTES="yes" IPV6_PEERDNS="yes" IPV6_PEERROUTES="yes" IPV6_PRIVACY="no" ``` ``` ip a s 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 64000 qdisc pfifo_fast state UP group default qlen 1000 link/ether 02:01:68:99:62:14 brd ff:ff:ff:ff:ff:ff inet 172.20.30.11/24 brd 172.20.30.255 scope global noprefixroute eth0 valid_lft forever preferred_lft forever inet6 fe80::1:68ff:fe99:6214/64 scope link noprefixroute valid_lft forever preferred_lft forever 3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default link/ether 02:42:37:39:b9:f7 brd ff:ff:ff:ff:ff:ff inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 valid_lft forever preferred_lft forever 4: br-c6f7fcf45b95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 02:42:fa:b9:64:01 brd ff:ff:ff:ff:ff:ff inet 172.18.0.1/16 brd 172.18.255.255 scope global br-c6f7fcf45b95 valid_lft forever preferred_lft forever inet6 fe80::42:faff:feb9:6401/64 scope link valid_lft forever preferred_lft forever 6: vethc2acd1f@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 1e:e4:78:a1:75:0c brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet6 fe80::1ce4:78ff:fea1:750c/64 scope link valid_lft forever preferred_lft forever 8: veth43291da@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 9e:63:93:c5:47:bc brd ff:ff:ff:ff:ff:ff link-netnsid 1 inet6 fe80::9c63:93ff:fec5:47bc/64 scope link valid_lft forever preferred_lft forever 10: vethab60ba6@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 02:9c:74:28:1c:e3 brd ff:ff:ff:ff:ff:ff link-netnsid 2 inet6 fe80::9c:74ff:fe28:1ce3/64 scope link valid_lft forever preferred_lft forever 12: veth83a3a1c@if11: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 72:1e:46:98:1e:e8 brd ff:ff:ff:ff:ff:ff link-netnsid 3 inet6 fe80::701e:46ff:fe98:1ee8/64 scope link valid_lft forever preferred_lft forever 14: vethb1e75d7@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether c6:f8:5f:98:03:d0 brd ff:ff:ff:ff:ff:ff link-netnsid 7 inet6 fe80::c4f8:5fff:fe98:3d0/64 scope link valid_lft forever preferred_lft forever 16: veth9e37587@if15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 7a:91:c6:b7:2a:82 brd ff:ff:ff:ff:ff:ff link-netnsid 4 inet6 fe80::7891:c6ff:feb7:2a82/64 scope link valid_lft forever preferred_lft forever 20: veth478cf98@if19: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 06:10:37:52:46:12 brd ff:ff:ff:ff:ff:ff link-netnsid 6 inet6 fe80::410:37ff:fe52:4612/64 scope link valid_lft forever preferred_lft forever 22: veth4a26f2a@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 8a:ff:2e:04:11:fd brd ff:ff:ff:ff:ff:ff link-netnsid 8 inet6 fe80::88ff:2eff:fe04:11fd/64 scope link valid_lft forever preferred_lft forever 26: vethba2c0ee@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-c6f7fcf45b95 state UP group default link/ether 6a:70:04:d9:15:eb brd ff:ff:ff:ff:ff:ff link-netnsid 5 inet6 fe80::6870:4ff:fed9:15eb/64 scope link valid_lft forever preferred_lft forever ```

Post an answer