Hi, >> It is possible that it is a certificate problem, can you give me the procedure to check the certificates on Gluu Docker please? It's not certificate but it's LDAP, when LDAP / OpenDJ failing... other Gluu components can't read data from datasource. You should check status of OpenDJ / LDAP.

``` root@auth.gluu ~ docker logs dca83b4b2741 INFO - pygluu.containerlib.wait - 2022-10-18 13:24:19,115 - Config is ready INFO - pygluu.containerlib.wait - 2022-10-18 13:24:19,228 - Secret is ready INFO - entrypoint - 2022-10-18 13:24:24,345 - Syncing OpenDJ certs. INFO - entrypoint - 2022-10-18 13:24:24,507 - Checking certificate's Subject Alt Name (SAN) INFO - entrypoint - 2022-10-18 13:24:24,567 - Loading Serf key from secrets ==> Starting Serf agent... ==> Starting Serf agent RPC... ==> Serf agent running! Node name: 'ldap' Bind addr: '0.0.0.0:7946' Advertise addr: '172.19.0.6:7946' RPC addr: '127.0.0.1:7373' Encrypted: true Snapshot: false Profile: lan Message Compression Enabled: true ==> Log data will now stream in as it occurs: INFO - ldap_peer - 2022-10-18 13:24:26,880 - Loading initial Serf peers from /etc/gluu/conf/serf-peers-static.json WARNING - ldap_peer - 2022-10-18 13:24:26,881 - Unable to load initial Serf peers from /etc/gluu/conf/serf-peers-static.json; reason=[Errno 2] No such file or directory: '/etc/gluu/conf/serf-peers-static.json' INFO - ldap_replicator - 2022-10-18 13:24:32,316 - Getting current server info INFO - ldap_replicator - 2022-10-18 13:24:32,396 - Checking replicated backends (attempt 1) [18/Oct/2022:13:24:38 +0000] category=com.forgerock.opendj.ldap.config.config severity=NOTICE msgID=571 msg=Loaded extension from file '/opt/opendj/lib/extensions/snmp-mib2605.jar' (build 4.4.12, revision eca841a137111c5f94ee3ac48ddaa5eebf2f68e0) [18/Oct/2022:13:24:39 +0000] category=CORE severity=NOTICE msgID=134 msg=OpenDJ Server 4.4.12 (build 20210928143226, revision number eca841a137111c5f94ee3ac48ddaa5eebf2f68e0) starting up INFO - ldap_replicator - 2022-10-18 13:24:42,423 - Checking replicated backends (attempt 2) [18/Oct/2022:13:24:43 +0000] category=JVM severity=NOTICE msgID=21 msg=Installation Directory: /opt/opendj [18/Oct/2022:13:24:43 +0000] category=JVM severity=NOTICE msgID=23 msg=Instance Directory: /opt/opendj [18/Oct/2022:13:24:43 +0000] category=JVM severity=NOTICE msgID=17 msg=JVM Information: 11.0.11+9-alpine-r0 by Alpine, 64-bit architecture, 1610612736 bytes heap size [18/Oct/2022:13:24:43 +0000] category=JVM severity=NOTICE msgID=18 msg=JVM Host: ldap, running Linux 5.4.0-121-generic amd64, 2147483648 bytes physical memory size, number of processors available 4 [18/Oct/2022:13:24:43 +0000] category=JVM severity=NOTICE msgID=19 msg=JVM Arguments: "-XX:+UseContainerSupport", "-XX:MaxRAMPercentage=75.0", "-Dorg.opends.server.scriptName=start-ds" [18/Oct/2022:13:24:47 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend site containing 2 entries has started [18/Oct/2022:13:24:52 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend metric containing 26266 entries has started INFO - ldap_replicator - 2022-10-18 13:24:52,571 - Checking replicated backends (attempt 3) [18/Oct/2022:13:24:54 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend userRoot containing 18790 entries has started [18/Oct/2022:13:24:55 +0000] category=EXTENSIONS severity=NOTICE msgID=221 msg=DIGEST-MD5 SASL mechanism using a server fully qualified domain name of: ldap INFO - ldap_replicator - 2022-10-18 13:25:02,651 - Checking replicated backends (attempt 4) [18/Oct/2022:13:25:10 +0000] category=CORE severity=NOTICE msgID=135 msg=The Directory Server has started successfully [18/Oct/2022:13:25:10 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Directory Server has started successfully [18/Oct/2022:13:25:11 +0000] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on LDAPS Connection Handler 0.0.0.0 port 1636 [18/Oct/2022:13:25:11 +0000] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on LDAP Connection Handler 0.0.0.0 port 1389 [18/Oct/2022:13:25:11 +0000] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on Administration Connector 0.0.0.0 port 4444 INFO - ldap_replicator - 2022-10-18 13:25:12,675 - Checking replicated backends (attempt 5) INFO - ldap_replicator - 2022-10-18 13:25:22,707 - Checking replicated backends (attempt 6) INFO - ldap_replicator - 2022-10-18 13:25:32,803 - Checking replicated backends (attempt 7) INFO - ldap_replicator - 2022-10-18 13:25:43,279 - Checking replicated backends (attempt 8) INFO - ldap_replicator - 2022-10-18 13:25:53,463 - Checking replicated backends (attempt 9) INFO - ldap_replicator - 2022-10-18 13:26:03,567 - Checking replicated backends (attempt 10) INFO - ldap_replicator - 2022-10-18 13:26:13,702 - Checking replicated backends (attempt 11) INFO - ldap_replicator - 2022-10-18 13:26:23,887 - Checking replicated backends (attempt 12) INFO - ldap_replicator - 2022-10-18 13:26:34,027 - Checking replicated backends (attempt 13) INFO - ldap_replicator - 2022-10-18 13:26:44,135 - Checking replicated backends (attempt 14) INFO - ldap_replicator - 2022-10-18 13:26:54,403 - Checking replicated backends (attempt 15) INFO - ldap_replicator - 2022-10-18 13:27:04,587 - Checking replicated backends (attempt 16) INFO - ldap_replicator - 2022-10-18 13:27:14,787 - Checking replicated backends (attempt 17) INFO - ldap_replicator - 2022-10-18 13:27:24,871 - Checking replicated backends (attempt 18) INFO - ldap_replicator - 2022-10-18 13:27:34,891 - Checking replicated backends (attempt 19) INFO - ldap_replicator - 2022-10-18 13:27:45,087 - Checking replicated backends (attempt 20) INFO - ldap_replicator - 2022-10-18 13:27:55,115 - Checking replicated backends (attempt 21) INFO - ldap_replicator - 2022-10-18 13:28:05,164 - Checking replicated backends (attempt 22) INFO - ldap_replicator - 2022-10-18 13:28:15,195 - Checking replicated backends (attempt 23) INFO - ldap_replicator - 2022-10-18 13:28:25,225 - Checking replicated backends (attempt 24) INFO - ldap_replicator - 2022-10-18 13:28:35,260 - Checking replicated backends (attempt 25) INFO - ldap_replicator - 2022-10-18 13:28:45,287 - Checking replicated backends (attempt 26) INFO - ldap_replicator - 2022-10-18 13:28:55,312 - Checking replicated backends (attempt 27) INFO - ldap_replicator - 2022-10-18 13:29:05,335 - Checking replicated backends (attempt 28) INFO - ldap_replicator - 2022-10-18 13:29:15,371 - Checking replicated backends (attempt 29) INFO - ldap_replicator - 2022-10-18 13:29:25,399 - Checking replicated backends (attempt 30) ```

Hi, Bruno. According to the log, LDAP server seems to successfully start. Then some warnings about replication backend checks appear (what can be normal behavior if you don't cluster it). Out of curiosity: why do you think it may be a certificate issue?

Hello, Before encountering the incident, there was an error in oxauth container : ```bash 2022-10-18 11:39:06,124 WARN [qtp6519275-17] [org.gluu.oxauth.model.crypto.AbstractCryptoProvider] (AbstractCryptoProvider.java:190) - WARNING! Expired Key is used, alias: f8556861-f87c-4b16-b828-0fbc1e45d4a6_sig_rs512 Expires On: 2022-10-14 09:54:51 ``` Today's Date: 2022-10-18 11:39:06 So i upgrade Nginx certifcate and I restarted the servers and now the server won't restart

Hmm.. I believe that log entry refers to oxAuth's own key set (JWKS) used to sign and encrypt tokens during auth flows, not a Nginx certificate certainly (oxAuth shouldn't be aware of the latter at all). What procedure did you use to upgrade the certificate? Can it be the cause of the crash?

Hello; In container gluufederation/nginx:4.3.0_01 Update certificates - /etc/certs/gluu_https.crt - /etc/certs/gluu_https.key and restart all services ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 58034e6e29a9 gliderlabs/registrator:master "/bin/registrator -i…" 2 hours ago Up 2 hours registrator c317e440ce27 gluufederation/scim:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 2 hours 8080/tcp scim 7873e7630cdd gluufederation/oxd-server:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up Less than a second 8443-8444/tcp oxd-server 0f52eb65d17b gluufederation/oxauth:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 2 hours 8080/tcp oxauth 950320d79f68 gluufederation/cr-rotate:4.3.0_01 "tini -g -- sh /app/…" 2 hours ago Up 2 hours cr-rotate 22c345cb53a0 gluufederation/oxpassport:4.3.0_01 "tini -g -- sh /app/…" 2 hours ago Up 29 minutes 8090/tcp oxpassport dc31731cd766 gluufederation/oxtrust:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 3 minutes 8080/tcp oxtrust 87ce53fc8e05 gluufederation/jackrabbit:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 2 hours 8080/tcp jackrabbit 7211e154c00a gluufederation/fido2:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 2 hours 8080/tcp fido2 fdd89b2858f7 gluufederation/opendj:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 2 hours 1389/tcp, 1636/tcp ldap 76558debe9b9 gluufederation/nginx:4.3.0_01 "tini -g -- sh /app/…" 2 hours ago Up 2 hours 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp nginx 84298d83fa14 gluufederation/oxshibboleth:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 2 hours 8080/tcp oxshibboleth cca03b7bef05 gluufederation/casa:4.3.0_01 "tini -e 143 -g -- s…" 2 hours ago Up 3 minutes 8080/tcp casa 8d8357dac4ad vault:1.0.1 "docker-entrypoint.s…" 2 hours ago Up 2 hours 8200/tcp vault 4c1e668c3b88 consul:1.6 "docker-entrypoint.s…" 2 hours ago Up 2 hours 8300-8302/tcp, 8500/tcp, 8301-8302/udp, 8600/tcp, 8600/udp consul ```

I need update certificate oxAuth's own key set (JWKS) ?

Hi Bruno, If you revert the cert in nginx container `/etc/certs/gluu_https.crt` and `/etc/certs/gluu_https.key` and restart nginx container, do you still encounter the issue with LDAP connection from oxAuth?

Guys, whispeak does not have any customer contract with us. That means they don't have either cloud native, or production support. They need to contact us asap about a support contract if this is really a prod issue.

Hello Michael, Isman Indeed, the incident concerns the pre-production env. I renewed the certificate again and restarted Nginx and oxAuth, the problem. Here are the oxAuth startup logs, there is a warning. Maybe OxAuth does not have the credentials to connect to LDAP? INFO - pygluu.containerlib.wait - 2022-10-19 09:16:31,683 - Config is ready INFO - pygluu.containerlib.wait - 2022-10-19 09:16:31,719 - Secret is ready INFO - pygluu.containerlib.wait - 2022-10-19 09:16:32,101 - LDAP is ready INFO - webdav - 2022-10-19 09:16:33,392 - Sync files from http://jackrabbit:8080/repository/default/opt/gluu/jetty/oxauth/custom WARNING - webdav - 2022-10-19 09:16:33,448 - Unable to sync files from http://jackrabbit:8080/repository/default/opt/gluu/jetty/oxauth/custom; reason=Remote resource: /opt/gluu/jetty/oxauth/custom not found INFO - webdav - 2022-10-19 09:16:33,448 - Sync /etc/certs/otp_configuration.json to http://jackrabbit:8080/repository/default/etc/certs/otp_configuration.json INFO - webdav - 2022-10-19 09:16:33,695 - Sync /etc/certs/super_gluu_creds.json to http://jackrabbit:8080/repository/default/etc/certs/super_gluu_creds.json 2022-10-19 09:16:37.687:INFO :oejs.Server:main: jetty-10.0.6; built: 2021-06-29T15:28:56.259Z; git: 37e7731b4b142a882d73974ff3bec78d621bd674; jvm 11.0.11+9-alpine-r0 2022-10-19 09:16:37.909:INFO :oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/gluu/jetty/oxauth/webapps/] WARNING - jks_sync - 2022-10-19 09:16:38,247 - JKS sync is disabled Regards Bruno