https://fed-auth-01.hartford.edu If you would like to see the behavior.

https://hartford0-my.sharepoint.com/:f:/g/personal/granholm_hartford_edu/EjmAMyzEofFEibZYTYsMvXEBdiO34x45u_NWEXbnEEOR1A?e=2zhzlM Link to both oxauth.log and oxtrust.log

Ben, It looks like the services oxTrust and oxAuth are having connectivity issued to LDAP. Can you send all the logs from both oxAuth and oxTrust/Identity?

So I got it back up and running with a revert to a snapshot. However, if I reboot it the issue crops up again.

I have posted the logs to the link in the post above yours. This didn't appear to have a way to attach files.

> I have posted the logs to the link in the post above yours. This didn't appear to have a way to attach files. Right. It only seemed to be the specific `oxauth.log` and `oxtrust.log`. There are a lot of logs in `/opt/gluu/jetty/$service/logs/` that would help me nail down the issue. Can you also run `ls -l /etc/init.d/rc2.d/` inside the chroot for me? It might be a service start-up order issue. We found an issue with that recently https://github.com/GluuFederation/community-edition-setup/issues/499 . That being said, your logs stopping here: ``` 2018-12-21 15:47:41,221 WARN [weld-worker-2] [org.jboss.weld.bootstrap.Validator] (Validator.java:443) - WELD-001440: Scope type @javax.enterprise.context.ApplicationScoped() used on injection point [UnbackedAnnotatedField] @Inject @ApplicationScoped private org.xdi.service.cache.CacheProviderFactory.instance at org.xdi.service.cache.CacheProviderFactory.instance(CacheProviderFactory.java:0) StackTrace 2018-12-21 15:47:41,561 INFO [main] [org.xdi.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:23) - Adding Bouncy Castle Provider ``` and here: ``` 2018-12-21 15:48:18,253 INFO [main] [org.xdi.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:23) - Adding Bouncy Castle Provider 2018-12-21 15:48:18,330 INFO [main] [org.gluu.oxtrust.ldap.service.AppInitializer] (AppInitializer.java:275) - Build date null. Code revision null on null. Build null 2018-12-21 15:48:18,339 INFO [main] [org.gluu.oxtrust.config.ConfigurationFactory] (ConfigurationFactory.java:134) - Creating oxTrustConfiguration 2018-12-21 15:48:18,339 INFO [main] [org.gluu.oxtrust.config.ConfigurationFactory] (ConfigurationFactory.java:420) - ########## ldapFileName = /etc/gluu/conf/ox-ldap.properties ``` Indicate that they can't access the local LDAP to pull their configuration and are halting because of it. Presumably it's a start-up order issue, but something else might be wrong with your LDAP. Are you using OpenDJ or OpenLDAP?

> This didn't appear to have a way to attach files. Community support doesn't allow attaching files. If you could `tar.gz` or zip the files on your sharepoint, I'll happily review them for you.

Now that I have reverted to the snapshot, it appears to be working. I am not sure the current logs will help you. I will upload them but I am not sure how much they will help. I am loathe to restart it as it is working and in about 3 hours we will be off until the new year. If the logs don't help you, we can revisit this in the new year. I will upload the logs shortly and put them in the folder above. Here is the output from the container's rc2.d: - lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean - -rw-r--r-- 1 root root 677 Feb 5 2016 README - lrwxrwxrwx 1 root root 18 Nov 13 2017 S01identity -> ../init.d/identity - lrwxrwxrwx 1 root root 13 Nov 13 2017 S01idp -> ../init.d/idp - lrwxrwxrwx 1 root root 16 Nov 13 2017 S01oxauth -> ../init.d/oxauth - lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog - lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 - lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus - lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached - lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync - lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver - lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron - lrwxrwxrwx 1 root root 18 Nov 13 2017 S05ondemand -> ../init.d/ondemand - lrwxrwxrwx 1 root root 18 Nov 13 2017 S05rc.local -> ../init.d/rc.local And we are using OpenLDAP as 3.1.1 it is the only option I believe.

[Logs](https://hartford0-my.sharepoint.com/:f:/g/personal/granholm_hartford_edu/EknHsPOnuM9FjTSdiWnjORMBbbPACnq3zOQWnX_t44ckrQ?e=P19a8a)

``` lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean -rw-r--r-- 1 root root 677 Feb 5 2016 README lrwxrwxrwx 1 root root 18 Nov 13 2017 S01identity -> ../init.d/identity lrwxrwxrwx 1 root root 13 Nov 13 2017 S01idp -> ../init.d/idp lrwxrwxrwx 1 root root 16 Nov 13 2017 S01oxauth -> ../init.d/oxauth lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron lrwxrwxrwx 1 root root 18 Nov 13 2017 S05ondemand -> ../init.d/ondemand lrwxrwxrwx 1 root root 18 Nov 13 2017 S05rc.local -> ../init.d/rc.local ``` This confirms my suspicion. As you can see `solserver` which is OpenLDAP is configured to start after (`S03`) `oxauth`, `idp` and `identity` (`S01`). That's the reason those services are failing to start, since they're dependent on OpenLDAP and `identity` requires `oxauth` and `idp` requires `oxauth` _and_ `identity`. In the link I sent there are examples on how to fix the start-up order, but let me provide you with a solution. It requires a simple modification to the `init.d` files for each service and running `update-rc.d $service_name defaults`. Modify the following `Required-Start` segment in each file: > /etc/init.d/oxauth ``` # Required-Start: $local_fs $network opendj ``` > /etc/init.d/identity ``` # Required-Start: $local_fs $network opendj oxauth ``` > /etc/init.d/idp ``` # Required-Start: $local_fs $network opendj oxauth identity ``` After that run `update-rc.d $service_name defaults` where `$service_name` is `oxauth`, then `identity`, then `idp`. > I am loathe to restart it as it is working and in about 3 hours we will be off until the new year. Happy holidays! We'll be here when you get back.

Oh also your new start order will looking something like this after you run the `update-rc.d` commands: ``` lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean -rw-r--r-- 1 root root 677 Feb 5 2016 README lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver lrwxrwxrwx 1 root root 16 Nov 13 2017 S04oxauth -> ../init.d/oxauth lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron lrwxrwxrwx 1 root root 18 Nov 13 2017 S05identity -> ../init.d/identity lrwxrwxrwx 1 root root 18 Nov 13 2017 S05ondemand -> ../init.d/ondemand lrwxrwxrwx 1 root root 18 Nov 13 2017 S05rc.local -> ../init.d/rc.local lrwxrwxrwx 1 root root 13 Nov 13 2017 S06idp -> ../init.d/idp ``` Check it again with `ls -l /etc/init.d/rc2.d/` (lower `Sxx` number equals first, higher ``Sxx`` number equals later.)

Do you mean it should be: `# Required-Start: $local_fs $network solserver` not opendj?

> Do you mean it should be: # Required-Start: $local_fs $network solserver not opendj? Yes. Sorry! I forgot to update for OpenLDAP/solserver

I am getting: ``` root@fed-auth-01:/etc/init.d# update-rc.d oxauth defaults insserv: Service solserver has to be enabled to start service oxauth insserv: exiting now! update-rc.d: error: insserv rejected the script header ``` The header looks like: ``` root@fed-auth-01:/etc/init.d# head oxauth #!/usr/bin/env bash # LSB Tags ### BEGIN INIT INFO # Provides: oxauth # Required-Start: $local_fs $network solserver # Required-Stop: $local_fs $network # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Jetty start script. ```

Can you show me the head of `/etc/init.d/solserver`?

``` #! /bin/sh # # Copyright (c) 2002-2015 by Symas Corporation # All rights reserved. # # The script will source the file pointed to by $SOL_CONF_FILE, # which can change any of several variables that control the # way this script behaves. # # LSB-style init information: ### BEGIN INIT INFO # Provides: slapd # Required-Start: $network $syslog # Required-Stop: $network $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Description: Symas OpenLDAP ### END INIT INFO ``` Just saw that. Does it need to be slapd?

Did it with slapd instead of opendj in your examples and now the /etc/rc2.d/ looks like: ``` drwxr-xr-x 2 root root 4096 Dec 21 18:53 . drwxr-xr-x 63 root root 4096 Apr 26 2018 .. lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean -rw-r--r-- 1 root root 677 Feb 5 2016 README lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron lrwxrwxrwx 1 root root 16 Dec 21 18:51 S04oxauth -> ../init.d/oxauth lrwxrwxrwx 1 root root 18 Dec 21 18:52 S05identity -> ../init.d/identity lrwxrwxrwx 1 root root 13 Dec 21 18:53 S06idp -> ../init.d/idp lrwxrwxrwx 1 root root 18 Dec 21 18:53 S07ondemand -> ../init.d/ondemand lrwxrwxrwx 1 root root 18 Dec 21 18:53 S07rc.local -> ../init.d/rc.local ```

Okay try `update-rc.d enable solserver` then `update-rc.d oxauth defaults` and so forth.

Ah okay the actual service name is `slapd` (# Provides: slapd) in the `/etc/init.d/solserver` LSB tags. My mistake. Everything looks good in your start order now.

I did the update and it is now in the order above. Just restarted the server to see if it will work. If not, I will drop back to the snapshot. Before I do that, if it is still giving me a 503, what files do you want to see?

Does not seem to be working. Still a 503 error. All of the services seem to be running.

Provide me all the logs from `/opt/gluu/jetty/identity/logs` and `/opt/gluu/jetty/oxauth/logs`. I need to confirm something. Looking at your other logs, `oxtrust_script.log` specifically, it looks like the SSL cert for OpenLDAP expired: ``` Caused by: com.unboundid.ldap.sdk.LDAPBindException: An error occurred while attempting to send the LDAP message to server localhost:1636: SSLHandshakeException(message='java.security.cert.CertificateExpiredException: NotAfter: Tue Nov 13 15:37:31 UTC 2018', trace='getSSLException(Alerts.java:192) / fatal(SSLSocketImpl.java:1949) / fatalSE(Handshaker.java:302) / fatalSE(Handshaker.java:296) / serverCertificate(ClientHandshaker.java:1509) / processMessage(ClientHandshaker.java:216) / processLoop(Handshaker.java:979) / process_record(Handshaker.java:914) / readRecord(SSLSocketImpl.java:1062) / performInitialHandshake(SSLSocketImpl.java:1375) / writeRecord(SSLSocketImpl.java:747) / write(AppOutputStream.java:123) / flushBuffer(BufferedOutputStream.java:82) / flush(BufferedOutputStream.java:140) / sendMessage(LDAPConnectionInternals.java:543) / sendMessage(LDAPConnection.java:4249) / process(SimpleBindRequest.java:551) / bind(LDAPConnection.java:2143) / createConnection(LDAPConnectionPool.java:1268) / createConnection(LDAPConnectionPool.java:1178) / getConnection(LDAPConnectionPool.java:1706) / search(AbstractConnectionPool.java:2012) / search(OperationsFacade.java:293) / findEntries(LdapEntryManager.java:357) / findEntries(LdapEntryManager.java:331) / findEntries(LdapEntryManager.java:323) / findEntries(LdapEntryManager.java:307) / findEntries(LdapEntryManager.java:303) / findEntries(null:unknown) / findCustomScripts(AbstractCustomScriptService.java:78) / reloadImpl(CustomScriptManager.java:149) / reload(CustomScriptManager.java:140) / reloadTimerEvent(CustomScriptManager.java:117) / reloadTimerEvent(null:unknown) / invoke(null:unknown) / invoke(DelegatingMethodAccessorImpl.java:43) / invoke(Method.java:498) / invoke(StaticMethodInjectionPoint.java:95) / invoke(StaticMethodInjectionPoint.java:85) / invoke(MethodInvocationStrategy.java:129) / sendEvent(ObserverMethodImpl.java:330) / sendEvent(ObserverMethodImpl.java:308) / notify(ObserverMethodImpl.java:286) / notify(ObserverMethod.java:124) / notify(Observers.java:166) / notifySyncObservers(ObserverNotifier.java:285) / notify(ObserverNotifier.java:273) / fireEvent(ObserverNotifier.java:177) / fireEvent(ObserverNotifier.java:159) / fireEvent(BeanManagerImpl.java:608) / fireEvent(ForwardingBeanManager.java:104) / execute(TimerJob.java:37) / execute(JobExecutionDelegate.java:29) / execute(null:unknown) / run(JobRunShell.java:202) / run(SimpleThreadPool.java:573)', cause=CertificateExpiredException(message='NotAfter: Tue Nov 13 15:37:31 UTC 2018', trace='valid(CertificateValidity.java:274) / checkValidity(X509CertImpl.java:629) / getTrustManagers(TrustStoreTrustManager.java:240) / checkServerTrusted(TrustStoreTrustManager.java:359) / checkServerTrusted(SSLContextImpl.java:984) / serverCertificate(ClientHandshaker.java:1491) / processMessage(ClientHandshaker.java:216) / processLoop(Handshaker.java:979) / process_record(Handshaker.java:914) / readRecord(SSLSocketImpl.java:1062) / performInitialHandshake(SSLSocketImpl.java:1375) / writeRecord(SSLSocketImpl.java:747) / write(AppOutputStream.java:123) / flushBuffer(BufferedOutputStream.java:82) / flush(BufferedOutputStream.java:140) / sendMessage(LDAPConnectionInternals.java:543) / sendMessage(LDAPConnection.java:4249) / process(SimpleBindRequest.java:551) / bind(LDAPConnection.java:2143) / createConnection(LDAPConnectionPool.java:1268) / createConnection(LDAPConnectionPool.java:1178) / getConnection(LDAPConnectionPool.java:1706) / search(AbstractConnectionPool.java:2012) / search(OperationsFacade.java:293) / findEntries(LdapEntryManager.java:357) / findEntries(LdapEntryManager.java:331) / findEntries(LdapEntryManager.java:323) / findEntries(LdapEntryManager.java:307) / findEntries(LdapEntryManager.java:303) / findEntries(null:unknown) / findCustomScripts(AbstractCustomScriptService.java:78) / reloadImpl(CustomScriptManager.java:149) / reload(CustomScriptManager.java:140) / reloadTimerEvent(CustomScriptManager.java:117) / reloadTimerEvent(null:unknown) / invoke(null:unknown) / invoke(DelegatingMethodAccessorImpl.java:43) / invoke(Method.java:498) / invoke(StaticMethodInjectionPoint.java:95) / invoke(StaticMethodInjectionPoint.java:85) / invoke(MethodInvocationStrategy.java:129) / sendEvent(ObserverMethodImpl.java:330) / sendEvent(ObserverMethodImpl.java:308) / notify(ObserverMethodImpl.java:286) / notify(ObserverMethod.java:124) / notify(Observers.java:166) / notifySyncObservers(ObserverNotifier.java:285) / notify(ObserverNotifier.java:273) / fireEvent(ObserverNotifier.java:177) / fireEvent(ObserverNotifier.java:159) / fireEvent(BeanManagerImpl.java:608) / fireEvent(ForwardingBeanManager.java:104) / execute(TimerJob.java:37) / execute(JobExecutionDelegate.java:29) / execute(null:unknown) / run(JobRunShell.java:202) / run(SimpleThreadPool.java:573)', revision=24201), revision=24201) ``` `SSLHandshakeException(message='java.security.cert.CertificateExpiredException: NotAfter: Tue Nov 13 15:37:31 UTC 2018'` specifically.

Is that a different cert then our webcert? https://fed-auth-01.hartford.edu As you can see, that is valid.

> Is that a different cert then our webcert? https://fed-auth-01.hartford.edu As you can see, that is valid. Yes, it's for the LDAP client, oxTrust, to access LDAP data over a TLS/SSL connection.

We have never purchased a cert for that. Could that be self signed? If so, is there any way to just renew it?

Correct me if I am wrong, but this is just a cert for it to talk to itself.

> Correct me if I am wrong, but this is just a cert for it to talk to itself. It's for a TLS connection from a client, oxTrust in this case, to access the LDAP data. > We have never purchased a cert for that. Could that be self signed? If so, is there any way to just renew it? Self-signed is fine.

Where do I find this particular cert?

So I have found the cert(s) and it would appear that everything has expired since Nov 13th of 2018, 1 year from when I built the server. Looks like every cert expired. Any way to bulk renew them all except the web cert?

Hello Ben, >> Any way to bulk renew them all except the web cert? No.

There are instructions on how to renew the apache cert, what about the other ones?

What if I run setup.py again on the server?

Hi Ben, You should only run setup.py during initial installation. You shouldn't run it again post installation. I'm going to close this ticket out for now. It's best to keep the tickets as specific as possible so they are easy for our staff and others to follow. If you have specific questions that you can't find answers for in the docs or elsewhere on support, please open a new ticket. Thanks, Will