By: Ben Granholm user 21 Dec 2018 at 9:55 a.m. CST

33 Responses
Ben Granholm gravatar
On reboot we lost the ability to use the gluu server for login. Restarting the gluu service or the individual services in the gluu container does not fix the issue.

By Ben Granholm user 21 Dec 2018 at 9:56 a.m. CST

Ben Granholm gravatar
https://fed-auth-01.hartford.edu If you would like to see the behavior.

By Ben Granholm user 21 Dec 2018 at 10:19 a.m. CST

Ben Granholm gravatar
https://hartford0-my.sharepoint.com/:f:/g/personal/granholm_hartford_edu/EjmAMyzEofFEibZYTYsMvXEBdiO34x45u_NWEXbnEEOR1A?e=2zhzlM Link to both oxauth.log and oxtrust.log

By Chris Blanton staff 21 Dec 2018 at 10:57 a.m. CST

Chris Blanton gravatar
Ben, It looks like the services oxTrust and oxAuth are having connectivity issued to LDAP. Can you send all the logs from both oxAuth and oxTrust/Identity?

By Ben Granholm user 21 Dec 2018 at 10:58 a.m. CST

Ben Granholm gravatar
So I got it back up and running with a revert to a snapshot. However, if I reboot it the issue crops up again.

By Ben Granholm user 21 Dec 2018 at 11:03 a.m. CST

Ben Granholm gravatar
I have posted the logs to the link in the post above yours. This didn't appear to have a way to attach files.

By Chris Blanton staff 21 Dec 2018 at 11:09 a.m. CST

Chris Blanton gravatar
> I have posted the logs to the link in the post above yours. This didn't appear to have a way to attach files. Right. It only seemed to be the specific `oxauth.log` and `oxtrust.log`. There are a lot of logs in `/opt/gluu/jetty/$service/logs/` that would help me nail down the issue. Can you also run `ls -l /etc/init.d/rc2.d/` inside the chroot for me? It might be a service start-up order issue. We found an issue with that recently https://github.com/GluuFederation/community-edition-setup/issues/499 . That being said, your logs stopping here: ``` 2018-12-21 15:47:41,221 WARN [weld-worker-2] [org.jboss.weld.bootstrap.Validator] (Validator.java:443) - WELD-001440: Scope type @javax.enterprise.context.ApplicationScoped() used on injection point [UnbackedAnnotatedField] @Inject @ApplicationScoped private org.xdi.service.cache.CacheProviderFactory.instance at org.xdi.service.cache.CacheProviderFactory.instance(CacheProviderFactory.java:0) StackTrace 2018-12-21 15:47:41,561 INFO [main] [org.xdi.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:23) - Adding Bouncy Castle Provider ``` and here: ``` 2018-12-21 15:48:18,253 INFO [main] [org.xdi.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:23) - Adding Bouncy Castle Provider 2018-12-21 15:48:18,330 INFO [main] [org.gluu.oxtrust.ldap.service.AppInitializer] (AppInitializer.java:275) - Build date null. Code revision null on null. Build null 2018-12-21 15:48:18,339 INFO [main] [org.gluu.oxtrust.config.ConfigurationFactory] (ConfigurationFactory.java:134) - Creating oxTrustConfiguration 2018-12-21 15:48:18,339 INFO [main] [org.gluu.oxtrust.config.ConfigurationFactory] (ConfigurationFactory.java:420) - ########## ldapFileName = /etc/gluu/conf/ox-ldap.properties ``` Indicate that they can't access the local LDAP to pull their configuration and are halting because of it. Presumably it's a start-up order issue, but something else might be wrong with your LDAP. Are you using OpenDJ or OpenLDAP?

By Chris Blanton staff 21 Dec 2018 at 11:10 a.m. CST

Chris Blanton gravatar
> This didn't appear to have a way to attach files. Community support doesn't allow attaching files. If you could `tar.gz` or zip the files on your sharepoint, I'll happily review them for you.

By Ben Granholm user 21 Dec 2018 at 11:18 a.m. CST

Ben Granholm gravatar
Now that I have reverted to the snapshot, it appears to be working. I am not sure the current logs will help you. I will upload them but I am not sure how much they will help. I am loathe to restart it as it is working and in about 3 hours we will be off until the new year. If the logs don't help you, we can revisit this in the new year. I will upload the logs shortly and put them in the folder above. Here is the output from the container's rc2.d: - lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean - -rw-r--r-- 1 root root 677 Feb 5 2016 README - lrwxrwxrwx 1 root root 18 Nov 13 2017 S01identity -> ../init.d/identity - lrwxrwxrwx 1 root root 13 Nov 13 2017 S01idp -> ../init.d/idp - lrwxrwxrwx 1 root root 16 Nov 13 2017 S01oxauth -> ../init.d/oxauth - lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog - lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 - lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus - lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached - lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync - lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver - lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron - lrwxrwxrwx 1 root root 18 Nov 13 2017 S05ondemand -> ../init.d/ondemand - lrwxrwxrwx 1 root root 18 Nov 13 2017 S05rc.local -> ../init.d/rc.local And we are using OpenLDAP as 3.1.1 it is the only option I believe.

By Ben Granholm user 21 Dec 2018 at 11:33 a.m. CST

Ben Granholm gravatar
[Logs](https://hartford0-my.sharepoint.com/:f:/g/personal/granholm_hartford_edu/EknHsPOnuM9FjTSdiWnjORMBbbPACnq3zOQWnX_t44ckrQ?e=P19a8a)

By Chris Blanton staff 21 Dec 2018 at 12:03 p.m. CST

Chris Blanton gravatar
``` lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean -rw-r--r-- 1 root root 677 Feb 5 2016 README lrwxrwxrwx 1 root root 18 Nov 13 2017 S01identity -> ../init.d/identity lrwxrwxrwx 1 root root 13 Nov 13 2017 S01idp -> ../init.d/idp lrwxrwxrwx 1 root root 16 Nov 13 2017 S01oxauth -> ../init.d/oxauth lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron lrwxrwxrwx 1 root root 18 Nov 13 2017 S05ondemand -> ../init.d/ondemand lrwxrwxrwx 1 root root 18 Nov 13 2017 S05rc.local -> ../init.d/rc.local ``` This confirms my suspicion. As you can see `solserver` which is OpenLDAP is configured to start after (`S03`) `oxauth`, `idp` and `identity` (`S01`). That's the reason those services are failing to start, since they're dependent on OpenLDAP and `identity` requires `oxauth` and `idp` requires `oxauth` _and_ `identity`. In the link I sent there are examples on how to fix the start-up order, but let me provide you with a solution. It requires a simple modification to the `init.d` files for each service and running `update-rc.d $service_name defaults`. Modify the following `Required-Start` segment in each file: > /etc/init.d/oxauth ``` # Required-Start: $local_fs $network opendj ``` > /etc/init.d/identity ``` # Required-Start: $local_fs $network opendj oxauth ``` > /etc/init.d/idp ``` # Required-Start: $local_fs $network opendj oxauth identity ``` After that run `update-rc.d $service_name defaults` where `$service_name` is `oxauth`, then `identity`, then `idp`. > I am loathe to restart it as it is working and in about 3 hours we will be off until the new year. Happy holidays! We'll be here when you get back.

By Chris Blanton staff 21 Dec 2018 at 12:07 p.m. CST

Chris Blanton gravatar
Oh also your new start order will looking something like this after you run the `update-rc.d` commands: ``` lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean -rw-r--r-- 1 root root 677 Feb 5 2016 README lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver lrwxrwxrwx 1 root root 16 Nov 13 2017 S04oxauth -> ../init.d/oxauth lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron lrwxrwxrwx 1 root root 18 Nov 13 2017 S05identity -> ../init.d/identity lrwxrwxrwx 1 root root 18 Nov 13 2017 S05ondemand -> ../init.d/ondemand lrwxrwxrwx 1 root root 18 Nov 13 2017 S05rc.local -> ../init.d/rc.local lrwxrwxrwx 1 root root 13 Nov 13 2017 S06idp -> ../init.d/idp ``` Check it again with `ls -l /etc/init.d/rc2.d/` (lower `Sxx` number equals first, higher ``Sxx`` number equals later.)

By Ben Granholm user 21 Dec 2018 at 12:32 p.m. CST

Ben Granholm gravatar
Do you mean it should be: `# Required-Start: $local_fs $network solserver` not opendj?

By Chris Blanton staff 21 Dec 2018 at 12:35 p.m. CST

Chris Blanton gravatar
> Do you mean it should be: # Required-Start: $local_fs $network solserver not opendj? Yes. Sorry! I forgot to update for OpenLDAP/solserver

By Ben Granholm user 21 Dec 2018 at 12:39 p.m. CST

Ben Granholm gravatar
I am getting: ``` root@fed-auth-01:/etc/init.d# update-rc.d oxauth defaults insserv: Service solserver has to be enabled to start service oxauth insserv: exiting now! update-rc.d: error: insserv rejected the script header ``` The header looks like: ``` root@fed-auth-01:/etc/init.d# head oxauth #!/usr/bin/env bash # LSB Tags ### BEGIN INIT INFO # Provides: oxauth # Required-Start: $local_fs $network solserver # Required-Stop: $local_fs $network # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Jetty start script. ```

By Chris Blanton staff 21 Dec 2018 at 12:49 p.m. CST

Chris Blanton gravatar
Can you show me the head of `/etc/init.d/solserver`?

By Ben Granholm user 21 Dec 2018 at 12:51 p.m. CST

Ben Granholm gravatar
``` #! /bin/sh # # Copyright (c) 2002-2015 by Symas Corporation # All rights reserved. # # The script will source the file pointed to by $SOL_CONF_FILE, # which can change any of several variables that control the # way this script behaves. # # LSB-style init information: ### BEGIN INIT INFO # Provides: slapd # Required-Start: $network $syslog # Required-Stop: $network $syslog # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Description: Symas OpenLDAP ### END INIT INFO ``` Just saw that. Does it need to be slapd?

By Ben Granholm user 21 Dec 2018 at 12:54 p.m. CST

Ben Granholm gravatar
Did it with slapd instead of opendj in your examples and now the /etc/rc2.d/ looks like: ``` drwxr-xr-x 2 root root 4096 Dec 21 18:53 . drwxr-xr-x 63 root root 4096 Apr 26 2018 .. lrwxrwxrwx 1 root root 29 Nov 28 2016 K01apache-htcacheclean -> ../init.d/apache-htcacheclean -rw-r--r-- 1 root root 677 Feb 5 2016 README lrwxrwxrwx 1 root root 17 Nov 28 2016 S01rsyslog -> ../init.d/rsyslog lrwxrwxrwx 1 root root 17 Nov 28 2016 S02apache2 -> ../init.d/apache2 lrwxrwxrwx 1 root root 14 Nov 28 2016 S03dbus -> ../init.d/dbus lrwxrwxrwx 1 root root 19 Nov 28 2016 S03memcached -> ../init.d/memcached lrwxrwxrwx 1 root root 15 Feb 3 2017 S03rsync -> ../init.d/rsync lrwxrwxrwx 1 root root 19 Nov 13 2017 S03solserver -> ../init.d/solserver lrwxrwxrwx 1 root root 14 Nov 13 2017 S04cron -> ../init.d/cron lrwxrwxrwx 1 root root 16 Dec 21 18:51 S04oxauth -> ../init.d/oxauth lrwxrwxrwx 1 root root 18 Dec 21 18:52 S05identity -> ../init.d/identity lrwxrwxrwx 1 root root 13 Dec 21 18:53 S06idp -> ../init.d/idp lrwxrwxrwx 1 root root 18 Dec 21 18:53 S07ondemand -> ../init.d/ondemand lrwxrwxrwx 1 root root 18 Dec 21 18:53 S07rc.local -> ../init.d/rc.local ```

By Chris Blanton staff 21 Dec 2018 at 12:54 p.m. CST

Chris Blanton gravatar
Okay try `update-rc.d enable solserver` then `update-rc.d oxauth defaults` and so forth.

By Chris Blanton staff 21 Dec 2018 at 12:56 p.m. CST

Chris Blanton gravatar
Ah okay the actual service name is `slapd` (# Provides: slapd) in the `/etc/init.d/solserver` LSB tags. My mistake. Everything looks good in your start order now.

By Ben Granholm user 21 Dec 2018 at 12:58 p.m. CST

Ben Granholm gravatar
I did the update and it is now in the order above. Just restarted the server to see if it will work. If not, I will drop back to the snapshot. Before I do that, if it is still giving me a 503, what files do you want to see?

By Ben Granholm user 21 Dec 2018 at 12:59 p.m. CST

Ben Granholm gravatar
Does not seem to be working. Still a 503 error. All of the services seem to be running.

By Chris Blanton staff 21 Dec 2018 at 1:06 p.m. CST

Chris Blanton gravatar
Provide me all the logs from `/opt/gluu/jetty/identity/logs` and `/opt/gluu/jetty/oxauth/logs`. I need to confirm something. Looking at your other logs, `oxtrust_script.log` specifically, it looks like the SSL cert for OpenLDAP expired: ``` Caused by: com.unboundid.ldap.sdk.LDAPBindException: An error occurred while attempting to send the LDAP message to server localhost:1636: SSLHandshakeException(message='java.security.cert.CertificateExpiredException: NotAfter: Tue Nov 13 15:37:31 UTC 2018', trace='getSSLException(Alerts.java:192) / fatal(SSLSocketImpl.java:1949) / fatalSE(Handshaker.java:302) / fatalSE(Handshaker.java:296) / serverCertificate(ClientHandshaker.java:1509) / processMessage(ClientHandshaker.java:216) / processLoop(Handshaker.java:979) / process_record(Handshaker.java:914) / readRecord(SSLSocketImpl.java:1062) / performInitialHandshake(SSLSocketImpl.java:1375) / writeRecord(SSLSocketImpl.java:747) / write(AppOutputStream.java:123) / flushBuffer(BufferedOutputStream.java:82) / flush(BufferedOutputStream.java:140) / sendMessage(LDAPConnectionInternals.java:543) / sendMessage(LDAPConnection.java:4249) / process(SimpleBindRequest.java:551) / bind(LDAPConnection.java:2143) / createConnection(LDAPConnectionPool.java:1268) / createConnection(LDAPConnectionPool.java:1178) / getConnection(LDAPConnectionPool.java:1706) / search(AbstractConnectionPool.java:2012) / search(OperationsFacade.java:293) / findEntries(LdapEntryManager.java:357) / findEntries(LdapEntryManager.java:331) / findEntries(LdapEntryManager.java:323) / findEntries(LdapEntryManager.java:307) / findEntries(LdapEntryManager.java:303) / findEntries(null:unknown) / findCustomScripts(AbstractCustomScriptService.java:78) / reloadImpl(CustomScriptManager.java:149) / reload(CustomScriptManager.java:140) / reloadTimerEvent(CustomScriptManager.java:117) / reloadTimerEvent(null:unknown) / invoke(null:unknown) / invoke(DelegatingMethodAccessorImpl.java:43) / invoke(Method.java:498) / invoke(StaticMethodInjectionPoint.java:95) / invoke(StaticMethodInjectionPoint.java:85) / invoke(MethodInvocationStrategy.java:129) / sendEvent(ObserverMethodImpl.java:330) / sendEvent(ObserverMethodImpl.java:308) / notify(ObserverMethodImpl.java:286) / notify(ObserverMethod.java:124) / notify(Observers.java:166) / notifySyncObservers(ObserverNotifier.java:285) / notify(ObserverNotifier.java:273) / fireEvent(ObserverNotifier.java:177) / fireEvent(ObserverNotifier.java:159) / fireEvent(BeanManagerImpl.java:608) / fireEvent(ForwardingBeanManager.java:104) / execute(TimerJob.java:37) / execute(JobExecutionDelegate.java:29) / execute(null:unknown) / run(JobRunShell.java:202) / run(SimpleThreadPool.java:573)', cause=CertificateExpiredException(message='NotAfter: Tue Nov 13 15:37:31 UTC 2018', trace='valid(CertificateValidity.java:274) / checkValidity(X509CertImpl.java:629) / getTrustManagers(TrustStoreTrustManager.java:240) / checkServerTrusted(TrustStoreTrustManager.java:359) / checkServerTrusted(SSLContextImpl.java:984) / serverCertificate(ClientHandshaker.java:1491) / processMessage(ClientHandshaker.java:216) / processLoop(Handshaker.java:979) / process_record(Handshaker.java:914) / readRecord(SSLSocketImpl.java:1062) / performInitialHandshake(SSLSocketImpl.java:1375) / writeRecord(SSLSocketImpl.java:747) / write(AppOutputStream.java:123) / flushBuffer(BufferedOutputStream.java:82) / flush(BufferedOutputStream.java:140) / sendMessage(LDAPConnectionInternals.java:543) / sendMessage(LDAPConnection.java:4249) / process(SimpleBindRequest.java:551) / bind(LDAPConnection.java:2143) / createConnection(LDAPConnectionPool.java:1268) / createConnection(LDAPConnectionPool.java:1178) / getConnection(LDAPConnectionPool.java:1706) / search(AbstractConnectionPool.java:2012) / search(OperationsFacade.java:293) / findEntries(LdapEntryManager.java:357) / findEntries(LdapEntryManager.java:331) / findEntries(LdapEntryManager.java:323) / findEntries(LdapEntryManager.java:307) / findEntries(LdapEntryManager.java:303) / findEntries(null:unknown) / findCustomScripts(AbstractCustomScriptService.java:78) / reloadImpl(CustomScriptManager.java:149) / reload(CustomScriptManager.java:140) / reloadTimerEvent(CustomScriptManager.java:117) / reloadTimerEvent(null:unknown) / invoke(null:unknown) / invoke(DelegatingMethodAccessorImpl.java:43) / invoke(Method.java:498) / invoke(StaticMethodInjectionPoint.java:95) / invoke(StaticMethodInjectionPoint.java:85) / invoke(MethodInvocationStrategy.java:129) / sendEvent(ObserverMethodImpl.java:330) / sendEvent(ObserverMethodImpl.java:308) / notify(ObserverMethodImpl.java:286) / notify(ObserverMethod.java:124) / notify(Observers.java:166) / notifySyncObservers(ObserverNotifier.java:285) / notify(ObserverNotifier.java:273) / fireEvent(ObserverNotifier.java:177) / fireEvent(ObserverNotifier.java:159) / fireEvent(BeanManagerImpl.java:608) / fireEvent(ForwardingBeanManager.java:104) / execute(TimerJob.java:37) / execute(JobExecutionDelegate.java:29) / execute(null:unknown) / run(JobRunShell.java:202) / run(SimpleThreadPool.java:573)', revision=24201), revision=24201) ``` `SSLHandshakeException(message='java.security.cert.CertificateExpiredException: NotAfter: Tue Nov 13 15:37:31 UTC 2018'` specifically.

By Ben Granholm user 21 Dec 2018 at 1:07 p.m. CST

Ben Granholm gravatar
Is that a different cert then our webcert? https://fed-auth-01.hartford.edu As you can see, that is valid.

By Chris Blanton staff 21 Dec 2018 at 1:08 p.m. CST

Chris Blanton gravatar
> Is that a different cert then our webcert? https://fed-auth-01.hartford.edu As you can see, that is valid. Yes, it's for the LDAP client, oxTrust, to access LDAP data over a TLS/SSL connection.

By Ben Granholm user 21 Dec 2018 at 1:13 p.m. CST

Ben Granholm gravatar
We have never purchased a cert for that. Could that be self signed? If so, is there any way to just renew it?

By Ben Granholm user 21 Dec 2018 at 1:15 p.m. CST

Ben Granholm gravatar
Correct me if I am wrong, but this is just a cert for it to talk to itself.

By Chris Blanton staff 21 Dec 2018 at 1:18 p.m. CST

Chris Blanton gravatar
> Correct me if I am wrong, but this is just a cert for it to talk to itself. It's for a TLS connection from a client, oxTrust in this case, to access the LDAP data. > We have never purchased a cert for that. Could that be self signed? If so, is there any way to just renew it? Self-signed is fine.

By Ben Granholm user 21 Dec 2018 at 1:21 p.m. CST

Ben Granholm gravatar
Where do I find this particular cert?

By Ben Granholm user 21 Dec 2018 at 1:40 p.m. CST

Ben Granholm gravatar
So I have found the cert(s) and it would appear that everything has expired since Nov 13th of 2018, 1 year from when I built the server. Looks like every cert expired. Any way to bulk renew them all except the web cert?

By Mohib Zico staff 02 Jan 2019 at 8:02 a.m. CST

Mohib Zico gravatar
Hello Ben, >> Any way to bulk renew them all except the web cert? No.

By Ben Granholm user 02 Jan 2019 at 8:03 a.m. CST

Ben Granholm gravatar
There are instructions on how to renew the apache cert, what about the other ones?

By Ben Granholm user 02 Jan 2019 at 2:06 p.m. CST

Ben Granholm gravatar
What if I run setup.py again on the server?

By William Lowe staff 03 Jan 2019 at 1:45 a.m. CST

William Lowe gravatar
Hi Ben, You should only run setup.py during initial installation. You shouldn't run it again post installation. I'm going to close this ticket out for now. It's best to keep the tickets as specific as possible so they are easy for our staff and others to follow. If you have specific questions that you can't find answers for in the docs or elsewhere on support, please open a new ticket. Thanks, Will