By: Eddy DP user 21 Jul 2022 at 2:28 p.m. CDT

8 Responses
Eddy DP gravatar
We recently installed Shibboleth IDP service to our gluu server, we followed the related documentation and everything was fine with the installation process. Now our idp service is running and active but we are experiencing some server outages, after 2-3 hours that we start the idp service the server is not responding and we have noticed that the free memory gradually decreases everytime we log in to our Gluu dashboard. There comes a time that we can no longer log in via ssh to our gluu server and we need to restart the instance. We have stopped the idp service to test the server outage and it seems that everything is back to normal when the idp service is stopped. We have checked the logs and we have noticed that the following error is constantly repeated: ``` ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:273] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20220719053518.235Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.searchImpl(LdapOperationServiceImpl.java:415) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.5.11) /opt/opendj/db/userRoot Latch timeout. BIN38 currentThread: Thread[Worker Thread 12,5,Directory Server Thread Group] currentTime: 1658207718010 > at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3803) 2022-07-19 05:45:18,237 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:273] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20220719054518.236Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.searchImpl(LdapOperationServiceImpl.java:415) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.5.11) /opt/opendj/db/userRoot Latch timeout. BIN38 currentThread: Thread[Worker Thread 12,5,Directory Server Thread Group] currentTime: 1658207718010 > at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3803) 2022-07-19 05:55:18,238 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:273] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20220719055518.236Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.searchImpl(LdapOperationServiceImpl.java:415) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.5.11) /opt/opendj/db/userRoot Latch timeout. BIN38 currentThread: Thread[Worker Thread 12,5,Directory Server Thread Group] currentTime: 1658207718010 > at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3803) 2022-07-19 06:05:18,237 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:273] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20220719060518.236Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.searchImpl(LdapOperationServiceImpl.java:415) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.5.11) /opt/opendj/db/userRoot Latch timeout. BIN38 currentThread: Thread[Worker Thread 12,5,Directory Server Thread Group] currentTime: 1658207718010 > at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3803) 2022-07-19 06:15:18,238 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:273] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20220719061518.237Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.searchImpl(LdapOperationServiceImpl.java:415) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.5.11) /opt/opendj/db/userRoot Latch timeout. BIN38 currentThread: Thread[Worker Thread 12,5,Directory Server Thread Group] currentTime: 1658207718010 > at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3803) 2022-07-19 06:25:18,239 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:273] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20220719062518.237Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.searchImpl(LdapOperationServiceImpl.java:415) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: com.sleepycat.je.EnvironmentFailureException: (JE 7.5.11) Environment must be closed, caused by: com.sleepycat.je.EnvironmentFailureException: Environment invalid because of previous exception: (JE 7.5.11) /opt/opendj/db/userRoot Latch timeout. BIN38 currentThread: Thread[Worker Thread 12,5,Directory Server Thread Group] currentTime: 1658207718010 > at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3803) ``` Could you please help us to understand what is happening?

By Mobarak Hosen Shakil staff 21 Jul 2022 at 8:24 p.m. CDT

Mobarak Hosen Shakil gravatar
Hi Eddy, Can you please browse opendj server and check the `ou=cache` entries? You can use `Apache Directory Studio` . It seems there might be some issues for which server failed to clean `ou=cache` entries. Kindly check this similar ticket: https://support.gluu.org/maintenance/9486/server-clean-up-problems-oucache-ogluu/ Thanks, Regards ~ Shakil

By Eddy DP user 25 Jul 2022 at 5:39 p.m. CDT

Eddy DP gravatar
We already checked the ou=cache entries but we do not see any data inside of it, we have the IDP service active and running. When we stop the IDP service, our gluu server remains with a constant amount of free memory and everything is ok but once we start the IDP service, the free memory begins to decrease to the point where the server goes down. Here is a screenshot of cache entries ![](https://firebasestorage.googleapis.com/v0/b/mis-pruebas-361b1.appspot.com/o/evidenceidp.png?alt=media&token=f628650d-1ee0-45c6-8d9e-62bbfe926a5d) We have checked the log files and we can not find any error at this moment but if you want we can share with any log file. Thank you!

By Eddy DP user 26 Jul 2022 at 4:56 p.m. CDT

Eddy DP gravatar
After monitoring the service and decrease the xmx flag to 512MB on the IDP service script, we continue experiencing free memory issues and server outages, related to this, we can now see some cache entries using apache directory studio ![](https://firebasestorage.googleapis.com/v0/b/mis-pruebas-361b1.appspot.com/o/image.png?alt=media&token=3d996296-f86c-4de0-a3f8-66385cf9e107)

By Mobarak Hosen Shakil staff 30 Jul 2022 at 2:06 a.m. CDT

Mobarak Hosen Shakil gravatar
please run below command inside the `gluu-server` and share each of the status: ``` ps aux | grep java systemctl list-units --type=service ``` Is this happening for `Shibboleth-IDP`? Did you check `idp-process.log`. Just to be confirmed here you are using `gluu 4.2.3`, right?

By Mohib Zico staff 01 Aug 2022 at 10:54 a.m. CDT

Mohib Zico gravatar
>> We recently installed Shibboleth IDP service to our gluu server... Post installation of Shibboleth IDP into your existing Gluu Server?

By Eddy DP user 01 Aug 2022 at 5:03 p.m. CDT

Eddy DP gravatar
@Mohib.Zicoobarak Hosen.Shakil This is what we see in the logs and the commands you mention and yes we use gluu 4.2.3 px aux | grep java results: ``` ldap 388 1.4 17.5 4020216 706476 ? Sl 14:07 4:08 /opt/jre/binjava -server -Xms512m -Xmx1280m -XX:+UseCompressedOops -Dorg.opends.server.scriptName=start-ds org.opends.server.core.DirectoryServer --configFile /opt/opendj/config/config.ldif jetty 557 1.3 18.2 3521976 734660 ? Sl 14:08 4:00 /opt/jre/binjava -server -Xms128m -Xmx742m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/oxauth -Dlog.base=/opt/gluu/jetty/oxauth -Dpython.home=/opt/jython -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/oxauth -Djava.io.tmpdir=/opt/jetty-9.4/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8081 jetty.state=/opt/gluu/jetty/oxauth/oxauth.state jetty-started.xml jetty 719 0.8 9.5 2773600 382652 ? Sl 14:08 2:22 /opt/jre/binjava -server -Xms128m -Xmx148m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/scim -Dlog.base=/opt/gluu/jetty/scim -Dpython.home=/opt/jython -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/scim -Djava.io.tmpdir=/opt/jetty-9.4/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8087 jetty.state=/opt/gluu/jetty/scim/scim.state jetty-started.xml jetty 747 0.6 13.0 3192820 525404 ? Sl 14:08 1:50 /opt/jre/binjava -server -Xms128m -Xmx512m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/idp -Dlog.base=/opt/gluu/jetty/idp -Dpython.home=/opt/jython -Dorg.eclipse.jetty.server.Request.maxFormContentSize=50000000 -Djava.io.tmpdir=/opt/jetty-9.4/temp -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/idp -Djava.io.tmpdir=/opt/jetty-9.4/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8086 jetty.state=/opt/gluu/jetty/idp/idp.state jetty-started.xml jetty 750 1.6 22.1 3955532 890788 ? Sl 14:08 4:41 /opt/jre/binjava -server -Xms128m -Xmx1113m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/identity -Dlog.base=/opt/gluu/jetty/identity -Dpython.home=/opt/jython -Dorg.eclipse.jetty.server.Request.maxFormContentSize=50000000 -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/identity -Djava.io.tmpdir=/opt/jetty-9.4/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8082 jetty.state=/opt/gluu/jetty/identity/identity.state jetty-started.xml root 12692 0.0 0.0 5200 2368 pts/1 S+ 18:55 0:00 grep --color=auto java ``` systemctl list-units --type=service results: ``` UNIT LOAD ACTIVE SUB DESCRIPTION apache2.service loaded active running The Apache HTTP Server apparmor.service loaded active exited Load AppArmor profiles console-getty.service loaded active running Console Getty console-setup.service loaded active exited Set console font and keymap cron.service loaded active running Regular background program processing daemon dbus.service loaded active running D-Bus System Message Bus identity.service loaded active running Identity service idp.service loaded active running Idp service keyboard-setup.service loaded active exited Set the console keyboard layout networkd-dispatcher.service loaded active running Dispatcher daemon for systemd-networkd opendj.service loaded active running OpenDJ Directory Service oxauth.service loaded active running Oxauth service passport.service loaded active running Passport service postfix.service loaded active exited Postfix Mail Transport Agent postfix@-.service loaded active running Postfix Mail Transport Agent (instance -) rsyslog.service loaded active running System Logging Service scim.service loaded active running scim service snapd.apparmor.service loaded active exited Load AppArmor profiles managed internally by snapd snapd.seeded.service loaded active exited Wait until snapd is fully seeded ssh.service loaded active running OpenBSD Secure Shell server systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage systemd-journald.service loaded active running Journal Service systemd-logind.service loaded active running Login Service systemd-remount-fs.service loaded active exited Remount Root and Kernel File Systems systemd-resolved.service loaded active running Network Name Resolution systemd-sysusers.service loaded active exited Create System Users systemd-tmpfiles-setup-dev.service loaded active exited Create Static Device Nodes in /dev systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown systemd-user-sessions.service loaded active exited Permit User Sessions user-runtime-dir@0.service loaded active exited User Runtime Directory /run/user/0 user@0.service loaded active running User Manager for UID 0 ``` the following error repeats continuously on the idp-process.log file: ``` 2022-08-01 12:52:45,181 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:273] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=202208011252> at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.searchImpl(LdapOperationServiceImpl.java:415) Caused by: com.unboundid.ldap.sdk.LDAPException: An error occurred while attempting to connect to server localhost:1636: IOException(LDAPException(resultCode=91 (con> at com.unboundid.ldap.sdk.LDAPConnection.connect(LDAPConnection.java:875) Caused by: java.io.IOException: LDAPException(resultCode=91 (connect error), errorMessage='An error occurred while attempting to establish a connection to server loca> at com.unboundid.ldap.sdk.LDAPConnectionInternals.<init>(LDAPConnectionInternals.java:185) Caused by: com.unboundid.ldap.sdk.LDAPException: An error occurred while attempting to establish a connection to server localhost/127.0.0.1:1636: ConnectException(Co> at com.unboundid.ldap.sdk.ConnectThread.getConnectedSocket(ConnectThread.java:269) Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.base/java.net.PlainSocketImpl.socketConnect(Native Method) ``` Related to @M question: Yes, we added the Shibboleth IDP service to our existing Gluu S erver following the FAQ section.

By Mohib Zico staff 01 Aug 2022 at 9:50 p.m. CDT

Mohib Zico gravatar
Ok, thanks. Check `ldap.properties` values. See if LDAP connectivity is okay or not. Compare with a freshly installed ( which has Shibboleth ) 4.2.3 system.

By Eddy DP user 10 Aug 2022 at 9:46 a.m. CDT

Eddy DP gravatar
Hi Mohib, Do you want to see the values of our gluu-ldap.properties file? We have no way to compare the values with a new freshly installed system. Could you please (if it is possible) share to us an example of these values on a new system integrated with Shibboleth? Thank you!