Unexplained OutOfMemoryError periodically invalidates LDAP environment

By: Zachary Doyle user 13 Dec 2020 at 6 p.m. CST

17 Responses
Zachary Doyle gravatar
## Expected behavior OpenDJ remains stable on a largely unused production server, if not using the default memory allocation, then with the generous 5GB heap suggested to Mr. Täffner in his similar ticket [here](https://support.gluu.org/outages/8775/using-421-ldap-server-outofmemoryerror-java-heap-space/). ## Actual behavior After retuning OpenDJ's -Xmx and -Xms to 5GB (OpenDJ's documentation suggested that the min and max heap size be equal, to avoid costly heap resize operations), the OpenDJ service began throwing back OutOfMemoryErrors to Identity/Oxtrust when we attempted to login to the Admin UI, ultimately leading to a 503 being served. The identity and opendj services remain seemingly healthy, and the actual OOM event seems to be transient. However, the LDAP environment is left unusable, with all connection attempts on port 1636 yielding a com.sleepycat.je.EnvironmentFailureException (for /opt/opendj/db/userRoot)until opendj.service is restarted, rendering much if not all gluu server functionality unavailable. ## Config While I cannot guarantee this config description is sufficient to reproduce, I believe it does satisfy the system requirements. Specifically it is a t3.xlarge aws ec2 instance in us-east-2 - 16GB RAM, 4 vCPUs. The following modules were selected at install time: 1. identity 2. oxauth 3. opendj 4. apache2 5. scim 6. oxd 7. shibboleth 8. passport 9. oxauth-rp ## Steps to reproduce We have not succeeded in deliberately reproducing the OpenDJ failure, however, it has occured twice since this server's deployment was completed - 12-02 and 12-07, which is in line with the 5-6 day interval reported on other tickets. Attached is the opendj and identity/oxtrust logs containing the actual environmental corruption error + stack trace. Other services' logs are filled with less informative stack traces with an inability to hit 1636 at all due to timeout. /opt/opendj/logs/error: ``` [08/Dec/2020:02:48:27 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space () [08/Dec/2020:02:48:27 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [08/Dec/2020:02:48:27 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 6 encountered an uncaught exception while processing operation SearchOperation(connID=16146, opID=49, baseDN=ou=configuration,o=gluu, scope=base, filter=(objectClass=*)): OutOfMemoryError: Java heap space () [08/Dec/2020:02:48:27 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Error Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [08/Dec/2020:02:48:27 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Error Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [08/Dec/2020:02:48:27 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(3) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [08/Dec/2020:02:48:27 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(3) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) ``` /opt/gluu/jetty/identity/logs/oxtrust_cache_refresh_12-08-01.log: ``` 2020-12-08 02:48:27,468 ERROR [ForkJoinPool.commonPool-worker-3] [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (CacheRefreshTimer.java:205) - Exception happened while executing cache refresh synchronization org.gluu.persist.exception.EntryPersistenceException: Failed to find entry: ou=configuration,o=gluu at org.gluu.persist.ldap.impl.LdapEntryManager.find(LdapEntryManager.java:409) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.persist.impl.BaseEntryManager.find(BaseEntryManager.java:624) ~[oxcore-persistence-core-4.2.1.Final.jar:?] at org.gluu.persist.impl.BaseEntryManager.find(BaseEntryManager.java:504) ~[oxcore-persistence-core-4.2.1.Final.jar:?] at jdk.internal.reflect.GeneratedMethodAccessor100.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.jboss.weld.bean.proxy.AbstractBeanInstance.invoke(AbstractBeanInstance.java:38) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.gluu.persist.EntityManager$PersistenceEntryManager$1037559104$Proxy$_$$_WeldClientProxy.find(Unknown Source) ~[weld-core-impl-3.1.4.Final.jar:?] at org.gluu.oxtrust.service.ConfigurationService.getConfiguration(ConfigurationService.java:125) ~[oxtrust-service-4.2.1.Final.jar:?] at org.gluu.oxtrust.service.ConfigurationService.getConfiguration(ConfigurationService.java:148) ~[oxtrust-service-4.2.1.Final.jar:?] at org.gluu.oxtrust.service.ConfigurationService$Proxy$_$$_WeldClientProxy.getConfiguration(Unknown Source) ~[oxtrust-service-4.2.1.Final.jar:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.processInt(CacheRefreshTimer.java:194) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer$Proxy$_$$_WeldSubclass.processInt(Unknown Source) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.process(CacheRefreshTimer.java:184) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer$Proxy$_$$_WeldSubclass.process$$super(Unknown Source) [classes/:?] at jdk.internal.reflect.GeneratedMethodAccessor320.invoke(Unknown Source) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.gluu.service.cdi.async.AsynchronousInterceptor$1.get(AsynchronousInterceptor.java:36) [oxcore-service-4.2.1.Final.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?] at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1692) [?:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?] at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?] at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) [?:?] Caused by: org.gluu.persist.exception.operation.ConnectionException: Failed to lookup entry at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.lookupImpl(LdapOperationServiceImpl.java:698) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.lookup(LdapOperationServiceImpl.java:682) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.persist.ldap.impl.LdapEntryManager.find(LdapEntryManager.java:403) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] ... 28 more Caused by: com.unboundid.ldap.sdk.LDAPSearchException: Worker Thread 6 encountered an uncaught exception while processing operation SearchOperation(connID=16146, opID=49, baseDN=ou=configuration,o=gluu, scope=base, filter=(objectClass=*)): OutOfMemoryError: Java heap space () at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3833) ~[unboundid-ldapsdk-5.1.0.jar:5.1.0] at com.unboundid.ldap.sdk.LDAPConnection.getEntry(LDAPConnection.java:1830) ~[unboundid-ldapsdk-5.1.0.jar:5.1.0] at com.unboundid.ldap.sdk.AbstractConnectionPool.getEntry(AbstractConnectionPool.java:630) ~[unboundid-ldapsdk-5.1.0.jar:5.1.0] at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.lookupImpl(LdapOperationServiceImpl.java:695) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.lookup(LdapOperationServiceImpl.java:682) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.persist.ldap.impl.LdapEntryManager.find(LdapEntryManager.java:403) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] ... 28 more ```
Gluu 4.2.1 Ubuntu 18.04
closed

Answers

By Mohib Zico Account Admin 16 Dec 2020 at 1 a.m. CST

Copy
Mohib Zico gravatar
We will try to build such environment and let it run for one week. Let's see how things go. Will update you.

By Zachary Doyle user 24 Dec 2020 at 8:35 a.m. CST

Copy
Zachary Doyle gravatar
Just an update, issue just reoccured on the same server for the first time since 12-07 (temporarily resolved by host, opendj, and identity restart). OpenDJ log: ``` [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 2 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 9 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 5 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 4 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LdapResponseMessageWriter.java:53 LdapResponseMessageWriter.java:28 FlowableMap.java:67 FlowableDoOnEach.java:89 LDAPClientConnection2.java:1754 LDAPClientConnection2.java:1804 LDAPClientConnection2.java:1679 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 4 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LdapResponseMessageWriter.java:53 LdapResponseMessageWriter.java:28 FlowableMap.java:67 FlowableDoOnEach.java:89 LDAPClientConnection2.java:1754 LDAPClientConnection2.java:1804 LDAPClientConnection2.java:1679 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 1 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 1 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 3 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 3 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 0 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 0 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 8 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 8 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 1 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 3 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 3 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(4) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 1 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 7 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(5) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(3) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 10 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 13 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 6 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Asynchronous Text Writer for cn=File-Based Access Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Asynchronous Text Writer for cn=File-Based Access Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(4) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Access Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Access Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 12 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 8 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 8 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 11 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 0 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 0 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 14 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(3) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(5) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(5) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(5) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Dec/2020:11:57:05 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 15 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) ```

By Zachary Doyle user 26 Dec 2020 at 8:56 p.m. CST

Copy
Zachary Doyle gravatar
by the way, had a quick clarification question regarding the cache-cleaning-by-LDAP-restore method documented [here](https://gluu.org/docs/gluu-server/4.2/operation/backup/#opendj) - this approach is part of the 4.2 docs, but the [issue](https://www.gluu.org/blog/managing-cache-in-the-gluu-server/) it links to as explanation is described as being fixed in 3.1.6. Is (possibly regular) cache filtered export/import still a possible fix for OutOfMemory errors in OpenDJ in Gluu 4.2.1? I haven't tried it yet. thanks, zach

By Mohib Zico Account Admin 26 Dec 2020 at 11:22 p.m. CST

Copy
Mohib Zico gravatar
Hi Zachary, I have built a similar environment ( but seems like it's installed a 4.2.2 as default ) in AWS and will keep it running till 7th Jan, 2021. Let's see if we face any issue or not. One quick question... did you face any question like attached one during installation of oxD? >> this approach is part of the 4.2 docs, but the issue it links to as explanation is described as being fixed in 3.1.6. Sorry, I haven't tried that either.
oxdLayer.PNG

By Zachary Doyle user 27 Dec 2020 at 11:21 a.m. CST

Copy
Zachary Doyle gravatar
Thanks for the quick response Mohib! and yes, we got that prompt and gave oxD its out-of-the-box h2 database.

By Mohib Zico Account Admin 28 Dec 2020 at 7:29 a.m. CST

Copy
Mohib Zico gravatar
Sure! Thanks much for your help. Just to keep an update. Just checked a bit and got my identity a 503. Screenshot attached. I adjusted file descriptor and rebooted server. Let's see how things go.
Dec28_629AM_ldap_error.PNG Dec28_629AM_identity_OOME.PNG

By Zachary Doyle user 29 Jan 2021 at 11:52 a.m. CST

Copy
Zachary Doyle gravatar
Hey Mohib, just want to update you on where our environment is at. After being stable for most of January, we got an OOME on the evening of the 19th, which I believe was precipitated by a cluster of host reboots we did while troubleshooting a cert problem. This reoccured on the 27th - unsure of the cause - but prompted me to disable both oxtrust logging and metrics, as per [this guide](https://gluu.org/docs/gluu-server/4.2/operation/fine-tuning/). I'll keep you posted as to how stable that got us. Thanks again for all your assistance. -zach

By Mohib Zico Account Admin 08 Feb 2021 at 1:13 a.m. CST

Copy
Mohib Zico gravatar
Hi Zach, Just fyi... we released 4.2.3 which has some memory leak fixes. Thanks much for your help and report on your Gluu instance.

By Zachary Doyle user 10 Feb 2021 at 8:52 a.m. CST

Copy
Zachary Doyle gravatar
Awesome! We're gonna give this upgrade a try today on a staging instance. Will let you know if there are any problems. Thanks again Mohib.

By Zachary Doyle user 10 Feb 2021 at 8:53 a.m. CST

Copy
Zachary Doyle gravatar
Is there a 4.2.1 -> 4.2.3 upgrade script by any chance? edit- found it: https://raw.githubusercontent.com/GluuFederation/community-edition-package/master/update/4.2.3/upg4xto423.py

By Mohib Zico Account Admin 20 Feb 2021 at 11:08 p.m. CST

Copy
Mohib Zico gravatar
>> https://raw.githubusercontent.com/GluuFederation/community-edition-package/master/update/4.2.3/upg4xto423.py Aye, that one. Apologies for late response. Couldn't manage much time for community issues in last week.

By Zachary Doyle user 21 Feb 2021 at 8:05 a.m. CST

Copy
Zachary Doyle gravatar
No worries at all- we did get the chance to try the 4.2.3 upgrade on a staging environment, and it looks good, with one issue- the sector identifier section of openid config in the adminui is missing. Our clients still have their SI ids, though- do you know if that's been moved elsewhere in the UI? Can get screenshots if I'm not describing this well. Many thanks for your continued support. zach

By Mohib Zico Account Admin 21 Feb 2021 at 8:30 a.m. CST

Copy
Mohib Zico gravatar
Thanks for your update, Zachary. >> with one issue- the sector identifier section of openid config in the adminui is missing. Would you mind to create a new ticket on that? I'll assign one of our oxTrust Dev there in that ticket for confirmation.

By Zachary Doyle user 21 Feb 2021 at 9:28 a.m. CST

Copy
Zachary Doyle gravatar
Done- new ticket is [here](https://support.gluu.org/upgrade/9430/missing-sector-identifier-section-of-admin-ui/). I'll go ahead and close this ticket out; thanks for all your help Mohib. Have a good sunday!

By Mohib Zico Account Admin 21 Feb 2021 at 9:33 a.m. CST

Copy
Mohib Zico gravatar
Thanks, received and assigned. You too have a great weekend!

By Levente Kintzel user 06 Apr 2021 at 11:18 a.m. CDT

Copy
Levente Kintzel gravatar
I have exactly same issue, with the difference that I'm using Gluu in dockerized environment. The docker image version I'm using is 4.2.3_02. Here is the issue I reported: https://support.gluu.org/installation/9460/out-of-memory/ At the end of the day the fix in your case was the upgrade? Thanks, Levente

By Zachary Doyle user 06 Apr 2021 at 11:33 a.m. CDT

Copy
Zachary Doyle gravatar
Hi Levente- Yeah, the 4.2.1 -> 4.2.3 upgrade fixed this and another LDAP issue for us. I think module overload is also a factor- we have some tight deadlines and can't rebuild our IDP, but if you also have say, oxauth-rp installed, you might want to prune your gluu install a bit. Best of luck!

Post an answer