opendj died, out of memory

By: Luke Miller user 24 Oct 2021 at 10:30 a.m. CDT

4 Responses
Luke Miller gravatar
Our GLUU server went down last night. I restart of services fixed it, looking in the the opendj error logs I see this: [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space (Integer.java:1059 EPollSelectorImpl.java:194 EPollSelectorImpl.java:137 SelectorImpl.java:124 SelectorImpl.java:136 DefaultSelectorHandler.java:90 SelectorRunner.java:315 SelectorRunner.java:255 AbstractThreadPool.java:569 AbstractThreadPool.java:549 Thread.java:834) [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 2 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [24/Oct/2021:06:50:21 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Audit Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 13 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDA PClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Oct/2021:06:50:21 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtExc eption, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 13 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CL OSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilt erChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java: 658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [24/Oct/2021:06:50:21 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Access Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Multifile Text Writer for cn=File-Based Access Logger,cn=Loggers,cn=config has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [24/Oct/2021:06:50:21 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerShutdown, alert ID org.opends.messages.core-141): The Directory Server has started the shutdown process. The shutdown was initiated by an instance of class org.opends.server.core.DirectoryServerShutdownHook and the reason provided for the shutdown was The Directory Server shutdown hook detected that the JVM is shutting down. This generally indicates that JVM received an external request to stop (e.g., through a kill signal) [24/Oct/2021:06:50:21 +0000] category=PROTOCOL severity=NOTICE msgID=277 msg=Stopped listening for new connections on LDAPS Connection Handler 0.0.0.0 port 1636 [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=Failure in publishing audit event to ldap-access : Failed to add event to queue [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=Failure in publishing audit event to ldap-access : Failed to add event to queue [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=Failure in publishing audit event to ldap-access : Failed to add event to queue [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=Failure in publishing audit event to ldap-access : Failed to add event to queue [24/Oct/2021:06:50:21 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=Failure in publishing audit event to ldap-access : Failed to add event to queue [24/Oct/2021:06:50:21 +0000] category=PROTOCOL severity=NOTICE msgID=277 msg=Stopped listening for new connections on Administration Connector 0.0.0.0 port 444
Gluu 4.2.3 Ubuntu 18.04
closed

Answers

By Michael Schwartz Account Admin 24 Oct 2021 at 12:56 p.m. CDT

Copy
Michael Schwartz gravatar
After how long? How much RAM? What other services running on the server?

By Luke Miller user 24 Oct 2021 at 2:46 p.m. CDT

Copy
Luke Miller gravatar
The server has 12GB of memory. Only GLUU is running on this server. I believe GLUU was last restarted when we did the upgrade on 8/8/2021. GLUU was down for 8 hours before I noticed this and restarted services. When I restarted opendj it took a while to shutdown which caused me to look at those logs. I saw the errors about it running out of memory there which I posted. According to our system monitoring the server itself did not run out of memory. At that time it had 3.7 GB of free memory. There is nothing in the OS system logs about running out of memory either. Thanks, Luke

By Michael Schwartz Account Admin 24 Oct 2021 at 6:51 p.m. CDT

Copy
Michael Schwartz gravatar
i recommend upgrading to 4.3.0 or later. It has latest OpenDJ code

By Luke Miller user 29 Oct 2021 at 6:20 p.m. CDT

Copy
Luke Miller gravatar
The release notes for 4.3 say that it is in beta. Is that correct or did that not get updated? Thanks, Luke

Post an answer