Out of memory: Metaspace

By: Sved Devs user 03 Aug 2020 at 10:47 a.m. CDT

2 Responses
Sved Devs gravatar
When performing some operations in Gluu's default admin portal, OxTrust, like try to open a scope or get into a user details, I get an OOPS, Something went wrong page. This is not a frequent issue, but something that happens occassionally (never happened on our 3.1.5 instance, but been seeing this often on the new 4.2 instance). Turning on Trace, I see the following log that says Metaspace is running out of memory. Can you please point me to where this setting is and if there is a thumb rule to how much it should be? Our server is not even loaded or used as much at the moment (so, a very mild load so to speak). ```bash 2020-08-03 15:34:11,109 TRACE [qtp296347592-18] [org.gluu.oxtrust.service.PermissionService] (PermissionService.java:72) - Checking permissions for target 'scope' an 'action'. Identity: access 2020-08-03 15:34:11,110 TRACE [qtp296347592-18] [org.gluu.oxtrust.service.PermissionService] (PermissionService.java:72) - Checking permissions for target 'scope' an 'action'. Identity: access 2020-08-03 15:34:11,110 TRACE [qtp296347592-18] [org.gluu.oxtrust.service.PermissionService] (PermissionService.java:72) - Checking permissions for target 'scope' an 'action'. Identity: access 2020-08-03 15:34:11,111 TRACE [qtp296347592-18] [org.gluu.oxtrust.service.PermissionService] (PermissionService.java:72) - Checking permissions for target 'scope' an 'action'. Identity: access 2020-08-03 15:34:11,113 TRACE [qtp296347592-18] [org.gluu.service.BaseCacheService] (BaseCacheService.java:37) - Request data, key 'l_02028353-b43b-4c4b-98ee-3f88b90396b6_08E2_0baa1718-72d1-456b-8611-8073b37bb030_3d74038a-42a2-46ff-84b0-184d6a494574_4703184c-bc17-44f4-8100-18cc8e074337_572699dc-b114-431f-b769-aea8c9fdc046_5a28cd0d-2307-4a48-983c-3e2706b0e1ad_75526443-1a60-4d90-9a02-038a1ba4c393_7966968b-0c73-4b52-946e-ff67c245d86a_86c99502-c37e-449a-a4c3-90b43e93e386_D03E_de9a9df5-0ccd-4805-816c-99d790531fe6' 2020-08-03 15:34:11,114 TRACE [qtp296347592-18] [org.gluu.service.BaseCacheService] (BaseCacheService.java:39) - Loaded data, key 'l_02028353-b43b-4c4b-98ee-3f88b90396b6_08E2_0baa1718-72d1-456b-8611-8073b37bb030_3d74038a-42a2-46ff-84b0-184d6a494574_4703184c-bc17-44f4-8100-18cc8e074337_572699dc-b114-431f-b769-aea8c9fdc046_5a28cd0d-2307-4a48-983c-3e2706b0e1ad_75526443-1a60-4d90-9a02-038a1ba4c393_7966968b-0c73-4b52-946e-ff67c245d86a_86c99502-c37e-449a-a4c3-90b43e93e386_D03E_de9a9df5-0ccd-4805-816c-99d790531fe6': 'null' 2020-08-03 15:34:11,757 TRACE [qtp296347592-18] [org.gluu.oxtrust.exception.GlobalExceptionHandler] (GlobalExceptionHandler.java:58) - Metaspace java.lang.OutOfMemoryError: Metaspace 2020-08-03 15:34:11,783 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:37) - Request data, key 'o=gluu' 2020-08-03 15:34:11,784 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:39) - Loaded data, key 'o=gluu': 'Entry [dn=o=gluu]' 2020-08-03 15:34:11,784 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:37) - Request data, key 'o=gluu' 2020-08-03 15:34:11,785 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:39) - Loaded data, key 'o=gluu': 'Entry [dn=o=gluu]' 2020-08-03 15:34:11,785 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:37) - Request data, key 'o=gluu' 2020-08-03 15:34:11,785 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:39) - Loaded data, key 'o=gluu': 'Entry [dn=o=gluu]' 2020-08-03 15:34:11,785 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:37) - Request data, key 'o=gluu' 2020-08-03 15:34:11,785 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:39) - Loaded data, key 'o=gluu': 'Entry [dn=o=gluu]' 2020-08-03 15:34:11,786 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:37) - Request data, key 'o=gluu' 2020-08-03 15:34:11,786 TRACE [qtp296347592-15] [org.gluu.service.BaseCacheService] (BaseCacheService.java:39) - Loaded data, key 'o=gluu': 'Entry [dn=o=gluu]' ```
Ubuntu 18.04
closed

Answers

By Sved Devs user 03 Aug 2020 at 11 a.m. CDT

Copy
Sved Devs gravatar
I believe it might be in /etc/defaults/identity file and the -XX:MaxMetaspaceSize (currently set to 128m) Increasing it to 512m and testing. The new options are as follows. Please post if there are any thumb rules to follow for allocating these memories. ```bash JAVA_OPTIONS="-server -Xms256m -Xmx1344m -XX:MaxMetaspaceSize=512m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/identity -Dlog.base=/opt/gluu/jetty/identity -Dpython.home=/opt/jython -Dorg.eclipse.jetty.server.Request.maxFormContentSize=50000000" ```

By Sved Devs user 03 Aug 2020 at 12:15 p.m. CDT

Copy
Sved Devs gravatar
Closing this out (since bumping up the metaspace memory to 512m seemed to have helped).

Post an answer