By: Miika Tuisku user 20 Nov 2020 at 4:55 a.m. CST

4 Responses
Miika Tuisku gravatar
Hello, I followed the Gluu Docs and Installation instructions for CentOS8. My environment is OpenStack OKD 3.11 with CentOS8 VM instance. I have Security Groups (OpenStack firewall) open for HTTP/S and a fixed floating IP-address and domain name for external connectivity. Installation script finishes with success, yet trying to point my browser to the Apache httpd root produces "Gluu template with HTTP ERROR 503, SERVICE UNAVAILABLE". How come?

By Mohit Mali staff 20 Nov 2020 at 10:25 p.m. CST

Mohit Mali gravatar
HI Miika Tuisku, Can you please check the status of your identity and oxauth services? try a stop and start because it looks like the service is down. Thanks and Regards Mohit Mali.

By 21 Nov 2020 at 10:38 a.m. CST

 gravatar
Hi Miika Tuisku and Mohit Mali, I have the same Problem with the 'normal' Debian 10 Installation (without docker) on a fresh image without a firewall. Rebooting the server fixes the issue and then the Website is reachable for a few days and after that it displays the 503 error. Reinstalling Gluu on a fresh Debian does not help. It is running on 8gb ram and can use 6gb of it and has 80gb disk space. Both identity and Oxauth service say everything is fine exept a warning (WARNING: Illegal reflective access by org.jboss.weld.util.bytecode.ClassFileUtils$1). Restarting the identity and oxauth service does not help either.

By 21 Nov 2020 at 11:13 a.m. CST

 gravatar
I will try to attach some logs, please tell me if I missed some important ones root@tegmentum:~# service oxauth status ``` ● oxauth.service - Oxauth service Loaded: loaded (/lib/systemd/system/oxauth.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2020-11-21 17:43:30 CET; 4min 7s ago Process: 5626 ExecStart=/opt/dist/scripts/oxauth start (code=exited, status=0/SUCCESS) Main PID: 5652 (java) CGroup: /system.slice/oxauth.service └─5652 /opt/jre/bin/java -server -Xms256m -Xmx1433m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gl Nov 21 17:42:38 tegmentum.mydomain.de systemd[1]: Starting Oxauth service... Nov 21 17:42:38 tegmentum.mydomain.de oxauth[5626]: Starting Jetty: 2020-11-21 16:42:38.921:INFO::main: Logging initialized @720ms Nov 21 17:42:39 tegmentum.mydomain.de oxauth[5626]: 2020-11-21 16:42:39.332:INFO::main: Console stderr/stdout captured to /opt/glu Nov 21 17:42:47 tegmentum.mydomain.de oxauth[5626]: . . WARNING: An illegal reflective access operation has occurred Nov 21 17:42:47 tegmentum.mydomain.de oxauth[5626]: WARNING: Illegal reflective access by org.jboss.weld.util.bytecode.ClassFileUt Nov 21 17:42:47 tegmentum.mydomain.de oxauth[5626]: WARNING: Please consider reporting this to the maintainers of org.jboss.weld.u Nov 21 17:42:47 tegmentum.mydomain.de oxauth[5626]: WARNING: Use --illegal-access=warn to enable warnings of further illegal refle Nov 21 17:42:47 tegmentum.mydomain.de oxauth[5626]: WARNING: All illegal access operations will be denied in a future release Nov 21 17:43:30 tegmentum.mydomain.de oxauth[5626]: . . . . . . . . . . OK Sat Nov 21 17:43:30 CET 2020 Nov 21 17:43:30 tegmentum.mydomain.de systemd[1]: Started Oxauth service. ``` root@tegmentum:~# service identity status ``` ● identity.service - Identity service Loaded: loaded (/lib/systemd/system/identity.service; enabled; vendor preset: enabl Active: active (running) since Sat 2020-11-21 17:44:26 CET; 5min ago Process: 5830 ExecStart=/opt/dist/scripts/identity start (code=exited, status=0/SUCC Main PID: 5856 (java) CGroup: /system.slice/identity.service └─5856 /opt/jre/bin/java -server -Xms256m -Xmx1433m -XX:+DisableExplicitGC Nov 21 17:43:34 tegmentum.mydomain.de systemd[1]: Starting Identity service... Nov 21 17:43:34 tegmentum.mydomain.de identity[5830]: Starting Jetty: 2020-11-21 16:43: Nov 21 17:43:35 tegmentum.mydomain.de identity[5830]: 2020-11-21 16:43:35.252:INFO::mai Nov 21 17:43:44 tegmentum.mydomain.de identity[5830]: . . WARNING: An illegal reflectiv Nov 21 17:43:44 tegmentum.mydomain.de identity[5830]: WARNING: Illegal reflective acces Nov 21 17:43:44 tegmentum.mydomain.de identity[5830]: WARNING: Please consider reportin Nov 21 17:43:44 tegmentum.mydomain.de identity[5830]: WARNING: Use --illegal-access=war Nov 21 17:43:44 tegmentum.mydomain.de identity[5830]: WARNING: All illegal access opera Nov 21 17:44:26 tegmentum.mydomain.de identity[5830]: . . . . . . . . . . OK Sat Nov 21 Nov 21 17:44:26 tegmentum.mydomain.de systemd[1]: Started Identity service. ``` root@tegmentum:~# cat /opt/gluu/jetty/identity/logs/2020_11_21.jetty.log ``` Nov 21, 2020 4:43:42 PM org.richfaces.webapp.ResourceServletContainerInitializer registerServlet INFO: Auto-registered servlet ResourceServlet with mapping '/org.richfaces.resources/*' SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/log4j-slf4j-impl-2.13.3.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2020-11-21 16:43:48,447 INFO [main] [org.gluu.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:24) - Adding Bouncy Castle Provider 2020-11-21 16:43:48,596 INFO [main] [org.gluu.oxtrust.util.BuildVersionService] (BuildVersionService.java:80) - Root element :beans 2020-11-21 16:43:48,599 INFO [main] [org.gluu.oxtrust.service.AppInitializer] (AppInitializer.java:253) - Build date 2020-09-24 08:27. Code revision 1557759f054e6dbc20bcc041ac045781586b4a1a on 2020-09-23T19:42:23+0000. Build 2218 2020-11-21 16:43:48,604 INFO [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:127) - Creating oxTrustConfiguration 2020-11-21 16:43:48,687 INFO [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:289) - Loading configuration from 'ldap' DB... 2020-11-21 16:44:25,670 ERROR [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:306) - Failed to create LDAP connection pool! Result code: '91 (connect error)' org.gluu.persist.exception.operation.ConfigurationException: Failed to create LDAP connection pool! Result code: '91 (connect error)' at org.gluu.persist.ldap.impl.LdapEntryManagerFactory.createEntryManager(LdapEntryManagerFactory.java:51) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.persist.ldap.impl.LdapEntryManagerFactory.createEntryManager(LdapEntryManagerFactory.java:23) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.persist.ldap.impl.LdapEntryManagerFactory$Proxy$_$$_WeldClientProxy.createEntryManager(Unknown Source) ~[oxcore-persistence-ldap-4.2.1.Final.jar:?] at org.gluu.oxtrust.service.AppInitializer.createPersistenceEntryManager(AppInitializer.java:296) ~[classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:85) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:103) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.contexts.AbstractContext.get(AbstractContext.java:96) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bean.ContextualInstanceStrategy$ApplicationScopedContextualInstanceStrategy.get(ContextualInstanceStrategy.java:140) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bean.proxy.ContextBeanInstance.getInstance(ContextBeanInstance.java:102) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:105) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.gluu.persist.EntityManager$PersistenceEntryManager$1037559104$Proxy$_$$_WeldClientProxy.find(Unknown Source) ~[weld-core-impl-3.1.4.Final.jar:?] at org.gluu.oxtrust.service.config.ConfigurationFactory.loadConfigurationFromDb(ConfigurationFactory.java:60) ~[classes/:?] at org.gluu.oxtrust.service.config.ConfigurationFactory$Proxy$_$$_WeldSubclass.loadConfigurationFromDb(Unknown Source) ~[classes/:?] at org.gluu.oxtrust.service.config.ConfigurationFactory.loadConfigurationFromDb(ConfigurationFactory.java:29) ~[classes/:?] at org.gluu.service.config.ConfigurationFactory.createFromDb(ConfigurationFactory.java:291) ~[oxtrust-service-4.2.1.Final.jar:?] at org.gluu.service.config.ConfigurationFactory.create(ConfigurationFactory.java:146) ~[oxtrust-service-4.2.1.Final.jar:?] at org.gluu.oxtrust.service.config.ConfigurationFactory$Proxy$_$$_WeldSubclass.create(Unknown Source) ~[classes/:?] at org.gluu.oxtrust.service.config.ConfigurationFactory$Proxy$_$$_WeldClientProxy.create(Unknown Source) ~[classes/:?] at org.gluu.oxtrust.service.AppInitializer.applicationInitialized(AppInitializer.java:180) ~[classes/:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:85) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.injection.MethodInvocationStrategy$SimpleMethodInvocationStrategy.invoke(MethodInvocationStrategy.java:168) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at javax.enterprise.inject.spi.ObserverMethod.notify(ObserverMethod.java:124) ~[jakarta.enterprise.cdi-api-2.0.2.jar:?] at org.jboss.weld.util.Observers.notify(Observers.java:166) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:177) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.bootstrap.BeanDeploymentModule.fireEvent(BeanDeploymentModule.java:93) ~[weld-core-impl-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.fireEventForApplicationScope(HttpContextLifecycle.java:161) ~[weld-web-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.module.web.servlet.HttpContextLifecycle.contextInitialized(HttpContextLifecycle.java:147) ~[weld-web-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.module.web.servlet.WeldInitialListener.contextInitialized(WeldInitialListener.java:99) ~[weld-web-3.1.4.Final.jar:3.1.4.Final] at org.jboss.weld.servlet.api.helpers.ForwardingServletListener.contextInitialized(ForwardingServletListener.java:34) ~[weld-spi-3.1.SP2.jar:3.1.SP2] at org.jboss.weld.environment.servlet.EnhancedListener.onStartup(EnhancedListener.java:66) ~[weld-servlet-core-3.1.4.Final.jar:3.1.4.Final] at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:145) ~[jetty-plus-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:64) ~[jetty-annotations-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:341) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) ~[jetty-webapp-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) ~[jetty-webapp-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) ~[jetty-servlet-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) ~[jetty-webapp-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) ~[jetty-deploy-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.server.Server.start(Server.java:408) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.server.Server.doStart(Server.java:372) ~[jetty-server-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) ~[jetty-util-9.4.31.v20200723.jar:9.4.31.v20200723] at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) ~[jetty-xml-9.4.31.v20200723.jar:9.4.31.v20200723] at java.security.AccessController.doPrivileged(Native Method) ~[?:?] at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) [jetty-xml-9.4.31.v20200723.jar:9.4.31.v20200723] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) [start.jar:9.4.31.v20200723] at org.eclipse.jetty.start.Main.start(Main.java:491) [start.jar:9.4.31.v20200723] at org.eclipse.jetty.start.Main.main(Main.java:77) [start.jar:9.4.31.v20200723] 2020-11-21 16:44:25,679 ERROR [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:147) - Failed to load configuration from LDAP. Please fix it!!!. 2020-11-21 16:44:25.679:WARN:oeja.ServletContainerInitializersStarter:main: org.gluu.exception.ConfigurationException: Failed to load configuration from LDAP. at org.gluu.service.config.ConfigurationFactory.create(ConfigurationFactory.java:148) at org.gluu.oxtrust.service.config.ConfigurationFactory$Proxy$_$$_WeldSubclass.create(Unknown Source) at org.gluu.oxtrust.service.config.ConfigurationFactory$Proxy$_$$_WeldClientProxy.create(Unknown Source) at org.gluu.oxtrust.service.AppInitializer.applicationInitialized(AppInitializer.java:180) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:85) at org.jboss.weld.injection.MethodInvocationStrategy$SimpleMethodInvocationStrategy.invoke(MethodInvocationStrategy.java:168) at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330) at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308) at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286) at javax.enterprise.inject.spi.ObserverMethod.notify(ObserverMethod.java:124) at org.jboss.weld.util.Observers.notify(Observers.java:166) at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285) at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273) at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:177) at org.jboss.weld.bootstrap.BeanDeploymentModule.fireEvent(BeanDeploymentModule.java:93) at org.jboss.weld.module.web.servlet.HttpContextLifecycle.fireEventForApplicationScope(HttpContextLifecycle.java:161) at org.jboss.weld.module.web.servlet.HttpContextLifecycle.contextInitialized(HttpContextLifecycle.java:147) at org.jboss.weld.module.web.servlet.WeldInitialListener.contextInitialized(WeldInitialListener.java:99) at org.jboss.weld.servlet.api.helpers.ForwardingServletListener.contextInitialized(ForwardingServletListener.java:34) at org.jboss.weld.environment.servlet.EnhancedListener.onStartup(EnhancedListener.java:66) at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:145) at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:64) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:341) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) 2020-11-21 16:44:25.680:WARN:oejw.WebAppContext:main: Failed startup of context o.e.j.w.WebAppContext@2c78324b{oxTrust Server,/identity,[file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/omnifaces-2.6.9.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/richfaces-4.5.18-gluu.Final.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/oxtrust-static-4.2.1.Final.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/scim-model-4.2.1.Final.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/richfaces-a4j-4.5.18-gluu.Final.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/primefaces-8.0.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/jakarta.faces-2.3.14.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/bootsfaces-1.5.0.jar!/META-INF/resources, jar:file:///opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-9110226917343654899.dir/webapp/WEB-INF/lib/richfaces-core-4.5.18-gluu.Final.jar!/META-INF/resources],UNAVAILABLE}{/opt/gluu/jetty/identity/webapps/identity.war} java.lang.RuntimeException: org.gluu.exception.ConfigurationException: Failed to load configuration from LDAP. at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:69) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:341) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.gluu.exception.ConfigurationException: Failed to load configuration from LDAP. at org.gluu.service.config.ConfigurationFactory.create(ConfigurationFactory.java:148) at org.gluu.oxtrust.service.config.ConfigurationFactory$Proxy$_$$_WeldSubclass.create(Unknown Source) at org.gluu.oxtrust.service.config.ConfigurationFactory$Proxy$_$$_WeldClientProxy.create(Unknown Source) at org.gluu.oxtrust.service.AppInitializer.applicationInitialized(AppInitializer.java:180) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:95) at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:85) at org.jboss.weld.injection.MethodInvocationStrategy$SimpleMethodInvocationStrategy.invoke(MethodInvocationStrategy.java:168) at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:330) at org.jboss.weld.event.ObserverMethodImpl.sendEvent(ObserverMethodImpl.java:308) at org.jboss.weld.event.ObserverMethodImpl.notify(ObserverMethodImpl.java:286) at javax.enterprise.inject.spi.ObserverMethod.notify(ObserverMethod.java:124) at org.jboss.weld.util.Observers.notify(Observers.java:166) at org.jboss.weld.event.ObserverNotifier.notifySyncObservers(ObserverNotifier.java:285) at org.jboss.weld.event.ObserverNotifier.notify(ObserverNotifier.java:273) at org.jboss.weld.event.ObserverNotifier.fireEvent(ObserverNotifier.java:177) at org.jboss.weld.bootstrap.BeanDeploymentModule.fireEvent(BeanDeploymentModule.java:93) at org.jboss.weld.module.web.servlet.HttpContextLifecycle.fireEventForApplicationScope(HttpContextLifecycle.java:161) at org.jboss.weld.module.web.servlet.HttpContextLifecycle.contextInitialized(HttpContextLifecycle.java:147) at org.jboss.weld.module.web.servlet.WeldInitialListener.contextInitialized(WeldInitialListener.java:99) at org.jboss.weld.servlet.api.helpers.ForwardingServletListener.contextInitialized(ForwardingServletListener.java:34) at org.jboss.weld.environment.servlet.EnhancedListener.onStartup(EnhancedListener.java:66) at org.eclipse.jetty.plus.annotation.ContainerInitializer.callStartup(ContainerInitializer.java:145) at org.eclipse.jetty.annotations.ServletContainerInitializersStarter.doStart(ServletContainerInitializersStarter.java:64) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:341) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1445) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1409) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:855) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:275) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:524) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:513) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:154) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:173) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:447) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:66) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:784) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:753) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:641) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:540) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:146) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:599) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:249) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) at org.eclipse.jetty.server.Server.start(Server.java:408) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:97) at org.eclipse.jetty.server.Server.doStart(Server.java:372) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) at org.eclipse.jetty.xml.XmlConfiguration.lambda$main$1(XmlConfiguration.java:1929) at java.base/java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1878) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.eclipse.jetty.start.Main.invokeMain(Main.java:218) at org.eclipse.jetty.start.Main.start(Main.java:491) at org.eclipse.jetty.start.Main.main(Main.java:77) 2020-11-21 16:44:25.700:INFO:oejs.AbstractConnector:main: Started ServerConnector@47dff7db{HTTP/1.1, (http/1.1)}{localhost:8082} 2020-11-21 16:44:25.701:INFO:oejs.Server:main: Started @51560ms ``` So it seems something is broken with the LDAP (similar error is in /opt/gluu/jetty/oxauth/logs/2020_11_21.jetty.log) root@tegmentum:/home/user# netstat -antlp | grep 1636 | grep LISTEN ``` tcp6 129 0 127.0.0.1:1636 :::* LISTEN 1409/java ``` root@tegmentum:/home/user# ps -ef | grep slapd ``` root 10607 5394 0 18:11 pts/2 00:00:00 grep slapd ``` root@tegmentum:~# cat /opt/opendj/logs/errors ``` [16/Nov/2020:22:57:09 +0000] category=PROTOCOL severity=NOTICE msgID=277 msg=Stopped listening for new connections on Administration Connector localhost port 4444 [16/Nov/2020:22:57:09 +0000] category=PROTOCOL severity=NOTICE msgID=277 msg=Stopped listening for new connections on LDAPS Connection Handler localhost port 1636 [17/Nov/2020:08:47:15 +0000] category=JVM severity=NOTICE msgID=21 msg=Installation Directory: /opt/opendj [17/Nov/2020:08:47:15 +0000] category=JVM severity=NOTICE msgID=23 msg=Instance Directory: /opt/opendj [17/Nov/2020:08:47:15 +0000] category=JVM severity=NOTICE msgID=17 msg=JVM Information: 11.0.8+10-LTS by Amazon.com Inc., 64-bit architecture, 2147483648 bytes heap size [17/Nov/2020:08:47:15 +0000] category=JVM severity=NOTICE msgID=18 msg=JVM Host: tegmentum.mydomain.de, running Linux 4.19.0-12-amd64 amd64, 8167116800 bytes physical memory size, number of processors available 2 [17/Nov/2020:08:47:15 +0000] category=JVM severity=NOTICE msgID=19 msg=JVM Arguments: "-Xms1g", "-Xmx2g", "-XX:+UseCompressedOops", "-Dorg.opends.server.scriptName=start-ds" [17/Nov/2020:08:47:16 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend site containing 2 entries has started [17/Nov/2020:08:47:16 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend metric containing 9 entries has started [17/Nov/2020:08:47:17 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend userRoot containing 179 entries has started [17/Nov/2020:08:47:17 +0000] category=EXTENSIONS severity=NOTICE msgID=221 msg=DIGEST-MD5 SASL mechanism using a server fully qualified domain name of: localhost [17/Nov/2020:08:47:17 +0000] category=CORE severity=NOTICE msgID=135 msg=The Directory Server has started successfully [17/Nov/2020:08:47:17 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.core.DirectoryServer (alert type org.opends.server.DirectoryServerStarted, alert ID org.opends.messages.core-135): The Directory Server has started successfully [17/Nov/2020:08:47:17 +0000] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on LDAPS Connection Handler localhost port 1636 [17/Nov/2020:08:47:17 +0000] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on Administration Connector localhost port 4444 [17/Nov/2020:13:39:28 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space (BuffersBuffer.java:153 BuffersBuffer.java:164 BuffersBuffer.java:633 BuffersBuffer.java:35 LdapCodec.java:64 ExecutorResolver.java:95 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 TCPNIOTransport.java:515 AbstractIOStrategy.java:89 SameThreadIOStrategy.java:79 AbstractIOStrategy.java:66 SelectorRunner.java:391 SelectorRunner.java:360 SelectorRunner.java:324 SelectorRunner.java:255 AbstractThreadPool.java:569 AbstractThreadPool.java:549 ...) [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0013: Exception during FilterChain execution exception=OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=ERROR msgID=1 msg=doSelect exception exception=OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(5),5,Directory Server Thread Group] exception=OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 4 encountered an uncaught exception while processng operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 13 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Monitor Provider State Updater has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Monitor Provider State Updater has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 10 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 3 encountered an uncaught exception while processing operation null: IllegalMonitorStateException: attempt to unlock read lock, not locked by current thread (ReentrantReadWriteLock.java:448 ReentrantReadWriteLock.java:432 AbstractQueuedSynchronizer.java:1382 ReentrantReadWriteLock.java:897 TraditionalWorkQueue.java:481 TraditionalWorkQueue.java:382 TraditionalWorkerThread.java:134) [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 9 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread LDAP Connection Finalizer for connection handler LDAPS Connection Handler localhost port 1636 0 has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread LDAP Connection Finalizer for connection handler LDAPS Connection Handler localhost port 1636 0 has caused it to terminate abnormally. The stack trace for that exception is: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 11 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 8 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 14 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 2 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 12 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 5 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 1 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 7 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 4 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 4 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=140 msg=An uncaught exception during processing for thread Worker Thread 4 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org.opends.server.api.DirectoryThread (alert type org.opends.server.UncaughtException, alert ID org.opends.messages.core-140): An uncaught exception during processing for thread Worker Thread 4 has caused it to terminate abnormally. The stack trace for that exception is: SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(1) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(1) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) [18/Nov/2020:13:54:33 +0000] category=org.opends.messages.external severity=WARNING msgID=1 msg=GRIZZLY0011: Uncaught exception on thread Thread[OpenDJ LDAP SDK Grizzly selector thread(2) SelectorRunner,5,Directory Server Thread Group] exception=SSLException: SSLEngine is CLOSED (SSLConnectionContext.java:293 SSLConnectionContext.java:224 SSLBaseFilter.java:457 SSLBaseFilter.java:365 SSLFilter.java:139 ExecutorResolver.java:87 DefaultFilterChain.java:260 DefaultFilterChain.java:177 DefaultFilterChain.java:109 DefaultFilterChain.java:88 ProcessorExecutor.java:53 DefaultFilterChain.java:390 NIOConnection.java:427 NIOConnection.java:401 LDAPServerFilter.java:599 LDAPServerFilter.java:518 LDAPClientConnection2.java:658 LDAPClientConnection2.java:239 LDAPServerFilter.java:574 LDAPServerFilter.java:239 LDAPServerFilter.java:195 ...) ``` So I conclude that some JVM processes of opendj are running out of memory causing a shutdown of the openldap server and then identity and oxauth services cant fetch their configs. How can we solve this issue? How do I restart the openid service? Is this some malconfiguration on the gluu side or a bug in OpenDJ? I cant find an issue on https://github.com/OpenIdentityPlatform/OpenDJ/issues.

By Mohib Zico staff 30 Nov 2020 at 6:31 a.m. CST

Mohib Zico gravatar
>> [18/Nov/2020:13:54:33 +0000] category=CORE severity=ERROR msgID=108 msg=Worker Thread 13 encountered an uncaught exception while processing operation null: OutOfMemoryError: Java heap space () That's the problem with OpenDJ. Give it more memory ( subject to google: "how to tune OpenDJ memory allocation ). See how things go please.