By: James Lao user 13 Feb 2021 at 1:23 a.m. CST

8 Responses
James Lao gravatar
After installation, the /identity/ page gives a 503 error. Identity and oxauth all fail to start due to timeout. Not exactly sure where to start with... ``` identity.service - Identity service Loaded: loaded (/lib/systemd/system/identity.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Sat 2021-02-13 02:08:25 EST; 12min ago Process: 28158 ExecStart=/opt/dist/scripts/identity start (code=killed, signal=TERM) Feb 13 02:06:44 auth.eighttalk.com systemd[1]: Starting Identity service... Feb 13 02:06:49 auth.eighttalk.com identity[28158]: Starting Jetty: Feb 13 02:06:49 auth.eighttalk.com identity[28189]: 2021-02-13 07:06:49.031:INFO::main: Logging initialized @1982ms to org.eclipse.jetty.util.log.StdErrLog Feb 13 02:06:49 auth.eighttalk.com identity[28189]: 2021-02-13 07:06:49.533:INFO::main: Console stderr/stdout captured to /opt/gluu/jetty/identity/logs/2021_02_13.jetty.log Feb 13 02:08:23 auth.eighttalk.com systemd[1]: identity.service: start operation timed out. Terminating. Feb 13 02:08:25 auth.eighttalk.com identity[28158]: . . . . . . . . . . . . Feb 13 02:08:25 auth.eighttalk.com systemd[1]: identity.service: Failed with result 'timeout'. Feb 13 02:08:25 auth.eighttalk.com systemd[1]: Failed to start Identity service. ``` ``` root@localhost:/opt/gluu/jetty/identity/logs# cat 2021_02_13.jetty.log 2021-02-13 07:06:49.616:INFO:oejs.Server:main: jetty-9.4.35.v20201120; built: 2020-11-20T21:17:03.964Z; git: bdc54f03a5e0a7e280fab27f55c3c75ee8da89fb; jvm 11.0.8+10-LTS 2021-02-13 07:06:49.636:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///opt/gluu/jetty/identity/webapps/] at interval 1 2021-02-13 07:06:49.683:INFO:oejsh.ContextHandler:main: Started o.e.j.s.h.ContextHandler@d23e042{/identity/ext/resources,null,AVAILABLE} ``` ``` root@localhost:/opt/gluu/jetty/identity/logs# tail 2021_02_13.jetty.log.070649500 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-12432551485116852959/webapp/WEB-INF/lib/log4j-slf4j-impl-2.14.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-12432551485116852959/webapp/WEB-INF/lib/slf4j-simple-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2021-02-13 06:57:20,474 INFO [main] [org.gluu.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:24) - Adding Bouncy Castle Provider 2021-02-13 06:57:21,474 INFO [main] [org.gluu.oxtrust.util.BuildVersionService] (BuildVersionService.java:80) - Root element :beans 2021-02-13 06:57:21,478 INFO [main] [org.gluu.oxtrust.service.AppInitializer] (AppInitializer.java:257) - Build date 2021-02-08 20:52. Code revision b41fa69ca1001a7de21061d808a52f4e69487b89 on 2021-02-08T17:01:14+0000. Build 148 2021-02-13 06:57:21,505 INFO [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:127) - Creating oxTrustConfiguration 2021-02-13 06:57:21,871 INFO [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:289) - Loading configuration from 'ldap' DB... ``` ``` oxauth.service - Oxauth service Loaded: loaded (/lib/systemd/system/oxauth.service; enabled; vendor preset: enabled) Active: failed (Result: timeout) since Sat 2021-02-13 01:56:01 EST; 19min ago Feb 13 01:55:40 auth.eighttalk.com oxauth[24070]: . . . . . . . . . . Feb 13 01:55:40 auth.eighttalk.com oxauth[24097]: WARNING: An illegal reflective access operation has occurred Feb 13 01:55:40 auth.eighttalk.com oxauth[24097]: WARNING: Illegal reflective access by org.jboss.weld.util.bytecode.ClassFileUtils$1 (file:/opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth> Feb 13 01:55:40 auth.eighttalk.com oxauth[24097]: WARNING: Please consider reporting this to the maintainers of org.jboss.weld.util.bytecode.ClassFileUtils$1 Feb 13 01:55:40 auth.eighttalk.com oxauth[24097]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations Feb 13 01:55:40 auth.eighttalk.com oxauth[24097]: WARNING: All illegal access operations will be denied in a future release Feb 13 01:55:49 auth.eighttalk.com systemd[1]: oxauth.service: start operation timed out. Terminating. Feb 13 01:56:01 auth.eighttalk.com oxauth[24070]: . . Feb 13 01:56:01 auth.eighttalk.com systemd[1]: oxauth.service: Failed with result 'timeout'. Feb 13 01:56:02 auth.eighttalk.com systemd[1]: Failed to start Oxauth service. ```

By Michael Schwartz staff 14 Feb 2021 at 4:48 p.m. CST

Michael Schwartz gravatar
Did you check the logs in `/opt/gluu/jetty/identity/logs/` and `/opt/gluu/jetty/oxauth/logs/` . It's probably a memory issue. How much memory to do you have allocated for the process? Check the files `/etc/default/identity` and `/etc/default/oxauth`, especially if you see an out of memory exception in oxtrust.log or oxauth.log. Also, make sure LDAP is running. Do a `ps -ef | grep ldap` from inside the chroot. You should also check the logs in `/opt/gluu/opendj/logs` and the memory settings in `/opt/opendj/config/java.properties`

By James Lao user 15 Feb 2021 at 3:40 p.m. CST

James Lao gravatar
Hi, thanks for the reply. Seems OpenDJ is running fine. ``` systemctl status opendj ● opendj.service - OpenDJ Directory Service Loaded: loaded (/etc/systemd/system/opendj.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2021-02-15 16:21:18 EST; 2min 29s ago Process: 29516 ExecStart=/opt/opendj/bin/start-ds (code=exited, status=0/SUCCESS) Main PID: 29562 (java) CGroup: /system.slice/opendj.service └─29562 /opt/jre/bin/java -server -Xms1g -Xmx2g -XX:+UseCompressedOops -Dorg.opends.server.scriptName=start-ds org.opends.server.core.DirectoryServer --configFile /opt/opendj/config/conf> Feb 15 16:20:54 auth.test.internal start-ds[29563]: [15/Feb/2021:21:20:54 +0000] category=JVM severity=NOTICE msgID=19 msg=JVM Arguments: "-Xms1g", "-Xmx2g", "-XX:+UseCompressedOops", "-Dorg.opends.s> Feb 15 16:20:56 auth.test.internal start-ds[29563]: [15/Feb/2021:21:20:56 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend site containing 2 entries has started Feb 15 16:20:56 auth.test.internal start-ds[29563]: [15/Feb/2021:21:20:56 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend metric containing 2 entries has started Feb 15 16:20:56 auth.test.internal start-ds[29563]: [15/Feb/2021:21:20:56 +0000] category=BACKEND severity=NOTICE msgID=513 msg=The database backend userRoot containing 179 entries has started Feb 15 16:20:56 auth.test.internal start-ds[29563]: [15/Feb/2021:21:20:56 +0000] category=EXTENSIONS severity=NOTICE msgID=221 msg=DIGEST-MD5 SASL mechanism using a server fully qualified domain name> Feb 15 16:20:58 auth.test.internal start-ds[29563]: [15/Feb/2021:21:20:58 +0000] category=CORE severity=NOTICE msgID=135 msg=The Directory Server has started successfully Feb 15 16:20:58 auth.test.internal start-ds[29563]: [15/Feb/2021:21:20:58 +0000] category=CORE severity=NOTICE msgID=139 msg=The Directory Server has sent an alert notification generated by class org> Feb 15 16:21:18 auth.test.internal start-ds[29563]: [15/Feb/2021:21:21:17 +0000] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on Administration Connector loca> Feb 15 16:21:18 auth.test.internal start-ds[29563]: [15/Feb/2021:21:21:17 +0000] category=PROTOCOL severity=NOTICE msgID=276 msg=Started listening for new connections on LDAPS Connection Handler loca> Feb 15 16:21:18 auth.test.internal systemd[1]: Started OpenDJ Directory Service. ``` ``` root@auth:/opt/gluu/jetty/identity/logs# ps -ef | grep ldap ldap 29562 1 19 16:20 ? 00:01:07 /opt/jre/bin/java -server -Xms1g -Xmx2g -XX:+UseCompressedOops -Dorg.opends.server.scriptName=start-ds org.opends.server.core.DirectoryServer --configFile /opt/opendj/config/config.ldif ``` yet with identity and oxauth, the last status seems to be stuck with loading configuration from 'ldap' DB... I haven't found anything related to memory in the logs. ``` root@auth:/opt/gluu/jetty/identity/logs# cat oxtrust.log 2021-02-13 06:57:20,474 INFO [main] [org.gluu.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:24) - Adding Bouncy Castle Provider 2021-02-13 06:57:21,474 INFO [main] [org.gluu.oxtrust.util.BuildVersionService] (BuildVersionService.java:80) - Root element :beans 2021-02-13 06:57:21,478 INFO [main] [org.gluu.oxtrust.service.AppInitializer] (AppInitializer.java:257) - Build date 2021-02-08 20:52. Code revision b41fa69ca1001a7de21061d808a52f4e69487b89 on 2021-02-08T17:01:14+0000. Build 148 2021-02-13 06:57:21,505 INFO [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:127) - Creating oxTrustConfiguration 2021-02-13 06:57:21,871 INFO [main] [org.gluu.service.config.ConfigurationFactory] (ConfigurationFactory.java:289) - Loading configuration from 'ldap' DB... ``` ``` root@auth:/opt/gluu/jetty/oxauth/logs# cat oxauth.log 2021-02-13 06:55:39,170 INFO [main] [org.jboss.weld.environment.servlet.EnhancedListener] (EnhancedListener.java:58) - WELD-ENV-001008: Initialize Weld using ServletContainerInitializer 2021-02-13 06:55:39,213 INFO [main] [org.jboss.weld.bootstrap.WeldStartup] (WeldStartup.java:149) - WELD-000900: 3.1.4 (Final) 2021-02-13 06:55:39,628 INFO [main] [jboss.weld.environment.deployment.discovery.ReflectionDiscoveryStrategy] (ReflectionDiscoveryStrategy.java:49) - WELD-ENV-000014: Falling back to Java Reflection for bean-discovery-mode="annotated" discovery. Add org.jboss:jandex to the classpath to speed-up startup. 2021-02-13 06:55:40,512 INFO [main] [org.jboss.weld.bootstrap.WeldStartup] (WeldStartup.java:231) - WELD-000101: Transactional services not available. Injection of @Inject UserTransaction not available. Transactional observers will be invoked synchronously. 2021-02-13 06:55:40,891 WARN [main] [org.jboss.weld.bootstrap.events.BeforeBeanDiscoveryImpl] (BeforeBeanDiscoveryImpl.java:118) - WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType) used for class com.sun.faces.flow.FlowDiscoveryCDIHelper is deprecated from CDI 1.1! 2021-02-13 06:55:42,079 INFO [main] [org.jboss.weld.environment.jetty.JettyLegacyContainer] (JettyLegacyContainer.java:59) - WELD-ENV-001200: Jetty 7.2+ detected, CDI injection will be available in Servlets and Filters. Injection into Listeners should work on Jetty 9.1.1 and newer. 2021-02-13 06:55:43,783 INFO [main] [org.gluu.oxauth.model.util.SecurityProviderUtility] (SecurityProviderUtility.java:24) - Adding Bouncy Castle Provider 2021-02-13 06:55:43,963 INFO [main] [org.gluu.oxauth.model.config.ConfigurationFactory] (ConfigurationFactory.java:381) - Loading configuration from 'ldap' DB... ```

By Mohit Mali staff 16 Feb 2021 at 5:58 a.m. CST

Mohit Mali gravatar
Hi James, Seems like components can't access LDAP server. You could try to restart it by stoping "opendj" service, and starting it again. Thanks and Regards Mohit Mali

By James Lao user 16 Feb 2021 at 11:41 a.m. CST

James Lao gravatar
Hi Mohit, I've already tried that several times... I can even run ldapsearch, but the identity and oxauth are still stuck... ``` root@auth:/opt/opendj/bin# ./ldapsearch -p 1636 -Z -X -D 'cn=directory manager' -w '****' -b o=gluu gluuGroupType=gluuManagerGroup 1.1 dn: inum=60B7,ou=groups,o=gluu ``` Best regards, James,

By Mohib Zico staff 16 Feb 2021 at 12:24 p.m. CST

Mohib Zico gravatar
Hi James, Quoting Mike from previous [comment](https://support.gluu.org/installation/9412/services-fail-to-start-after-ubuntu-package-installation/#at67818): What components you installed ( oxauth, identity, ldap etc ) when you install your Gluu Server? And how much memory + CPU you have in your machine?

By James Lao user 16 Feb 2021 at 12:30 p.m. CST

James Lao gravatar
Hi, Machine has 6 x Intel(R) Xeon(R) CPU Platinum 8268 @ 2.9 GHz and 10 GB memory. When installing Gluu it seemed to recognize 8GB. I have identity / oxauth / oxauth-rp under /opt/gluu/jetty. I also have oxd-server and opendj under /opt. I haven't selected Casa or Scim or IDP. Best regards, James,

By Mohit Mali staff 18 Feb 2021 at 5:05 a.m. CST

Mohit Mali gravatar
Hi James, Can you restart the gluu container and provide the latest logs . ```/sbin/gluu-serverd restart ``` Thanks and Regards Mohit Mali

By James Lao user 24 Feb 2021 at 2:42 a.m. CST

James Lao gravatar
Hi thanks obviously I've done that numerous times... Sometimes oxauth starts fine, but identity doesn't... really just weird...