Hi, Ben. >when I browse to see the metadata file /idp/shibboleth/ I get a 503 service unavailable No need for the trailing slash, it's just `/idp/shibboleth`. If you installed IdP component, it should be there. >I got the "-su: keytool: command not found" error You can do search inside container for `keytool`, and use the full path to it. May be it's not added to PATH, but it should be there.

I definitely installed the IDP component and you can see: https://fed-auth-01.hartford.edu/idp/shibboleth I will find keytool.

Found keytool Put the new certs on there but there is no cacerts keystore. So stopped and started all services. cert shows up properly but I get an error when I go the the main page. Oops Something wrong happened. Return to the application using below button. Button returns to same page. https://fed-auth-01.hartford.edu/identity/error

I took a snapshot of the server before I ran setup.py. Should I revert and run it again?

That's caused by the fact you didn't add certificate to `cacerts`. You can search for it again, should be something like `/opt/jdk1.8.0_112/jre/lib/security/cacerts`. When it's added and old one is removed or overwritten, restart services and it should work again.

Won't let me overwrite it. Asking for a password and none of the passwords I used to create the server are working.

You can use next commands, Ben: ``` /opt/jre/jre/bin/keytool -delete -alias <hostname_of_your_Gluu_Server>_httpd -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit /opt/jre/jre/bin/keytool -importcert -file httpd.der -keystore /opt/jre/jre/lib/security/cacerts -alias <hostname_of_your_Gluu_Server>_httpd -storepass changeit ```

>I definitely installed the IDP component and you can see: https://fed-auth-01.hartford.edu/idp/shibboleth Please check whether idp service is running. Try to restart it and see what may be logged to `/opt/shibboleth-idp/logs/idp-process.log`

Cleared the certificate issue and that also seemed to clear the idp issue. Thank you for your assistance.

I guess I should have not closed this. Any way to reopen it? I just rebooted the server and the IDP service is running but the error is back. The logs say this: > 2017-11-14 14:00:45,129 - ERROR [org.springframework.web.context.ContextLoader:351] - Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'shibboleth.metrics.RegisterMetricSets$child#0' defined in URL [file:/opt/shibboleth-idp/conf/admin/metrics .xml]: Cannot resolve reference to bean 'shibboleth.metrics.AttributeResolverGaugeSet' while setting bean property 'arguments' with key [7]; nested exception is org.springframework.beans.factory .BeanCreationException: Error creating bean with name 'shibboleth.metrics.AttributeResolverGaugeSet' defined in URL [file:/opt/shibboleth-idp/system/conf/general-admin-system.xml]: Invocation of init method failed; nested exception is net.shibboleth.utilities.java.support.component.ComponentInitializationException: Injected service was null or not an AttributeResolver at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveReference(BeanDefinitionValueResolver.java:359) Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'shibboleth.metrics.AttributeResolverGaugeSet' defined in URL [file:/opt/shibboleth-idp/system/c onf/general-admin-system.xml]: Invocation of init method failed; nested exception is net.shibboleth.utilities.java.support.component.ComponentInitializationException: Injected service was null o r not an AttributeResolver at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1578) Caused by: net.shibboleth.utilities.java.support.component.ComponentInitializationException: Injected service was null or not an AttributeResolver at net.shibboleth.idp.attribute.resolver.impl.AttributeResolverServiceGaugeSet.doInitialize(AttributeResolverServiceGaugeSet.java:104) An idp restart clears up the issue.

>I guess I should have not closed this. Any way to reopen it? You can still post to it even if it's closed, it's ok. >An idp restart clears up the issue. Could you elaborate a bit? Each time you restart Gluu service idp service fails to start, but when you restart just idp service after that, the issue is resolved?

I have only done it the one time, and it was a full server reboot. It is not in production currently. Did you want me to try and restart the gluu service? Upon reboot, the Gluu service started, no problem. And inside the Gluu container, a 'service --status-all' showed idp as up, but the 503 error was happening. Restarting the idp service fixed that, as it did the first time, pre-reboot.

>> 2017-11-14 14:00:45,129 - ERROR [org.springframework.web.context.ContextLoader:351] - Context initialization failed org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'shibboleth.metrics.RegisterMetricSets$child#0' defined in URL [file:/opt/shibboleth-idp/conf/admin/metrics .xml]: Cannot resolve reference to bean 'shibboleth.metrics.AttributeResolverGaugeSet' while setting bean property 'arguments' with key [7]; nested exception is org.springframework.beans.factory .BeanCreationException: Error creating bean with name This error is misleading and coming from Shib v3 core code; hopefully v3 should have a better exception handing in future. For now if you face any issue; check for some other error messages than this... that's what I am doing as well.

Any particular logs you want me to look at? Also, did you want me to try and reproduce the issue?

Were you able to confirm it's systematic issue? I.e. that it happens at each restart, or at least frequently enough?

I have verified that every stop and start of the gluu-server-3.1.1 service causes the 503 error and that a 'service idp restart' from within the gluu container fixes the issue.

It takes some time to load configuration and setup. I generally use 8GB memory + 2 CPU units even for my test instances. Plus feel free to 'allocate' memory for every services ( idp, identity, oxauth ). You can get them inside /etc/defaults/

I am using 8GB of memory and 2 cpu. And unless it takes over many hours, for sure it stays on the 503 error until I restart the service.

Please consider the following: 1. Stop the main Gluu service 2. Remove IDP log: `rm /opt/gluu-server-3.1.1/opt/shibboleth-idp/logs/idp-process.log` 3. Start the main service and wait for 10 minutes. Make sure the issue is triggered again this time. In that case retrieve the whole re-created `idp-process.log` and share it with us.

I added the link to the log in the link URL.

Any word on the log?

Not much so far. There are some errors, but whether it may prevent it from starting and why restarting fixes it is unclear. We'll try to QA Xenial package for some time, I'll update you on results of it.

I was able to confirm the issue. It happens because for runlevel we start container at for the `idp` service we don't set startup priority the same way as for others, and it starts prior to OpenLDAP, failing to establish connection to it. If corrected, it begin to startup fine. Steps to correct it (inside container): 1. `# mv /etc/rc3.d/S83apache2 /etc/rc3.d/S84apache2` 2. `# mv /etc/rc3.d/S01idp /etc/rc3.d/S83idp` Thanks for letting us know, Ben. [Issue](https://github.com/GluuFederation/community-edition-setup/issues/371) is reported.