Shibboleth idP URLs periodically stop working

By: Thomas Maerz user 20 Apr 2016 at 5:54 p.m. CDT

19 Responses
Thomas Maerz gravatar
I previously opened a ticket 2442 because `https://<yourhostname>/idp/shibboleth` was loading a blank page. [https://support.gluu.org/integrations/metadata-collection-for-idp-not-working-2442](https://support.gluu.org/integrations/metadata-collection-for-idp-not-working-2442) Once, I was able to follow John Feulner's suggestion: ``` I had this issue as well. IF you are using chrome. Try navigating to that site and then hit the F12 key. This will open the Chrome Dev Tools. IF you can, on the bottom right where it says console. (If it doesn't there is a console tab. You may need to reload the page) If it says 404. Try these steps. If it's clear, a re-install may be necessary. It's been a super weird issue I see now and then. Log into Gluu-server24 service stop Tomcat service stop httpd Wait a minute or so. service start tomcat service start httpd give it a minute or so again and verify the regular login page and services are up, and reload the page. It shouldn't say 404 anymore. ``` This made my idP URLs start working again `https://<yourhostname>/idp/shibboleth` & `https://<yourhostname>/idp/profile/SAML2/POST/SSO`. After that one time though, the only way I've been able to get the URLs to work again is to restore a snapshot of the entire server to a time when it was working. After restoring a snapshot, the URLs will work for some period of time (at least a few days) before dying again. I'm monitoring the page with a check_http_ssl check in nagios and there seems to be no rhyme or reason to why it stops; I am unable to find anything suspicious in the logs around the time it stops working, but it could be that I don't know what I'm looking for. As per Mohib's suggestion in the previous ticket, I upgraded from v. 2.4 to the latest stable CE 2.4.2 following this guide: [https://www.gluu.org/docs/deployment/upgrading/](https://www.gluu.org/docs/deployment/upgrading/). The upgrade is successful but after the upgrade, the idP URLs are still coming up blank. Any suggestions?
None
closed

Answers

By Thomas Maerz user 20 Apr 2016 at 6:49 p.m. CDT

Copy
Thomas Maerz gravatar
I am using CentOS 6.7 x64. The VM has 6GB RAM. I just upgraded from 2.4.0 to 2.4.2. In 2.4.0's setup.properties.last, tomcat is set to 4GB: `tomcat_max_ram=4096`. After I upgraded to 2.4.2, there is no value for `tomcat_max_ram` in `setup.properties.last`, but it was definitely doing this before I upgraded today. The `setup.py` script for 2.4.2 didn't ask me how much RAM to give tomcat, and apparently the [import.py](https://github.com/GluuFederation/community-edition-setup/blob/master/static/scripts/import24.py) script didn't copy that over? Either way, in my current running instance, `/opt/tomcat/conf/gluuTomcatWrapper.conf` has `wrapper.java.maxmemory` set to `4096`: ``` wrapper.java.initmemory=512 wrapper.java.maxmemory=4096 wrapper.filter.message.1000=The JVM has run out of memory. ``` I believe these are the settings that setup.py manipulates anyway. Does this information help? Is there a specific log entry I could be looking for that would tell me why the idP is dying or a log entry that is made when the JVM or tomcat runs out of memory for whatever reason?

By Thomas Maerz user 20 Apr 2016 at 7:22 p.m. CDT

Copy
Thomas Maerz gravatar
`wrapper.log` has some java errors: `ERROR [org.gluu.oxtrust.ldap.service.TemplateService] Failed to write IDP configuration file '/opt/idp/metadata//BD1B744969B688420001F23AD70A-idp-metadata.xml'` This happened when gluu-server first started on the new version 2.4.2 after upgrade. It doesn't say why, though. Could this be related? Here's more from the logfile: [http://pastebin.com/raw/WG96LsBD](http://pastebin.com/raw/WG96LsBD)

By Mohib Zico Account Admin 21 Apr 2016 at 1:39 a.m. CDT

Copy
Mohib Zico gravatar
>> /opt/idp/metadata//BD1B744969B688420001F23AD70A-idp-metadata.xml Problem is in this path. See two '//' in this location.

By Thomas Maerz user 21 Apr 2016 at 9:50 a.m. CDT

Copy
Thomas Maerz gravatar
Any idea what is causing this or what can be done to fix it and prevent it from happening again? This looks like the idP metadata that is generated by the system. Is this a bug?

By Thomas Maerz user 22 Apr 2016 at 1:46 p.m. CDT

Copy
Thomas Maerz gravatar
Should I report this as a bug on GitHub?

By Mohib Zico Account Admin 22 Apr 2016 at 2:06 p.m. CDT

Copy
Mohib Zico gravatar
Please allow us to check this situation in our lab, we will inform you. Here is what we will check ( correct us if we miss anything ): - Upgrade from 2.4.0 ( gluu-server24 ) to gluu-server-2.4.3 - Check the status of IDP metadata

By Thomas Maerz user 22 Apr 2016 at 2:22 p.m. CDT

Copy
Thomas Maerz gravatar
It was also doing it in 2.4.0 before the upgrade after some time passed, so I don't think it is directly related to the upgrade. It is still doing it after the upgrade though. If you'd like, I can send zip up the output of export24.py and send it to you. Doing a fresh install of 2.4.2 and importing the existing config and ldap will trigger the issue. I don't want to post the zip publicly on here though in case it contains sensitive data, so if you can provide me with contact info to send it to or a secure file transfer that would be great.

By Mohib Zico Account Admin 22 Apr 2016 at 2:43 p.m. CDT

Copy
Mohib Zico gravatar
>> It was also doing it in 2.4.0 before the upgrade after some time passed, so I don't think it is directly related to the upgrade. It is still doing it after the upgrade though. 100% of our production servers are using SAML along with OpenID Connect, haven't heard any problem like this from them. That's why we need to know if there is any way we can reproduce the problem. If it's an issue in upgrade script / upgrade procedure we need to push that properly in upgrade doc for everyone who will do upgrade by themselves.

By Thomas Maerz user 22 Apr 2016 at 2:46 p.m. CDT

Copy
Thomas Maerz gravatar
It may be specific to my configuration. Would you like me to send you the tarball of my export file?

By Mohib Zico Account Admin 27 Apr 2016 at 4:33 a.m. CDT

Copy
Mohib Zico gravatar
Thomas, Just to update you on status... I tried 2.4.1 --> 2.4.3 and Shib section is not working properly. I have created an internal ticket. We will update you on status.

By Thomas Maerz user 27 Apr 2016 at 2:05 p.m. CDT

Copy
Thomas Maerz gravatar
If I wipe and start over without importing settings (manually reconfigure everything), will that prevent the idP page from failing again?

By Mohib Zico Account Admin 27 Apr 2016 at 2:32 p.m. CDT

Copy
Mohib Zico gravatar
>> If I wipe and start over without importing settings (manually reconfigure everything), will that prevent the idP page from failing again? I would use files and configuration from 1st script ( export24.py ) - Run export23/export24.py, whichever your current server is - Grab 'backup' directory - Backup LDAP - Make a clean installation of Gluu Server - 2.4.3 - Try to import configurations files for new ldap and certificates Yes, that won't be easy but you will be able to grab a new running Gluu Server with all previous configurations.

By Thomas Maerz user 27 Apr 2016 at 3:08 p.m. CDT

Copy
Thomas Maerz gravatar
I think that is actually what I already did (almost): gluu-server24 --> gluu-server-2.4.2. Has something been fixed in 2.4.3 since then? I'm not sure how what you're describing is much different, unless you're suggesting that I don't use the import24.py script to restore the backup and do that step manually. I will do a fresh install and manually reconfigure from scratch with no backup or import and see if the shib stays online then. My LDAP contents are synced from AD anyway so I will just take screenshots of my configuration and start over.

By Arunmozhi P user 28 Apr 2016 at 9:57 a.m. CDT

Copy
Arunmozhi P gravatar
Hi Thomas, To update you on the status of the export and import. Our testing has shown that the directory ownership for the Shibboleth files aren't properly set as the import is done by the root user and ownership changes to the root user. We are working on a fix for this and would update you once the script is fixed. Thank you for the efforts.

By Thomas Maerz user 28 Apr 2016 at 10:10 a.m. CDT

Copy
Thomas Maerz gravatar
Is there a manual workaround I can perform in the meanwhile to get up and running?

By Mohib Zico Account Admin 16 May 2016 at 8:57 a.m. CDT

Copy
Mohib Zico gravatar
[Upgrade](https://gluu.org/docs/deployment/upgrading/) script upgraded. Please feel free to test.

Post an answer