By: Todd Higgins user 10 Dec 2019 at 2:37 p.m. CST

5 Responses
Todd Higgins gravatar
I am unable to view or edit any trust relationship that relies on the InCommon Federation Trust relationship. The last time this happened the server had run out of disk space (run away log file) and the metadata file could not be loaded. No disk space issues this time. I know a restart caused my all my federated trusts to stop working, so I am working on a clone of the current production server. On the cloned server I have done the following: - Restarted the server - increased the JAVA OPTIONS= from` -Xms1536m -Xmx4096m` to `-Xmx3072m -Xmx8192m` in /etc/default/idp (server has 16GB allocated - same as prod) - Verified that the latest InCommon metadata file has loaded (65 m in size!) - Checked that the LDAP database (/opt/gluu/data/main_db/data.md) is less that 1GB in size (currently 321 m) Still cannot edit or create new trusts with InCommon and my existing InCommon trusts no longer work ( from my workstation.) snippet of the oxtrust.log: ``` 2019-12-10 09:54:56,117 ERROR [pool-2-thread-6] [org.gluu.oxtrust.ldap.service.EntityIDMonitoringService] (EntityIDMonitoringService.java:149) - Exception happened while checking LDAP connections java.lang.NullPointerException: null ``` snippet from oxauth.log: ``` Caused by: org.gluu.site.ldap.exception.ConnectionException: Failed to authenticate dn --> 80090308: LdapErr: DSID-0C0903D9, comment: AcceptSecurityContext error, data 52e, v2580 at org.gluu.site.ldap.OperationsFacade.authenticate(OperationsFacade.java:112) ~[oxLdap-3.0.1.jar:?] at org.gluu.site.ldap.persistence.LdapEntryManager.authenticate(LdapEntryManager.java:666) ~[oxLdap-3.0.1.jar:?] ... 148 more Caused by: com.unboundid.ldap.sdk.LDAPBindException: 80090308: LdapErr: DSID-0C0903D9, comment: AcceptSecurityContext error, data 52e, v2580 at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2171) ~[unboundid-ldapsdk-3.2.0.jar:3.2.0] at com.unboundid.ldap.sdk.LDAPConnection.bind(LDAPConnection.java:2087) ~[unboundid-ldapsdk-3.2.0.jar:3.2.0] at org.gluu.site.ldap.OperationsFacade.authenticateBindConnectionPoolImpl(OperationsFacade.java:161) ~[oxLdap-3.0.1.jar:?] at org.gluu.site.ldap.OperationsFacade.authenticateImpl(OperationsFacade.java:124) ~[oxLdap-3.0.1.jar:?] at org.gluu.site.ldap.OperationsFacade.authenticate(OperationsFacade.java:110) ~[oxLdap-3.0.1.jar:?] at org.gluu.site.ldap.persistence.LdapEntryManager.authenticate(LdapEntryManager.java:666) ~[oxLdap-3.0.1.jar:?] ``` I can link to the log files once pastebin.com is online

By Aliaksandr Samuseu staff 10 Dec 2019 at 4:31 p.m. CST

Aliaksandr Samuseu gravatar
Hi, Todd. Are you sure these LDAP errors are related to the issue with metadata? So far, your description resembles another federation metadata issue we encountered recently - but it wasn't related to LDAP server. When you browse the list of your TRs in web UI, how does it look? May I assume that's Incommon TR is in "active" state, but all its child TRs are inactive? Please also run next query inside container: 1. Move into Gluu's container 2. Put your LDAP password in `/tmp/.dpw` (it's the same as default admin's password was right after installation) 3. Run this: `# /opt/opendj/bin/ldapsearch -h 127.0.0.1 -p 1636 -s sub -T -Z -X -D 'cn=directory manager,o=gluu' -j /tmp/.dpw -b 'o=gluu' -z 10 '&(objectclass=gluusamlconfig)(gluuIsFederation=true)'` Assuming InCommon is the only federation you use, it should return a single entry. If you'll get a huge entry with hundreds of `gluuEntityId` attribuite values, then we should look somewhere else. Otherwise, please share the output of it.

By Aliaksandr Samuseu staff 10 Dec 2019 at 4:32 p.m. CST

Aliaksandr Samuseu gravatar
And btw, you should also increase heap limits in `/etc/default/identity` file - perhaps your oxTrust just doesn't have enough and crashes when trying to parse that huge metadata blob?

By Todd Higgins user 11 Dec 2019 at 8:31 a.m. CST

Todd Higgins gravatar
Hi Aliaksandr, No I not sure that the LDAP errors are related, but the last time we experienced this issue (in July 2018) that was the road we were traveling down so I figure I would start there. You are correct, the InCommon TR status is "Active" while all of the child TRs are "Inactive" Only one federation, output below: ``` dn: inum=@!4ACB.9863.7345.C9B9!0002!858F.21C6!0006!ED4F.522E,ou=trustRelationships,inum=@!4ACB.9863.7345.C9B9!0002!858F.21C6,ou=appliances,o=gluu description: InCommon Federation TR displayName: InCommon Trust Relationship gluuEntityType: Single SP gluuSpecificRelyingPartyConfig: false inum: @!4ACB.9863.7345.C9B9!0002!858F.21C6!0006!ED4F.522E gluuSAMLmaxRefreshDelay: PT8H o: o=@!4ACB.9863.7345.C9B9!0001!7E87.2A29,o=gluu gluuSAMLspMetaDataSourceType: uri gluuSAMLspMetaDataURL: http://md.incommon.org/InCommon/InCommon-metadata.xml objectClass: top objectClass: gluuSAMLconfig gluuIsFederation: true gluuSAMLspMetaDataFN: 4ACB98637345C9B90002858F21C60006ED4F522E-sp-metadata.xml gluuStatus: active gluuValidationStatus: validation_success ``` I made the same changes to /etc/default/identity last night - no changes other than increased memory utilization

By Aliaksandr Samuseu staff 12 Dec 2019 at 5:20 p.m. CST

Aliaksandr Samuseu gravatar
Hi, Todd. That really looks like the case I mentioned. That was a bug in oxTrust which was triggered when incorrect metadata was downloaded by IDP at some point. It was fixed in 3.1.7 and 4.x packages later. There is no easy way to fix it, restarting anything won't help definetely. The most straighforward way of fixing it is to remove all affected child TRs and the Incommon partent (via direct LDAP writes, if needed), and recreate them manually. That would be the easiest way if you would just have a few child TRs If you have dozens of child TRs, fixing the existing Incommon TR is more time-efficient. 1. Move into container and stop "idp" and "identity" services 2. Make sure that metadata file refrenced from your Incommon TR contains a valid Incommon metadata; file should be at this location: `/opt/shibboleth-idp/metadata/4ACB98637345C9B90002858F21C60006ED4F522E-sp-metadata.xml` 3. Compose LDIF file based on the output you shared in [this post](https://support.gluu.org/authentication/7757/system-error-when-attempting-to-view-trust-relationship-for-sites-federated-with-incommon/#at54768); it will look like this: dn: inum=@!4ACB.9863.7345.C9B9!0002!858F.21C6!0006!ED4F.522E,ou=trustRelationships,inum=@!4ACB.9863.7345.C9B9!0002!858F.21C6,ou=appliances,o=gluu changetype: modify add: gluuEntityId gluuEntityId: https://entityid1 gluuEntityId: https://entityid2 4. Put your LDAP admin password to `/tmp/.dpw` 5. Apply the LDIF: `# /opt/opendj/bin/ldapmodify -h 127.0.0.1 -p 1636 -Z -X -D 'cn=directory manager,o=gluu' -j /tmp/.dpw -f /path/to/ldif.file` At that point you should have a more or less valid fed TR. Start "identity" service, ideally it should re-validate your real metadata on disk and substitute those bogus entityids you used in LDIF with full list from the metadata. Even if it won't, you should be able to access Incommon TR from web UI now. Switch the method of metadata provisioning to "File" and upload the valid metadata file manually. This should fix "oops" error when accessing child TRs. After all TRs are accessible in web UI, you can start "idp" service again. I would recommend upgrading anyway, as 3.0.2 is pretty old version now. Until you'll be able to upgrade, I would suggest to keep Incommon's TR metadata source at "File", and upload fresh metadata manually in controlled fashion, when you'll be able to restore previous configuration, if needed.

By Todd Higgins user 16 Dec 2019 at 2:30 p.m. CST

Todd Higgins gravatar
Hi Alaksandr, Thanks for the the detailed response. I followed your directions for the second option and restored functionality to the InCommon Trust (after I restarted the whole server it works.) Thanks for the assistance. Regards, Todd