Periodic ldap error

By: Ivan Carrion user 03 May 2018 at 5:23 a.m. CDT

8 Responses
Ivan Carrion gravatar
Hi. We have 2 nodes (2cpu's, 4Gb RAM) with Red Hat 7.4, Gluu 3.1.2 and OpenDJ with cluster config. We're getting "periodically" the same error on oxauth: 2018-05-03 08:43:04,834 ERROR [Thread-2106788] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap org.gluu.site.ldap.persistence.exception.EntryPersistenceException: Failed to find entry: inum=@!A6F2.6807.ADCA.FBE4!0002!F17D.3F16,ou=appliances,o=gluu I mean "periodically" because is allways shown at same time (x3 or x7 minutes). Node 1: 2018-05-03 00:27:57,587 ERROR [Thread-822930] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 00:37:57,587 ERROR [Thread-823065] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 00:47:57,587 ERROR [Thread-823227] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 00:57:57,587 ERROR [Thread-823380] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 01:17:57,587 ERROR [Thread-823688] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 01:27:57,587 ERROR [Thread-823811] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 01:57:57,587 ERROR [Thread-824256] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 02:27:57,587 ERROR [Thread-824640] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 02:37:57,587 ERROR [Thread-824766] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 02:47:57,587 ERROR [Thread-824892] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 02:57:57,587 ERROR [Thread-825091] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 03:07:57,587 ERROR [Thread-825269] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 03:27:57,587 ERROR [Thread-825569] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 03:37:57,587 ERROR [Thread-825721] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 03:57:57,587 ERROR [Thread-826126] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 04:07:57,587 ERROR [Thread-826262] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 04:17:57,586 ERROR [Thread-826419] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 04:27:57,587 ERROR [Thread-826699] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 04:37:57,587 ERROR [Thread-827116] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 04:57:57,587 ERROR [Thread-828048] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 05:17:57,587 ERROR [Thread-829883] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 05:27:57,587 ERROR [Thread-830872] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 06:17:57,587 ERROR [Thread-837163] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 06:47:57,587 ERROR [Thread-842692] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 07:37:57,587 ERROR [Thread-852115] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap Node 2: 2018-05-03 03:13:04,823 ERROR [Thread-2069432] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 06:13:04,823 ERROR [Thread-2080278] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 08:03:04,823 ERROR [Thread-2099929] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 08:23:04,823 ERROR [Thread-2103324] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 08:43:04,834 ERROR [Thread-2106788] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 08:53:04,822 ERROR [Thread-2108918] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap 2018-05-03 09:33:04,826 ERROR [Thread-2117612] [org.xdi.oxauth.service.AppInitializer] (AppInitializer.java:554) - Failed to load appliance entry from Ldap As you can see for the hours (please, add 2 hours more for real time), we think there shouldn't be resource reason to increase +4Gb RAM. We have read from other posts that this way does not resolve the issue, but we don't find how to fix it. We have disabled AD refresh and OpenDJ replication, but it still appears. Please, we need some advice. Yes, anyway, we are deploying another cluster with 8GbRAM-nodes, but we think it'll not fix it. Thanks for your support.
Rhel 7.4
closed

Answers

By Mohib Zico staff 04 May 2018 at 10:53 a.m. CDT

Copy
Mohib Zico gravatar
Yes, let us know how 8GB instances are going. You can allocate certain amount of memory for identity,idp,oxauth from /etc/defaults/ location. Also, feel free to allocated 2Gb for OpenDJ only.

By Ivan Carrion user 09 May 2018 at 6:04 a.m. CDT

Copy
Ivan Carrion gravatar
Hi again. We just deployed 2 new nodes with 8Gb in Amazon. 2 nodes are in same VPC and subnet. Now, we receive 401 errors (invalid grant) when the 2 nodes reply on same flow: node 1: 2018-05-09 07:50:40,212 INFO [qtp2008017533-13] [org.xdi.oxauth.service.AuthenticationServ ice] (AuthenticationService.java:494) - Attempting to redirect user: SessionUser: SessionSt ate {dn='oxAuthSessionId=8a7c5683-1ff7-44b5-9b84-fd10ef20a6f6,ou=session,o=@!81C8.B862.947C .FC91!0001!D858.3824,o=gluu', id='8a7c5683-1ff7-44b5-9b84-fd10ef20a6f6', lastUsedAt=Wed May 09 07:50:40 UTC 2018, userDn='inum=@!81C8.B862.947C.FC91!0001!D858.3824!0000!2554.88F0,ou= people,o=@!81C8.B862.947C.FC91!0001!D858.3824,o=gluu', authenticationTime=Wed May 09 07:50: 40 UTC 2018, state=authenticated, sessionState='cbcd1d7c-ed15-42e7-8ff6-81b067a5bce7', perm issionGranted=null, isJwt=false, jwt=null, permissionGrantedMap=org.xdi.oxauth.model.common .SessionIdAccessMap@24dba048, involvedClients=null, sessionAttributes={auth_step=1, acr=aut h_ldap_server, remote_ip=195.55.82.133, 10.50.1.19, auth_external_attributes=[], scope=open id profile email memberOf user_name company, response_type=code, redirect_uri=http://localh ost:9090/login, state=PAyf1g, client_id=@!81C8.B862.947C.FC91!0001!D858.3824!0008!6C2D.DA39 .B78D.41D1}, persisted=true} 2018-05-09 07:50:40,214 INFO [qtp2008017533-13] [org.xdi.oxauth.service.AuthenticationServ ice] (AuthenticationService.java:502) - Attempting to redirect user: User: org.xdi.oxauth.m odel.common.User@356a0a00 2018-05-09 07:50:40,215 INFO [qtp2008017533-13] [org.xdi.oxauth.auth.Authenticator] (Authe nticator.java:357) - Authentication success for User: 'asuch' node 2: 2018-05-09 07:50:40,624 INFO [qtp2008017533-13] [org.xdi.oxauth.auth.Authenticator] (Authe nticator.java:217) - Authentication success for Client: '@!81C8.B862.947C.FC91!0001!D858.38 24!0008!6C2D.DA39.B78D.41D1' After those logs, it throws us 401 error. The 4Gb nodes don't throw those 401 errors in test. We think there's a Race Condition. When we debug the tests, allways runs ok. When we work with only one node, it works. When we work with 2 cluster nodes, it throws 401 errors when 2 nodes work on the response.

By Chris Blanton user 09 May 2018 at 10:45 a.m. CDT

Copy
Chris Blanton gravatar
Hey Ivan, Can you give us a better image of everything you've done to set up your cluster? I'm assuming you have a proxy. Do you also have twemproxy handle cache which points to redis servers?

By Ivan Carrion user 09 May 2018 at 11:56 a.m. CDT

Copy
Ivan Carrion gravatar
A deep view of our AWS Infraestructure: - 1 Classic Load Balancer ELB AWS with sticky session (86400s) - 2 instances EC2 with 2vcpu+8GbRAM Red Hat 7.4 - 1 Redis as a service (Elastic Cache AWS Service) with 1 node (0 replicas, 0 shards) - ELB, 2 instances and redis inside the same VPC and subnet. - Security Groups reviewed to accomplish with port viewing (1636, 443, 4444, 6379, 8989) We have followed the doc page to setup vm's, install gluu 3.1.2 and config: - Cache refresh with AD Ldaps (it works great) - Modify schema with more attributes feeded by cache refresh (it works great) - OpenID Connect configuration with 7200s for unused tokens and 86400s for token lifetime - authentication added to allow "ad authentication" - Json config modified to work with redis - OpenDJ replication (it works great) for o=gluu (validated with Apache Directory Studio and dsreplica status inside Gluu) After that, as I said: - If there's only one Gluu instance in Load Balancer, it works. - If there's two Gluu instances in Load Balancer, it works only when only one node replies in the flow. - When the two instances replies in the flow, it throws 401 errors as: error="invalid_grant", error_description="The provided authorization grant is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client."

By Ivan Carrion user 09 May 2018 at 11:58 a.m. CDT

Copy
Ivan Carrion gravatar
We're going to test with Memcached, because we've read that it could be latency problems that provoke "race condition" with Redis. As soon as we test, I'll write the results.

By Chris Blanton user 09 May 2018 at 5:12 p.m. CDT

Copy
Chris Blanton gravatar
Ivan, > We have followed the doc page to setup vm's, install gluu 3.1.2 Can you point me to what documentation you followed? > 1 Classic Load Balancer ELB AWS with sticky session (86400s) I'm not familiar with this particular load balancer, so I'll have to see if I can provision some resources to try and replicate your issue. Please stand by.

By Ivan Carrion user 10 May 2018 at 1:35 a.m. CDT

Copy
Ivan Carrion gravatar
Thanks for your support, Chris. We have followed only official doc (where applies, because Amazon AWS let us deploy Load Balance as a service instead of setup our own nginx): https://gluu.org/docs/ce/3.1.2/installation-guide/ https://gluu.org/docs/ce/3.1.2/installation-guide/install/ https://gluu.org/docs/ce/3.1.2/installation-guide/cluster/ https://gluu.org/docs/ce/3.1.2/admin-guide/attribute/ https://gluu.org/docs/ce/3.1.2/reference/JSON-oxauth-prop/ and so on... Today we're going to test with Memcached.

By Ivan Carrion user 10 May 2018 at 10:35 a.m. CDT

Copy
Ivan Carrion gravatar
Our initial tests with Memcached say us there's no problem with invalid grant (it seems there's no Race Condition). This week we're going to lunch performance tests to check the config.

Post an answer