By: Lukasz Golinski named 20 Jun 2022 at 4:41 a.m. CDT

18 Responses
Lukasz Golinski gravatar
## Expected behavior Querying scim user devices by keyhandle should not make opendj unresponsive. ## Actual behavior We have found an issue during querying fido devices by keyhandle using scim-client. When the user is created and no devices are registered and we perform a get query on the following enpoint: /scim/v2/FidoDevices we noticed that that scim service throws an LdapSearchException mentioning that the base entry does not exist. This is probably expected, but any further get queries to /scim/v2/FidoDevices are timed out. It seems like opendj is unresponsive. In the logs I attach exception being logged by scim component. We use scim-client version 4.2.3.Final ## Minimized example 1. Create a user 2. Search device by keyhandle 3. Search device by keyhandle (this query is timed out and at this point ldap is unresponsive)

By Thomas Gasmyr Mougang staff 20 Jun 2022 at 5:40 a.m. CDT

Thomas Gasmyr Mougang gravatar
Hello Lukasz, Can you provide the full log file?

By Thomas Gasmyr Mougang staff 21 Jun 2022 at 11:18 a.m. CDT

Thomas Gasmyr Mougang gravatar
We are not able to replicate this issue. Our suggestion is this: As you are using Gluu server 4.4.0, can you use scim-client 4.4.0 instead of 4.2.3?

By Lukasz Golinski named 24 Jun 2022 at 4:49 a.m. CDT

Lukasz Golinski gravatar
I did some additional tests. The outcome is that actually ldap is working fine, but further scim service queries takes more than 20 seconds to complete. Here is a simplified scenario: ``` public void test() { log.info("test"); for (int i = 0; i < 4; i++) { long startTime = System.currentTimeMillis(); log.info("Iteration: " + i); ClientSideService client = ScimClientFactory.getClient(...); try { String inum = "dd539fac-088d-4923-826e-9d1449c75528"; // This user should exist without any registered devices String query = "urn:ietf:params:scim:schemas:core:2.0:FidoDevice:deviceKeyHandle eq \"somekeyhandle\""; Integer startIndex = 1; Integer count = 200; String sortBy = "meta.created"; String order = "descending"; log.info("Performing ClientSideService query"); Response response = client.searchDevices(inum, query, 1, 200, sortBy, order, (String) null, (String) null); log.info("Success"); } catch (Exception e) { log.error("Unable to fetch"); } finally { long endTime = System.currentTimeMillis(); long duration = (endTime - startTime); log.info("Lasted: " + duration); client.close(); } } } ``` I also attach the logs of the scim service and the above simplified application. I also confirm that ldap is working properly... during running above test querying ldap via ldapsearch works as expected.

By Lukasz Golinski named 24 Jun 2022 at 5:15 a.m. CDT

Lukasz Golinski gravatar
Tested with following versions of scim-client: 4.2.3.Final 4.3.0.Final 4.4.0-SNAPSHOT The problem occurs using all above libraries.

By Thomas Gasmyr Mougang staff 27 Jun 2022 at 5:52 a.m. CDT

Thomas Gasmyr Mougang gravatar
Hi Lukasz, Everything looks good when i execute that example. Here is the output: ``` =====================TEST======= Iteration: 0 2022-06-27 11:39:01.736 INFO 14444 --- [nio-8080-exec-1] o.g.u.security.SecurityProviderUtility : Adding Bouncy Castle Provider 2022-06-27 11:39:02.336 INFO 14444 --- [nio-8080-exec-1] o.g.u.security.SecurityProviderUtility : Provider 'BC' with version 1.7 is added Performing ClientSideService query Success Lasted: 4152 2022-06-27 11:39:05.898 INFO 14444 --- [nio-8080-exec-1] gluu.scim2.client.TestModeScimClient : Closing RestEasy client Iteration: 1 Performing ClientSideService query Success Lasted: 380 2022-06-27 11:39:06.280 INFO 14444 --- [nio-8080-exec-1] gluu.scim2.client.TestModeScimClient : Closing RestEasy client Iteration: 2 Performing ClientSideService query Success Lasted: 205 2022-06-27 11:39:06.486 INFO 14444 --- [nio-8080-exec-1] gluu.scim2.client.TestModeScimClient : Closing RestEasy client Iteration: 3 Performing ClientSideService query Success Lasted: 498 2022-06-27 11:39:06.984 INFO 14444 --- [nio-8080-exec-1] gluu.scim2.client.TestModeScimClient : Closing RestEasy client ``` As you can see the first request took 4152, the next one 380 then 205. So no performance issue after the first request.

By Thomas Gasmyr Mougang staff 27 Jun 2022 at 6:02 a.m. CDT

Thomas Gasmyr Mougang gravatar
Here the result is way better from the previous one due to the reuse of the client connexion. ``` =====================TEST======= 2022-06-27 11:58:57.869 INFO 21360 --- [nio-8080-exec-1] o.g.u.security.SecurityProviderUtility : Adding Bouncy Castle Provider 2022-06-27 11:58:58.621 INFO 21360 --- [nio-8080-exec-1] o.g.u.security.SecurityProviderUtility : Provider 'BC' with version 1.7 is added Iteration: 0 Performing ClientSideService query Success Lasted: 237 Iteration: 1 Performing ClientSideService query Success Lasted: 146 Iteration: 2 Performing ClientSideService query Success Lasted: 122 Iteration: 3 Performing ClientSideService query Success Lasted: 70 2022-06-27 11:59:01.203 INFO 21360 --- [nio-8080-exec-1] gluu.scim2.client.TestModeScimClient : Closing RestEasy client ``` I think you should reuse the client connexion as much as possible.

By Lukasz Golinski named 27 Jun 2022 at 8:39 a.m. CDT

Lukasz Golinski gravatar
Hello Thomas, We actually do reuse the connection in our production code. I was thinking that it might be the cause of the issue we experience, but it seems it is not. Do you confirm that you ran the test **against a user inum that do not have any registered devices** and that you see the following LDAP exception in the scim logs? > Caused by: com.unboundid.ldap.sdk.LDAPSearchException: The search base entry 'ou=fido,inum=dd539fac-088d-4923-826e-9d1449c75528,ou=people,o=gluu' does not exist > Best regards, Lukasz Golinski

By Thomas Gasmyr Mougang staff 28 Jun 2022 at 4:02 a.m. CDT

Thomas Gasmyr Mougang gravatar
Yes, i'm getting the same error: ``` Caused by: com.unboundid.ldap.sdk.LDAPSearchException: The search base entry 'ou=fido,inum=7f84ad59-cfd4-4730-b5da-64361440c120,ou=people,o=gluu' does not exist at com.unboundid.ldap.sdk.LDAPConnection.search(LDAPConnection.java:3966) ~[unboundid-ldapsdk-6.0.4.jar:6.0.4] at org.gluu.persist.ldap.operation.impl.LdapOperationServiceImpl.nextSearchResult(LdapOperationServiceImpl.java:515) ~[gluu-orm-ldap-4.4.0.Final.jar:?] ``` That user exist in LDAP but doesn't have a fido devices registered.

By Lukasz Golinski named 04 Jul 2022 at 9:25 a.m. CDT

Lukasz Golinski gravatar
Hi Thomas, Do you have any idea regarding what configuration may cause such behavior? This problem occurs on all of our kubernetes environments. Today I tested changing the scim component and opendj component image to "base gluu" image, but it didn't reslve the problem. I also noticed that after scim pod restart this problem happen after 10th query. Best regards, Lukasz Golinski

By Lukasz Golinski named 04 Jul 2022 at 9:30 a.m. CDT

Lukasz Golinski gravatar
Looking at the gluu.ldap.properties.tmpl it would indicate that the maxConnection pool is exhausted.

By Lukasz Golinski named 04 Jul 2022 at 10:03 a.m. CDT

Lukasz Golinski gravatar
I can confirm that changing the maxConnection property in scim component to 15 changed the initial number of queries without failure to 15.

By Thomas Gasmyr Mougang staff 05 Jul 2022 at 3:55 a.m. CDT

Thomas Gasmyr Mougang gravatar
Hi, This a section from the documentation that i think will fix your issue ``` The following lists the steps required to switch the java client to support access in a multithreaded environment: Set a Java variable of name httpclient.multithreaded with any value. This will make the scim-client use the PoolingHttpClientConnectionManager. To override the default maximum number of total connections the manager uses, supply variable httpclient.multithreaded.maxtotal with the value of your choosing. To override the default maximum number of connections per route, supply variable httpclient.multithreaded.maxperroute. (Applicable when using UMA protection mode) Check RPT connection pooling is enabled in oxTrust. Login to oxTrust and go to Configuration > JSON Configuration. Scroll down to rptConnectionPoolUseConnectionPooling and set the flag to true. To finish press Save configuration at the bottom of the page. ``` Reference: https://gluu.org/docs/gluu-server/4.4/user-management/scim2/

By Lukasz Golinski named 05 Jul 2022 at 8:59 a.m. CDT

Lukasz Golinski gravatar
Hello Thomas, I tried the above configuration updates, but it seems that it doesn't fix the issue. I think that issue lies not within the scim-client configuration, but in the scim component configuration (scim deployment). I've noticed following behavior upon changing gluu-ldap.properties file inside scim deployment. 1. When I change maxConnection to 15 first 15 queries takes as little as 500 ms to process 2. All other queries above 15 which runs the same query takes connection.max-wait-time-millis value (20 seconds by default). The interesting point is that you are not able to replicate this problem on your environment.

By Mohib Zico staff 01 Aug 2022 at 12:45 a.m. CDT

Mohib Zico gravatar
Gasmyr is in sick leave. I am re-assigning to @Mobarak Hosen.Shakil for double confirmation.

By Mobarak Hosen Shakil staff 05 Aug 2022 at 10:09 a.m. CDT

Mobarak Hosen Shakil gravatar
Hello Lukasz, Thanks for those points. I will try to replicate them. Regards ~ Shakil

By Mobarak Hosen Shakil staff 17 Aug 2022 at 11:03 p.m. CDT

Mobarak Hosen Shakil gravatar
Hi Lukasz, Just for an update. I have tested in my setup through `curl`, Couldn't able to replicate yet. I will try using scim-client. Once I am done, will share the update. Thanks, Regards ~ Shakil

By Lukasz Golinski named 18 Aug 2022 at 2:21 a.m. CDT

Lukasz Golinski gravatar
We found a workaround from our side. We modified all scim device queries and removed the inum parameter in the query. The result is that we actually query all users and not a single person. For sure this in the performance wise this is not really good but we ommit LDAPSearchException to be thrown and therefore connections from scim component to opendj to be "hanging" Nevertheless I think this issue should be covered by a fix in scim service I believe. Regards, Lukasz

By Mobarak Hosen Shakil staff 19 Aug 2022 at 8:35 p.m. CDT

Mobarak Hosen Shakil gravatar
Noted Sir. Please allow us some time to replicate this issue properly. Once We replicated it, will be fixed then soon. Thanks