Hi, Matt. Let's start with a no-brainer that may save us some time: please try to delete all contents of CR's snapshot directory (by default it's `/var/ox/oxtrust/vds-snapshots/` in the container). Then wait for next pulling cycle and see whether it will help. It may be reasonable to set pulling interval to, say, 2 mins, while we are troubleshooting (unless you have hundreds of thousands user entries in your backend) Meanwhile, could you please provide either screenshots of all your CR web UI pages with all your current settings visible, or output of the next 2 commands? 1. `# /opt/opendj/bin/ldapsearch -h 127.0.0.1 -p 1636 -s sub -T -Z -X -D 'cn=directory manager' -w 'YOUR_LDAP_PASS' -b 'o=gluu' -z 5 '(objectclass=oxtrustconfiguration)' oxTrustConfCacheRefresh ` 2. `# /opt/opendj/bin/ldapsearch -h 127.0.0.1 -p 1636 -s sub -T -Z -X -D 'cn=directory manager' -w 'YOUR_LDAP_PASS' -b 'o=gluu' -z 5 '(objectclass=gluuappliance)' gluuVdsCacheRefreshPollingInterval gluuVdsCacheRefreshEnabled gluuIpAddress` Whichever is easier for you. Could you also describe your setup briefly? Is it a single instance, or a cluster setup? Anything unusual about environment it's placed into? Best regards, Alex.

This is a single instance. There is nothing special about it. It's in a sandbox type environment that is not locked down in any kind of way. Here's the output from those 2 commands. ``` GLUU.root@dev-ecp:/var/ox/oxtrust/vds-snapshots# /opt/opendj/bin/ldapsearch -h 127.0.0.1 -p 1636 -s sub -T -Z -X -D 'cn=directory manager' -w inventrio -b 'o=gluu' -z 5 '(objectclass=oxtrustconfiguration)' oxTrustConfCacheRefresh dn: ou=oxtrust,ou=configuration,inum=@!D400.1B10.7F90.F198!0002!7AFB.4452,ou=appliances,o=gluu oxTrustConfCacheRefresh: {"sourceConfigs":[{"configId":"vitl ldap server","bindDN":"cn=gridadmin,ou=users,dc=trio,dc=giab","bindPassword":"XXXXXXXXXX","servers":["10.10.10.16:1024"],"maxConnections":5,"useSSL":false,"baseDNs":["ou=users,dc=trio,dc=giab"],"primaryKey":null,"localPrimaryKey":null,"useAnonymousBind":false,"enabled":false,"version":0,"level":1}],"inumConfig":{"configId":"local_inum","bindDN":"cn=directory manager","bindPassword":"XXXXXXXXXXXX","servers":["localhost:1636"],"maxConnections":10,"useSSL":true,"baseDNs":["o=site"],"primaryKey":null,"localPrimaryKey":null,"useAnonymousBind":false,"enabled":true,"version":0,"level":0},"targetConfig":{"configId":null,"bindDN":null,"bindPassword":null,"servers":[],"maxConnections":0,"useSSL":false,"baseDNs":[],"primaryKey":null,"localPrimaryKey":null,"useAnonymousBind":false,"enabled":false,"version":0,"level":0},"ldapSearchSizeLimit":100,"keyAttributes":["uid"],"keyObjectClasses":["person"],"sourceAttributes":["uid","sn","cn","mail","givenName","o","employeeNumber"],"customLdapFilter":"","updateMethod":"copy","defaultInumServer":false,"keepExternalPerson":true,"useSearchLimit":true,"attributeMapping":[{"source":"uid","destination":"uid"},{"source":"sn","destination":"sn"},{"source":"cn","destination":"cn"},{"source":"mail","destination":"mail"},{"source":"givenName","destination":"givenName"},{"source":"o","destination":"o"},{"source":"employeeNumber","destination":"nickname"}],"snapshotFolder":"/var/ox/oxtrust/vds-snapshots","snapshotMaxCount":10} ``` ``` GLUU.root@dev-ecp:/var/ox/oxtrust/vds-snapshots# /opt/opendj/bin/ldapsearch -h 127.0.0.1 -p 1636 -s sub -T -Z -X -D 'cn=directory manager' -w inventrio -b 'o=gluu' -z 5 '(objectclass=gluuappliance)' gluuVdsCacheRefreshPollingInterval gluuVdsCacheRefreshEnabled gluuIpAddress dn: inum=@!D400.1B10.7F90.F198!0002!7AFB.4452,ou=appliances,o=gluu gluuVdsCacheRefreshPollingInterval: 1 gluuVdsCacheRefreshEnabled: enabled gluuIpAddress: 10.10.10.5 ```

I found my problem, when I unchecked Load source data with limited search it loaded all the users. Apparently Gluu doesn't do paged searches well with apache-ds. Our actual staging and production environments use Active Directory so hopefully this won't be a problem in those environments but we will have to test that.

Thanks, Matt. Let's try to do a command line search with filter similar to the one CR would use with settings you chose for it, and see whether it will return something too. I take it that this command you provided - `# ldapsearch -x -h 10.10.10.16:1024 -b "ou=users,dc=trio,dc=giab"` - works well for your case. Let's upgrade it a bit to this: `# ldapsearch -x -h 10.10.10.16:1024 -b "ou=users,dc=trio,dc=giab" '(&(uid=*)(objectclass=person))'` Let us know what is the result. From the configs you provided it also appears you have CR's "Load source data with limited search" feature enabled. Could you try to disable it in web UI, and see whether this will help?

> I found my problem, when I unchecked Load source data with limited search it loaded all the users. Apparently Gluu doesn't do paged searches well with apache-ds Yes, you were one step ahead of me, it seems. This feature operates over hardcoded limited selection of characters it will iteratively put into first 2 positions of key attribute, attempting to narrow down amount of entries that will be returned in response to each LDAP request. AFAICR, it's a workaround from the past for some old directories.

The search string you gave me sends back all the users... I will not post them all but the very end of the response was like this ``` # LukeSkywalker3009, users, trio.giab dn: cn=LukeSkywalker3009,ou=users,dc=trio,dc=giab postalCode: 32401 uid: LukeSkywalker3009 userPassword:: XXXXXXXXXXXX employeeNumber: 8a748b8456b4413d0156b44ef8710025 objectclass: organizationalPerson objectclass: person objectclass: inetorgperson objectclass: top givenName: Luke3009 cn: LukeSkywalker3009 sn: Skywalker3009 telephoneNumber: 555-555-5555 street: 3009 Street LN l: Panama City mail: LukeSkywalker3009@test.com o: 8a748b8456b4413d0156b44eec1f000d st: FL # search result search: 2 result: 0 Success # numResponses: 27 # numEntries: 26 ```

Must comment on this: >I found my problem, when I unchecked Load source data with limited search it loaded all the users. Apparently Gluu doesn't do paged searches well with apache-ds The feature you disabled has nothing to do with paged searches, actually. It's sort of alternative to it, which works as I described above. You control paging used for your LDAP requests with the "Search size limit" web UI field alone (it's set to 100 in your case, as I can see in configs you provided). I.e. CR should use paging in your setup without a problem. Is picture different for you?

Oh ok, I guess I just misunderstood the meaning of the checkbox. Thank you very much for your help.

You are welcome, Matt. I've cleared encrypted passwords that were in configs you posted, just in case. Closing ticket now, please feel free to open another one in case of further issues.