Cache Refresh throwing Null Reference Exception

By: Daniel Ascione user 27 Nov 2016 at 10:01 p.m. CST

14 Responses
Daniel Ascione gravatar
I have setup Cache Refresh for our internal LDAP server, but I'm receiving the following Null Reference Exception in the refresh logs at each interval. I can send any relevant connection information, but using the same variables and the ldapsearch command in opendj in the command line, it successfully connected and pulled records. > 2016-11-27 22:55:40,790 ERROR [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (pool-2-thread-2) Exception happened while executing cache refresh synchronization java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:514) at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.toLdapProperties(CacheRefreshTimer.java:1196) at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.prepareLdapServerConnection(CacheRefreshTimer.java:1050) at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.prepareLdapServerConnection(CacheRefreshTimer.java:1040) at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.prepareLdapServerConnections(CacheRefreshTimer.java:1030) at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.processImpl(CacheRefreshTimer.java:247) at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.process(CacheRefreshTimer.java:178) at sun.reflect.GeneratedMethodAccessor804.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.seam.util.Reflections.invoke(Reflections.java:22) at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56) at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.jboss.seam.async.AsynchronousInterceptor.aroundInvoke(AsynchronousInterceptor.java:52) at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107) at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196) at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114) at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer_$$_javassist_seam_34.process(CacheRefreshTimer_$$_javassist_seam_34.java) at sun.reflect.GeneratedMethodAccessor803.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.jboss.seam.util.Reflections.invoke(Reflections.java:22) at org.jboss.seam.util.Reflections.invokeAndWrap(Reflections.java:144) at org.jboss.seam.Component.callComponentMethod(Component.java:2313) at org.jboss.seam.core.Events.raiseEvent(Events.java:85) at org.jboss.seam.async.AsynchronousEvent$1.process(AsynchronousEvent.java:33) at org.jboss.seam.async.Asynchronous$ContextualAsynchronousRequest.run(Asynchronous.java:80) at org.jboss.seam.async.AsynchronousEvent.execute(AsynchronousEvent.java:27) at org.jboss.seam.async.ThreadPoolDispatcher$RunnableAsynchronous.run(ThreadPoolDispatcher.java:142) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)
CentOS72
closed

Answers

By Michael Schwartz Account Admin 27 Nov 2016 at 10:27 p.m. CST

Copy
Michael Schwartz gravatar
Post screenshots of your cache refresh config tabs in oxTrust. Likely you have something misconfigured.

By Daniel Ascione user 27 Nov 2016 at 10:39 p.m. CST

Copy
Daniel Ascione gravatar
Thanks for the quick response, I'm hoping it's something simple. https://postimg.org/gallery/1re06zm8e/ There are the screenshots. Two things to note: 1. On the LDAP config, ignore the level, it is 0, I accidentally put 97 in the text field before taking the screenshot 2. For the ldap host, we tried that host without a port but ended up leaving the default ldap port specified

By Michael Schwartz Account Admin 27 Nov 2016 at 10:50 p.m. CST

Copy
Michael Schwartz gravatar
A few comments: * Anonymous BIND? You can test if this will work using the CLI ``` # service gluu-server-2.4.4 login # cd /opt/opendj/bin # ./ldapsearch -h (host) -p 389 -b "dc=example,dc=com" "(&(sn=s*)(objectclass=user))" dn ``` Does that return anything? * You don't need to map attributes names that are the same. (i.e. cn --> cn) * Set polling interval to 3 for testing... then set it higher when it's workign Otherwise it dones't look too bad. I bet its a connection problem with the backend ldap server.

By Daniel Ascione user 27 Nov 2016 at 10:58 p.m. CST

Copy
Daniel Ascione gravatar
Michael, Yep, in my initial request I said I ran the ldapsearch command, but I ran your example and it returned a list of DNs just fine. We allow anonymous binding for internal machines, so I'm not worried about permissions. I did remove the attributes that are mapped to the same names. I did have the polling interval at 5 for testing, but upped it to 120 when the errors kept piling up. After letting it run a couple more times, it's still producing the same errors.

By Daniel Ascione user 27 Nov 2016 at 11 p.m. CST

Copy
Daniel Ascione gravatar
One thing I was unsure about is the server IP address. It's currently the external IP of the server Gluu is running on. Is this correct? Should I revert back to the original broadcast IP (255.255.255.255)?

By Michael Schwartz Account Admin 27 Nov 2016 at 11:01 p.m. CST

Copy
Michael Schwartz gravatar
It should be the IP address of eth0. Try adding credentials. Not many people use anonymous, so that could be the problem. It seems to be throwing on error on connection.

By Daniel Ascione user 27 Nov 2016 at 11:18 p.m. CST

Copy
Daniel Ascione gravatar
The server IP is the IP of eth0 on the server. I changed to an authenticated bind (tested with ldapsearch) but it's giving a more generic could not connect error > 2016-11-28 00:15:40,793 ERROR [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (pool-2-thread-3) Failed to connect to LDAP server using configuration PPPL.GOV 2016-11-28 00:15:40,833 ERROR [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (pool-2-thread-3) Skipping cache refresh due to invalid server configuration I have the DN in correctly and bind password set as well. I'll continue to try variants, but if it's working in the command line, I'm not sure what the difference would be.

By Michael Schwartz Account Admin 27 Nov 2016 at 11:29 p.m. CST

Copy
Michael Schwartz gravatar
Yes, definitely try again using ldapsearch to make sure there is connectivity from the Gluu Server. ``` # echo {password} > ~/.pw # ./ldapsearch -h {host} -p 389 -D "{cn=manager}" -j ~/.pw -b "dc=example,dc=com" "(&(sn=s*)(objectclass=user))" dn ... Replace everyting in {} with real values for your system... ``` Let's see if Alex has any more ideas tomorrow. Is there many a seperate cache-refresh log with more detail in `/opt/tomcat/logs`?

By Daniel Ascione user 28 Nov 2016 at 10:11 a.m. CST

Copy
Daniel Ascione gravatar
We are still unable to get the interface to connect to our LDAP server. We're using the exact same variables in the web interface in test queries using ldapsearch, and the command line always works. The previously provided ldapsearch string works perfectly but fails when oxtrust cache refresh tries to use them. Looking in the logs, I am seeing nothing more than: > 2016-11-28 10:28:40,792 ERROR [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (pool-2-thread-7) Failed to connect to LDAP server using configuration PPPL.GOV 2016-11-28 10:28:40,838 ERROR [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (pool-2-thread-7) Skipping cache refresh due to invalid server configuration

By Aliaksandr Samuseu staff 28 Nov 2016 at 10:31 a.m. CST

Copy
Aliaksandr Samuseu gravatar
Hi, Daniel. Are you using a clean 2.4.4 CE package? Or may be the latest 2.4.4.2 one (with SP2 integrated in it)? You could check with `# rpm -qi gluu-server-2.4.4` (outside of container) A couple more notes: 1. You use DNS name when specifying your backend. Are you sure it's resolvable **from inside the container**? Have you tried to use ip address instead? You also may try to ping it by that name from within the container. Please check `/etc/resolv.conf` file in there too, does it contain nameservers you need to resolve names in your network? 2. Configuration screenshots you shared are from time you tried to use anonymous bind. Could you provide the current ones, as I believe you are using password authentication now, correct? 3. Could you also share that console search command you run against your backend successfully, for reference?

By Daniel Ascione user 28 Nov 2016 at 10:39 a.m. CST

Copy
Daniel Ascione gravatar
Aliaksandr, We are using this version of the package: gluu-server-2.4.4-1-1.centos7.src.rpm 1. We have our two DNS servers in the resolv.conf file and we can ping the server by hostname, verifying hostname lookup 2. https://postimg.org/gallery/1p3o30ry0/ 3. ./ldapsearch -h pppl-rodc.pppl.gov -p 389 -D "CN=Daniel R Ascione,OU=Domain Users,DC=pppl,DC=gov" -j .pw -b "dc=pppl,dc=gov" "(&(cn=D*)(objectclass=user))" dn I obviously cannot provide the contents of the .pw file, but it is the same password we have set in the 'Change Bind Password' feature under the LDAP server tab

By Aliaksandr Samuseu staff 28 Nov 2016 at 10:47 a.m. CST

Copy
Aliaksandr Samuseu gravatar
You have "Max connections" at "Source backend" tab set to "0". Let's try to set it to, say, "3" and see whether it will help.

By Daniel Ascione user 28 Nov 2016 at 10:55 a.m. CST

Copy
Daniel Ascione gravatar
Aliaksandr, I set the connections to 100, because I'm not worried about load. With a 1 minute interval, it didn't work for a few iterations. However, I backed out of the Gluu container and restarted the container entirely. After it came back up, it seems to have pulled in our users. I'm not sure ultimately what fixed it, but I'll leave the configs as is for now. We can close this out if you think it's OK. Otherwise, any other config changes I'd be happy to hear about. Thanks for all your help. Please pass along my appreciation to Michael.

By Aliaksandr Samuseu staff 28 Nov 2016 at 11:03 a.m. CST

Copy
Aliaksandr Samuseu gravatar
> I set the connections to 100, because I'm not worried about load. With a 1 minute interval, it didn't work for a few iterations. For CR it doesn't do much difference, limit of 3-5 will do. It's more important for authentication-related part of functionality, though it's not recommended to set it above 100 connections without proper benchmarking there too. >However, I backed out of the Gluu container and restarted the container entirely. After it came back up, it seems to have pulled in our users. I'm not sure ultimately what fixed it, but I'll leave the configs as is for now. Some delays before new settings are applied may happen, indeed, but usually it's about one or max 2 CR cycles, after that they should be in effect, no restart usually is needed. Perhaps something you changed in last few minutes prior to that restart fixed it. Very likely it still was that "0" value in the connections' limit field, I don't think I've ever tried this, though I understand logic behind your decision, in many cases "0" would mean "no limit". Perhaps in case of this setting it literally means zero, I'll need to check it out for myself. Ok, thanks for confirmation, closing it now.

Post an answer