By: Jozef Babjak user 20 Mar 2017 at 5:08 a.m. CDT

12 Responses

As discussed by our and your architects last week, we are trying to set up Gluu in IP-agnostic way to make it able to run behind Amazon's Elastic Load Balancer. In our setup, the application will sit behind load balancer. The 'oxauth' application -- which should be stateless, right -- is going be auto-scaled, so we do not want to be coupled with an IP address of the transient instances. If local server IP address mapping to external hostname is removed from /etc/hosts file inside the chroot(1) environment, the 'oxauth' application logs the following NullPointerException on startup; I will add another symptom in comments, but it seems that this is the point where problems start: ``` 2017-03-20 09:50:45,611 INFO [qtp242131142-11] [org.xdi.oxauth.auth.AuthenticationFilter] (AuthenticationFilter.java:365) - JWT authentication failed: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException javax.servlet.ServletException: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:96) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.resteasy.ResteasyResourceAdapter.getResource(ResteasyResourceAdapter.java:120) ~[jboss-seam-resteasy-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamResourceServlet.service(SeamResourceServlet.java:80) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[servlet-api-3.1.jar:3.1.0] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:845) ~[?:?] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) ~[?:?] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:83) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.web.RewriteFilter.doFilter(RewriteFilter.java:63) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:60) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.xdi.oxauth.auth.AuthenticationFilter.processJwtAuth(AuthenticationFilter.java:361) [classes/:?] at org.xdi.oxauth.auth.AuthenticationFilter.access$200(AuthenticationFilter.java:67) [classes/:?] at org.xdi.oxauth.auth.AuthenticationFilter$1.process(AuthenticationFilter.java:113) [classes/:?] at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:65) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.xdi.oxauth.auth.AuthenticationFilter.doFilter(AuthenticationFilter.java:89) [classes/:?] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) [jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751) [jetty-servlet-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110) [jetty-io-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.15.v20161220.jar:9.3.15.v20161220] at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.15.v20161220.jar:9.3.15.v20161220] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] Caused by: org.jboss.resteasy.spi.UnhandledException: java.lang.NullPointerException at org.jboss.resteasy.core.SynchronousDispatcher.handleApplicationException(SynchronousDispatcher.java:365) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.handleException(SynchronousDispatcher.java:233) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.handleInvokerException(SynchronousDispatcher.java:209) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:557) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.seam.resteasy.ResteasyResourceAdapter$1.process(ResteasyResourceAdapter.java:145) ~[jboss-seam-resteasy-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:65) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] ... 50 more Caused by: java.lang.NullPointerException at org.xdi.net.InetAddressUtility.getMACAddressOrNullImpl(InetAddressUtility.java:57) ~[oxUtil-3.0.1.jar:?] at org.xdi.net.InetAddressUtility.getMACAddressOrNull(InetAddressUtility.java:45) ~[oxUtil-3.0.1.jar:?] at org.xdi.oxauth.model.audit.OAuth2AuditLog.<init>(OAuth2AuditLog.java:28) ~[classes/:?] at org.xdi.oxauth.token.ws.rs.TokenRestWebServiceImpl.requestAccessToken(TokenRestWebServiceImpl.java:108) ~[classes/:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112] at org.jboss.seam.util.Reflections.invoke(Reflections.java:22) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.resteasy.ResteasyContextInjectionInterceptor.aroundInvoke(ResteasyContextInjectionInterceptor.java:59) ~[jboss-seam-resteasy-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] at org.xdi.oxauth.token.ws.rs.TokenRestWebServiceImpl_$$_javassist_seam_43.requestAccessToken(TokenRestWebServiceImpl_$$_javassist_seam_43.java) ~[classes/:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_112] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_112] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112] at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:167) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:269) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:227) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:216) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:542) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:524) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:126) ~[resteasy-jaxrs-2.3.7.Final.jar:?] at org.jboss.seam.resteasy.ResteasyResourceAdapter$1.process(ResteasyResourceAdapter.java:145) ~[jboss-seam-resteasy-2.3.1.Final.jar:2.3.1.Final] at org.jboss.seam.servlet.ContextualHttpServletRequest.run(ContextualHttpServletRequest.java:65) ~[jboss-seam-2.3.1.Final.jar:2.3.1.Final] ... 50 more ```

U1404

closed

By Jozef Babjak user 20 Mar 2017 at 5:13 a.m. CDT

Copy

The next symptom is that when accessing Gluu 'identity' application login page, an error is displayed: ``` java.lang.IllegalStateException: Counter for nested ContextualHttpServletRequest was removed before it should be! ``` Unfortunately, it seems that this error is not logged into any logging file under /opt/gluu/jetty/*/logs/, so I cannot provide the stack trace.

By Mohib Zico staff 20 Mar 2017 at 6:31 a.m. CDT

Copy

Hi Jozef, When you installed Gluu server, which IP address you used?

By Jozef Babjak user 20 Mar 2017 at 6:45 a.m. CDT

Copy

During the installation, the IP address of the only network interface has been used. Basically, the setup.py script was left to use the IP address it detected. That IP address was apparently used in 3 places: 1) HTTP -- port 80 -- VirtualHost directive in Apache config. 2) HTTPS -- port 443 -- VirtualHost directive in Apache config. 3) /etc/hosts where that local IP address is mapped to the hostname specified on setup time. Replacing 1) and 2) by "*" works pretty well, apache correctly listens on one and only interface, even if local IP changes. Anyway, removing mapping from /etc/hosts causes the problem described, even if IP stays the same. Please note that our Elastic Load Balancer is configured as "TCP", so it passes plain TCP traffic through it, it is _not_ configured as HTTP(S) "man-in-the-middle". In another words, any traffic sent directly to local IP address should be pretty same as if passed through ELB.

By Mohib Zico staff 20 Mar 2017 at 8:58 a.m. CDT

Copy

Thanks. I think I'll give a shot locally to check what's happening actually. Please correct me if I am wrong but here is what we will check: - Install a Gluu Server in 14.04 AWS - ELB will be infront of this instance - Change IP address in instance to check how it is doing: - provide asterisk in apache configuration - Add new IP ( local ) in /etc/hosts

By Jozef Babjak user 20 Mar 2017 at 9:08 a.m. CDT

Copy

I think it is even easier. * Install Gluu server in AWS. * ELB is set up; HTTP (or TCP) 80<->80, TCP 443<->443 * Hostname in DNS set up as CNAME for ELB's hostname. In that setup, everything works. Now what to change: * Remove local IP address mapping to external hostname from /etc/hosts; restart the AWS instance. After this change, you have to see the error I mentioned.

By Mohib Zico staff 20 Mar 2017 at 2:37 p.m. CDT

Copy

Thanks much. Moving forward...

By Mohib Zico staff 21 Mar 2017 at 10:03 a.m. CDT

Copy

Hi Jozef, Base environment setup is completed here, the next step is testing by removal of local IP address mapping and vm reboot. But before that I am going to send you the procedure which I followed to create this setup over email ( as this is public ticket.. so not sharing info here ).

By Mohib Zico staff 29 Mar 2017 at 12:58 a.m. CDT

Copy

**Status** Trying to replicate the issue locally. Communication going on with Jozef Babjak over email.

By Mohib Zico staff 06 Apr 2017 at 8:32 a.m. CDT

Copy

Hi Jozef, Anything we want here in this ticket?

By Jozef Babjak user 13 Apr 2017 at 5:35 a.m. CDT

Copy

Why is this ticket closed? I cannot even see any confirmation whether the issue was reproduced or not.

By Jozef Babjak user 13 Apr 2017 at 6:01 a.m. CDT

Copy

Hmmm, maybe I'm little bit lost in results of your investigation. You are telling me that for you application works correctly even when you remove mapping from /etc/hosts file and let the components communicate through load balancer?

By Mohib Zico staff 13 Apr 2017 at 6:09 a.m. CDT

Copy

Correct.

You need to Login in order to post an answer

JWT authentication failed with NullPointerException when IP mapping is removed from /etc/hosts

By: Jozef Babjak user 20 Mar 2017 at 5:08 a.m. CDT

Answers

By Jozef Babjak user 20 Mar 2017 at 5:13 a.m. CDT

By Mohib Zico staff 20 Mar 2017 at 6:31 a.m. CDT

By Jozef Babjak user 20 Mar 2017 at 6:45 a.m. CDT

By Mohib Zico staff 20 Mar 2017 at 8:58 a.m. CDT

By Jozef Babjak user 20 Mar 2017 at 9:08 a.m. CDT

By Mohib Zico staff 20 Mar 2017 at 2:37 p.m. CDT

By Mohib Zico staff 21 Mar 2017 at 10:03 a.m. CDT

By Mohib Zico staff 29 Mar 2017 at 12:58 a.m. CDT

By Mohib Zico staff 06 Apr 2017 at 8:32 a.m. CDT

By Jozef Babjak user 13 Apr 2017 at 5:35 a.m. CDT

By Jozef Babjak user 13 Apr 2017 at 6:01 a.m. CDT

By Mohib Zico staff 13 Apr 2017 at 6:09 a.m. CDT

Post an answer