Invalid stream header: No user access

By: Neemesh Patel user 16 Jan 2017 at 10:31 a.m. CST

4 Responses
Neemesh Patel gravatar
We're using Gluu as an IdP for both SAML and openID Connect. Within the last hour it appears that Gluu wasn't able to log any user in (be that admin or otherwise). Restarting the Gluu service the first time it happened, allowed everyone to login once more, but it was short-lived and Gluu became unavailable once more about 20 mins later. I've restarted it again, and it appears to be back up once more but I'm keen to find what the issue is. The first time it was restarted, apache didn't shut down but that normally seems to happen for us so I just started Gluu back up. The second time, apache also didn't want to shutdown, but this time I forcefully killed it with a "kill -9" and started it back up. In both instances it would display the login page, but once you try to login you received an apache HTTP 500 error "invalid stream header" (example further below). Could anyone shed light on what happened as it seemed to come out of nowhere? The server has 16GB RAM, Ubuntu 14.04, had been running fine for two weeks since it's last restart, no updates/upgrades have been applied recently, it had 400Gb of storage free, 9GB of RAM free and the CPU wasn't being taxed. ### Below is the output you see in the browser when trying to login: > HTTP Status 500 - invalid stream header: FD73972C > type Exception report > > message invalid stream header: FD73972C > > description The server encountered an internal error that prevented it from fulfilling this request. > > exception > > java.io.StreamCorruptedException: invalid stream header: FD73972C > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804) > java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > com.sun.faces.renderkit.ApplicationObjectInputStream.<init>(ApplicationObjectInputStream.java:81) > com.sun.faces.spi.SerializationProviderFactory$JavaSerializationProvider.createObjectInputStream(SerializationProviderFactory.java:211) > com.sun.faces.renderkit.ResponseStateManagerImpl.getTreeStructureToRestore(ResponseStateManagerImpl.java:170) > javax.faces.render.ResponseStateManager.getState(ResponseStateManager.java:226) > com.sun.faces.application.StateManagerImpl.restoreTree(StateManagerImpl.java:650) > com.sun.faces.application.StateManagerImpl.restoreView(StateManagerImpl.java:131) > org.jboss.seam.jsf.SeamStateManager.restoreView(SeamStateManager.java:76) > com.sun.faces.application.ViewHandlerImpl.restoreView(ViewHandlerImpl.java:311) > org.jboss.seam.jsf.SeamViewHandler.restoreView(SeamViewHandler.java:93) > com.sun.facelets.FaceletViewHandler.restoreView(FaceletViewHandler.java:317) > com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:175) > com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100) > com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:103) > com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118) > javax.faces.webapp.FacesServlet.service(FacesServlet.java:265) > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > org.jboss.seam.web.RewriteFilter.process(RewriteFilter.java:98) > org.jboss.seam.web.RewriteFilter.doFilter(RewriteFilter.java:57) > org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) > org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:60) > org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) > org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73) > org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40) > org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) > org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90) > org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) > org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) > org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) > org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) > org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) > org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) > > note The full stack trace of the root cause is available in the Apache Tomcat/7.0.65 logs. ### Below is a log snippet from the wrapper.log > INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:31 PM org.richfaces.cache.lru.LRUMapCacheFactory createCache > INFO | jvm 1 | 2017/01/16 16:09:32 | INFO: Creating LRUMap cache instance of 512 items capacity > INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:31 PM org.richfaces.application.InitializationListener onStart > INFO | jvm 1 | 2017/01/16 16:09:32 | INFO: RichFaces Core Implementation by JBoss by Red Hat, version 4.3.7.Final > INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:32 PM org.apache.catalina.session.StandardManager doLoad > INFO | jvm 1 | 2017/01/16 16:09:32 | SEVERE: IOException while loading persisted sessions: java.io.EOFException > INFO | jvm 1 | 2017/01/16 16:09:32 | java.io.EOFException > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2332) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2801) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:56) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:244) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.load(StandardManager.java:202) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.startInternal(StandardManager.java:489) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5526) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:677) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1942) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.FutureTask.run(FutureTask.java:262) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.lang.Thread.run(Thread.java:745) > INFO | jvm 1 | 2017/01/16 16:09:32 | > INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:32 PM org.apache.catalina.session.StandardManager startInternal > INFO | jvm 1 | 2017/01/16 16:09:32 | SEVERE: Exception loading sessions from persistent storage > INFO | jvm 1 | 2017/01/16 16:09:32 | java.io.EOFException > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2332) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2801) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:56) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:244) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.load(StandardManager.java:202) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.startInternal(StandardManager.java:489) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5526) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:677) > INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1942) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.FutureTask.run(FutureTask.java:262) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > INFO | jvm 1 | 2017/01/16 16:09:32 | at java.lang.Thread.run(Thread.java:745) > INFO | jvm 1 | 2017/01/16 16:09:32 | > INFO | jvm 1 | 2017/01/16 16:09:32 | 2017-01-16 16:09:32,052 INFO [org.jboss.seam.servlet.SeamFilter] Initializing filter: org.jboss.seam.web.contextFilter > INFO | jvm 1 | 2017/01/16 16:09:32 | 2017-01-16 16:09:32,052 INFO [org.jboss.seam.servlet.SeamFilter] Initializing filter: org.jboss.seam.web.redirectFilter Any help or insight would be greatly appreciated
U1404
closed

Answers

By Michael Schwartz Account Admin 16 Jan 2017 at 10:33 a.m. CST

Copy
Michael Schwartz gravatar
This is a JSF error we've seen. What version of the Gluu Server are you using? You might want to update to 2.4.4.2

By Aliaksandr Samuseu staff 16 Jan 2017 at 10:43 a.m. CST

Copy
Aliaksandr Samuseu gravatar
Hi, Neemesh Not sure what to add, Michael is right, that looks very similar to known issue older packages had. If you indeed use 2.4.3 like you specified in tags, you should consider upgrading to the CE 2.4.4.2 asap.

By Neemesh Patel user 16 Jan 2017 at 10:50 a.m. CST

Copy
Neemesh Patel gravatar
Thanks for the prompt responses, I'll see if there's a good time we can schedule this. Two quick questions on the upgrade (sorry to switch topics): - Is our data in 2.4.3 compatible with 2.4.4.2 or will this require data migration of any kind? - We customised the login screen pages and some of the UI to reflect our branding, in an upgrade do you know if this is typically overwritten (these are xhtml files on the server we would have customised)? Thanks

By Mohib Zico staff 26 Jan 2017 at 8:07 a.m. CST

Copy
Mohib Zico gravatar
>> Is our data in 2.4.3 compatible with 2.4.4.2 or will this require data migration of any kind? If I remember correctly, it should work. However feel free to snapshot backup your existing 2.4.3. >> We customised the login screen pages and some of the UI to reflect our branding, in an upgrade do you know if this is typically overwritten (these are xhtml files on the server we would have customised)? There are couple of ways to migrated customization files there in new server. The easiest way is copy-paste.

Post an answer