By: Neemesh Patel user 16 Jan 2017 at 10:31 a.m. CST

4 Responses
Neemesh Patel gravatar

We're using Gluu as an IdP for both SAML and openID Connect.

Within the last hour it appears that Gluu wasn't able to log any user in (be that admin or otherwise). Restarting the Gluu service the first time it happened, allowed everyone to login once more, but it was short-lived and Gluu became unavailable once more about 20 mins later.

I've restarted it again, and it appears to be back up once more but I'm keen to find what the issue is.

The first time it was restarted, apache didn't shut down but that normally seems to happen for us so I just started Gluu back up. The second time, apache also didn't want to shutdown, but this time I forcefully killed it with a "kill -9" and started it back up.

In both instances it would display the login page, but once you try to login you received an apache HTTP 500 error "invalid stream header" (example further below).

Could anyone shed light on what happened as it seemed to come out of nowhere? The server has 16GB RAM, Ubuntu 14.04, had been running fine for two weeks since it's last restart, no updates/upgrades have been applied recently, it had 400Gb of storage free, 9GB of RAM free and the CPU wasn't being taxed.

Below is the output you see in the browser when trying to login:

HTTP Status 500 - invalid stream header: FD73972C

type Exception report

message invalid stream header: FD73972C

description The server encountered an internal error that prevented it from fulfilling this request.

exception

java.io.StreamCorruptedException: invalid stream header: FD73972C java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804) java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) com.sun.faces.renderkit.ApplicationObjectInputStream.<init>(ApplicationObjectInputStream.java:81) com.sun.faces.spi.SerializationProviderFactory$JavaSerializationProvider.createObjectInputStream(SerializationProviderFactory.java:211) com.sun.faces.renderkit.ResponseStateManagerImpl.getTreeStructureToRestore(ResponseStateManagerImpl.java:170) javax.faces.render.ResponseStateManager.getState(ResponseStateManager.java:226) com.sun.faces.application.StateManagerImpl.restoreTree(StateManagerImpl.java:650) com.sun.faces.application.StateManagerImpl.restoreView(StateManagerImpl.java:131) org.jboss.seam.jsf.SeamStateManager.restoreView(SeamStateManager.java:76) com.sun.faces.application.ViewHandlerImpl.restoreView(ViewHandlerImpl.java:311) org.jboss.seam.jsf.SeamViewHandler.restoreView(SeamViewHandler.java:93) com.sun.facelets.FaceletViewHandler.restoreView(FaceletViewHandler.java:317) com.sun.faces.lifecycle.RestoreViewPhase.execute(RestoreViewPhase.java:175) com.sun.faces.lifecycle.Phase.doPhase(Phase.java:100) com.sun.faces.lifecycle.RestoreViewPhase.doPhase(RestoreViewPhase.java:103) com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:118) javax.faces.webapp.FacesServlet.service(FacesServlet.java:265) org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) org.jboss.seam.web.RewriteFilter.process(RewriteFilter.java:98) org.jboss.seam.web.RewriteFilter.doFilter(RewriteFilter.java:57) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:60) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73) org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.MultipartFilter.doFilter(MultipartFilter.java:90) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158)

note The full stack trace of the root cause is available in the Apache Tomcat/7.0.65 logs.

Below is a log snippet from the wrapper.log

INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:31 PM org.richfaces.cache.lru.LRUMapCacheFactory createCache INFO | jvm 1 | 2017/01/16 16:09:32 | INFO: Creating LRUMap cache instance of 512 items capacity INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:31 PM org.richfaces.application.InitializationListener onStart INFO | jvm 1 | 2017/01/16 16:09:32 | INFO: RichFaces Core Implementation by JBoss by Red Hat, version 4.3.7.Final INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:32 PM org.apache.catalina.session.StandardManager doLoad INFO | jvm 1 | 2017/01/16 16:09:32 | SEVERE: IOException while loading persisted sessions: java.io.EOFException INFO | jvm 1 | 2017/01/16 16:09:32 | java.io.EOFException INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2332) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2801) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:56) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:244) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.load(StandardManager.java:202) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.startInternal(StandardManager.java:489) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5526) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:677) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1942) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.FutureTask.run(FutureTask.java:262) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.lang.Thread.run(Thread.java:745) INFO | jvm 1 | 2017/01/16 16:09:32 | INFO | jvm 1 | 2017/01/16 16:09:32 | Jan 16, 2017 4:09:32 PM org.apache.catalina.session.StandardManager startInternal INFO | jvm 1 | 2017/01/16 16:09:32 | SEVERE: Exception loading sessions from persistent storage INFO | jvm 1 | 2017/01/16 16:09:32 | java.io.EOFException INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2332) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2801) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.CustomObjectInputStream.<init>(CustomObjectInputStream.java:56) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.doLoad(StandardManager.java:244) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.load(StandardManager.java:202) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.session.StandardManager.startInternal(StandardManager.java:489) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5526) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:677) INFO | jvm 1 | 2017/01/16 16:09:32 | at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1942) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.FutureTask.run(FutureTask.java:262) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) INFO | jvm 1 | 2017/01/16 16:09:32 | at java.lang.Thread.run(Thread.java:745) INFO | jvm 1 | 2017/01/16 16:09:32 | INFO | jvm 1 | 2017/01/16 16:09:32 | 2017-01-16 16:09:32,052 INFO [org.jboss.seam.servlet.SeamFilter] Initializing filter: org.jboss.seam.web.contextFilter INFO | jvm 1 | 2017/01/16 16:09:32 | 2017-01-16 16:09:32,052 INFO [org.jboss.seam.servlet.SeamFilter] Initializing filter: org.jboss.seam.web.redirectFilter

Any help or insight would be greatly appreciated

By Michael Schwartz staff 16 Jan 2017 at 10:33 a.m. CST

Michael Schwartz gravatar

This is a JSF error we've seen. What version of the Gluu Server are you using? You might want to update to 2.4.4.2

By Aliaksandr Samuseu staff 16 Jan 2017 at 10:43 a.m. CST

Aliaksandr Samuseu gravatar

Hi, Neemesh

Not sure what to add, Michael is right, that looks very similar to known issue older packages had. If you indeed use 2.4.3 like you specified in tags, you should consider upgrading to the CE 2.4.4.2 asap.

By Neemesh Patel user 16 Jan 2017 at 10:50 a.m. CST

Neemesh Patel gravatar

Thanks for the prompt responses, I'll see if there's a good time we can schedule this.

Two quick questions on the upgrade (sorry to switch topics):

  • Is our data in 2.4.3 compatible with 2.4.4.2 or will this require data migration of any kind?
  • We customised the login screen pages and some of the UI to reflect our branding, in an upgrade do you know if this is typically overwritten (these are xhtml files on the server we would have customised)?

Thanks

By Mohib Zico staff 26 Jan 2017 at 8:07 a.m. CST

Mohib Zico gravatar

Is our data in 2.4.3 compatible with 2.4.4.2 or will this require data migration of any kind?

If I remember correctly, it should work. However feel free to snapshot backup your existing 2.4.3.

We customised the login screen pages and some of the UI to reflect our branding, in an upgrade do you know if this is typically overwritten (these are xhtml files on the server we would have customised)?

There are couple of ways to migrated customization files there in new server. The easiest way is copy-paste.