By: Gene Liverman user 31 Aug 2016 at 9:26 a.m. CDT

28 Responses
Gene Liverman gravatar
I can't seem to get logged into Gluu anymore... now I get a page saying "Oops Something wrong happened. Login failed, oxTrust wasn't allow to access user data" after authenticating. Below is what I am seeing in my tomcat logs: ``` INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:54,998 INFO [net.sf.ehcache.util.UpdateChecker] New update(s) found: 2.4.7 [http://www.terracotta.org/confluence/display/release/Release+Notes+Ehcache+Core+2.4]. Please check http://ehcache.org for the latest version. INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,017 INFO [org.gluu.oxtrust.action.Authenticator] authorizationCode : 628ac3c6-c0fc-4c1d-b0c9-b04740d10f20 INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,017 INFO [org.gluu.oxtrust.action.Authenticator] scopes : user_name email openid profile INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,017 INFO [org.gluu.oxtrust.action.Authenticator] clientID : @!1A75.F512.73A4.10CF!0001!7EA6.2F19!0008!3257.42B4 INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,017 INFO [org.gluu.oxtrust.action.Authenticator] getting accessToken INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,017 INFO [org.gluu.oxtrust.action.Authenticator] tokenURL : https://gluu-test.westga.edu/oxauth/seam/resource/restv1/oxauth/token INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,021 INFO [org.gluu.oxtrust.action.Authenticator] Sending request to token endpoint INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,021 INFO [org.gluu.oxtrust.action.Authenticator] redirectURI : https://gluu-test.westga.edu/identity/authentication/authcode INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,257 ERROR [org.xdi.oxauth.client.TokenClient] Received fatal alert: handshake_failure INFO | jvm 1 | 2016/08/31 10:09:55 | javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1991) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1098) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1344) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1371) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1355) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:535) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:403) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.client.core.executors.ApacheHttpClient4Executor.execute(ApacheHttpClient4Executor.java:182) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.core.interception.ClientExecutionContextImpl.proceed(ClientExecutionContextImpl.java:39) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.plugins.interceptors.encoding.AcceptEncodingGZIPInterceptor.execute(AcceptEncodingGZIPInterceptor.java:40) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.core.interception.ClientExecutionContextImpl.proceed(ClientExecutionContextImpl.java:45) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.client.ClientRequest.execute(ClientRequest.java:444) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.client.ClientRequest.httpMethod(ClientRequest.java:688) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.client.ClientRequest.post(ClientRequest.java:572) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.resteasy.client.ClientRequest.post(ClientRequest.java:577) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.xdi.oxauth.client.TokenClient.exec(TokenClient.java:306) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.xdi.oxauth.client.TokenClient.execAuthorizationCode(TokenClient.java:112) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.gluu.oxtrust.action.Authenticator.requestAccessToken(Authenticator.java:548) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.gluu.oxtrust.action.Authenticator.oAuthGetAccessToken(Authenticator.java:536) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1 | 2016/08/31 10:09:55 | at java.lang.reflect.Method.invoke(Method.java:606) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.util.Reflections.invoke(Reflections.java:22) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.RootInvocationContext.proceed(RootInvocationContext.java:32) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:56) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.transaction.RollbackInterceptor.aroundInvoke(RollbackInterceptor.java:28) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.core.BijectionInterceptor.aroundInvoke(BijectionInterceptor.java:79) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.core.MethodContextInterceptor.aroundInvoke(MethodContextInterceptor.java:44) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.core.SynchronizationInterceptor.aroundInvoke(SynchronizationInterceptor.java:35) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.SeamInvocationContext.proceed(SeamInvocationContext.java:68) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.RootInterceptor.invoke(RootInterceptor.java:107) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.JavaBeanInterceptor.interceptInvocation(JavaBeanInterceptor.java:196) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.intercept.JavaBeanInterceptor.invoke(JavaBeanInterceptor.java:114) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.gluu.oxtrust.action.Authenticator_$$_javassist_seam_44.oAuthGetAccessToken(Authenticator_$$_javassist_seam_44.java) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) INFO | jvm 1 | 2016/08/31 10:09:55 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) INFO | jvm 1 | 2016/08/31 10:09:55 | at java.lang.reflect.Method.invoke(Method.java:606) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:335) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.el.util.ReflectionUtil.invokeMethod(ReflectionUtil.java:348) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.el.parser.AstPropertySuffix.invoke(AstPropertySuffix.java:58) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.el.parser.AstValue.invoke(AstValue.java:96) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:276) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.core.Expressions$2.invoke(Expressions.java:222) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.navigation.Page.preRender(Page.java:311) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.navigation.Pages.preRender(Pages.java:351) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.jsf.SeamPhaseListener.preRenderPage(SeamPhaseListener.java:565) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.jsf.SeamPhaseListener.beforeRenderResponse(SeamPhaseListener.java:476) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.jsf.SeamPhaseListener.beforeServletPhase(SeamPhaseListener.java:147) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.jsf.SeamPhaseListener.beforePhase(SeamPhaseListener.java:117) INFO | jvm 1 | 2016/08/31 10:09:55 | at com.sun.faces.lifecycle.Phase.handleBeforePhase(Phase.java:228) INFO | jvm 1 | 2016/08/31 10:09:55 | at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:99) INFO | jvm 1 | 2016/08/31 10:09:55 | at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139) INFO | jvm 1 | 2016/08/31 10:09:55 | at javax.faces.webapp.FacesServlet.service(FacesServlet.java:594) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:748) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:486) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:411) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:338) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.web.RewriteFilter.process(RewriteFilter.java:98) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.web.RewriteFilter.doFilter(RewriteFilter.java:57) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.web.LoggingFilter.doFilter(LoggingFilter.java:60) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.web.IdentityFilter.doFilter(IdentityFilter.java:40) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.web.ExceptionFilter.doFilter(ExceptionFilter.java:64) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.web.RedirectFilter.doFilter(RedirectFilter.java:45) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:69) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter$FilterChainImpl.doFilter(SeamFilter.java:73) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.jboss.seam.servlet.SeamFilter.doFilter(SeamFilter.java:158) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:423) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:190) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316) INFO | jvm 1 | 2016/08/31 10:09:55 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) INFO | jvm 1 | 2016/08/31 10:09:55 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) INFO | jvm 1 | 2016/08/31 10:09:55 | at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) INFO | jvm 1 | 2016/08/31 10:09:55 | at java.lang.Thread.run(Thread.java:745) INFO | jvm 1 | 2016/08/31 10:09:55 | 2016-08-31 10:09:55,260 ERROR [org.gluu.oxtrust.action.Authenticator] Get empty token response. User rcan't log into application ```

By Michael Schwartz Account Admin 31 Aug 2016 at 10:40 a.m. CDT

Michael Schwartz gravatar
The logs are complaining about the certificate for https://gluu-test.westga.edu This could be for a few reasons: 1. The SSL certificate expired for that server? 2. Somehow oxTrust was configured to trust all, but is not configured to trust all anymore? 3. The java certificate truststore got replaced, and the imported self-signed certificate was lost or overwritten. Did anything change about that certificate? Or does one of the above three look possible?

By Gene Liverman user 31 Aug 2016 at 10:55 a.m. CDT

Gene Liverman gravatar
I setup clustering yesterday... is there an easy way from the server itself to see the certs used by oxTrust? Trust all is what I think I would want it set to btw.

By Michael Schwartz Account Admin 31 Aug 2016 at 10:58 a.m. CDT

Michael Schwartz gravatar
Alex, can you provide Gene with a link to instructions on how to import the certificate into his JVM or how to configure trust all?

By Aliaksandr Samuseu staff 31 Aug 2016 at 11:05 a.m. CDT

Aliaksandr Samuseu gravatar
Hi, Gene. Ho did this happen? From you last ticket I remember you completed csync's configuration. Were you able to log in to web UI after that?

By Gene Liverman user 31 Aug 2016 at 11:11 a.m. CDT

Gene Liverman gravatar
I was. It started out that I had the issue but using a Chrome incognito window bypassed the error. It then progressed to this. I am truly at a loss here... I THOUGHT I had checked this every step of the way but maybe I missed something. I did not touch the certs while clustering and I am only logging into the box that was working before, not the new cluster member.

By Gene Liverman user 31 Aug 2016 at 11:14 a.m. CDT

Gene Liverman gravatar
btw, the bypass above was pre-clustering too.

By Aliaksandr Samuseu staff 31 Aug 2016 at 11:23 a.m. CDT

Aliaksandr Samuseu gravatar
> I did not touch the certs while clustering Could you elaborate? You are supposed to sync most of contents of `/etc/certs` directory in the container, except, perhaps for OpenDJ's cert. Guide mentions this step too. > the bypass above was pre-clustering too. What exactly did you have to bypass?

By Aliaksandr Samuseu staff 31 Aug 2016 at 11:37 a.m. CDT

Aliaksandr Samuseu gravatar
I've found out that certificate/key syncing chapter of the guide wasn't clear enough that this is mandatory step, I've updated it a bit to be more assertive. You must copy specified certificates/keys manually from one node to another. In case you didn't use `setup.properties.last` method of installation, you **must** use the same node which was selected as source for initial csync's run as the source for certs/keys files.

By Gene Liverman user 31 Aug 2016 at 12:25 p.m. CDT

Gene Liverman gravatar
I did use `setup.properties.last`... does that mean that I do or don't need to sync certs? The "bypass" was > using a Chrome incognito window bypassed the error. Basically, what I was trying to say with that was that prior to the clustering I had this same issue. Pre-clustering, if I used a Chrome incognito window it did not give me the error.

By Aliaksandr Samuseu staff 31 Aug 2016 at 12:48 p.m. CDT

Aliaksandr Samuseu gravatar
Gene, here is extensive description of certificate update procedures Michael asked to provide you before: [link](https://www.gluu.org/docs/how-to/update-certificate/)

By Aliaksandr Samuseu staff 31 Aug 2016 at 1:43 p.m. CDT

Aliaksandr Samuseu gravatar
> did use setup.properties.last... does that mean that I do or don't need to sync certs? Yes, you still need to sync certs and keystores by hand even in this case. Regardless of that issue you had even before you went for cluster, it still must be done to complete all steps guide requires from you. You need to take certs and keystores from the `/etc/certs/` (within container) of the node you used as a source in first csync's run, and move to the `/etc/certs/` on the other node, over-writting its contents (better backup the whole directory first, just in case). You'll find full list of files to transfer [here](https://www.gluu.org/docs/cluster/#certificate-management), but in a nutshell it's everything with extensions .key, .csr, .crt, .jks, .pkcs12. Don't forget to check ownership on newly transferred files, it must be `tomcat:tomcat`. Then you must update the default java keystore (within container) on the node where certificates and keystores in `/etc/certs/` got over-written - that's what that link I provided above will help you with. Still, this bothers me: > Basically, what I was trying to say with that was that prior to the clustering I had this same issue. Pre-clustering, if I used a Chrome incognito window it did not give me the error. Do you mean you couldn't access web UI and saw the same error entries in `wrapper.log` too back then?

By Aliaksandr Samuseu staff 31 Aug 2016 at 1:47 p.m. CDT

Aliaksandr Samuseu gravatar
> Then you must update the default java keystore (within container) on the node where certificates and keystores in /etc/certs/ got over-written To be specific - you need to update at least Apache's and Shib's certificates. You can skip updating Asimba certificate unless you are installed this component too.

By Gene Liverman user 31 Aug 2016 at 1:53 p.m. CDT

Gene Liverman gravatar
> Do you mean you couldn't access web UI and saw the same error entries in wrapper.log too back then? I saw the same error on the web UI but did not look at the wrapper.log. Going back and looking at the archived logs I do not see any `SSLHandshakeException` messages before yesterday.

By Aliaksandr Samuseu staff 31 Aug 2016 at 2:49 p.m. CDT

Aliaksandr Samuseu gravatar
That's strange, but it seems we can't tie it to our current situation with certainty. Let's concentrate on resolving the current trustsore issue first. Do you understand all steps clearly? The link provided before - [here it is](https://www.gluu.org/docs/how-to/update-certificate/#default-jvm-keystore) again - contains steps you'll need for updating the default java keystore within the container on the node to which you'll be moving contents of `/etc/certs` directory. Please feel free to ask if anything is still not completely clear to you.

By Michael Schwartz Account Admin 31 Aug 2016 at 2:59 p.m. CDT

Michael Schwartz gravatar
Just to be clear, like ALex is suggesting: 1. Make sure you are using the same apache HTTPS certificate on both servers. 2. Make sure `/usr/java/latest/lib/security/cacerts` has the self-signed certificate imported. This is a java JKS keystore. The default password is `changeit` If these two things are true, you should not be getting an SSL handshake error because the HTTP client (oxTrust) should have access to the self-signed certificate for the server (oxAuth).

By Aliaksandr Samuseu staff 31 Aug 2016 at 3:40 p.m. CDT

Aliaksandr Samuseu gravatar
Here are exact commands you'll need for updating your default java keystore in your case (please ignore everything else on the certificate updating docs page): For Apache cert: 1. Create a copy of your certificate encoded in DER format: `# openssl x509 -in pem-formatted-cert.crt -outform der -out der-formatted-cert.der` 2. Find out the exact alias name of your current (self-signed) Apache certificate in the cacerts file: `# keytool -list -v -keystore /usr/java/latest/lib/security/cacerts -storepass changeit | grep -i '_httpd'` It should have an alias of sort “your-instance-hostname_httpd” 3. Remove your old certificate from the store: `# keytool -delete -alias your-instance-hostname_httpd -keystore /usr/java/latest/lib/security/cacerts \ -storepass changeit` 4. Import the new one with the same alias: `# keytool -import -alias your-instance-hostname_httpd --trustcacerts -file /etc/certs/der-formatted-cert.der \ -keystore /usr/java/latest/lib/security/cacerts -storepass changeit` 5. Restart Tomcat service: `# /etc/init.d/tomcat restart` 6. Restart Apache service: `# /etc/init.d/apache2 restart` For updating Shibboleth certificate: 1. Create a copy of your Shibboleth certificate encoded in DER format: `# openssl x509 -in /etc/certs/shibIDP.crt -outform der -out /etc/certs/shibIDP.der` 2. Find out the exact alias name of your current Shibboleth's certificate in the cacerts file: `# keytool -list -v -keystore /usr/java/latest/lib/security/cacerts -storepass changeit | grep -i '_shibidp'`. It should have an alias of sort “your-instance-hostname_shibidp” 3. Remove your old certificate from the store: `# keytool -delete -alias your-instance-hostname_shibidp -keystore /usr/java/latest/lib/security/cacerts \ -storepass changeit` 4. Import the new one with the same alias: `# keytool -import -alias your-instance-hostname_shibidp --trustcacerts -file /etc/certs/shibIDP.der \ -keystore /usr/java/latest/lib/security/cacerts -storepass changeit` 5. Restart Tomcat service: `# /etc/init.d/tomcat restart` 6. Restart Apache service: `# /etc/init.d/apache2 restart`

By Gene Liverman user 31 Aug 2016 at 3:43 p.m. CDT

Gene Liverman gravatar
Awesome! I got pulled into some meetings but will work on this first thing in the morning.

By Gene Liverman user 01 Sep 2016 at 7:56 a.m. CDT

Gene Liverman gravatar
None of this made a differance so I decided to roll back to a VMware snapshot from pre-clustering. I'll post an update as soon as I see where things are.

By Gene Liverman user 01 Sep 2016 at 9:53 a.m. CDT

Gene Liverman gravatar
Rolling back to pre-clustering seems to have resolved my issue for the moment, going to re-cluster and see what happens. I'd appreciate it if we could keep this ticket open for the moment while I give it another go.

By William Lowe user 01 Sep 2016 at 10:51 a.m. CDT

William Lowe gravatar
No problem, Gene. Let us know how it goes.

By Aliaksandr Samuseu staff 01 Sep 2016 at 10:51 a.m. CDT

Aliaksandr Samuseu gravatar
Hi, Gene. >None of this made a differance so I decided to roll back to a VMware snapshot from pre-clustering. That's a pity it didn't help. Do you mean even after coping `/etc/certs` from node1 to node2, and adding them to cacerts truststore on node2, you still kept seeing this `SSLHandshakeException` exception in logs? I've also tried to use Chrome browser when accessing web UI of my own 2.4.4 instance, and didn't see any errors along the way. I can confirm that a newly installed cluster of 2.4.4 instances I'm working on also works fine. Let's hope your second attempt will be successful.

By Aliaksandr Samuseu staff 01 Sep 2016 at 10:52 a.m. CDT

Aliaksandr Samuseu gravatar
One thing to note, though: for that cluster I've mentioned I didn't use setup.properties.last method of install.

By Gene Liverman user 02 Sep 2016 at 7:06 a.m. CDT

Gene Liverman gravatar
I found what broke it! The instructions [here](https://www.gluu.org/docs/cluster/#csync2-configuration-for-host-1) show using just the host names in the hosts file. When I open my hosts file post-install in the container it has an entry for the cluster name assigned to the host's IP like so: ``` 10.10.5.129 gluu-test.westga.edu ``` If I change it to the setup below it breaks Gluu: ``` 10.10.5.129 gluu-idp-t01.westga.edu 10.10.5.130 gluu-idp-t02.westga.edu ``` If I instead do this it seems to keep working (I have not tested the second node yet): ``` 10.10.5.129 gluu-test.westga.edu gluu-idp-t01.westga.edu 10.10.5.130 gluu-idp-t02.westga.edu ```

By Aliaksandr Samuseu staff 02 Sep 2016 at 7:27 a.m. CDT

Aliaksandr Samuseu gravatar
That's most likely it, indeed. Now I understand that I've never actually removed original entry that is added here on Gluu's initial installation, I've just been adding mappings for both nodes to the end of the file, without a second thought.. Seems like a lot of people have been doing the same, that's why it hasn't been reported yet.. Please accept our apologies for that, Gene. To a large extent, it's because of input of insightful community users like you we are able to make our products better and better. For the second node I believe you must use something like this: ``` 10.10.5.130 gluu-test.westga.edu gluu-idp-t02.westga.edu 10.10.5.129 gluu-idp-t01.westga.edu ``` I'll make sure this mistake is corrected in the doc. Thank you for your time and input again, we really appreciate it.

By Gene Liverman user 02 Sep 2016 at 7:42 a.m. CDT

Gene Liverman gravatar
I am glad to help! That "the open source way" in my opinion :) In going back through all this I noticed that it would be MUCH more clear if you made a couple of other changes too: * Add a heading to the part about doing the initial sync * Move the initial sync part up to just under configuring host2 * Remove the cron configs from the host1 and host2 parts and add them at the end of the initial sync part. Something else I noticed is that there is no logrotate setup for `/var/log/csync2_action.log`. I think that should get added along with renaming the cron output to `/var/log/csync2_cron_error.log` and changing the cron line from `>` to `>>`. Naturally, this change would also require an entry in logrotate. Below is a sample of what could be put in `/etc/logrotate.d/csync2`: ``` /var/log/csync2*.log { daily rotate 7 copytruncate compress missingok notifempty } ```

By Aliaksandr Samuseu staff 02 Sep 2016 at 10:05 a.m. CDT

Aliaksandr Samuseu gravatar
How's it going with your 2nd node, Gene? Were you able finally to make it work? Btw, did you use setup.properties.last file method of install for the 2nd node this time, too? I've corrected sections mentioning contents of hosts files in doc. I've also passed your proposal regarding log rotation for csync's logs to devs via github, you can check it [here](https://github.com/GluuFederation/community-edition-setup/issues/171) Your proposals on restructuring the doc are quite reasonable too, but taking into account how complex it already is, and that we have a couple of similar proposals from other user already, we need some time to figure out what will be the best way to format it. I've added yours to the list too, thank you

By Gene Liverman user 02 Sep 2016 at noon CDT

Gene Liverman gravatar
Thanks for the info. Yes, my test cluster is finally fully up and running. I used the setup.properties.last method + forcing initial sync before creating cron jobs. I am now working to build my production cluster. Thanks again for the help. We can safely close this ticket out.

By Aliaksandr Samuseu staff 02 Sep 2016 at 12:56 p.m. CDT

Aliaksandr Samuseu gravatar
That's a relief, thanks for confirming. > I used the setup.properties.last method + forcing initial sync before creating cron jobs Please just be aware that if you don't use this method, local LDAP passwords (for user "cn=directory manager") on both nodes won't guaranteed to be the same (and solution expects they are). As they are stored in LDAP context that is not replicated by default, you'll need either to set it manually at 2nd node to be the same as at 1st, or, better, during running `setup.py` in interactive mode at 2nd node just provide LDAP pass of the 1st node explicitly. We need to add it to the doc also.