By: Marlon Martínez user 09 Nov 2020 at 5:58 p.m. CST

19 Responses
Marlon Martínez gravatar
Since we upgraded our gluu version from 3.1.5 to 4.1 we are getting some server outages. We opened a ticket and you recommended us to make some changes in the xms and xmx parameters in /etc/default/oxauth file, before this change we experienced server outages each 20 - 24 hours. After bumping up this parameters a bit we are still experimenting server outages each 5 or 6 days. Our current values for oxauth file are: JAVA_OPTIONS="-server -Xms512m -Xmx2048m -XX:MaxMetaspaceSize=552m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dserver.base=/opt/gluu/jetty/oxauth -Dlog.base=/opt/gluu/jetty/oxauth -Dpython.home=$ Is there anything else we can do to avoid these server outages, do we need to bumping up this parameters (xms, xmx) in identity or idp file? This is our AWS instance configuration: RAM: 4GB HDD: 40GB SWAP: 6GB CPU: 2 Thank you for your support and help!

By Michael Schwartz staff 09 Nov 2020 at 8:32 p.m. CST

Michael Schwartz gravatar
Did you do a `top` on the server to see what was going on during the crash? Was one of the processes using 100% of the CPU? Also, it's hard to say what the problem is if you're not sure what process was going haywire. 6GB is pretty tight with Gluu these days. You should look at the memory allocation and logs for all the services. Also, make sure you set the file descriptors > 65k, per the "Preparing the VM" docs.

By Marlon Martínez user 10 Nov 2020 at 9:28 a.m. CST

Marlon Martínez gravatar
Thanks for your response Michael, I executed the top command and I noticed that jetty service was using 170 - 180 % of the CPU. We have already followed and checked the "VM Preparation", "Performance tunning" and "Security tips" docs. Do you need any logs for more information? Thank you!

By Mobarak Hosen Shakil staff 10 Nov 2020 at 11:02 a.m. CST

Mobarak Hosen Shakil gravatar
Hi, about memory allocation? is it normal? What are the services running currently? Please, share oxauth, identity logs. and also if any other services are running.

By Michael Schwartz staff 10 Nov 2020 at 12:06 p.m. CST

Michael Schwartz gravatar
Which jetty service is using that much CPU ? Is it the the Shibboleth IDP ? Do a `ps -ef | grep ___` and use the pid you get from `top` to get more info.

By Marlon Martínez user 10 Nov 2020 at 6:05 p.m. CST

Marlon Martínez gravatar
About the services we are using: oxauth, oxtrust, identity and passport. About the top command detail: After execute the ps ef | grep "pid", we found that the jetty service is the oxauth service. The same that we modified some time ago to increase the xmx and xms parameters. As I said before, some time ago we had been experienced server outages each day, when the outage came, the CPU usage increased to 170-180%. Now, we have been experienced server outages each 5-6 days, when the outage comes, we check the top command and we see that jetty service (oxauth) is using 170-180% cpu. After we restart the services, the jetty service backs to normal. It's like the jetty service is accumulating the usage of cpu, each day is higher than the last one until the server outage occurs. [Image of top command](https://drive.google.com/file/d/1_GoX5I_Zx8RXqiKbbOVTs5--jJsrlEaO/view?usp=sharing)

By Michael Schwartz staff 11 Nov 2020 at 11:24 a.m. CST

Michael Schwartz gravatar
``` 4GB HDD: 40GB SWAP: 6GB CPU: 2 ``` This is the minimal requirements for the service. You may need to double this for a prod server. It may be using a lot of CPU when it does garbage collection. Are you running any person authentication interception scripts? Per the suggestion of @Mobarak Hosen.Shakil , you should also make sure you collect the logs to see if there are any errors or other indications of what could be wrong (in `/opt/gluu/jetty/oxauth/logs`) Also, run these commands and share the files: ``` # jmap -clstats PID > memory_sage.txt # jstack -l PID > threads.txt ``` Finally, how much swap is configured on your server? You can check with ``` # free -m ``` It's advisable to have at least 16GB of swap configured...

By Yuriy Movchan staff 11 Nov 2020 at 12:05 p.m. CST

Yuriy Movchan gravatar
In 4.2.x we removed `-XX:MaxMetaspaceSize` options from java command line. It causes memory issues. Can you update `JAVA_OPTIONS` in `/etc/default/oxauth` and remove this option?

By Marlon Martínez user 11 Nov 2020 at 12:24 p.m. CST

Marlon Martínez gravatar
We are running the "Social Login" authentication interception script, particularly the facebook login and apple id login. Also we are using the basic authentication. I executed the top command today and the jetty service is still increasing using more than 100% of cpu. [Image of top command](https://drive.google.com/file/d/12sGCNBEA1vCaibSFEM6LuDtbBewk5407/view?usp=sharing) I share to you the following logs: [Oxauth log](https://drive.google.com/file/d/1PIOJUMZREr8c6fcd6OLKOlYP-EZt54vw/view?usp=sharing) [Oxauth script log](https://drive.google.com/file/d/1ybJHYuJdA1_hbiOKm26htWu6YRGVpe2l/view?usp=sharing) [Oxauth script log for 2020-11-09 - the day we had the server outage](https://drive.google.com/file/d/11FK6zYKr9GGZNA4RPcYzDfiNraD5yV1l/view?usp=sharing) Thanks again!

By Marlon Martínez user 11 Nov 2020 at 12:54 p.m. CST

Marlon Martínez gravatar
Ok @Yuriy.Movchan I'll remove the -XX:MaxMetaspaceSize parameter, thank you!

By Marlon Martínez user 11 Nov 2020 at 1:13 p.m. CST

Marlon Martínez gravatar
@Michael.Schwartz about the swap memory, after we executed the free -m command it shows us the following: Mem: 7881 (total) - 6403 (used) - 699 (free) Swap: 6183 (total) - 292 (used) - 5851 (free) About the files for the jmap and jstack: [jmap file](https://drive.google.com/file/d/1mjUKsXrnkf7_jpVY95s2hGpdvkY5G45W/view?usp=sharing) [jstack file](https://drive.google.com/file/d/1T2GaKQX1RYvfePb2tWgrpoicb33pN_jk/view?usp=sharing) When I executed the jstack command it shows me the following: 7863: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding. And when I use the -F option after some time I need to stop the process since it does not show me anything, after I stop the process it shows me the following: ``` root@ip-172-31-11-163:/# sudo rm threads.txt -su: sudo: command not found root@ip-172-31-11-163:/# rm threads.txt root@ip-172-31-11-163:/# ls bin dev home lib media mnt proc run srv tmp var boot etc install lib64 memory_sage.txt opt root sbin sys usr root@ip-172-31-11-163:/# jstack -l -F 7863 > threads.txt ^CException in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) at sun.tools.jstack.JStack.main(JStack.java:106) Caused by: java.lang.RuntimeException: VM.initialize() was not yet called at sun.jvm.hotspot.runtime.VM.getVM(VM.java:401) at sun.jvm.hotspot.oops.Oop.getKlass(Oop.java:79) at sun.jvm.hotspot.oops.ObjectHeap$2.canInclude(ObjectHeap.java:289) at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:352) ``` Thank you!

By Michael Schwartz staff 11 Nov 2020 at 1:36 p.m. CST

Michael Schwartz gravatar
It does seem like there are many objects awaiting garbage collection. 2 CPU's is not a lot. I would at least double the resources on this server: 8GB 4CPU.

By Marlon Martínez user 11 Nov 2020 at 5:02 p.m. CST

Marlon Martínez gravatar
@Michael.Schwartz we'll consider increasing our pc requirements. Also, do you know any services that we can use to monitor our Gluu server and reset it automatically if its outage? Thank you!

By Michael Schwartz staff 11 Nov 2020 at 10:13 p.m. CST

Michael Schwartz gravatar
Kubernetes

By Marlon Martínez user 13 Nov 2020 at 11:39 a.m. CST

Marlon Martínez gravatar
Is there a way or is it possible to migrate to kubernetes after we have launched our gluu production server? Thank you

By Michael Schwartz staff 13 Nov 2020 at 11:41 a.m. CST

Michael Schwartz gravatar
Yes of course. But keep in mind that support for Cloud Native edition of Gluu is only available for VIP customers.

By Marlon Martínez user 13 Nov 2020 at 1:02 p.m. CST

Marlon Martínez gravatar
Could you please facilitate to me the url for the Cloud Native edition of Gluu page and the prices for the VIP customers service? Thank you!

By Michael Schwartz staff 13 Nov 2020 at 2:12 p.m. CST

Michael Schwartz gravatar
https://gluu.org/docs/gluu-server/4.2/installation-guide/install-kubernetes/ https://gluu.org/docs/gluu-server/4.2/reference/container-configs/ https://gluu.org/docs/gluu-server/4.2/tutorials/install-rancher/

By Mobarak Hosen Shakil staff 13 Nov 2020 at 2:32 p.m. CST

Mobarak Hosen Shakil gravatar
For `VIP` services checkout this link, please: https://www.gluu.org/pricing/

By Mobarak Hosen Shakil staff 23 Nov 2020 at 6:16 a.m. CST

Mobarak Hosen Shakil gravatar
Please, reopen this ticket if required. Thanks