Did you do a `top` on the server to see what was going on during the crash? Was one of the processes using 100% of the CPU? Also, it's hard to say what the problem is if you're not sure what process was going haywire. 6GB is pretty tight with Gluu these days. You should look at the memory allocation and logs for all the services. Also, make sure you set the file descriptors > 65k, per the "Preparing the VM" docs.

Thanks for your response Michael, I executed the top command and I noticed that jetty service was using 170 - 180 % of the CPU. We have already followed and checked the "VM Preparation", "Performance tunning" and "Security tips" docs. Do you need any logs for more information? Thank you!

Hi, about memory allocation? is it normal? What are the services running currently? Please, share oxauth, identity logs. and also if any other services are running.

Which jetty service is using that much CPU ? Is it the the Shibboleth IDP ? Do a `ps -ef | grep ___` and use the pid you get from `top` to get more info.

About the services we are using: oxauth, oxtrust, identity and passport. About the top command detail: After execute the ps ef | grep "pid", we found that the jetty service is the oxauth service. The same that we modified some time ago to increase the xmx and xms parameters. As I said before, some time ago we had been experienced server outages each day, when the outage came, the CPU usage increased to 170-180%. Now, we have been experienced server outages each 5-6 days, when the outage comes, we check the top command and we see that jetty service (oxauth) is using 170-180% cpu. After we restart the services, the jetty service backs to normal. It's like the jetty service is accumulating the usage of cpu, each day is higher than the last one until the server outage occurs. [Image of top command](https://drive.google.com/file/d/1_GoX5I_Zx8RXqiKbbOVTs5--jJsrlEaO/view?usp=sharing)

``` 4GB HDD: 40GB SWAP: 6GB CPU: 2 ``` This is the minimal requirements for the service. You may need to double this for a prod server. It may be using a lot of CPU when it does garbage collection. Are you running any person authentication interception scripts? Per the suggestion of @Mobarak Hosen.Shakil , you should also make sure you collect the logs to see if there are any errors or other indications of what could be wrong (in `/opt/gluu/jetty/oxauth/logs`) Also, run these commands and share the files: ``` # jmap -clstats PID > memory_sage.txt # jstack -l PID > threads.txt ``` Finally, how much swap is configured on your server? You can check with ``` # free -m ``` It's advisable to have at least 16GB of swap configured...

In 4.2.x we removed `-XX:MaxMetaspaceSize` options from java command line. It causes memory issues. Can you update `JAVA_OPTIONS` in `/etc/default/oxauth` and remove this option?

We are running the "Social Login" authentication interception script, particularly the facebook login and apple id login. Also we are using the basic authentication. I executed the top command today and the jetty service is still increasing using more than 100% of cpu. [Image of top command](https://drive.google.com/file/d/12sGCNBEA1vCaibSFEM6LuDtbBewk5407/view?usp=sharing) I share to you the following logs: [Oxauth log](https://drive.google.com/file/d/1PIOJUMZREr8c6fcd6OLKOlYP-EZt54vw/view?usp=sharing) [Oxauth script log](https://drive.google.com/file/d/1ybJHYuJdA1_hbiOKm26htWu6YRGVpe2l/view?usp=sharing) [Oxauth script log for 2020-11-09 - the day we had the server outage](https://drive.google.com/file/d/11FK6zYKr9GGZNA4RPcYzDfiNraD5yV1l/view?usp=sharing) Thanks again!

Ok @Yuriy.Movchan I'll remove the -XX:MaxMetaspaceSize parameter, thank you!

@Michael.Schwartz about the swap memory, after we executed the free -m command it shows us the following: Mem: 7881 (total) - 6403 (used) - 699 (free) Swap: 6183 (total) - 292 (used) - 5851 (free) About the files for the jmap and jstack: [jmap file](https://drive.google.com/file/d/1mjUKsXrnkf7_jpVY95s2hGpdvkY5G45W/view?usp=sharing) [jstack file](https://drive.google.com/file/d/1T2GaKQX1RYvfePb2tWgrpoicb33pN_jk/view?usp=sharing) When I executed the jstack command it shows me the following: 7863: Unable to open socket file: target process not responding or HotSpot VM not loaded The -F option can be used when the target process is not responding. And when I use the -F option after some time I need to stop the process since it does not show me anything, after I stop the process it shows me the following: ``` root@ip-172-31-11-163:/# sudo rm threads.txt -su: sudo: command not found root@ip-172-31-11-163:/# rm threads.txt root@ip-172-31-11-163:/# ls bin dev home lib media mnt proc run srv tmp var boot etc install lib64 memory_sage.txt opt root sbin sys usr root@ip-172-31-11-163:/# jstack -l -F 7863 > threads.txt ^CException in thread "main" java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.tools.jstack.JStack.runJStackTool(JStack.java:140) at sun.tools.jstack.JStack.main(JStack.java:106) Caused by: java.lang.RuntimeException: VM.initialize() was not yet called at sun.jvm.hotspot.runtime.VM.getVM(VM.java:401) at sun.jvm.hotspot.oops.Oop.getKlass(Oop.java:79) at sun.jvm.hotspot.oops.ObjectHeap$2.canInclude(ObjectHeap.java:289) at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:352) ``` Thank you!

It does seem like there are many objects awaiting garbage collection. 2 CPU's is not a lot. I would at least double the resources on this server: 8GB 4CPU.

@Michael.Schwartz we'll consider increasing our pc requirements. Also, do you know any services that we can use to monitor our Gluu server and reset it automatically if its outage? Thank you!

Kubernetes

Is there a way or is it possible to migrate to kubernetes after we have launched our gluu production server? Thank you

Yes of course. But keep in mind that support for Cloud Native edition of Gluu is only available for VIP customers.

Could you please facilitate to me the url for the Cloud Native edition of Gluu page and the prices for the VIP customers service? Thank you!

https://gluu.org/docs/gluu-server/4.2/installation-guide/install-kubernetes/ https://gluu.org/docs/gluu-server/4.2/reference/container-configs/ https://gluu.org/docs/gluu-server/4.2/tutorials/install-rancher/

For `VIP` services checkout this link, please: https://www.gluu.org/pricing/

Please, reopen this ticket if required. Thanks