By: Hernan Quevedo user 03 Oct 2017 at 9:41 a.m. CDT

7 Responses
Hernan Quevedo gravatar
I have Gluu server installed on a dedicated machine on AWS, and, as I understand, the Gluu server execution comprises two java processes for jetty. Since yesterday, one of those processes started to get killed, presumably by the OOM killer, although I don't have proof of that. I presume this because the machine has only 4GB of RAM which in reality is just only 3729020 bytes, and the fact that the top command tells me that one of those processes is using 2GB of RAM and the other one more than 1GB. As of one right now, this is the amount of memory Gluu server is grabbing (ps aux | less): Gluu process 1: %RAM:56.4, VSZ:4179872 RSS:2104668 Gluu process 2: %RAM:31.2, VSZ:3646880 RSS:1167148 I think the OOM killer is going to act on the first process. Any thoughts on this? UPDATE: just now, the second process started to increase it memory usage and the OS killed the first one. Why would this happen? Nothing in particular was being tested on the system. Thank you.

By Aliaksandr Samuseu staff 03 Oct 2017 at 10:44 a.m. CDT

Aliaksandr Samuseu gravatar
Hi, Herman. 4GB RAM is bare minimum for Gluu to function in a test environment. For a production you should go with at least 6GB, better 8. If you know the PID of the process in question, you could try to find out which exactly Gluu's service it belongs to. Try to run this in the container: `# ps -aux | grep -i java`. Find out line corresponding to your PID, you should see some clues in there. Reasons for what you see may be many. For one, if you employ federation TR of a big federation, like Incommon, and supply its metadata by url, it may refresh it, from time to time, and this process is extremely memory-hungry.

By Hernan Quevedo user 03 Oct 2017 at 11:02 a.m. CDT

Hernan Quevedo gravatar
All right. Well, we don't employ any federation, we just use Gluu server for authentication and managing users. Now, when I run setup.py, it says a number for the RAM that is goint to be always the same [3072]? Or it analyzes the machine and comes up with a number adjusted to it? If I installed it with 4096, more than the machine physical memory, I can expect this process killing behaviour? Another question: is it more expensive, memory-wise, to create and update an user or to authenticate it and return a token? What we have is a simple installation of Gluu 3.0.2, one client with scopes and grant types, and a java app using it. Without any use from the app, the server is growing its memory usage, so eventually, the OOM killer is goint to end the process. I believe that if we use a machine with more RAM, it will simply consume its memory until the end, which is alarming. UPDATE: it appears that it has stabilized memory usage after installing it with (seems obvious) only 3GB. Of course the console shows only 15% of free memory. Now, is it possible to initialize Gluu with just the necessary processes to authenticate and manage users, and no other service whatsoever, so it consumes less memory?

By Aliaksandr Samuseu staff 04 Oct 2017 at 11:16 a.m. CDT

Aliaksandr Samuseu gravatar
Hi, Herman. >Now, when I run setup.py, it says a number for the RAM that is goint to be always the same [3072]? Or it analyzes the machine and comes up with a number adjusted to it? No, default of 3GB is chosen so that a new users aiming to evaluate Gluu in some test environment wouldn't be met with excessively demanding requirements. We provide our recommendations for production setups [here](https://gluu.org/docs/ce/3.1.0/installation-guide/). It doesn't try to assume how much memory it may take for yourself, but it tries to divide the amount you specified between all its subcomponents, using some hardcoded ratios. Note, that you can adjust memory settings for each service after installation by editing corresponding file here (within container): `/etc/default/` (so, for oxAuth, edit `/etc/default/oxauth`) >If I installed it with 4096, more than the machine physical memory, I can expect this process killing behaviour? Yes, nothing good can come out of this. JVMs will be set with overall limits exceeding maximum size of RAM, that at best will lead to a lot of swapping, most likely to constant OOMs >Another question: is it more expensive, memory-wise, to create and update an user or to authenticate it and return a token? Can't say I understand the question completely, but in any case a user entry for each user Gluu handles is always created in its LDAP directory, regardless of what entity actually handles authentication - that's how it's designed. >I believe that if we use a machine with more RAM, it will simply consume its memory until the end, which is alarming. That shouldn't be the case. You always have an option to set upper limits for each JVM's heap using defaults files mentioned above, if they are not set already. >Now, is it possible to initialize Gluu with just the necessary processes to authenticate and manage users, and no other service whatsoever, so it consumes less memory? That's hard to answer a question like this, as it will heavily depend on your environment and loads. But, as mentioned, you have an option to tweak each JVM's memory allocations in whatever way it allows to do so. Profiling Gluu in such way is not covered by free community support, and is not different from a general task of profiling a JVM to accommodate a web app. If you may be interested in hiring a help from a 3rd party, we have some partners dealing with system integrations we could recommend.

By Michael Schwartz Account Admin 04 Oct 2017 at 4:20 p.m. CDT

Michael Schwartz gravatar
``` Is it possible to initialize Gluu with just the necessary processes to authenticate and manage users, and no other service whatsoever, so it consumes less memory? ``` We already have stopped any non-essential container services. The most important thing is that you don't install any extra components in setup.py (e.g. don't install the Asimba SAML proxy, which will instantiate it's own jetty JVM). - Mike

By Aliaksandr Samuseu staff 04 Oct 2017 at 8:02 p.m. CDT

Aliaksandr Samuseu gravatar
Michael is correct. You also shouldn't install Shibboleth IdP as well, if you don't need to serve SAML requests (Shibboleth's additional memory consumption was what I meant when I was talking about huge federations' metadata before, I should have been more clear on that). Though please also note that adding components later on (after initial installation is completed) is not supported at the moment, so you may need to think through your requirements properly in advance. Generally, 4GB RAM should be enough for a test/small production setup, especially if you indeed don't need to use anything except OIDC/UMA/SCIM, which all are provided by oxAuth service. Just note that those 4GB must be allocated to Gluu exclusively, it's not the total memory of the host machine.

By Hernan Quevedo user 05 Oct 2017 at 9:19 a.m. CDT

Hernan Quevedo gravatar
Everybody, thanks for your prompt answers. The memory consumption reached a steady level that has not been surpassed by now, but of course, I will take all those considerations to the letter. Thank you again.

By Hernan Quevedo user 09 Oct 2017 at 10:50 a.m. CDT

Hernan Quevedo gravatar
I think this can be closed.