By: Stephane Chulak user 15 Feb 2019 at 8:13 p.m. CST

18 Responses
Stephane Chulak gravatar
Hi. Without modification, the gluu server starts to slow down and the slapd process starts to have a high utilization rate of the CPU. 100%. Until httpd no longer holds a query result. I tried to do a db_recover -h. With a good result but only for the first 30 minutes ... Tx for your help.

By Sahil Arora user 15 Feb 2019 at 8:24 p.m. CST

Sahil Arora gravatar
Hi Stephane, I am looking into it.

By Sahil Arora user 15 Feb 2019 at 8:32 p.m. CST

Sahil Arora gravatar
Stephane, Could you please oxauth and ldap logs during the time when CPU was 100%. Is the CPU still at 100% and how is the current traffic looking? Thanks Sahil

By Stephane Chulak user 15 Feb 2019 at 8:50 p.m. CST

Stephane Chulak gravatar
Hi Sahil. The CPU still use at 100% by the slapd. Can you tel me exactly which log you need and the full path? Tx,

By Sahil Arora user 15 Feb 2019 at 8:55 p.m. CST

Sahil Arora gravatar
Please see below the logs full path: /var/log/openldap/ldap.log /opt/gluu/jetty/oxauth/logs/oxauth.log Thanks Sahil

By Stephane Chulak user 15 Feb 2019 at 9:12 p.m. CST

Stephane Chulak gravatar
!!! I found nothing for openldap!!! -bash-4.2# ls /var/log/openldap/ -bash-4.2# Did you how I can enable it? Here the oxauth.log file.

By Stephane Chulak user 15 Feb 2019 at 9:22 p.m. CST

Stephane Chulak gravatar
It's stil slow. My coworker told me it takes more than 2 seconds to get an access token.

By Sahil Arora user 15 Feb 2019 at 9:30 p.m. CST

Sahil Arora gravatar
While I am analysing the logs, could you please try restarting the services in following order to see if it provides relief to users Restart solserver Restart oxauth Restart identity

By Mohib Zico staff 16 Feb 2019 at 2:08 a.m. CST

Mohib Zico gravatar
Hi Stephane, oxAuth log showing that it's timing out to localhost:1636 and/or ldap server not reachable. Can you please check the status of your `slapd.conf` for logging? Here is what my status looks like and I am getting openldap logging in /var/log/openldap/ ``` [root@localhost openldap]# cat /opt/symas/etc/openldap/slapd.conf | grep loglevel loglevel stats sync [root@localhost openldap]# ls /var/log/openldap/ ldap.log [root@localhost openldap]# ```

By Stephane Chulak user 16 Feb 2019 at 8:55 a.m. CST

Stephane Chulak gravatar
This is the config. ``` *"cat /opt/symas/etc/openldap/slapd.conf | grep loglevel #loglevel stats sync loglevel stats sync stats" ``` I don't know what is the difference between: ``` loglevel stats sync stats and loglevel stats sync ``` The "openldap" folder is empty!!! ``` -bash-4.2# ls /var/log/openldap/ -bash-4.2# ``` This morning the process slapd is quiet...and I do nothing... ``` PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 154 ldap 20 0 3913260 1,060g 873404 S 0,3 9,0 178:11.67 slapd 283 jetty 20 0 7220636 1,938g 24000 S 0,3 16,5 2:57.41 java 1 root 20 0 43108 4580 3856 S 0,0 0,0 0:04.12 systemd 16 root 20 0 168584 122200 121988 S 0,0 1,0 7:38.30 systemd-journal 28 root 20 0 118900 2664 2664 S 0,0 0,0 0:00.00 lvmetad 36 root 20 0 24288 3328 3324 S 0,0 0,0 0:00.04 smartd ``` Do you know how I can track and log the CPU usage of this process to make association whit some outside events?

By Mohib Zico staff 16 Feb 2019 at 12:02 p.m. CST

Mohib Zico gravatar
You might wanna change to `loglevel stats sync` and stop/start `slapd` service. >> Do you know how I can track and log the CPU usage of this process to make association whit some outside events? Linux [sar](https://linux.die.net/man/1/sar) command should help in this case.

By Stephane Chulak user 18 Feb 2019 at 2:05 p.m. CST

Stephane Chulak gravatar
Hi. I was set the loglevel and restart the slapd service. I do not find the log in /var/log/openldap/ !!!

By Mohib Zico staff 18 Feb 2019 at 2:43 p.m. CST

Mohib Zico gravatar
Can you please share your slapd.conf file? We will compare with our 3.0.2+OpenLDAP Gluu Server one.

By Stephane Chulak user 18 Feb 2019 at 3:14 p.m. CST

Stephane Chulak gravatar
Voila. Tx.

By Mohammad Abudayyeh staff 19 Feb 2019 at 8:37 a.m. CST

Mohammad Abudayyeh gravatar
Hi Stephan, You preforming the recover might have momentarily fixed your issue. And sometimes it does take time to see the effect. I want you to verify you did the following : - Stop the ldap - Make a copy of the ldab data /opt/gluu/data/main_db - run the dp_recover using slapd - db files should be gone - restart ldap Also please send us what you have in your slapd configuration file without sensitive information ? We need to see more. We can't help you without taking a look at it. Note: Think about upgrading*

By Stephane Chulak user 26 Feb 2019 at 6:20 p.m. CST

Stephane Chulak gravatar
Here thay are. Sorry for the late return. The problem has not come back but I would like to understand for next time. Tx in advance.

By Aliaksandr Samuseu staff 28 Feb 2019 at 9:14 p.m. CST

Aliaksandr Samuseu gravatar
Hi, Stephane. Thanks for the file. Log level setting seems to be correct. That's strange that you don't have the log file itself. Just to make things clear: you were checking for it while inside container, correct? One thing worth checking here are permissions. First, let's check which user slapd runs under in your setup: `# ps -aux | grep -i slapd` It's most likely "ldap" user, though it may be "root" as well. If it's "ldap", then try running this inside container: `# ll /var/log/openldap/`, then `# ll /var/log/` Assess the current file system's permissions for "/var/log/openldap/" directory. Can "ldap" user can write into it? To be sure, make it an owner of it: `# chown ldap:ldap /var/log/openldap/` After that restart "solserver" service, and check for the log now.

By Stephane Chulak user 01 Mar 2019 at 8:21 a.m. CST

Stephane Chulak gravatar
I think I followed the steps well (inside the container). Indeed, the "openldap" directory was held by Root. But everyone had access. I still change the ownership. Unfortunately, the log file is not created. -bash-4.2# ll /var/log/openldap/ total 0 -bash-4.2# ll /var/log/ total 764 -rw-r--r-- 1 root root 0 Feb 15 17:58 boot.log -rw------- 1 root utmp 0 Mar 1 03:46 btmp -rw------- 1 root utmp 0 Feb 1 03:45 btmp-20190301 drwxr-xr-x 1 root root 0 Nov 21 2016 chrony -rw-r--r-- 1 root root 32691 Mar 12 2016 dmesg -rw-r--r-- 1 root root 32691 Mar 12 2016 dmesg.old drwxr-xr-x 1 root root 334 Feb 24 03:47 httpd drwxr-sr-x 1 root systemd-journal 64 Aug 3 2017 journal -rw-r--r-- 1 root root 292584 Mar 1 08:33 lastlog **drwxrwxrwx 1 root root 0 Sep 18 14:21 openldap** drwxr-xr-x 1 root root 522 Feb 28 23:53 sa -rw-r--r-- 1 root root 87 Apr 26 2018 service_check -rw------- 1 root root 0 Mar 11 2016 tallylog -rw-rw-r-- 1 root utmp 367104 Mar 1 08:33 wtmp -rw------- 1 root root 0 Jan 1 2018 yum.log -rw------- 1 root root 46445 Feb 3 2017 yum.log-20180101 -bash-4.2# chown ldap:ldap /var/log/openldap/ -bash-4.2# ll /var/log total 764 -rw-r--r-- 1 root root 0 Feb 15 17:58 boot.log -rw------- 1 root utmp 0 Mar 1 03:46 btmp -rw------- 1 root utmp 0 Feb 1 03:45 btmp-20190301 drwxr-xr-x 1 root root 0 Nov 21 2016 chrony -rw-r--r-- 1 root root 32691 Mar 12 2016 dmesg -rw-r--r-- 1 root root 32691 Mar 12 2016 dmesg.old drwxr-xr-x 1 root root 334 Feb 24 03:47 httpd drwxr-sr-x 1 root systemd-journal 64 Aug 3 2017 journal -rw-r--r-- 1 root root 292584 Mar 1 08:33 lastlog **drwxrwxrwx 1 ldap ldap 0 Sep 18 14:21 openldap** drwxr-xr-x 1 root root 522 Feb 28 23:53 sa -rw-r--r-- 1 root root 87 Apr 26 2018 service_check -rw------- 1 root root 0 Mar 11 2016 tallylog -rw-rw-r-- 1 root utmp 367104 Mar 1 08:33 wtmp -rw------- 1 root root 0 Jan 1 2018 yum.log -rw------- 1 root root 46445 Feb 3 2017 yum.log-20180101 -bash-4.2# systemctl restart solserver -bash-4.2# ps -aux | grep -i slapd ldap 22642 0.3 1.4 2356588 179384 ? Ssl 08:39 0:00 /opt/symas/lib64/slapd -u ldap -g ldap -h ldaps://0.0.0.0:1636/ ldaps:/// -bash-4.2# ll /var/log/openldap/ total 0 ****

By Aliaksandr Samuseu staff 01 Mar 2019 at 6:19 p.m. CST

Aliaksandr Samuseu gravatar
That's strange. I've checked in my locate test CentOS7 3.0.2 instance, and this log file is present there out of the box, and logs are being pushed into it without any reconfiguration neede. I've compared `slapd.conf` Stephane provided with the one used there, and there are no substantial differences between them.