By: Tomas Larsson user 15 Mar 2021 at 12:58 p.m. CDT

8 Responses
Tomas Larsson gravatar
Hi We have been running Gluu for some time now (approx 6 months). Last couple of weeks we have had problems that the server suddenly stops respond. So far it seems that ou=cache, o=glue grows above reasonable limits. For now we have about 1 400 000 entrys which seems cause problems for the clean-up process. I made an export of ou=cache and found that the attribute 'exp' almost always has a strange value. For example: ``` del: true exp: 20880926172951.742Z createTimestamp: 20200908141544Z ``` As you can see the attribute `exp` is 68 years and a couple of days in the future. And the search from clean-up process are looking for entries that are one hour old or more, which never occurs in this situation. And as a result we now have so many entries that timeout occurs which seems to get the server confused, I think? So I guess the question is. What causes the attribute `exp` to get this strange value? Hope someone can help with this. /Tomas Errormessage looks like this: ``` 2021-03-10 09:49:47,908 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:262] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20210310084447.907Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.searchImpl(LdapOperationsServiceImpl.java:412) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: A client-side timeout was encountered while waiting 300000ms for a response to search request with message ID 8, base DN 'ou=cache,o=gluu', scope SUB, and filter '(&(&(objectClass=cache))(&(del=true)(exp<=20210310084447.907Z)))' from server localhost:1636. at com.unboundid.ldap.sdk.SearchRequest.process(SearchRequest.java:1206) ```

By Michael Schwartz Account Admin 15 Mar 2021 at 1:19 p.m. CDT

Michael Schwartz gravatar
@Mobarak Hosen.Shakil, can you figure out why the expiration time is so far in the future? Is there a JSON property for oxAuth that controls that? Can you dig into this, to make sure it's not a misconfig or typo out of the box?

By Tomas Larsson user 19 Mar 2021 at 5:18 a.m. CDT

Tomas Larsson gravatar
Hi Some additional info. If I take the 'dat' attribute from cache-entries with strange expire date and run the value through a base64 decoder I will find the following. I assume the datablob represents some kind of java object. The first part in the decoded blob is always **org.gluu.oxauth.model.common.SessionIdPv** which maybe says something about what module/part in the server that has generated the cache-entry. All entries with correct expiredate says something else. Just a thought. The complete decoded blob itself. ?sr&org.gluu.oxauth.model.common.SessionIdPv);.???L?authenticationTimet?Ljava/util/Date;L?dnt?Ljava/lang/String;L?idq~?L?involvedClientst1Lorg/gluu/oxauth/model/common/SessionIdAccessMap;L?isJwtt?Ljava/lang/Boolean;L?jwtq~?L lastUsedAtq~?L?permissionGrantedq~?L?permissionGrantedMapq~?L?sessionAttributest?Ljava/util/Map;L?sessionStateq~?L?statet-Lorg/gluu/oxauth/model/common/SessionIdState;L?usert#Lorg/gluu/oxauth/model/common/User;L?userDnq~?xpsr?java.util.Datehj?KYt??xpw??x@l-Zxt$04032aea-911a-4931-a1f6-b014aaa486b7q~?psr?java.lang.Boolean r՜??Z?valuexppsq~ w??x@{^xpsr/org.gluu.oxauth.model.common.SessionIdAccessMap;#???L?permissionGrantedq~?xpsr?java.util.HashMap???`??F loadFactorI thresholdxp?@?w???t)1001.37d6ebf8-85a5-4460-b4ce-b5e021198e26sq~??t)1101.e3ffdefc-c344-4bdc-bab8-bb9fb8f15477sq~??xsq~??@?w? ?t?auth_external_attributespt?opbst$a900686d-5854-49ad-ab8f-e0591cc14fd1t response_typet?codet?noncet UzY8qVeFJ4t client_idq~?t auth_stept?1t?acrt?basict remote_ipt 10.1.100.9t auth_usert?tlht?scopet?openid email user_namet?redirect_urit#https://idp.oru.se/idp/Authn/oxAutht?statet`eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdGF0ZSI6Ind6M29ScnJOSkIiLCJjb252ZXJzYXRpb24iOiJlM3MxIn0.xteeee8c7ce3f7297d9f5649434c2891f65e9717f119d3453b96d2c0c9e38871ca5.4f33aeee-bfee-4ea7-96db-53884a17faa0~r+org.gluu.oxauth.model.common.SessionIdState?xr?java.lang.Enum?xpt AUTHENTICATEDsr!org.gluu.oxauth.model.common.User\?a|H?xr'org.gluu.oxauth.model.common.SimpleUserqx!G8??xr&org.gluu.persist.model.base.SimpleUserR/ȫ8???L?customAttributest?Ljava/util/List;[?customObjectClassest?[Ljava/lang/String;L?dnq~?[?oxAuthPersistentJwtq~7L?userIdq~?xpsr?java.util.ArrayListx?a??I?sizexp?w??sr+org.gluu.persist.model.base.CustomAttribute?`?*??L?nameq~?L?valuesq~6xpt?cnsq~9?w??t?Tomas Liljeberghxsq~;t?displayNamesq~9?w??t?Tomas Liljeberghxsq~;t?eduPersonAffiliationsq~9?w??t?employeet?memberxsq~;t?eduPersonEntitlementsq~9?w??t%urn:mace:terena.org:tcs:personal-userxsq~;t?eduPersonOrcidsq~9?w??t%https://orcid.org/0000-0001-6451-1270xsq~;t?eduPersonOrgDNsq~9?w??t?cn=oru-org, dc=oru, dc=sexsq~;t?eduPersonOrgUnitDNsq~9?w??t"cn=8500, cn=oru-org, dc=oru, dc=sexsq~;t?eduPersonPrimaryOrgUnitDNsq~9?w??t"cn=8500, cn=oru-org, dc=oru, dc=sexsq~;t?eduPersonPrincipalNamesq~9?w??t tlh@oru.sexsq~;t?eduPersonTargetedIDsq~9?w??t tlh@oru.sexsq~;t givenNamesq~9?w??t?Tomasxsq~;t?gluuSLAManagersq~9?w??t?truexsq~;t gluuStatussq~9?w??t?activexsq~;t?inumsq~9?w??t?0000!35E5.C660xsq~;t?mailsq~9?w??t?Tomas.liljebergh@oru.sexsq~;t?memberOfsq~9?w??t?inum=60B7,ou=groups,o=gluuxsq~;t?norEduPersonNINsq~9?w??t?196407286753xsq~;t?ousq~9?w??t?IT-avdelningenxsq~;t?oxCreationTimestampsq~9?w??t?20200515142745.506Zxsq~;t?oxTrustEmailsq~9?w??tN{"value":"Tomas.liljebergh@oru.se","display":null,"type":null,"primary":false}xsq~;t?snsq~9?w??t Liljeberghxsq~;t updatedAtsq~9?w??t?20210216140405.262Zxxur?[Ljava.lang.String;V?{G?xp?t?topt eduPersont?gluuCustomPersont$inum=0000!35E5.C660,ou=people,o=gluupt?tlhq~

By Mobarak Hosen Shakil staff 29 Mar 2021 at 8:31 a.m. CDT

Mobarak Hosen Shakil gravatar
Hi Tomas Larsson, I tried to replicate this issue. Can you please share the below information? - cleanServiceInterval - cleanServiceBaseDns - cleanServiceBatchChunkSize All of this properties are in: `Gluu UI > JSON Configuration > oxAuth Configuration`. Please follow this link to better understand: https://gluu.org/docs/gluu-server/4.1/operation/cleanup/#properties Thanks & Regards ~ Shakil

By Tomas Larsson user 29 Mar 2021 at 8:53 a.m. CDT

Tomas Larsson gravatar
Hi Shakil Thank You for your response. Below the requested info. cleanServiceInterval 60 cleanServiceBaseDns No additional base DNs defined cleanServiceBatchChunkSize 10000 Regards /Tomas

By Mohib Zico staff 03 Apr 2021 at 11:32 p.m. CDT

Mohib Zico gravatar
Hi, Attaching my first findings. It's a default 4.1.1 installation, no customization applied. I couldn't found any cache which has such big lifetime. A GIF attached. I'll see how these cache entries behave after 48 hours.

By Tomas Larsson user 09 Apr 2021 at 8:24 a.m. CDT

Tomas Larsson gravatar
Hi Mohib Nice GIF :) Thank you for your dedication. In our case the cache entries is created with the strange expiration date from start. I was thinking, maybe the GUI fools us. Where is the actuall values stored? A specific configuration file? In the LDAP itself? Or is it hard coded? Regards /Tomas

By Mohib Zico staff 17 May 2021 at 9:49 a.m. CDT

Mohib Zico gravatar
Hi Tomas, >> Where is the actuall values stored? A specific configuration file? In the LDAP itself? Or is it hard coded? They are stored in LDAP. Screenshot attached. And being removed periodically.

By Tomas Larsson user 18 May 2021 at 2:17 a.m. CDT

Tomas Larsson gravatar
Interesting answer.