By: Tomas Larsson user 15 Mar 2021 at 12:58 p.m. CDT

6 Responses
Tomas Larsson gravatar
Hi We have been running Gluu for some time now (approx 6 months). Last couple of weeks we have had problems that the server suddenly stops respond. So far it seems that ou=cache, o=glue grows above reasonable limits. For now we have about 1 400 000 entrys which seems cause problems for the clean-up process. I made an export of ou=cache and found that the attribute 'exp' almost always has a strange value. For example: ``` del: true exp: 20880926172951.742Z createTimestamp: 20200908141544Z ``` As you can see the attribute `exp` is 68 years and a couple of days in the future. And the search from clean-up process are looking for entries that are one hour old or more, which never occurs in this situation. And as a result we now have so many entries that timeout occurs which seems to get the server confused, I think? So I guess the question is. What causes the attribute `exp` to get this strange value? Hope someone can help with this. /Tomas Errormessage looks like this: ``` 2021-03-10 09:49:47,908 - - ERROR [org.gluu.service.cache.NativePersistenceCacheProvider:262] - Failed to perform clean up. org.gluu.persist.exception.EntryDeleteException: Failed to delete entries with baseDN: ou=cache,o=gluu, filter: (&(&(objectClass=cache))(&(del=true)(exp<=20210310084447.907Z))) at org.gluu.persist.ldap.impl.LdapEntryManager.remove(LdapEntryManager.java:343) Caused by: org.gluu.persist.exception.operation.SearchException: Failed to scroll to specified start at org.gluu.persist.ldap.operation.impl.LdapOperationsServiceImpl.searchImpl(LdapOperationsServiceImpl.java:412) Caused by: com.unboundid.ldap.sdk.LDAPSearchException: A client-side timeout was encountered while waiting 300000ms for a response to search request with message ID 8, base DN 'ou=cache,o=gluu', scope SUB, and filter '(&(&(objectClass=cache))(&(del=true)(exp<=20210310084447.907Z)))' from server localhost:1636. at com.unboundid.ldap.sdk.SearchRequest.process(SearchRequest.java:1206) ```

By Michael Schwartz staff 15 Mar 2021 at 1:19 p.m. CDT

Michael Schwartz gravatar
@Mobarak Hosen.Shakil, can you figure out why the expiration time is so far in the future? Is there a JSON property for oxAuth that controls that? Can you dig into this, to make sure it's not a misconfig or typo out of the box?

By Tomas Larsson user 19 Mar 2021 at 5:18 a.m. CDT

Tomas Larsson gravatar
Hi Some additional info. If I take the 'dat' attribute from cache-entries with strange expire date and run the value through a base64 decoder I will find the following. I assume the datablob represents some kind of java object. The first part in the decoded blob is always **org.gluu.oxauth.model.common.SessionIdPv** which maybe says something about what module/part in the server that has generated the cache-entry. All entries with correct expiredate says something else. Just a thought. The complete decoded blob itself. ?sr&org.gluu.oxauth.model.common.SessionIdPv);.???L?authenticationTimet?Ljava/util/Date;L?dnt?Ljava/lang/String;L?idq~?L?involvedClientst1Lorg/gluu/oxauth/model/common/SessionIdAccessMap;L?isJwtt?Ljava/lang/Boolean;L?jwtq~?L lastUsedAtq~?L?permissionGrantedq~?L?permissionGrantedMapq~?L?sessionAttributest?Ljava/util/Map;L?sessionStateq~?L?statet-Lorg/gluu/oxauth/model/common/SessionIdState;L?usert#Lorg/gluu/oxauth/model/common/User;L?userDnq~?xpsr?java.util.Datehj?KYt??xpw??x@l-Zxt$04032aea-911a-4931-a1f6-b014aaa486b7q~?psr?java.lang.Boolean r՜??Z?valuexppsq~ w??x@{^xpsr/org.gluu.oxauth.model.common.SessionIdAccessMap;#???L?permissionGrantedq~?xpsr?java.util.HashMap???`??F loadFactorI thresholdxp?@?w???t)1001.37d6ebf8-85a5-4460-b4ce-b5e021198e26sq~??t)1101.e3ffdefc-c344-4bdc-bab8-bb9fb8f15477sq~??xsq~??@?w? ?t?auth_external_attributespt?opbst$a900686d-5854-49ad-ab8f-e0591cc14fd1t response_typet?codet?noncet UzY8qVeFJ4t client_idq~?t auth_stept?1t?acrt?basict remote_ipt 10.1.100.9t auth_usert?tlht?scopet?openid email user_namet?redirect_urit#https://idp.oru.se/idp/Authn/oxAutht?statet`eyJ0eXAiOiJKV1QiLCJhbGciOiJub25lIn0.eyJzdGF0ZSI6Ind6M29ScnJOSkIiLCJjb252ZXJzYXRpb24iOiJlM3MxIn0.xteeee8c7ce3f7297d9f5649434c2891f65e9717f119d3453b96d2c0c9e38871ca5.4f33aeee-bfee-4ea7-96db-53884a17faa0~r+org.gluu.oxauth.model.common.SessionIdState?xr?java.lang.Enum?xpt AUTHENTICATEDsr!org.gluu.oxauth.model.common.User\?a|H?xr'org.gluu.oxauth.model.common.SimpleUserqx!G8??xr&org.gluu.persist.model.base.SimpleUserR/ȫ8???L?customAttributest?Ljava/util/List;[?customObjectClassest?[Ljava/lang/String;L?dnq~?[?oxAuthPersistentJwtq~7L?userIdq~?xpsr?java.util.ArrayListx?a??I?sizexp?w??sr+org.gluu.persist.model.base.CustomAttribute?`?*??L?nameq~?L?valuesq~6xpt?cnsq~9?w??t?Tomas Liljeberghxsq~;t?displayNamesq~9?w??t?Tomas Liljeberghxsq~;t?eduPersonAffiliationsq~9?w??t?employeet?memberxsq~;t?eduPersonEntitlementsq~9?w??t%urn:mace:terena.org:tcs:personal-userxsq~;t?eduPersonOrcidsq~9?w??t%https://orcid.org/0000-0001-6451-1270xsq~;t?eduPersonOrgDNsq~9?w??t?cn=oru-org, dc=oru, dc=sexsq~;t?eduPersonOrgUnitDNsq~9?w??t"cn=8500, cn=oru-org, dc=oru, dc=sexsq~;t?eduPersonPrimaryOrgUnitDNsq~9?w??t"cn=8500, cn=oru-org, dc=oru, dc=sexsq~;t?eduPersonPrincipalNamesq~9?w??t tlh@oru.sexsq~;t?eduPersonTargetedIDsq~9?w??t tlh@oru.sexsq~;t givenNamesq~9?w??t?Tomasxsq~;t?gluuSLAManagersq~9?w??t?truexsq~;t gluuStatussq~9?w??t?activexsq~;t?inumsq~9?w??t?0000!35E5.C660xsq~;t?mailsq~9?w??t?Tomas.liljebergh@oru.sexsq~;t?memberOfsq~9?w??t?inum=60B7,ou=groups,o=gluuxsq~;t?norEduPersonNINsq~9?w??t?196407286753xsq~;t?ousq~9?w??t?IT-avdelningenxsq~;t?oxCreationTimestampsq~9?w??t?20200515142745.506Zxsq~;t?oxTrustEmailsq~9?w??tN{"value":"Tomas.liljebergh@oru.se","display":null,"type":null,"primary":false}xsq~;t?snsq~9?w??t Liljeberghxsq~;t updatedAtsq~9?w??t?20210216140405.262Zxxur?[Ljava.lang.String;V?{G?xp?t?topt eduPersont?gluuCustomPersont$inum=0000!35E5.C660,ou=people,o=gluupt?tlhq~

By Mobarak Hosen Shakil staff 29 Mar 2021 at 8:31 a.m. CDT

Mobarak Hosen Shakil gravatar
Hi Tomas Larsson, I tried to replicate this issue. Can you please share the below information? - cleanServiceInterval - cleanServiceBaseDns - cleanServiceBatchChunkSize All of this properties are in: `Gluu UI > JSON Configuration > oxAuth Configuration`. Please follow this link to better understand: https://gluu.org/docs/gluu-server/4.1/operation/cleanup/#properties Thanks & Regards ~ Shakil

By Tomas Larsson user 29 Mar 2021 at 8:53 a.m. CDT

Tomas Larsson gravatar
Hi Shakil Thank You for your response. Below the requested info. cleanServiceInterval 60 cleanServiceBaseDns No additional base DNs defined cleanServiceBatchChunkSize 10000 Regards /Tomas

By Mohib Zico staff 03 Apr 2021 at 11:32 p.m. CDT

Mohib Zico gravatar
Hi, Attaching my first findings. It's a default 4.1.1 installation, no customization applied. I couldn't found any cache which has such big lifetime. A GIF attached. I'll see how these cache entries behave after 48 hours.

By Tomas Larsson user 09 Apr 2021 at 8:24 a.m. CDT

Tomas Larsson gravatar
Hi Mohib Nice GIF :) Thank you for your dedication. In our case the cache entries is created with the strange expiration date from start. I was thinking, maybe the GUI fools us. Where is the actuall values stored? A specific configuration file? In the LDAP itself? Or is it hard coded? Regards /Tomas