Hi Lukasz, What redis cluster are you using ? We recommend using a managed service like AWS. All you have to do there is provide the url and specify redis type `CLUSTER`. I personally never seen that error before.

We are using manually deployed redis cluster within the same kubernetes cluster. I do not think it is a matter of Redis we are using. I suppose if it was an issue with redis cluster it would be failing in the same fassion with a single oxAuth and oxTrust pod. Are there any additional components needed to enable scaling oxAuth and oxTrust? Is the nginx ingress controller (deployed automaticaly through pykubernetes script) enough to manage user sessions in different pods?

yes of course. Nothing has to be done to scale oxAuth or oxTrust. Its a cloud native application. Yes the nginx ingress controller is enough. You can try a normal setup without redis and scale to observe. We have never had issues in scaling oxAuth or oxTrust but I do think it is related to the cache layer. Many struggle with manually managing redis . Please try a managed service and check.

I've switched to NATIVE_PERSISTENCE cache. Performed login with single oxAuth pod is successful. Scaled up to 3 oxAuth pods. Performing the login again shows **ERROR: MAC did not verify!** in the logs. Seems that this issue is not related to cache type.

ok thats odd. Can you send me the EKS instance type that you used. I will attempt to replicate the issue.

I found that this error is related to PrimeFaces library. https://stackoverflow.com/a/35626681 I also looked at ingress configuration and noticed that for oxAuth there is no sticky sessions enabled in nginx annotations. I've added following annotation to oxAuth ingress definition: _nginx.ingress.kubernetes.io/affinity: "cookie" nginx.ingress.kubernetes.io/session-cookie-name: "route" nginx.ingress.kubernetes.io/session-cookie-hash: "sha1"_ and it seems to fix the above issue. Question is if it is a bug in the ingress configuration than?

No oxAuth should not have sticky sessions. I just tried the latest installer 1.4.4 that uses images `4.2.3` and I did not run into the issue on EKS. Can you please try it and let me know. For reference: Installer --> https://github.com/GluuFederation/cloud-native-edition/releases/tag/v1.4.4 EKS command: ```bash eksctl create cluster \ --name gluutest \ --version 1.19 \ --nodegroup-name standard-workers \ --node-type t2.xlarge \ --zones eu-central-1a,eu-central-1b,eu-central-1c \ --nodes 4 \ --region eu-central-1 \ --node-ami auto \ --ssh-public-key "~/.ssh/id_rsa.pub" ``` eksctl version : `0.41.0` Installer command: ``` pygluu-kubernetes-linux-amd64.pyz helm-install ``` Waited for the service to be a ready state and scaled. ``` kubectl scale deploy gluu-oxauth -n gluu --replicas=3 kubectl scale statefulset gluu-oxtrust -n gluu --replicas=3 ``` ``` NAME READY STATUS RESTARTS AGE gluu-config-pggxq 0/1 Completed 0 17m gluu-jackrabbit-0 1/1 Running 0 17m gluu-opendj-0 1/1 Running 0 17m gluu-oxauth-ddbbdbbd7-4j4c7 1/1 Running 1 17m gluu-oxauth-ddbbdbbd7-9p8vz 1/1 Running 0 12m gluu-oxauth-ddbbdbbd7-qkpwh 1/1 Running 0 12m gluu-oxtrust-0 1/1 Running 1 17m gluu-oxtrust-1 1/1 Running 0 3m17s gluu-oxtrust-2 1/1 Running 0 2m18s gluu-persistence-796db 0/1 Completed 0 17m ```

Unfortunately I am unable to create new EKS cluster. But I can replicate this issue also using my minikube local environment and helm installation.

Just to make things clear. Scaling process works fine. This error occurs when logging into gluu UI from browser (after scaled to 3 oxAuth pods).

Something is off. I can't replicate. Can you provide your `values.yaml`. You can attempt to login in the test environment that I just created. Add the following to your hosts file ``` 18.198.87.101 demoexample.gluu.org ``` And head to `https://demoexample.gluu.org` I tried to login and logout a bunch of times but I didn't run into the issue. ``` admin Test1234# ``` Also did you use the latest installer ? If so try changing your image tag for oxauth to `4.2.3_dev`.

I've tried to replicate the issue on your environment and indeed it works fine. In the attached file you can find values.yaml that I'm using in my local minikube environment with helm installation.

Can you try to use our images. A pure installation of Gluu on your minikube using the installer. I think this might be originating from your copy of our images.

Indeed. The problem seems to be related to our customization layer. Tested with base gluu images. Worked correctly. Thank you for your help.

Thanks for confirming.