CR-Rotate Pod does not update Server IP after upgrade

By: Kee Wee Wong Account Admin 18 Mar 2020 at 6:38 a.m. CDT

4 Responses
Kee Wee Wong gravatar
Hi Support, We recently upgraded the EKS Gluu from 4.0 to 4.1, following the upgrade instruction in the documentation. https://gluu.org/docs/ce/4.1/ After upgrading, the cache refresh server IP is no longer updated by the CR-Rotate pod. The logs in the CR-Rotate pods are showing that the oxtrust container cannot be found ``` [root@ip-10-248-0-238 gluu_eks_yamls]# kubectl logs cr-rotate-4f6rm -n gluu -f INFO - pygluu.containerlib.wait - 2020-03-18 10:51:58,866 - Config is ready INFO - pygluu.containerlib.wait - 2020-03-18 10:51:59,068 - Secret is ready INFO - pygluu.containerlib.wait - 2020-03-18 10:52:00,546 - LDAP is ready INFO - entrypoint - 2020-03-18 10:52:05,749 - No oxtrust containers found on this node. Provisioning other oxtrust containers at other nodes... INFO - entrypoint - 2020-03-18 10:52:07,247 - Signal has been sent INFO - entrypoint - 2020-03-18 10:52:07,247 - Waiting for response...It may take up to 5 mins INFO - entrypoint - 2020-03-18 10:57:15,939 - No nodes found.Cache Refresh updated ip to default. Please add oxtrust containers INFO - entrypoint - 2020-03-18 10:57:19,538 - No oxtrust containers found on this node. Provisioning other oxtrust containers at other nodes... INFO - entrypoint - 2020-03-18 10:57:21,237 - Signal has been sent INFO - entrypoint - 2020-03-18 10:57:21,237 - Waiting for response...It may take up to 5 mins INFO - entrypoint - 2020-03-18 11:02:34,839 - No nodes found.Cache Refresh updated ip to default. Please add oxtrust containers INFO - entrypoint - 2020-03-18 11:02:38,836 - No oxtrust containers found on this node. Provisioning other oxtrust containers at other nodes... INFO - entrypoint - 2020-03-18 11:02:40,739 - Signal has been sent INFO - entrypoint - 2020-03-18 11:02:40,739 - Waiting for response...It may take up to 5 mins INFO - entrypoint - 2020-03-18 11:07:49,943 - No nodes found.Cache Refresh updated ip to default. Please add oxtrust containers INFO - entrypoint - 2020-03-18 11:07:53,548 - No oxtrust containers found on this node. Provisioning other oxtrust containers at other nodes... INFO - entrypoint - 2020-03-18 11:07:55,247 - Signal has been sent INFO - entrypoint - 2020-03-18 11:07:55,247 - Waiting for response...It may take up to 5 mins INFO - entrypoint - 2020-03-18 11:13:03,749 - No nodes found.Cache Refresh updated ip to default. Please add oxtrust containers INFO - entrypoint - 2020-03-18 11:13:08,038 - No oxtrust containers found on this node. Provisioning other oxtrust containers at other nodes... INFO - entrypoint - 2020-03-18 11:13:09,335 - Signal has been sent INFO - entrypoint - 2020-03-18 11:13:09,335 - Waiting for response...It may take up to 5 mins INFO - entrypoint - 2020-03-18 11:18:16,652 - No nodes found.Cache Refresh updated ip to default. Please add oxtrust containers INFO - entrypoint - 2020-03-18 11:18:20,637 - No oxtrust containers found on this node. Provisioning other oxtrust containers at other nodes... INFO - entrypoint - 2020-03-18 11:18:21,851 - Signal has been sent INFO - entrypoint - 2020-03-18 11:18:21,851 - Waiting for response...It may take up to 5 mins INFO - entrypoint - 2020-03-18 11:23:29,540 - No nodes found.Cache Refresh updated ip to default. Please add oxtrust containers INFO - entrypoint - 2020-03-18 11:23:33,142 - No oxtrust containers found on this node. Provisioning other oxtrust containers at other nodes... ``` There is only a single worker node as this is a development environment. ``` [root@ip-10-248-0-238 gluu_eks_yamls]# kubectl get po -n gluu -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES config-init-load-job-txx22 0/1 Completed 0 20d 10.248.8.48 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> cr-rotate-4f6rm 1/1 Running 0 46m 10.248.8.62 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> gluu-upgrade-job-gw26x 0/1 Completed 0 5d2h 10.248.8.60 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> key-rotation-6bc98564b7-74vgn 1/1 Running 0 5d2h 10.248.8.49 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> opendj-0 1/1 Running 0 25h 10.248.8.48 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> opendj-backup-create-1584529800-qjdw5 0/1 Completed 0 28m 10.248.8.60 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> opendj-backup-create-1584530400-ghfhs 0/1 Completed 0 18m 10.248.8.40 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> opendj-backup-create-1584531000-7dtrg 0/1 Completed 0 8m14s 10.248.8.61 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> oxauth-c45f5cbcb-lm999 1/1 Running 0 5d2h 10.248.8.50 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> oxpassport-5dddc99785-p5fb4 1/1 Running 0 5d2h 10.248.8.56 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> oxshibboleth-0 1/1 Running 0 5d2h 10.248.8.52 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> oxtrust-0 1/1 Running 0 74m 10.248.8.54 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> persistence-load-job-r8nwt 0/1 Completed 0 20d 10.248.8.52 ip-10-248-8-47.ap-southeast-1.compute.internal <none> <none> ```
AWS EKS 1.14
closed

Answers

By Mohammad Abudayyeh staff 18 Mar 2020 at 1:43 p.m. CDT

Copy
Mohammad Abudayyeh gravatar
Hey Kee Wee, I think we discovered the issue with a missing label on oxtrust pod. Please try running `kubectl label pod oxtrust-0 -n gluu APP_NAME=oxtrust` and the checking cr-pod logs. We are [updating](https://github.com/GluuFederation/enterprise-edition/issues/85) please wait for an update from us but the workaround is above for you

By Kee Wee Wong Account Admin 18 Mar 2020 at 9:01 p.m. CDT

Copy
Kee Wee Wong gravatar
Hi Mohammad, I've added the label to the pod, however, cr-rotate pod keeps crashing now with this error ``` [root@ip-10-248-0-238 ec2-user]# kubectl logs cr-rotate-4f6rm -n gluu -f INFO - pygluu.containerlib.wait - 2020-03-19 01:59:49,340 - Config is ready INFO - pygluu.containerlib.wait - 2020-03-19 01:59:49,479 - Secret is ready INFO - pygluu.containerlib.wait - 2020-03-19 01:59:50,850 - LDAP is ready INFO - entrypoint - 2020-03-19 01:59:54,845 - Oxtrust containers found after resetting to defaults. INFO - entrypoint - 2020-03-19 01:59:56,145 - Current oxTrustCacheRefreshServerIpAddress: 255.255.255.255 INFO - entrypoint - 2020-03-19 01:59:56,145 - Cleaning cache folders for oxtrust-0 with IP 10.248.8.54 Traceback (most recent call last): File "/app/scripts/entrypoint.py", line 391, in <module> main() File "/app/scripts/entrypoint.py", line 377, in main logger.info("Updating oxTrustCacheRefreshServerIpAddress to {} with IP {}".format(container.name, ip)) AttributeError: 'V1Pod' object has no attribute 'name' ```

By Mohammad Abudayyeh staff 19 Mar 2020 at 12:39 a.m. CDT

Copy
Mohammad Abudayyeh gravatar
Hey Kee Wee, You need to update CR pod image to `4.1.0_02`

By Kee Wee Wong Account Admin 19 Mar 2020 at 2:10 a.m. CDT

Copy
Kee Wee Wong gravatar
Hi Mohammad, It's working now after upgrading to `4.1.0_02` Thank you!

Post an answer