oxauth key rotation job failures

By: paras joshi user 30 Nov 2022 at 1:29 a.m. CST

2 Responses
paras joshi gravatar
expected behaviour - jwks keys should be rotated. Actual behaviour - Keys were not rotated and job had failures . File "/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 994, in connect_get_namespaced_pod_exec return self.connect_get_namespaced_pod_exec_with_http_info(name, namespace, **kwargs) # noqa: E501 File "/usr/lib/python3.8/site-packages/kubernetes/client/api/core_v1_api.py", line 1101, in connect_get_namespaced_pod_exec_with_http_info return self.api_client.call_api( File "/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 348, in call_api return self.__call_api(resource_path, method, File "/usr/lib/python3.8/site-packages/kubernetes/client/api_client.py", line 180, in __call_api response_data = self.request( File "/usr/lib/python3.8/site-packages/kubernetes/stream/ws_client.py", line 515, in websocket_call raise ApiException(status=0, reason=str(e)) kubernetes.client.exceptions.ApiException: (0) Reason: Handshake status 500 Internal Server Error Current status - this is fixed after the oxauth pods were in running state. Root cause - Pods were in evicted state due to which job was not able to connect to pods , hence, internal server error. Could you please investigate to have check for key rotation job to first verify the status of the pods and before the job is run.
Gluu 4.2.3 Ubuntu 18.0
closed

Answers

By Isman Firmansyah staff 05 Dec 2022 at 2:37 a.m. CST

Copy
Isman Firmansyah gravatar
Hi Paras Joshi, The `certmanager` image used in oxauth-key-rotation cronjob doesn't check the readiness of other components (i.e. oxAuth) before running command. However you can intercept pod entrypoint (`command`) to ensure required components. Example of using `curl` to check for oxAuth readiness: ``` spec: jobTemplate: spec: template: spec: containers: - name: oxauth-key-rotation command: - tini - -g - -- - /bin/sh - -c - | if [ $(curl http://oxauth:8080/oxauth/restv1/health-check -i -w "%{http_code}" -o /dev/null -s) != "200" ]; then echo "oxauth not ready ... exiting" && exit 1; fi /app/scripts/entrypoint.sh patch oxauth --opts interval:48 envFrom: - configMapRef: name: gluu-config-cm ``` Or using `python`: ``` spec: jobTemplate: spec: template: spec: containers: - name: oxauth-key-rotation command: - tini - -g - -- - /bin/sh - -c - | python3 -c "from pygluu.containerlib import get_manager, wait_for; wait_for(get_manager(), deps=['oxauth'])" /app/scripts/entrypoint.sh patch oxauth --opts interval:48 envFrom: - configMapRef: name: gluu-config-cm # add the environment variable below env: - name: GLUU_OXAUTH_BACKEND value: "oxauth:8080" # format: oxauth-service-name:port ```

By Isman Firmansyah staff 19 Dec 2022 at 4:13 p.m. CST

Copy
Isman Firmansyah gravatar
Feel free to re-open this ticket if required. Best regards,

Post an answer