Hey Brandon, Do you have openebs installed ? Thanks, Mohammad

Thanks Mohammad. Installing `openebs` got me passed the pvc issue but I still see several pods that never make it to the `Ready` state. Below is what I’m seeing now: ``` ❯ k get pods NAME READY STATUS RESTARTS AGE gluu-casa-75455b8774-p929n 0/1 Running 6 12m gluu-config-hhpfh 0/1 Completed 0 12m gluu-cr-rotate-2w8gc 1/1 Running 1 12m gluu-cr-rotate-htfsl 1/1 Running 1 12m gluu-cr-rotate-rmxfx 1/1 Running 1 12m gluu-cr-rotate-tj6w7 1/1 Running 1 12m gluu-jackrabbit-0 1/1 Running 0 12m gluu-ldap-backup-1598458800-vtcr9 1/1 Running 0 3m16s gluu-opendj-0 1/1 Running 0 12m gluu-oxauth-58474cb669-nrr9w 0/1 Running 5 12m gluu-oxd-server-78c9bc84b7-l74q6 0/1 Running 5 12m gluu-oxpassport-d9ff569d6-zhnk4 0/1 Running 1 12m gluu-oxshibboleth-0 0/1 Running 5 12m gluu-oxtrust-0 0/1 Running 4 12m gluu-persistence-cdmxt 0/1 Completed 0 12m gluu-scim-54fc5d6d75-t9dzv 0/1 Running 5 12m ``` Is this expected?

No. But it takes time for persistence to finish and openDJ to initialize. Can you please check logs of oxAuth and share. `kubectl get logs -f gluu-oxauth-58474cb669-nrr9w`

Output below. Thanks! ``` ❯ kubectl logs -f pod/gluu-oxauth-58474cb669-nrr9w INFO - 2020-08-26 16:55:11,140 - Backing up hosts file to /etc/hosts.back ... INFO - 2020-08-26 16:55:11,140 - Writing new hosts file INFO - pygluu.containerlib.wait - 2020-08-26 16:55:11,894 - Config is ready INFO - pygluu.containerlib.wait - 2020-08-26 16:55:11,928 - Secret is ready INFO - pygluu.containerlib.wait - 2020-08-26 16:55:12,338 - LDAP is ready INFO - webdav - 2020-08-26 16:55:13,150 - Sync files with remote directory http://localhost:8080/repository/default/opt/gluu/jetty/oxauth/custom ```

Is that all the logs ? Can you attach your `settings.json` please and omit all sensitive information.

Redacted settings below. Thanks again ```json { "ACCEPT_GLUU_LICENSE": "Y", "GLUU_VERSION": "4.2", "TEST_ENVIRONMENT": "", "GLUU_UPGRADE_TARGET_VERSION": "", "GLUU_HELM_RELEASE_NAME": "gluu", "NGINX_INGRESS_RELEASE_NAME": "gluu-ingress", "NGINX_INGRESS_NAMESPACE": "gluu-ingress", "INSTALL_GLUU_GATEWAY": "N", "POSTGRES_NAMESPACE": "postgres", "KONG_NAMESPACE": "", "GLUU_GATEWAY_UI_NAMESPACE": "", "KONG_PG_USER": "", "KONG_PG_PASSWORD": "", "GLUU_GATEWAY_UI_PG_USER": "", "GLUU_GATEWAY_UI_PG_PASSWORD": "", "KONG_DATABASE": "", "GLUU_GATEWAY_UI_DATABASE": "", "POSTGRES_REPLICAS": 3, "POSTGRES_URL": "postgres.postgres.svc.cluster.local", "KONG_HELM_RELEASE_NAME": "", "GLUU_GATEWAY_UI_HELM_RELEASE_NAME": "", "USE_ISTIO": "N", "USE_ISTIO_INGRESS": "N", "ISTIO_SYSTEM_NAMESPACE": "", "NODES_IPS": [ "172.19.2.38", "172.19.2.30", "172.19.2.33", "172.19.2.36", "172.19.2.37" ], "NODES_ZONES": [], "NODES_NAMES": [ "gluu-master-gro-8d472f6c78", "gluu-node-group-1c2c31bb70", "gluu-node-group-9595ecc8d3", "gluu-node-group-b34e745cfd", "gluu-node-group-c434ddbdc1" ], "NODE_SSH_KEY": "omitted", "HOST_EXT_IP": "22.22.22.22", "VERIFY_EXT_IP": "", "AWS_LB_TYPE": "", "USE_ARN": "", "ARN_AWS_IAM": "", "LB_ADD": "gluu-ingress-nginx-ingress-controller.gluu-ingress.svc.cluster.local", "REDIS_URL": "", "REDIS_TYPE": "", "REDIS_PW": "", "REDIS_USE_SSL": "false", "REDIS_SSL_TRUSTSTORE": "", "REDIS_SENTINEL_GROUP": "", "REDIS_MASTER_NODES": "", "REDIS_NODES_PER_MASTER": "", "REDIS_NAMESPACE": "", "INSTALL_REDIS": "", "INSTALL_JACKRABBIT": "Y", "JACKRABBIT_STORAGE_SIZE": "4Gi", "JACKRABBIT_URL": "http://jackrabbit:8080", "JACKRABBIT_ADMIN_ID": "admin", "JACKRABBIT_ADMIN_PASSWORD": "omitted", "JACKRABBIT_CLUSTER": "Y", "JACKRABBIT_PG_USER": "jackrabbit", "JACKRABBIT_PG_PASSWORD": "omitted", "JACKRABBIT_DATABASE": "jackrabbit", "DEPLOYMENT_ARCH": "local", "PERSISTENCE_BACKEND": "ldap", "INSTALL_COUCHBASE": "", "COUCHBASE_NAMESPACE": "", "COUCHBASE_VOLUME_TYPE": "", "COUCHBASE_CLUSTER_NAME": "", "COUCHBASE_URL": "", "COUCHBASE_USER": "", "COUCHBASE_PASSWORD": "", "COUCHBASE_CRT": "", "COUCHBASE_CN": "", "COUCHBASE_SUBJECT_ALT_NAME": "", "COUCHBASE_CLUSTER_FILE_OVERRIDE": "", "COUCHBASE_USE_LOW_RESOURCES": "", "COUCHBASE_DATA_NODES": "", "COUCHBASE_QUERY_NODES": "", "COUCHBASE_INDEX_NODES": "", "COUCHBASE_SEARCH_EVENTING_ANALYTICS_NODES": "", "COUCHBASE_GENERAL_STORAGE": "", "COUCHBASE_DATA_STORAGE": "", "COUCHBASE_INDEX_STORAGE": "", "COUCHBASE_QUERY_STORAGE": "", "COUCHBASE_ANALYTICS_STORAGE": "", "COUCHBASE_INCR_BACKUP_SCHEDULE": "", "COUCHBASE_FULL_BACKUP_SCHEDULE": "", "COUCHBASE_BACKUP_RETENTION_TIME": "", "COUCHBASE_BACKUP_STORAGE_SIZE": "", "LDAP_BACKUP_SCHEDULE": "*/30 * * * *", "NUMBER_OF_EXPECTED_USERS": "", "EXPECTED_TRANSACTIONS_PER_SEC": "", "USING_CODE_FLOW": "", "USING_SCIM_FLOW": "", "USING_RESOURCE_OWNER_PASSWORD_CRED_GRANT_FLOW": "", "DEPLOY_MULTI_CLUSTER": "", "HYBRID_LDAP_HELD_DATA": "", "LDAP_JACKRABBIT_VOLUME": "", "APP_VOLUME_TYPE": 26, "LDAP_STATIC_VOLUME_ID": "", "LDAP_STATIC_DISK_URI": "", "GLUU_CACHE_TYPE": "NATIVE_PERSISTENCE", "GLUU_NAMESPACE": "gluu", "GLUU_FQDN": "omitted", "COUNTRY_CODE": "US", "STATE": "TX", "EMAIL": "support@gluu.org", "CITY": "omitted", "ORG_NAME": "Gluu", "GMAIL_ACCOUNT": "", "GOOGLE_NODE_HOME_DIR": "", "IS_GLUU_FQDN_REGISTERED": "N", "LDAP_PW": "omitted", "ADMIN_PW": "omitted", "OXD_APPLICATION_KEYSTORE_CN": "oxd-server", "OXD_ADMIN_KEYSTORE_CN": "oxd-server", "LDAP_STORAGE_SIZE": "4Gi", "OXAUTH_REPLICAS": 1, "OXTRUST_REPLICAS": 1, "LDAP_REPLICAS": 1, "OXSHIBBOLETH_REPLICAS": 1, "OXPASSPORT_REPLICAS": 1, "OXD_SERVER_REPLICAS": 1, "CASA_REPLICAS": 1, "RADIUS_REPLICAS": "", "FIDO2_REPLICAS": "", "SCIM_REPLICAS": 1, "ENABLE_OXTRUST_API": "Y", "ENABLE_OXTRUST_TEST_MODE": "N", "ENABLE_CACHE_REFRESH": "Y", "ENABLE_OXD": "Y", "ENABLE_FIDO2": "N", "ENABLE_SCIM": "Y", "ENABLE_RADIUS": "N", "ENABLE_OXPASSPORT": "Y", "ENABLE_OXSHIBBOLETH": "Y", "ENABLE_CASA": "Y", "ENABLE_OXAUTH_KEY_ROTATE": "Y", "ENABLE_OXTRUST_API_BOOLEAN": "true", "ENABLE_OXTRUST_TEST_MODE_BOOLEAN": "false", "ENABLE_RADIUS_BOOLEAN": "false", "ENABLE_OXPASSPORT_BOOLEAN": "true", "ENABLE_CASA_BOOLEAN": "true", "ENABLE_SAML_BOOLEAN": "true", "ENABLED_SERVICES_LIST": [ "config", "oxauth", "oxtrust", "persistence", "jackrabbit", "cr-rotate", "oxauth-key-rotation", "oxpassport", "oxshibboleth", "casa", "scim", "oxd-server", "ldap" ], "OXAUTH_KEYS_LIFE": 48, "EDIT_IMAGE_NAMES_TAGS": "N", "CASA_IMAGE_NAME": "gluufederation/casa", "CASA_IMAGE_TAG": "4.2.0_02", "CONFIG_IMAGE_NAME": "gluufederation/config-init", "CONFIG_IMAGE_TAG": "4.2.0_01", "CACHE_REFRESH_ROTATE_IMAGE_NAME": "gluufederation/cr-rotate", "CACHE_REFRESH_ROTATE_IMAGE_TAG": "4.2.0_01", "CERT_MANAGER_IMAGE_NAME": "gluufederation/certmanager", "CERT_MANAGER_IMAGE_TAG": "4.2.0_01", "LDAP_IMAGE_NAME": "gluufederation/wrends", "LDAP_IMAGE_TAG": "4.2.0_02", "JACKRABBIT_IMAGE_NAME": "gluufederation/jackrabbit", "JACKRABBIT_IMAGE_TAG": "4.2.0_01", "OXAUTH_IMAGE_NAME": "gluufederation/oxauth", "OXAUTH_IMAGE_TAG": "4.2.0_02", "FIDO2_IMAGE_NAME": "gluufederation/fido2", "FIDO2_IMAGE_TAG": "4.2.0_01", "SCIM_IMAGE_NAME": "gluufederation/scim", "SCIM_IMAGE_TAG": "4.2.0_02", "OXD_IMAGE_NAME": "gluufederation/oxd-server", "OXD_IMAGE_TAG": "4.2.0_01", "OXPASSPORT_IMAGE_NAME": "gluufederation/oxpassport", "OXPASSPORT_IMAGE_TAG": "4.2.0_01", "OXSHIBBOLETH_IMAGE_NAME": "gluufederation/oxshibboleth", "OXSHIBBOLETH_IMAGE_TAG": "4.2.0_02", "OXTRUST_IMAGE_NAME": "gluufederation/oxtrust", "OXTRUST_IMAGE_TAG": "4.2.0_02", "PERSISTENCE_IMAGE_NAME": "gluufederation/persistence", "PERSISTENCE_IMAGE_TAG": "4.2.0_02", "RADIUS_IMAGE_NAME": "gluufederation/radius", "RADIUS_IMAGE_TAG": "4.2.0_01", "GLUU_GATEWAY_IMAGE_NAME": "gluufederation/gluu-gateway", "GLUU_GATEWAY_IMAGE_TAG": "4.2.0_01", "GLUU_GATEWAY_UI_IMAGE_NAME": "gluufederation/gluu-gateway-ui", "GLUU_GATEWAY_UI_IMAGE_TAG": "4.2.0_01", "UPGRADE_IMAGE_NAME": "gluufederation/upgrade", "UPGRADE_IMAGE_TAG": "4.2.0_01", "CONFIRM_PARAMS": "Y" } ```

There doesn't seem to be any issues there. As I try to replicate please try the following : 1. Get the ssl cert and place it in file `gluu_https.crt` ```bash kubectl get secret gluu -o json -n gluu | grep '"ssl_cert":' | sed -e 's#.*:\(\)#\1#' | tr -d '"' | tr -d "," | tr -d '[:space:]' > gluu_https.crt ``` 1. Get the key and place it in file `gluu_https.key` ```bash kubectl get secret gluu -o json -n gluu | grep '"ssl_key":' | sed -e 's#.*:\(\)#\1#' | tr -d '"' | tr -d "," | tr -d '[:space:]' > gluu_https.key ``` 1. Add both to secrets ```bash kubectl create secret generic crt -n gluu --from-file=gluu_https.crt kubectl create secret generic key -n gluu --from-file=gluu_https.key ``` 1. Mount secrets inside `oxAuth` deployment. A helm upgrade would be the normal choice here but for testing you may add it directly to the oxAuth deployment. ```bash kubectl edit deployment gluu-oxauth -n gluu ``` Add the following to the `volumeMounts` and `volumes` sections: ```yaml volumeMounts: - name: gluukey mountPath: "/etc/certs/gluu_https.key" subPath: gluu_https.key - name: gluucrt mountPath: "/etc/certs/gluu_https.crt" subPath: gluu_https.crt volumes: - name: gluucrt secret: secretName: crt - name: gluukey secret: secretName: key ``` 1. Save `:wq` If the above works, repeat steps 4 and 5 for oxTrust. Step 4 would be `kubectl edit statefulset gluu-oxtrust -n gluu` Let me know how it goes as we dig in here on our side. Thanks

I’ve applied the changes suggested below. It appears oxauth is now running but oxtrust still appears to be crashing. Thanks! ```bash ❯ kubectl get pods NAME READY STATUS RESTARTS AGE gluu-casa-75455b8774-p929n 0/1 CrashLoopBackOff 1552 4d22h gluu-config-hhpfh 0/1 Completed 0 4d22h gluu-cr-rotate-2w8gc 1/1 Running 1 4d22h gluu-cr-rotate-htfsl 1/1 Running 1 4d22h gluu-cr-rotate-rmxfx 1/1 Running 1 4d22h gluu-cr-rotate-tj6w7 1/1 Running 1 4d22h gluu-jackrabbit-0 1/1 Running 0 4d22h gluu-ldap-backup-1598881200-2bf8j 0/1 Error 0 41m gluu-ldap-backup-1598881200-52s4n 0/1 Error 0 15m gluu-ldap-backup-1598881200-8br6d 0/1 Error 0 51m gluu-ldap-backup-1598881200-c5gsb 0/1 Error 0 35m gluu-ldap-backup-1598881200-cppqf 0/1 Error 0 23m gluu-ldap-backup-1598881200-m5r5s 0/1 Error 0 46m gluu-ldap-backup-1598881200-zkkmk 0/1 Error 0 29m gluu-ldap-backup-1598883600-dctrc 0/1 Error 0 10m gluu-ldap-backup-1598883600-s7ds4 1/1 Running 0 5m20s gluu-opendj-0 1/1 Running 0 4d22h gluu-oxauth-5f7fb69587-x5ppp 1/1 Running 0 17m gluu-oxauth-key-rotation-1598659200-mpgcn 0/1 Completed 0 2d14h gluu-oxauth-key-rotation-1598745600-kv98c 0/1 Completed 0 38h gluu-oxauth-key-rotation-1598832000-fznxz 0/1 Completed 0 14h gluu-oxd-server-78c9bc84b7-l74q6 0/1 Running 1499 4d22h gluu-oxpassport-d9ff569d6-zhnk4 0/1 Running 645 4d22h gluu-oxshibboleth-0 0/1 CrashLoopBackOff 1499 4d22h gluu-oxtrust-0 0/1 Running 2 8m36s gluu-persistence-cdmxt 0/1 Completed 0 4d22h gluu-scim-54fc5d6d75-t9dzv 0/1 Running 1501 4d22h ```

Yea please repeat steps 4 and 5 for oxTrust as listed [here](https://support.gluu.org/8759/#at63249). oxTrust depends on oxAuth so after the edit it should work as well. The rest of the services should work as well without any edits.

Sorry I should have specified that I applied the changes to both oxauth and oxtrust as you suggested in your email. The screenshot I shared was after applying the changes to both.

Hey Brandon, Can you attach oxtrust logs please.

Hey Brandon, Any news on your deployment? Best,

So sorry for the delay. I was looking through the logs and it looks like it’s having issues resolving the hostname I configured during setup. I’ll dig into this and will upload the logs if I still can’t get it to work. Thanks!

Thanks for letting us know. I wanted to let you know that you may want to use the latest installer https://github.com/GluuFederation/cloud-native-edition/releases/tag/v1.2.6 as it does not call the FQDN but rather pulls the ssl from secret layer. Hence, won't require the workaround we did [here](https://support.gluu.org/installation/8759/kubernetes-gluu-install-failing/#at63249). Thanks, Mohammad

I attempted a re-deploy using the latest cloud-native-edition release; however, I’m still not able to complete the install. Below are a few kubectl outputs. Please let me know if there’s any additional logs I can grab. Thank you so much. ```bash ❯ k get pods NAME READY STATUS RESTARTS AGE gluu-casa-7dd55c7f47-9tnrt 0/1 Running 9 26m gluu-config-snhwk 0/1 Completed 0 26m gluu-cr-rotate-5qcp7 1/1 Running 0 26m gluu-cr-rotate-qc9gt 1/1 Running 0 26m gluu-cr-rotate-ql8jx 1/1 Running 0 26m gluu-cr-rotate-z6kvr 1/1 Running 0 26m gluu-jackrabbit-0 1/1 Running 0 26m gluu-ldap-backup-1601485200-hdfw6 0/1 Error 0 5m21s gluu-ldap-backup-1601485200-np8bw 0/1 Error 0 21m gluu-ldap-backup-1601485200-r2bx7 0/1 Error 0 16m gluu-ldap-backup-1601485200-wstfc 0/1 Error 0 11m gluu-opendj-0 1/1 Running 0 26m gluu-oxauth-cdd459456-m85tk 0/1 CrashLoopBackOff 8 26m gluu-oxd-server-69cfb866c4-hv7hg 0/1 CrashLoopBackOff 8 26m gluu-oxpassport-9f69f69f-9tdjq 0/1 Running 2 26m gluu-oxshibboleth-0 0/1 Running 8 26m gluu-oxtrust-0 0/1 CrashLoopBackOff 7 26m gluu-persistence-v9zqn 0/1 Completed 0 26m gluu-scim-69b4db7f78-mfwjf 0/1 Running 8 26m ``` ```bash ❯ k logs pod/gluu-oxauth-cdd459456-m85tk INFO - 2020-09-30 17:15:27,385 - Backing up hosts file to /etc/hosts.back ... INFO - 2020-09-30 17:15:27,386 - Writing new hosts file INFO - pygluu.containerlib.wait - 2020-09-30 17:15:28,110 - Config is ready INFO - pygluu.containerlib.wait - 2020-09-30 17:15:28,148 - Secret is ready INFO - pygluu.containerlib.wait - 2020-09-30 17:15:28,559 - LDAP is ready INFO - webdav - 2020-09-30 17:15:29,403 - Sync files with remote directory http://localhost:8080/repository/default/opt/gluu/jetty/oxauth/custom Traceback (most recent call last): File "/app/scripts/entrypoint.py", line 149, in module> main() File "/app/scripts/entrypoint.py", line 90, in main get_server_certificate(manager.config.get("hostname"), 443, "/etc/certs/gluu_https.crt") File "/src/pygluu-containerlib/pygluu/containerlib/utils.py", line 183, in get_server_certificate with socket.create_connection((host, port)) as conn: File "/usr/lib/python3.8/socket.py", line 808, in create_connection raise err File "/usr/lib/python3.8/socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Operation timed out ```

Hey Brandon, add an ENV to the main configmap. `k edit cm gluu-config-cm` ``` GLUU_SSL_CERT_FROM_SECRETS: "true" ``` and restart your oxauth pod by deleting it.

I applied the change and now see the following in the pod logs. Thanks! ```bash ❯ k logs pod/gluu-oxauth-cdd459456-cjdw7 INFO - 2020-09-30 19:14:30,348 - Backing up hosts file to /etc/hosts.back ... INFO - 2020-09-30 19:14:30,349 - Writing new hosts file INFO - pygluu.containerlib.wait - 2020-09-30 19:14:31,050 - Config is ready INFO - pygluu.containerlib.wait - 2020-09-30 19:14:31,077 - Secret is ready INFO - pygluu.containerlib.wait - 2020-09-30 19:14:31,472 - LDAP is ready INFO - webdav - 2020-09-30 19:14:32,350 - Sync files with remote directory http://localhost:8080/repository/default/opt/gluu/jetty/oxauth/custom Traceback (most recent call last): File "/app/scripts/entrypoint.py", line 149, in module> main() File "/app/scripts/entrypoint.py", line 90, in main get_server_certificate(manager.config.get("hostname"), 443, "/etc/certs/gluu_https.crt") File "/src/pygluu-containerlib/pygluu/containerlib/utils.py", line 183, in get_server_certificate with socket.create_connection((host, port)) as conn: File "/usr/lib/python3.8/socket.py", line 808, in create_connection raise err File "/usr/lib/python3.8/socket.py", line 796, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Operation timed out ```

Thats odd. Whats the oxauth image here ?

`Image: gluufederation/oxauth:4.2.0_02`

That explains it. I wonder if you used the latest installer then because we released `4.2.1` in Sept. Anyways please use the latest which would be todays release and let us know. https://github.com/GluuFederation/cloud-native-edition/releases/tag/v1.2.11 Thanks!!

Since the upgrade I’m now seeing this ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 79s (x61 over 78m) default-scheduler 0/9 nodes are available: 4 Insufficient cpu, 5 node(s) had volume node affinity conflict. ``` Did the CPU requirements change? Thanks

No. Its the same. Since its a test setup , enable test enviorment which would strip away all resource definitions.