Docker Restart required in order for Vault Unsealing steps to occur after calling pygluu-compose.pyz up

By: Travis Salyer user 22 Apr 2021 at 2:04 p.m. CDT

5 Responses
Travis Salyer gravatar
Clarification on OS: Using Docker-Desktop version 3.3.0 (62916) within a Windows 10 environment using the WSL2 based engine. The starting container is running in a python:3-alpine3.13 container. During the rapid construction and destruction of docker containers and images to test out new configurations, a docker restart is required in order to have the `pygluu-compose.pyz up` command result in the successful creation and configuration of the consul and vault containers with auto-unsealing to work correctly. Subsequent attempts of running `pygluu-compose.pyz up` after deleting old containers and images and rerunning docker build / run will result in a failure of the Vault initialization step in the "pygluu-compose" binary. It seems to skip the initialization call and tries to access vault while sealed. The vault_key_token.txt file is not produced at this point so there is no way to unseal the vault. Steps to reproduce: 1. create a local gluu directory on a system with docker-desktop installed/running and that contains this [dockerfile](https://pastebin.com/QW6hfGju), this [settings.py](https://pastebin.com/dWCPRL6F), and the pygluu-compose.pyz from release version 1.4.2. 2. in a console, run the following `docker build --no-cache --pull -t gluu-image . ; docker run -v "/var/run/docker.sock:/var/run/docker.sock:rw" --name gluu_container -t -d gluu-image` to produce the starting container. 3. Access the terminal from within the gluu_container instance and run `./pygluu-compose.pyz up` 4. Note that the "Initializing Vault with 1 recovery key and token" record exists in the log. 5. Abort the current command and call `./pygluu-compose.pyz down` to bring down the consul and vault containers 6. Close the terminal and delete the gluu-container container and all hanging images. 7. Repeat steps 2 and 3 and note that the "Initializing Vault" console entry does not occur and the `up` command fails due to vault being sealed. The only way to recover beyond this point is to preform a docker restart and start over. This is the console output after a fresh Docker Restart and rerunning the docker build/run commands: https://pastebin.com/PaL70RcA This is the console output without a Docker Restart but still wiping the containers/images and rerunning the docker build/run commands: https://pastebin.com/ZhS89pNs
Gluu 4.2.3 Docker 3.3
closed

Answers

By Aliaksandr Samuseu staff 22 Apr 2021 at 3:29 p.m. CDT

Copy
Aliaksandr Samuseu gravatar
Hey, Travis. We'll try to look into it, but so far I can't see mentions of us supporting Windows-based deployments in the docs. I may need to have a word with the responsible developer as well.

By Travis Salyer user 22 Apr 2021 at 3:34 p.m. CDT

Copy
Travis Salyer gravatar
Thanks for the response! I will say that we aren't using windows as the deployment environment. As mentioned we are using docker-desktop to setup these containers in the development environment which *should* be agnostic to the host environment since it uses the WSL2 engine which is Linux based

By Isman Firmansyah staff 26 Apr 2021 at 3:06 a.m. CDT

Copy
Isman Firmansyah gravatar
Hi Travis, I have tested your Dockerfile build. > Note that the "Initializing Vault with 1 recovery key and token" record exists in the log. This is where Vault is initialized and its key and token will be saved as `vault_key_token.txt` in the working directory `/usr/local/gluu`. Note that you need to create a volume by mounting your working directory into host to persist the files, otherwise they will be deleted when container is destroyed (hence you will lost your `vault_key_token.txt`). > Abort the current command and call ./pygluu-compose.pyz down to bring down the consul and vault containers > Close the terminal and delete the gluu-container container and all hanging images. > Repeat steps 2 and 3 and note that the "Initializing Vault" console entry does not occur and the up command fails due to vault being sealed. Initialization does not occur because it's already initialized and its state is persisted in Consul. As I have mentioned before, you cannot unseal Vault because you're unable to get the unseal key which is saved in `vault_key_token.txt` inside your previously deleted `gluu_container` container. One way to reset them all is by deleting Consul and Vault containers and their volumes that defined in `docker-compose.yml`. As you're using `/usr/local/gluu` as working directory on `pygluu-compose`, you may find your volumes at `/usr/local/gluu/volumes` inside the host (not the container). This unexpected path is probably because you're mounting host's `docker.sock` into container. I would recommend that you're running `pygluu-compose` directly in the host and follow our official docs.

By Travis Salyer user 26 Apr 2021 at 9:44 a.m. CDT

Copy
Travis Salyer gravatar
Hello Isman, Thanks for the response. > As I have mentioned before, you cannot unseal Vault because you're unable to get the unseal key which is saved in vault_key_token.txt inside your previously deleted gluu_container container. > As mentioned, I removed all containers and images as stated in the steps to reproduce, including the consul and vault instances. On the second attempt to build and run the dockerfile, the previous container instances of consul and vault have been wiped. This time when bringing up the containers by calling the `up` command, the `Initialize Vault` step is skipped and the vault_key_token file is not produced inside the parent container. This seems like a bug because these are freshly created containers and images and it is not running the same way depending on if Docker was just restarted or not. Its almost like there is something being cached in docker in between build and run attempts. We are exploring alternative methods of standing up a Gluu server instance that do not involve the pygluu-compose method now based on your documentation.

By Isman Firmansyah staff 26 Apr 2021 at 11:13 a.m. CDT

Copy
Isman Firmansyah gravatar
Hi Travis, > As mentioned, I removed all containers and images as stated in the steps to reproduce, including the consul and vault instances. On the second attempt to build and run the dockerfile, the previous container instances of consul and vault have been wiped. This time when bringing up the containers by calling the up command, the Initialize Vault step is skipped and the vault_key_token file is not produced inside the parent container. When you run `pygluu-compose up` command for the first time, Consul and Vault will be initialized and its state will be saved into volumes at `/usr/local/gluu/volumes`. If you re-deploy Consul and Vault, they will load data/states/etc from those volumes. This is why Vault is not re-initialized and there's no process to save key/token into `/usr/local/gluu/vault_key_token.txt` again. My suggestion: - remove existing Consul and Vault containers - remove existing Consul and Vault volumes (check `/usr/local/gluu/volumes/consul` and `/usr/local/gluu/vault` in host filesystem) - ensure your working directory (`/usr/local/gluu`) can be mounted correctly as volume in host filesystem (because you will need to persist files there, including `vault_key_token.txt`) - redeploy containers > This seems like a bug because these are freshly created containers and images and it is not running the same way depending on if Docker was just restarted or not. Its almost like there is something being cached in docker in between build and run attempts. Not sure if this is a bug because we heavily tested `pygluu-compose` on host directly. > We are exploring alternative methods of standing up a Gluu server instance that do not involve the pygluu-compose method now based on your documentation. You can use our Kubernetes-based installation.

Post an answer