Built-in key rotation does not seem to be working

By: Ryan Nicholls named 23 Nov 2021 at 7:01 p.m. CST

11 Responses
Ryan Nicholls gravatar
Im having trouble getting keys to appear in the oxauth/restv1/jwks endpoint. Ive followed the steps over at https://gluu.org/docs/gluu-server/4.3/admin-guide/certificate/#generating-cryptographic-keys to generate the new key, and placed it in /etc/certs/oxauth-keys.jks but the key has not shown up at the jwks endpoint. The json files described at https://gluu.org/docs/gluu-server/4.3/admin-guide/certificate/ dont seem to exist in my installation, so im a bit lost as to how to get the key to show up. Im also unsure how to get the key rotation to work in oxauth, i have keyRegenerationEnabled set to true but none of the keys seem to ever be regenerating.
Gluu 4.3 Rhel 8.4
closed

Answers

By Michael Schwartz Account Admin 24 Nov 2021 at 10:15 p.m. CST

Copy
Michael Schwartz gravatar
They have deployed one server, `keyRegenerationEnabled = true`, but still the keys are not rotating. Also, as CITEC is publishing an additional encryption key, will they have to disable internal key rotation, and implement a different process?

By Milton Ch. staff 25 Nov 2021 at 9:19 a.m. CST

Copy
Milton Ch. gravatar
What is the value for `keyRegenerationInterval`? In the middle you should see these logs: `Updating JWKS keys ...` `Updated JWKS successfully` This process is handled via Job executor and at startup time of the application you should see also these logs (just to verify that the executor is running) `Initializing Key Generator Timer` `Initialized Key Generator Timer` Could you verify if you had some error/exception in the middle?

By Ryan Nicholls named 25 Nov 2021 at 4:34 p.m. CST

Copy
Ryan Nicholls gravatar
My keyRegenerationInterval is currently set to 48. I can see both Initializing Key Generator Timer and Initialized Key Generator Timer appear a number of times in my logs with no errors in the middle. Updating JWKS keys appears in my logs fine until the 31st of October, after that point it never appears in the logs. They keys i have are all expired November 2.

By Ryan Nicholls named 25 Nov 2021 at 5:12 p.m. CST

Copy
Ryan Nicholls gravatar
I Updated my keyRegenerationInterval to 1 and the key has rotated. Is the rotation interval tied to uptime? If the server is restarted, does that mean that the server will wait until the regeneration interval elapses from the restart time even if the keys were generated well before?

By Milton Ch. staff 26 Nov 2021 at 6:58 a.m. CST

Copy
Milton Ch. gravatar
Yes, it is, but in case your keys have rotated, then it's tied to the last rotation. 48 hours is a good interval, maybe there was another issue in the middle after log `Updating JWKS keys ...`, perhaps you have an error/exception if you don't see `Updated JWKS successfully` log at the end. Anyway, `keyRegenerationInterval` is only an interval to review all keys, in case a key hasn't expired yet, it would get reviewed during next execution. Good to see that your keys have rotated once you updated it, but still wonder why it failed before, let me know if you want to review it further.

By Ryan Nicholls named 01 Dec 2021 at 11:58 p.m. CST

Copy
Ryan Nicholls gravatar
Im wondering if there is possibly a bug in the keyRegenerationInterval. If the gluu server is restarted right before the keyRegenerationInterval is triggered, it appears to reset when the next check of they keys expiry will be. This can lead to the expired keys that I was experiencing (I.e If the interval is 48 hours but the server is restarted every 36 hours the keys may never rotate)

By Milton Ch. staff 02 Dec 2021 at 7:15 a.m. CST

Copy
Milton Ch. gravatar
Actually, how `keyRegenerationInternal` works, when app starts, after the whole app is initialized, timers start, so unless there was an error/exception in the middle, this should be executed after that smoothly. Another properly that might affect this process is `keyAlgsAllowedForGeneration`, here you should have listed all algorithms allowed to regenerate. Anyway, perhaps you could share your logs in order to review regeneration process and try to find root reason.

By Michael Schwartz Account Admin 02 Dec 2021 at 9:52 a.m. CST

Copy
Michael Schwartz gravatar
I created a bug for this in Janssen Auth Server: [Corner Case: Key Rotation timer may trigger regeneration after keys expire #359](https://github.com/JanssenProject/jans-auth-server/issues/359)

By Ryan Nicholls named 05 Dec 2021 at 4:45 p.m. CST

Copy
Ryan Nicholls gravatar
There are no errors in the logs at all. But say if the server has the keyRegenerationInternal set to 48 hours, then if the server is restarted every 24 hours, the timer will reset each time the server is restarted then the 48 hour duration will never trigger. My logs have alot of Initializing Key Generator Timer and Initialized Key Generator Timer when i had to restart my computer a bunch of times for unrelated work. Updating JWKS keys ... never appears in my logs until i set the regeneration interval to 1 hour and left the server up long enough.

By Milton Ch. staff 06 Dec 2021 at 9:01 a.m. CST

Copy
Milton Ch. gravatar
I see, well, after talking about it internally, I think this is not an issue, even though we renew all keys just after server is started, might be problematic in case an issue, bad configuration or simple server stopped suddenly. This is a very special case restarting AS every day on prod, currently we have many timers running with different intervals and they are using memory to keep data about next and last execution, in your case I'd suggest to handle it using an appropriate interval, might be a value lower than 24 hours. Let me know what do you think, thanks!

By Michael Schwartz Account Admin 06 Dec 2021 at 1:11 p.m. CST

Copy
Michael Schwartz gravatar
Definitely won't be fixed in 4.x for a while. Perhaps we should close this issue and track it in the future in [Github](https://github.com/JanssenProject/jans-auth-server/issues/359)

Post an answer