By: Stephen Jewell user 12 Mar 2018 at 4:10 a.m. CDT

18 Responses
Stephen Jewell gravatar
On Tues 6th March at approx 19:25hrs (GMT) our existing OXD license failed to validate from each of the application servers. All application Servers being configured to use the Secure Information Assurance Ltd OXD license (License ID: 630e8b45-c6c1-4904-9f8f-90d9debe63c5). The license status on the portal showed as active and the billing details also appeared to be ok, last payments taken on 7th Feb 18 and the same card remains valid. The OXD log files on the application servers showing: [org.xdi.oxd.server.ServerLauncher] Shutdown oxd server. org.xdi.oxd.server.ShutdownException: Failed to validate license, shutdown server ... at org.xdi.oxd.server.license.LicenseService.start(LicenseService.java:69) at org.xdi.oxd.server.service.SocketService.listenSocket(SocketService.java:65) at org.xdi.oxd.server.ServerLauncher.startOxd(ServerLauncher.java:85) at org.xdi.oxd.server.ServerLauncher.start(ServerLauncher.java:62) at org.xdi.oxd.server.ServerLauncher.main(ServerLauncher.java:51) QUESTION: Does the OXD license have a validity period? It is approx 12 months since this license was generated. QUESTION: Can you confirm why this license failed to validate from 19:25 (GMT) on 6th March 2018. To resolve this problem, a new OXD account was created for UKFast Ltd (SecureIA is now a division of UKFast). This new license was applied to each of the application servers, which then all started correctly and generated their new OXD license. On Saturday 10th March at approx the New UKFast OXD License (License ID:ee4677ab-e999-4e61-90ae-39232b007372) failed to validate from 16:20 hrs (GMT). All of the appliation servers then started to fail their OXD license checks and each OXD service stopped - causing a second series of outages. On first checking the License portal - it appeared that this license was active and OK, but this may be inaccurate. When rechecked later on 10th March - the UKFast license was 'Deactivated' - without User intervention. The license was re-activated and each OXD service restarted, some requiring a number of restarts - some reconnected, some have still not re-licensed. On Monday morning 12th March at approx 07:00 (GMT) the OXD licensing failed again on a number of the application servers. On checking the GLUU OXD licensing portal - the UKFast OXD license was Deactivated again. This was set to active again and the OXD services restarted - again some recovered, some are still unstable. Looking through the log files of all of our servers - there a numerous instances of the OXD license check failing, but then OK after 2 or 3 tries. QUESTION: Why is the UKFast OXD license becoming inactive? QUESTION: Is there any current problem with the OXD License Server such as an intermittent service - If so is there any ongoing work to rectify this? QUESTION: Is there anyway to prevent service outages when the license check has failed - we need this SSO service to be resilient, and the external checks of the licenses is our most significant cause of service outages. Perhaps under a different licensing model? Note: We have de-activated the Secure Information Assurance Ltd OXD License now, as this is no longer in use.

By Mohib Zico staff 12 Mar 2018 at 4:12 a.m. CDT

Mohib Zico gravatar
We will get back to you as soon as we can with report from your oxD license account.

By William Lowe user 12 Mar 2018 at 9:47 a.m. CDT

William Lowe gravatar
Stephen, We had an issue with our licensing server, so sorry for the interruption. I just manually updated your UKFast account. Can you restart your oxd server? Thanks, Will

By William Lowe user 12 Mar 2018 at 10:07 a.m. CDT

William Lowe gravatar
The command is `/etc/init.d/oxd-server start` and in case that doesn't work, please also send your oxd-server.log file to me at will at gluu.org

By Stephen Jewell user 12 Mar 2018 at 10:34 a.m. CDT

Stephen Jewell gravatar
Hi Will, Thanks you for the prompt response to the ticket. We will restart on two of the application servers now and confirm the result. Has the issue with the licensing server been over a period of time as our first issues started on Tuesday 6th? Best regards, Steve

By William Lowe user 12 Mar 2018 at 10:37 a.m. CDT

William Lowe gravatar
Yes, it looks like it has. For some reason the license expiration date did not get updated after the most recent billing cycle. I manually updated both the SIA and UK Fast licenses in the backend. They both should be active again. We're trying to get to the root of the issue, but first want to make sure everyone's server's can be started as expected.

By Stephen Jewell user 12 Mar 2018 at 11:29 a.m. CDT

Stephen Jewell gravatar
Hi Will, We have restarted OXD on a couple of the Application Servers - which started first time. The oxd.log entries being: 2018-03-12 16:02:32,254 INFO [org.xdi.oxd.server.ServerLauncher] Starting... 2018-03-12 16:02:32,254 INFO [org.xdi.oxd.server.ServerLauncher] commit: 0eda70e831db6c16919ee7cbee4aa90a5fbeb6fa, branch: origin/version_3.0.1, build time:24.02.2017 @ 08:50:18 EST 2018-03-12 16:02:32,707 TRACE [org.xdi.oxd.server.service.ConfigurationService] Try to load configuration from system property: oxd.server.config, value: ../conf/oxd-conf.json 2018-03-12 16:02:32,832 TRACE [org.xdi.oxd.server.service.ConfigurationService] Configuration loaded successfully from system property: oxd.server.config. 2018-03-12 16:02:32,832 TRACE [org.xdi.oxd.server.service.ConfigurationService] Configuration: Configuration{port=8098, timeOutInSeconds=0, jettyPort=0, startJetty=false, localhostOnly=false, useClientAuthenticationForPat=true, useClientAuthenticationForAat=true, trustAllCerts=true, keyStorePath='', keyStorePassword='', licenseId='ee4677ab-e999-4e61-90ae-39232b007372', publicKey='EQ1aCA07CSEezql9GsrBmyRuGsaJH35ndySjFDM5cUp3lF0iL24HtaBDnsSqocDfIb7CBtGOoFznBc5AAlEK36SbeM6NNiQ6pqUF3EYIYnXyM35I46EmQzLdPSJWGIX2PpD/awvvlgnSaBJ99bwuTMV3PFIk6142Vi0aeCosxCaIKqxPbbDkLBF1AnCqoB5KqbtfAt+yUB+8eKHt3TBftQ2EB9tx9YPUtwdgIwxTObMMQw0bL82+pBW7qkyuckSqog2IL4O39QP0RfNjy168Y5WzAAPiiyNoXmPQrqcoyBOcuxCWiLqmQGvHy+aGAZKOgHjvBeW9O06GNQg7jUUUlJ4STec/sYhXvL3OULL3HSjLs5DPLuf+Q/vpiFUIDJsBMjXMqZ6/8870uE9yA8Oe8g==', publicPassword='nqtwIrXf63iW1NxRfYkv', licensePassword='lOfpIGrFGaGSH9jYTf8K', supportGoogleLogout='true', stateExpirationInMinutes='5', nonceExpirationInMinutes='5'} 2018-03-12 16:02:32,926 ERROR [org.xdi.oxd.server.service.ConfigurationService] String index out of range: -1 java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(Unknown Source) at org.xdi.oxd.server.service.ConfigurationService.getConfDirectoryPath(ConfigurationService.java:57) at org.xdi.oxd.server.service.ConfigurationService.getConfDirectoryFile(ConfigurationService.java:44) at org.xdi.oxd.server.service.SiteConfigurationService.load(SiteConfigurationService.java:61) at org.xdi.oxd.server.ServerLauncher.startOxd(ServerLauncher.java:84) at org.xdi.oxd.server.ServerLauncher.start(ServerLauncher.java:62) at org.xdi.oxd.server.ServerLauncher.main(ServerLauncher.java:51) 2018-03-12 16:02:32,926 INFO [org.xdi.oxd.server.service.ConfigurationService] Configuration directory: C:\oxd-server-3.0.1-distribution\bin\..\conf 2018-03-12 16:02:32,926 TRACE [org.xdi.oxd.server.service.SiteConfigurationService] Loading site file name: oxd-default-site-config.json 2018-03-12 16:02:32,973 DEBUG [org.xdi.oxd.server.license.LicenseFile] License file location: C:\oxd-server-3.0.1-distribution\bin\.oxd-license 2018-03-12 16:02:32,973 TRACE [org.xdi.oxd.server.license.LicenseFile] MAC ADDRESS set to : 00-50-56-AA-6E-97 2018-03-12 16:02:32,973 TRACE [org.xdi.oxd.server.license.LicenseService] Validating license ... 2018-03-12 16:02:32,973 DEBUG [org.xdi.oxd.server.license.LicenseFile] License file location: C:\oxd-server-3.0.1-distribution\bin\.oxd-license 2018-03-12 16:02:32,973 TRACE [org.xdi.oxd.server.license.LicenseFile] MAC ADDRESS set to : 00-50-56-AA-6E-97 2018-03-12 16:02:33,035 TRACE [org.xdi.oxd.server.service.HttpService] Created TRUST_ALL client. 2018-03-12 16:02:34,488 TRACE [org.xdi.oxd.server.license.LicenseService] License is validated successfully. 2018-03-12 16:02:34,488 TRACE [org.xdi.oxd.server.license.LicenseService] License data: LicenseMetadata{creationDate=Wed Mar 07 07:02:12 GMT 2018, licenseId=ee4677ab-e999-4e61-90ae-39232b007372, autoupdate=true, active=true, product=oxd, licenseName='UKFast', expirationDate=Fri Mar 06 07:02:12 GMT 2020, licenseCountLimit=9999, customerName=, emails=[]} 2018-03-12 16:02:34,488 TRACE [org.xdi.oxd.server.license.LicenseService] Validating license ... 2018-03-12 16:02:34,488 DEBUG [org.xdi.oxd.server.license.LicenseFile] License file location: C:\oxd-server-3.0.1-distribution\bin\.oxd-license 2018-03-12 16:02:34,488 TRACE [org.xdi.oxd.server.license.LicenseFile] MAC ADDRESS set to : 00-50-56-AA-6E-97 2018-03-12 16:02:34,488 TRACE [org.xdi.oxd.server.service.HttpService] Created TRUST_ALL client. 2018-03-12 16:02:34,879 TRACE [org.xdi.oxd.server.license.LicenseService] License is validated successfully. 2018-03-12 16:02:34,879 TRACE [org.xdi.oxd.server.license.LicenseService] License data: LicenseMetadata{creationDate=Wed Mar 07 07:02:12 GMT 2018, licenseId=ee4677ab-e999-4e61-90ae-39232b007372, autoupdate=true, active=true, product=oxd, licenseName='UKFast', expirationDate=Fri Mar 06 07:02:12 GMT 2020, licenseCountLimit=9999, customerName=, emails=[]} 2018-03-12 16:02:34,879 INFO [org.xdi.oxd.server.service.SocketService] Server socket is bound to port: 8098, with timeout: 0 seconds. Start listening for notifications.

By Stephen Jewell user 12 Mar 2018 at 11:33 a.m. CDT

Stephen Jewell gravatar
Hi Will, Before we do anything with the other application servers - is there any further work planned on the license server that may affect the service? I need to communicate a Service interruption to our User community in order to restart the other services. Steve

By Yuriy Zabrovarnyy staff 12 Mar 2018 at 11:35 a.m. CDT

Yuriy Zabrovarnyy gravatar
This log indicates that oxd-server is started successfully.

By Stephen Jewell user 12 Mar 2018 at 11:38 a.m. CDT

Stephen Jewell gravatar
As a related Question - are there any options for the OXD licensing so that it is not dependant on the validation checks, such as Enterprise licensing or different support models? THank you Steve

By William Lowe user 12 Mar 2018 at 11:38 a.m. CDT

William Lowe gravatar
Stephen, No further work is planned for the license server right now. We're investigating the cause of the issue, but your licenses have been manually updated and will not expire. I'll follow up when we find the root cause of the issue and with details about our resolution. But net-net, your license and server should again be stable. Sorry again for the inconvenience. Also, do you want to leave the call on the calendar for a few hours from now, or should we go ahead and cancel that now? Thanks, Will

By William Lowe user 12 Mar 2018 at 11:50 a.m. CDT

William Lowe gravatar
> As a related Question - are there any options for the OXD licensing so that it is not dependant on the validation checks, such as Enterprise licensing or different support models? Yes, we offer site licenses as an add-on to our [VIP support packages](https://gluu.org/pricing). If you want to keep the call on the calendar today, would be happy to discuss then. I know the timing of the existing call is late for you though, so perhaps it would be better to reschedule for tomorrow morning at 3pm your time (10am our time)? Thanks, Will

By Stephen Jewell user 12 Mar 2018 at 11:58 a.m. CDT

Stephen Jewell gravatar
Hi Will, Thank you for the series of answers - your assistance has been excellent. Could we please reschedule this afternoons call to tomorrow - 10:00 (15:00GMT) is good - particularly around the support packages and site license. Best regards, Steve

By William Lowe user 12 Mar 2018 at 12:03 p.m. CDT

William Lowe gravatar
Yes, rescheduled for 15:00 GMT. Thanks, Stephen. Will

By Stephen Jewell user 13 Mar 2018 at 12:21 p.m. CDT

Stephen Jewell gravatar
Good afternoon Will, We have just had two more OXD failures - 3 attempts to validate license then service stopped on two different application servers. When restarting the logs show license not valid. I changed the status of the license on the Gluu eCommerce Portal to Active - when we then restarted OXD it was successful. Everything is currently running again, but we are monitoring closely for the next few hours. We had not restarted the OXD services on these applications since Monday 12th. Just keeping you informed as it may help with your internal investigations.

By William Lowe user 13 Mar 2018 at 12:54 p.m. CDT

William Lowe gravatar
OK, everything looks right on the backend license portal. Via the ecom dashboard, you can still deactivate / activate your license. But the change we made ensures that no automatic changes will be incurred due to billing. i..e if you do it, it's OK. But the ecom will not do anything automatically until we have the issue resolved.

By Mohib Zico staff 21 Mar 2018 at 5:38 a.m. CDT

Mohib Zico gravatar
Hi Stephen, How is it going? Want to keep this ticket for anything else?

By Stephen Jewell user 21 Mar 2018 at 6:11 a.m. CDT

Stephen Jewell gravatar
No, thank you Mohib.

By Mohib Zico staff 21 Mar 2018 at 6:28 a.m. CDT

Mohib Zico gravatar
Thanks for quick response and update!