By: Arthur Barrett user 12 Dec 2021 at 5:02 a.m. CST

24 Responses
Arthur Barrett gravatar
Can you recommend on how to mitigate CVE-2021-44228 on a Gluu server 4.2.3 (Ubuntu 18.04.5)? Or how to upgrade the installed log4j-*-2.14.0.jar to 2.15.0? Security notes below: CVE-2021-44228: Apache Log4j2 JNDI features do not protect against attacker controlled LDAP and other JNDI related endpoints. https://logging.apache.org/log4j/2.x/security.html Severity: Critical Base CVSS Score: 10.0 CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H Versions Affected: all versions from 2.0-beta9 to 2.14.1 affected files below: /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-over-slf4j-1.7.30.jar /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-1.2-api-2.13.3.jar.gluu-4.2.3-1~ /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-core-2.13.3.jar /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-core-2.13.3.jar.gluu-4.2.3-1~ /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-1.2-api-2.13.3.jar /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-api-2.13.3.jar /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-over-slf4j-1.7.30.jar.gluu-4.2.3-1~ /opt/shibboleth-idp/webapp/WEB-INF/lib/log4j-api-2.13.3.jar.gluu-4.2.3-1~ /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-4004568737330869594/webapp/WEB-INF/lib/log4j-plugin-fluency-1.3.2.jar /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-4004568737330869594/webapp/WEB-INF/lib/log4j-core-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-4004568737330869594/webapp/WEB-INF/lib/log4j-1.2-api-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-4004568737330869594/webapp/WEB-INF/lib/log4j-jul-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-4004568737330869594/webapp/WEB-INF/lib/log4j-slf4j-impl-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-4004568737330869594/webapp/WEB-INF/lib/log4j-api-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-4004568737330869594/webapp/WEB-INF/classes/log4j2.xml /opt/jetty-9.4/temp/jetty-localhost-8099-casa_war-_casa-any-15631844310170244673/webapp/WEB-INF/lib/log4j-1.2.8.jar /opt/jetty-9.4/temp/jetty-localhost-8099-casa_war-_casa-any-15631844310170244673/webapp/WEB-INF/lib/log4j-core-2.11.2.jar /opt/jetty-9.4/temp/jetty-localhost-8099-casa_war-_casa-any-15631844310170244673/webapp/WEB-INF/lib/log4j-1.2-api-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8099-casa_war-_casa-any-15631844310170244673/webapp/WEB-INF/lib/log4j-slf4j-impl-2.11.2.jar /opt/jetty-9.4/temp/jetty-localhost-8099-casa_war-_casa-any-15631844310170244673/webapp/WEB-INF/lib/log4j-api-2.11.2.jar /opt/jetty-9.4/temp/jetty-localhost-8099-casa_war-_casa-any-15631844310170244673/webapp/WEB-INF/classes/log4j2.xml /opt/jetty-9.4/temp/jetty-localhost-8086-idp_war-_idp-any-560876776726205793/webapp/WEB-INF/lib/log4j-core-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8086-idp_war-_idp-any-560876776726205793/webapp/WEB-INF/lib/log4j-1.2-api-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8085-oxauth-rp_war-_oxauth-rp-any-709950414183117628/webapp/WEB-INF/lib/log4j-core-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8085-oxauth-rp_war-_oxauth-rp-any-709950414183117628/webapp/WEB-INF/lib/log4j-1.2-api-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8085-oxauth-rp_war-_oxauth-rp-any-709950414183117628/webapp/WEB-INF/lib/log4j-jul-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8085-oxauth-rp_war-_oxauth-rp-any-709950414183117628/webapp/WEB-INF/lib/log4j-slf4j-impl-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8085-oxauth-rp_war-_oxauth-rp-any-709950414183117628/webapp/WEB-INF/lib/log4j-api-2.13.3.jar /opt/jetty-9.4/temp/jetty-localhost-8085-oxauth-rp_war-_oxauth-rp-any-709950414183117628/webapp/WEB-INF/classes/log4j2.xml /opt/jetty-9.4/temp/jetty-localhost-8087-scim_war-_scim-any-11684237553890558522/webapp/WEB-INF/lib/log4j-api-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8087-scim_war-_scim-any-11684237553890558522/webapp/WEB-INF/lib/log4j-plugin-fluency-1.3.2.jar /opt/jetty-9.4/temp/jetty-localhost-8087-scim_war-_scim-any-11684237553890558522/webapp/WEB-INF/lib/log4j-core-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8087-scim_war-_scim-any-11684237553890558522/webapp/WEB-INF/lib/log4j-slf4j-impl-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8087-scim_war-_scim-any-11684237553890558522/webapp/WEB-INF/lib/log4j-1.2-api-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8087-scim_war-_scim-any-11684237553890558522/webapp/WEB-INF/classes/log4j2.xml /opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-17274585625456637347/webapp/WEB-INF/classes/log4j2.xml /opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-3034897520083220938/webapp/WEB-INF/lib/log4j-api-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-3034897520083220938/webapp/WEB-INF/lib/log4j-plugin-fluency-1.3.2.jar /opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-3034897520083220938/webapp/WEB-INF/lib/log4j-core-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-3034897520083220938/webapp/WEB-INF/lib/log4j-slf4j-impl-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-3034897520083220938/webapp/WEB-INF/lib/log4j-1.2-api-2.14.0.jar /opt/jetty-9.4/temp/jetty-localhost-8082-identity_war-_identity-any-3034897520083220938/webapp/WEB-INF/classes/log4j2.xml /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j2-api.mod /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j-impl /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j-impl/resources/log4j.xml /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j2-impl /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j2-impl/resources/log4j2.xml /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j-impl.mod /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j2-slf4j.mod /opt/jetty-9.4/jetty-distribution-9.4.35.v20201120/modules/log4j2-impl.mod

By Zachary Doyle user 12 Dec 2021 at 12:57 p.m. CST

Zachary Doyle gravatar
Extremely interested in this as well.

By Hao Bin Kwan Account Admin 12 Dec 2021 at 8:47 p.m. CST

Hao Bin Kwan gravatar
Monitoring this as well, any update on this?

By Vijayabalaji Sekar user 12 Dec 2021 at 9:59 p.m. CST

Vijayabalaji Sekar gravatar
ANy update here ?

By Kee Wee Wong Account Admin 12 Dec 2021 at 10:40 p.m. CST

Kee Wee Wong gravatar
Hi everyone, While waiting for Gluu to remediate the vulnerability from their side, we are applying the `LOG4J_FORMAT_MSG_NO_LOOKUPS=true` environment variable to all our servers to mitigate the vulnerability. https://logging.apache.org/log4j/2.x/security.html Hope this helps anyone who are getting pressured to fix this vulnerability

By Hao Bin Kwan Account Admin 12 Dec 2021 at 10:57 p.m. CST

Hao Bin Kwan gravatar
> While waiting for Gluu to remediate the vulnerability from their side, we are applying the LOG4J_FORMAT_MSG_NO_LOOKUPS=true environment variable to all our servers to mitigate the vulnerability. > Thanks for the tip! I'll take a look at this.

By Arthur Barrett user 12 Dec 2021 at 11:39 p.m. CST

Arthur Barrett gravatar
> While waiting for Gluu to remediate the vulnerability from their side, we are applying the LOG4J_FORMAT_MSG_NO_LOOKUPS=true environment variable to all our servers to mitigate the vulnerability. > Where are you setting that? In /sbin/gluu-serverd ?

By Kee Wee Wong Account Admin 12 Dec 2021 at 11:44 p.m. CST

Kee Wee Wong gravatar
>Where are you setting that? In /sbin/gluu-serverd ? We are editing the `/etc/profile` in chroot and adding the env var into .yaml files for kubernetes

By Praveen Srinivasan user 13 Dec 2021 at 6:36 a.m. CST

Praveen Srinivasan gravatar
How can we set the env in docker version? Is it fine if we set as below in stack file (oxauth.yml, oxtrust.yml, etc,..) ? Also, what are all the services we need to have this env variable? As I saw from the log, shibboleth, oxauth, scim, identity we need to set. How about other services like, consul, registrator, opendj? ``` environment: - LOG4J_FORMAT_MSG_NO_LOOKUPS=true ```

By Arthur Barrett user 13 Dec 2021 at 9:49 p.m. CST

Arthur Barrett gravatar
We've applied mitigation using systemd: From within the chroot jail: ``` systemctl edit oxd-server systemctl edit casa systemctl edit idp systemctl edit scim systemctl edit oxauth-rp systemctl edit oxauth systemctl edit opendj systemctl edit identity systemctl edit passport ``` The text of the mitigation is: ``` [Service] Environment=LOG4J_FORMAT_MSG_NO_LOOKUPS="true" ``` I can test any process running ‘java’ has the environment variable set with: ``` cat /proc/2556/environ | tr '\0' '\n' | grep LOG4J_FORMAT_MSG_NO_LOOKUPS LOG4J_FORMAT_MSG_NO_LOOKUPS=true ``` We couldn't verify the environment variable was set in the java processes using any of the other techniques mentioned by the community. Also I found [this](https://stackoverflow.com/questions/70315727/where-to-put-formatmsgnolookups-in-log4j-xml-config-file) helpful, eg: ``` lsof | grep log4j ``` Feedback welcome...

By Arthur Barrett user 15 Dec 2021 at 5:13 p.m. CST

Arthur Barrett gravatar
I found some notes in github for [casa](https://github.com/GluuFederation/casa/pull/173) and [oxAuth](https://github.com/GluuFederation/oxAuth/pull/1605) (there are actually several more for oxAuth). I can't find any statement anywhere for customers though, particularly regarding mitigations.

By Shaun Walker named 16 Dec 2021 at 5:41 p.m. CST

Shaun Walker gravatar
From our testing, adding ***LOG4JFORMATMSGNOLOOKUPS=true*** has not worked, and we can still replicate jndi lookup requests We are looking at implementing **_-Dlog4j2.formatMsgNoLookups=true_** across the services as that appears to be working to stop the lookup requests Is there evidence to suggest Gluu is vulneraable to CVE-2021-45046 ?(https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046)

By Matt Andersen named 16 Dec 2021 at 5:43 p.m. CST

Matt Andersen gravatar
Just an FYI, the system property *log4j2.formatMsgNoLookups* and environment variable *LOG4JFORMATMSGNOLOOKUPS* mitigations are now being deemed as insufficient: https://logging.apache.org/log4j/2.x/security.html.

By Arthur Barrett user 16 Dec 2021 at 7:01 p.m. CST

Arthur Barrett gravatar
Shaun - can you share what scripts you are using to test? Matt - thanks, I hadn't spotted that. I'm going to use tonights downtime window to try repacking the oauth.war with a modified log4j-core-.jar with the JndiLookup class deleted as suggested by the current notice on the apache web site: ``` zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ``` I'll post here if I have a successful recipe I can share. But it would be great if anyone has any scripts for testing they can share.

By Shaun Walker named 16 Dec 2021 at 9:09 p.m. CST

Shaun Walker gravatar
Hi Arthur, On one of your hosts, run ***nc -nvlp 8888*** to open port 8888 then try and login with the username or user-agent as ***${jndi:ldap://localhost:8888/a}*** if log4j tries to call home you should see similar to ``` Ncat: Version 7.50 ( https://nmap.org/ncat ) Ncat: Listening on :::8888 Ncat: Listening on 0.0.0.0:8888 Ncat: Connection from 127.0.0.1. Ncat: Connection from 127.0.0.1:52238. 0 `▒ ``` This is what we noticed even after env change for LOG4J_FORMAT_MSG_NO_LOOKUPS Since adding -Dlog4j2.formatMsgNoLookups=true manually to each service in /opt/dist/scripts/ we are not seeing it call home Can verify that this is set via ``` ps axf | grep --color=always log4j2.formatMsgNoLookups | grep -v grep ```

By Arthur Barrett user 16 Dec 2021 at 9:42 p.m. CST

Arthur Barrett gravatar
Shaun: Thanks! I'm not seeing that result here. So I guess the variable is working for us. I tried testing [using this tool](https://github.com/fullhunt/log4j-scan), and it shows that our install is not vulnerable. I also tried changing the python script to use the payload you provided, and I also got nothing. I'd love to try turning the changes off just to test, but I suspect that's a bad idea. I've successfully unpacked and repacked the war to remove JndiLookup.class I'll post instructions for that soon.

By Shaun Walker named 16 Dec 2021 at 9:45 p.m. CST

Shaun Walker gravatar
Very good. I suspect we were seeing mixed results as we have two different versions running 4.2.3 seemed fine with the env, but 4.2.1 did not, even though the log4j libraries seemed pretty simillar. Once we see a new release from Gluu we will roll these up to latest

By Praveen Srinivasan user 16 Dec 2021 at 9:51 p.m. CST

Praveen Srinivasan gravatar
Hello Everyone, For the docker image, I've removed JndiLookup class by using below commands in Dockerfile ``` FROM oxauth-image RUN apk add zip RUN zip -q -d /opt/gluu/jetty/oxauth/webapps/oxauth/WEB-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ``` After building docker image from above docker file, I can be able to deploy without any issue. And when I run the command inside container, I am getting the below which means that the JndiLookup class is removed ``` bash-5.0# zip -q -d /opt/gluu/jetty/oxauth/webapps/oxauth/WEB-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class zip error: Nothing to do! (/opt/gluu/jetty/oxauth/webapps/oxauth/WEB-INF/lib/log4j-core-2.13.3.jar) ``` ### Adding update: Below are the commands to remove JndiLookup.class for other services **Oxauth** ``` zip -q -d /opt/gluu/jetty/oxauth/webapps/oxauth/WEB-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ``` **Oxshiboleth** ``` zip -q -d /opt/gluu/jetty/idp/webapps/idp/WEB-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ``` **Oxtrust** ``` zip -q -d /opt/gluu/jetty/identity/webapps/identity/WEB-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ``` **Scim** ``` zip -q -d /opt/gluu/jetty/scim/webapps/scim/WEB-INF/lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class ```

By Arthur Barrett user 16 Dec 2021 at 11:39 p.m. CST

Arthur Barrett gravatar
**Ubuntu 18.04 guide to patching Gluu 4.2.3 log4j ** Here is the process I have used. First I have to find all the log4j-core libraries in use - as root (not in chroot jail): ``` lsof 2> /dev/null | grep log4j-core >/tmp/log4j.txt ``` I then wanted to remove the first 92 character of each line, and only show unique lines - so I used: ``` sed 's/^.\{,92\}//' /tmp/log4j.txt | sort | uniq ``` So this list of impacted war’s and the log4j version in my installation is: * oxauth.war log4j-core-2.13.3.jar * identity.war log4j-core-2.14.0.jar * oxauth_rp.war log4j-core-2.13.3.jar * idp.war log4j-core-2.13.3.jar * scim.war log4j-core-2.14.0.jar * casa.war log4j-core-2.11.2.jar So now the environment variable is not deemed sufficient to mitigate the vulnerability, so let me try and repack the CASA war (in chroot jail): ``` cd /tmp mkdir casa-log4j cd /tmp/casa-log4j jar -xf /opt/gluu/jetty/casa/webapps/casa.war WEB-INF/lib/log4j-core-2.11.2.jar zip -q -d WEB-INF/lib/log4j-core-2.11.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class jar -uf /opt/gluu/jetty/casa/webapps/casa.war WEB-INF/lib/log4j-core-2.11.2.jar systemctl restart casa ``` So now I just need to repeat that for each of the others: SCIM: ``` cd /tmp mkdir scim-log4j cd /tmp/scim-log4j jar -xf /opt/gluu/jetty/scim/webapps/scim.war WEB-INF/lib/log4j-core-2.14.0.jar zip -q -d WEB-INF/lib/log4j-core-2.14.0.jar org/apache/logging/log4j/core/lookup/JndiLookup.class jar -uf /opt/gluu/jetty/scim/webapps/scim.war WEB-INF/lib/log4j-core-2.14.0.jar systemctl restart scim ``` IDENTITY: ``` cd /tmp mkdir identity-log4j cd /tmp/identity-log4j jar -xf /opt/gluu/jetty/identity/webapps/identity.war WEB-INF/lib/log4j-core-2.14.0.jar zip -q -d WEB-INF/lib/log4j-core-2.14.0.jar org/apache/logging/log4j/core/lookup/JndiLookup.class jar -uf /opt/gluu/jetty/identity/webapps/identity.war WEB-INF/lib/log4j-core-2.14.0.jar systemctl restart identity ``` OXAUTH: ``` cd /tmp mkdir oxauth-log4j cd /tmp/oxauth-log4j jar -xf /opt/gluu/jetty/oxauth/webapps/oxauth.war WEB-INF/lib/log4j-core-2.13.3.jar zip -q -d WEB-INF/lib/log4j-core-2.13.3.jar org/apache/logging/log4j/core/lookup/JndiLookup.class jar -uf /opt/gluu/jetty/oxauth/webapps/oxauth.war WEB-INF/lib/log4j-core-2.13.3.jar systemctl restart oxauth ``` OXAUTH_RP: ``` cd /tmp mkdir oxauth_rp-log4j cd /tmp/oxauth_rp-log4j jar -xf /opt/gluu/jetty/oxauth-rp/webapps/oxauth-rp.war WEB-INF/lib/log4j-core-2.13.3.jar zip -q -d WEB-INF/lib/log4j-core-2.13.3.jar org/apache/logging/log4j/core/lookup/JndiLookup.class jar -uf /opt/gluu/jetty/oxauth-rp/webapps/oxauth-rp.war WEB-INF/lib/log4j-core-2.13.3.jar systemctl restart oxauth_rp ``` IDP: ``` cd /tmp mkdir idp-log4j cd /tmp/idp-log4j jar -xf /opt/gluu/jetty/idp/webapps/idp.war WEB-INF/lib/log4j-core-2.13.3.jar zip -q -d WEB-INF/lib/log4j-core-2.13.3.jar org/apache/logging/log4j/core/lookup/JndiLookup.class jar -uf /opt/gluu/jetty/idp/webapps/idp.war WEB-INF/lib/log4j-core-2.13.3.jar systemctl restart idp ``` So the big test is to do a reboot and check that the log4j that gets unpacked to run the servers are now the modified versions. First I need to find what their ‘temporary’ locations are: ``` lsof 2> /dev/null | grep log4j-core >/tmp/log4j.txt ``` I then wanted to remove the first 92 character of each line, and only show unique lines - so I used: ``` sed 's/^.\{,92\}//' /tmp/log4j.txt | sort | uniq ``` TEST OXAUTH: ``` cd /tmp mkdir oxauth-log4j-test cd /tmp/oxauth-log4j-test zip -sf -d /opt/jetty-9.4/temp/jetty-localhost-8081-oxauth_war-_oxauth-any-3819058360256726813/webapp/WEB-INF/lib/log4j-core-2.13.3.jar org/apache/logging/log4j/core/lookup/JndiLookup.class zip warning: name not matched: org/apache/logging/log4j/core/lookup/JndiLookup.class Would Delete: Total 0 entries (0 bytes) ``` So there you have it. Still running an old release of log4j, but now with a mitigation for both CVE-2021-45046 and CVE-2021-44228

By Arthur Barrett user 20 Dec 2021 at 7:42 p.m. CST

Arthur Barrett gravatar
I see that there is finally a [statement on the web site](https://gluu.org/log4j-response/). Note: In the past few hours I've seen the text on that page change at least once, so I don't know what to make of that. Since the statement recommends that Community support users open a support ticket - I assume we've got that covered here in this support ticket, and we are waiting on the promised instructions on how to get early access to the patch. > The patch is a script that will update libraries for an in-place deployment. > Since that is exactly what we've been asking for, I think that will be a good resolution.

By Michael Schwartz Account Admin 20 Dec 2021 at 7:52 p.m. CST

Michael Schwartz gravatar
upgrade to 4.3.1 when it's available

By Arthur Barrett user 22 Dec 2021 at 9:53 p.m. CST

Arthur Barrett gravatar
Thanks for the reply Michael. Security notices are not something to mess about with. Please do the right thing and release the patch for all users. Besides being the right thing to do it will also create goodwill. Forcing users to upgrade to new versions when the implications of the hundreds of changes involved have not been considered, let alone tested, is a bad idea, and rightly an idea that the Gluu developers have consistently argued against. Finally, releasing this patch is exactly what your [web page](https://gluu.org/log4j-response/) says you are going to do (see screenshot): > The patch is a script that will update libraries for an in-place deployment. > > Community support users may open a support ticket, and the Gluu team will be in touch with instructions on how to get early access to the patch when available. > I look forward to you updating this support ticket with the patch soon.