Special Characters in Cache Refresh Custom Script

By: Ivan Carrion user 15 Dec 2017 at 7:18 a.m. CST

14 Responses
Ivan Carrion gravatar
I'm trying to manipulate AD info mapped to Gluu before set them in fields, like UAC, Photo, displayName and so on... My problem is that we are from Spain and there's a lot of special characters (á,é,í,ó,ú,ñ,...) that function getValue from CustomAttribute doesn't work well with. i.e.: attrDisNam = user.getGluuCustomAttribute("displayName") #AD displayName field print attrDisNam type(attrDisNam) #trying to figure out the type returned if attrDisName is not None: disName = attrDisNam.getValue() #this function throws a Java Exception!!!! -------------- In /opt/gluu/jetty/identity/logs/oxtrust_script.log I get this info... 2017-12-15 13:32:16,389 INFO [Thread-10535] [org.xdi.service.PythonService$PythonLoggerOutputStream] (PythonService.java:208) - Attribute [name=displayname, values=[DAVID ORTU?O GOMEZ], metadata=null] 2017-12-15 13:32:16,389 INFO [Thread-10535] [org.xdi.service.PythonService$PythonLoggerOutputStream] (PythonService.java:208) - Nombre: 2017-12-15 13:32:16,389 ERROR [Thread-10535] [org.gluu.oxtrust.service.external.ExternalCacheRefreshService] (ExternalCacheRefreshService.java:43) - null org.python.core.PyException: null at org.python.core.Py.NameError(Py.java:284) ~[jython-2.7.0.jar:?] at org.python.core.PyFrame.getglobal(PyFrame.java:265) ~[jython-2.7.0.jar:?] at org.python.pycode._pyx18.updateUser$5(<iostream>:135) ~[?:?] at org.python.pycode._pyx18.call_function(<iostream>) ~[?:?] at org.python.core.PyTableCode.call(PyTableCode.java:167) ~[jython-2.7.0.jar:?] at org.python.core.PyBaseCode.call(PyBaseCode.java:307) ~[jython-2.7.0.jar:?] at org.python.core.PyBaseCode.call(PyBaseCode.java:198) ~[jython-2.7.0.jar:?] at org.python.core.PyFunction.__call__(PyFunction.java:482) ~[jython-2.7.0.jar:?] at org.python.core.PyMethod.instancemethod___call__(PyMethod.java:237) ~[jython-2.7.0.jar:?] at org.python.core.PyMethod.__call__(PyMethod.java:228) ~[jython-2.7.0.jar:?] at org.python.core.PyMethod.__call__(PyMethod.java:218) ~[jython-2.7.0.jar:?] at org.python.core.PyMethod.__call__(PyMethod.java:213) ~[jython-2.7.0.jar:?] at org.python.core.PyObject._jcallexc(PyObject.java:3626) ~[jython-2.7.0.jar:?] at org.python.core.PyObject._jcall(PyObject.java:3658) ~[jython-2.7.0.jar:?] at org.python.proxies.__builtin__$CacheRefresh$18.updateUser(Unknown Source) ~[?:?] at org.gluu.oxtrust.service.external.ExternalCacheRefreshService.executeExternalUpdateUserMethod(ExternalCacheRefreshService.java:41) [classes/:?] at org.gluu.oxtrust.service.external.ExternalCacheRefreshService.executeExternalUpdateUserMethods(ExternalCacheRefreshService.java:52) [classes/:?] at org.gluu.oxtrust.service.external.ExternalCacheRefreshService$Proxy$_$$_WeldClientProxy.executeExternalUpdateUserMethods(Unknown Source) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.updateTargetEntryViaCopy(CacheRefreshTimer.java:669) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.updateTargetEntriesViaCopy(CacheRefreshTimer.java:569) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.detectChangedEntries(CacheRefreshTimer.java:392) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.processImpl(CacheRefreshTimer.java:283) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.processInt(CacheRefreshTimer.java:194) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer$Proxy$_$$_WeldSubclass.processInt(Unknown Source) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer.process(CacheRefreshTimer.java:177) [classes/:?] at org.gluu.oxtrust.ldap.cache.service.CacheRefreshTimer$Proxy$_$$_WeldSubclass.process$$super(Unknown Source) [classes/:?] at sun.reflect.GeneratedMethodAccessor611.invoke(Unknown Source) ~[?:?] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_112] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_112] at org.jboss.weld.interceptor.proxy.TerminalAroundInvokeInvocationContext.proceedInternal(TerminalAroundInvokeInvocationContext.java:51) [weld-core-impl-3.0.0.Final.jar:3.0.0.Final] at org.jboss.weld.interceptor.proxy.AroundInvokeInvocationContext.proceed(AroundInvokeInvocationContext.java:78) [weld-core-impl-3.0.0.Final.jar:3.0.0.Final] at org.xdi.service.cdi.async.AsynchronousInterceptor$1.get(AsynchronousInterceptor.java:36) [oxcore-service-3.1.1.Final.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) [?:1.8.0_112] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112] ------------ As you can view, value shows as "DAVID ORTU?O GOMEZ", but when we see it in Gluu Admin UI, it shows it correct, as "DAVID ORTUÑO GOMEZ". I need to know if there's a way to work with that special characters on the right way. Thanks in advance!
CentOS 7.0
closed

Answers

By Yuriy Movchan staff 15 Dec 2017 at 9:45 a.m. CST

Copy
Yuriy Movchan gravatar
I'm not sure that this is Unicode issue. The method which you show returns java String. I think Jython uses java String internally too. Can you check if your customized script loaded without errors? Are there message about it load in same log file? Also, you can add print statement prior line which you think it might be wrong and check log? Can you attach your customized method or script to this ticket?

By Ivan Carrion user 18 Dec 2017 at 3:13 a.m. CST

Copy
Ivan Carrion gravatar
Thanks for the answer. The code that throws the exception is: disName = attrDisNam.getValue() When I print the attribute returned by user.getGluuCustomAttribute("displayName"), it shows: Attribute [name=displayname, values=[DAVID ORTU?O GOMEZ], metadata=null] My script is loaded without errors. In Gluu UI it shows this info in the correct way "DAVID ORTUÑO GOMEZ" Is there a function to manipulate the info before user.getGluuCustomAttribute or attr.getValue()?

By Aliaksandr Samuseu staff 18 Dec 2017 at 11:21 a.m. CST

Copy
Aliaksandr Samuseu gravatar
Hi, Ivan. >When I print the attribute returned by user.getGluuCustomAttribute("displayName"), it shows: Attribute [name=displayname, values=[DAVID ORTU?O GOMEZ], metadata=null] Where is it shown like this? Is it only shown in logs like that (as you mention it's shown correctly in web UI)? Or you also see incorrect values showing up in some other places (like, after some RP/SP receives them from Gluu etc)?

By Ivan Carrion user 18 Dec 2017 at 11:43 a.m. CST

Copy
Ivan Carrion gravatar
When I put this piece of code in my cache refresh script, I see errors (a high quantity) in Gluu UI during cache refresh execution. So, I connect through ssh and see opt/gluu/jetty/identity/logs/oxtrust_script.log. It shows prints like I put early (Attribute [name=displayname, values=[DAVID ORTU?O GOMEZ], all the accented names and those that contains ñ-letter) and if I want to get the value from the Attribute, it throws an exception like first post (this is the reason of all the errors). I tried putting in python script a first-coding-line "# -*- coding:utf-8 -*-" and "# -*- coding: cp1252 -*-", but it doesn't work. The reason to put that print code and register it on oxtrust_script.log is that I need to manipulate this info to put it on the same or another field, and when I try to get his value, it raises the error mentioned.

By Aliaksandr Samuseu staff 18 Dec 2017 at 12:31 p.m. CST

Copy
Aliaksandr Samuseu gravatar
>When I put this piece of code in my cache refresh script, I see errors (a high quantity) in Gluu UI during cache refresh execution. Could you please elaborate on this? Do you see the "errors during last execution" CR counter increasing? Or something else perhaps? I'm not aware of any error messages displayed in web UI when CR fails. Are you sure those errors you observe indeed has something to do with those characters in your attribute values? Could you perhaps do a quick experiment? 1. If you have a lot of user entries in your backend, limit the scope CR uses by changing "Base DN" to some OU containing few users. Wait for the next CR pool so it will other users from Gluu's LDAP 2. Make sure the users now still visible to CR do not contain any special characters. 3. See whether your issue still will persist.

By Aliaksandr Samuseu staff 18 Dec 2017 at 12:37 p.m. CST

Copy
Aliaksandr Samuseu gravatar
Have you also checked `oxtrust_cache_refresh.log`? Any clues are there?

By Ivan Carrion user 19 Dec 2017 at 1:42 a.m. CST

Copy
Ivan Carrion gravatar
I'll give you more data: - We have 2705 users synchronized from Microsoft Active Directory. - When we don't manipulate in our custom cache refresh script info like displayName or memberOf (attributes that contain ñ-letters and accents), all the syncs works. - When we try to manipulate info in displayName or memberOf, it throws 252 errors. CR counter increases from 0 to 252. - To check those errors, as oxtrust_cache_refresh.log doesn't show enough info, we print/log in our cache refresh script log all the process. We see all the errors come from displayName and memberOf containing ñ-letter or accents, at the moment we call for getValue() function of GluuCustomAttribute. Sure, the errors come from that function. - We print on cache refresh script log every line we write... the java exception is thrown from the function early mentioned. Those exceptions can't be catched in "try-except" with Python. So, it seems it throws directly from Java without any way to mitigate it from our script. - oxtrust_cache_refresh.log only shows that had been errors on users with messages like this: 2017-12-18 11:39:55,463 ERROR [Thread-1165] [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (CacheRefreshTimer.java:671) - Failed to execute Cache Refresh scripts for person '@!1F14.B01A.2B94.44F3!0001!EDBE.3490!0000!72D9.A615' - At the end of the sync process, oxtrust_cache_refresh.log shows: 2017-12-18 11:39:55,694 INFO [Thread-1165] [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (CacheRefreshTimer.java:395) - Updated '2453' entries 2017-12-18 11:39:55,695 INFO [Thread-1165] [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (CacheRefreshTimer.java:397) - Failed to update '252' entries 2017-12-18 11:39:55,712 INFO [Thread-1165] [gluu.oxtrust.ldap.cache.service.CacheRefreshTimer] (CacheRefreshTimer.java:437) - Removed '0' persons from target server - I'm pretty sure there's another way to receive info from LDAP as is used in Gluu UI, because that "spanish info" is well-represented without errors on it. - Aliaksandr, we have spent a whole week only working in sync process. Now, I could teach Gluu sync process hehehehe (have you certificate path? hehehehe). I mean, all the steps you have wrote, I've already made it. I filtered with multiple scopes (Base DNs) to sync little groups. Always when we don't manipulate "spanish-letters-people", it works. Sure, the big problem is accents and ñ-letters with getValue() function.

By Ivan Carrion user 19 Dec 2017 at 1:49 a.m. CST

Copy
Ivan Carrion gravatar
Please, is there any way to simulate all I'm writing in your labs? You only need to write an accent on a user name and try to get the value in a custom script.

By Aliaksandr Samuseu staff 19 Dec 2017 at 1:10 p.m. CST

Copy
Aliaksandr Samuseu gravatar
Hi, Ivan. Please review a code fragment below I used in attempt to reproduce this issue: ``` for attribute in attributes: attrName = attribute.getName() if (("givenname" == StringHelper.toLowerCase(attrName)) and StringHelper.isNotEmpty(attribute.getValue())): attribute.setValue(StringHelper.removeMultipleSpaces(attribute.getValue()) + " (updated)") try: if (("displayname" == StringHelper.toLowerCase(attrName)) and StringHelper.isNotEmpty(attribute.getValue())): #print "displayName: ", attribute.getValue() attribute.setValue(StringHelper.removeMultipleSpaces(attribute.getValue()) + " (updated2)") except Exception, ex: print "Catching point #1" print "Unexpected error:", ex raise except: print "Catching point #2" print "Unexpected error:", sys.exc_info()[0] raise ``` As you can see, it uses call to `attribute.getValue()` method for "displayName" - my test backend (AD) user has "DAVID ORTUÑO GOMEZ" value assigned to this one. This code works without issues and actually modify "displayName" of the user before adding it to Gluu's LDAP. Also note the commented out string `#print "displayName: ", attribute.getValue()` If uncommented, this one starts to throw exceptions of next type: ``` 2017-12-19 14:00:33,069 INFO [Thread-440] [org.xdi.service.PythonService$PythonLoggerOutputStream] (PythonService.java:208) - 'ascii' codec can't encode character u'\xd1' in position 10: ordinal not in range(128) 2017-12-19 14:00:33,069 ERROR [Thread-440] [org.gluu.oxtrust.service.external.ExternalCacheRefreshService] (ExternalCacheRefreshService.java:43) - null org.python.core.PyException: null at org.python.core.PyException.doRaise(PyException.java:198) ~[jython-2.7.0.jar:?] at org.python.core.Py.makeException(Py.java:1337) ~[jython-2.7.0.jar:?] ... ``` So it only seems to create issues when you attempt to feed it to Python's method for printing to logs. It doesn't seem like it's related to Gluu at the moment. Please provide some code fragments and an explanation why it should be considered as such, if you think otherwise. Please also note that usually we don't provide support for script writing to community users, so if it's a Python/Jython-related question, please try to find some help at their dedicated resources.

By Ivan Carrion user 20 Dec 2017 at 2:37 a.m. CST

Copy
Ivan Carrion gravatar
Hi Aliaksandr. I was making changes in my code after read you. Finally, a working-and-clean code is: ``` attrDisNam = user.getGluuCustomAttribute("displayname") print attrDisNam #shows all attribute values in log. Working. if attrDisNam is not None: attrDisNam.setValue(StringHelper.removeMultipleSpaces(attrDisNam.getValue()) + " updated") ``` This code works. So, there's no error in getValue() function. BUT, I must use StringHelper.removeMultipleSpaces to get to work. If not, it throws the same exception. I need to work with AD groups string manipulation. I'll use that function to see if it works with groups too. Thanks a lot!

By Aliaksandr Samuseu staff 20 Dec 2017 at 10:57 a.m. CST

Copy
Aliaksandr Samuseu gravatar
That doesn't seem to me like issue with Gluu's code, more like some Python/Java compatibility/type casting issue. Good, I believe we can close it, after Yuriy will review.

By Yuriy Movchan staff 20 Dec 2017 at 1:18 p.m. CST

Copy
Yuriy Movchan gravatar
I think there is problem with logging only. Aleks, can you try this build http://ox.gluu.org/maven/org/xdi/oxauth-server/3.2.0-SNAPSHOT/oxauth-server-3.2.0-SNAPSHOT.war ?

By Aliaksandr Samuseu staff 21 Dec 2017 at 12:10 p.m. CST

Copy
Aliaksandr Samuseu gravatar
I can confirm that I don't receive the same exception as before when I try to directly `print` some string with accented characters to logs any more with 3.2.0 oxTrust. I also must note that I don't see exceptions even if I don't pass a string like this through `StringHelper.removeMultipleSpaces()`. I think we can conclude there is little can (and need to) be done from our side in this regard. We can close the ticket now, I suppose.

By Ivan Carrion user 21 Dec 2017 at 12:32 p.m. CST

Copy
Ivan Carrion gravatar
Ok, thanks both for the answers! Next I must do is clustering,... if I success it with HAProxy and Keepalived, I'll write you :) I close this ticket! thanks a lot!

Post an answer