By: Simon Devlin user 20 Apr 2017 at 5:26 a.m. CDT

11 Responses
Simon Devlin gravatar
Hi folks, I'm trying to import users from an XLS into the gluu LDAP but the import is failing with duplicate users. I can't find anywhere which users are actually causing the issue? There were a few duplicates in the original spreadsheet but these have been removed using Find Duplicates and yet the problem still persists. Is it possible to turn extra debugging on to diagnose the import process? Thanks

By Mohib Zico Account Admin 20 Apr 2017 at 5:37 a.m. CDT

Mohib Zico gravatar
Hi, >> I can't find anywhere which users are actually causing the issue? oxtrust.log should show the reason. But we will highly suggest you not to use this feature because it's too slow. You should use Cache Refresh to pull your users from backend server or SCIM to push from your side.

By Simon Devlin user 20 Apr 2017 at 2:55 p.m. CDT

Simon Devlin gravatar
Are you certain about oxtrust.log? I checked before posting hoping for a reference to the duplicate record and have since re-checked and there does not appear to be anything of relevance (literally no errors on or around the time of import). I don't have a master server to cache refresh from, the OpenDJ instance embedded in Gluu is the source. If I could find a straightforward SCIM client to use I'd be happy to, but at least from the ones I've found I'd suggest C, for complex would be a better first letter :-/ Thanks

By Mohib Zico Account Admin 26 Apr 2017 at 2:17 p.m. CDT

Mohib Zico gravatar
>> Are you certain about oxtrust.log? I checked before posting hoping for a reference to the duplicate record and have since re-checked and there does not appear to be anything of relevance (literally no errors on or around the time of import). Yes. Here is a sample output from my log: ``` 2017-04-26 18:22:25,753 INFO [qtp274064559-12] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:405) - Found 4 persons in input Excel file 2017-04-26 18:22:25,760 INFO [qtp274064559-12] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:282) - Prepared 4 persons for creation 2017-04-26 18:22:36,726 INFO [qtp274064559-14] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:137) - Attempting to add 4 persons 2017-04-26 18:22:36,919 INFO [qtp274064559-14] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:158) - All 4 persons added successfully ```

By Simon Devlin user 11 May 2017 at 4:06 a.m. CDT

Simon Devlin gravatar
Sorry it's taken a little while to reply. All I'm getting is ``` INFO | jvm 1 | 2017/05/11 09:58:17 | Warning: External sheet record for Biff 7 not supported INFO | jvm 1 | 2017/05/11 09:58:17 | Warning: Usage of a local non-builtin name INFO | jvm 1 | 2017/05/11 09:58:20 | 2017-05-11 09:58:20,753 INFO [org.gluu.oxtrust.action.PersonImportAction] Found 460 persons in input Excel file INFO | jvm 1 | 2017/05/11 09:58:20 | May 11, 2017 9:58:20 AM com.sun.faces.renderkit.html_basic.MessageRenderer encodeEnd INFO | jvm 1 | 2017/05/11 09:58:20 | WARNING: 'for' attribute cannot be null ``` Then nothing. Page stays on validate button with no option to proceed. If I try to re-validate I get the similar uid message.

By Simon Devlin user 11 May 2017 at 4:20 a.m. CDT

Simon Devlin gravatar
So I cut down the file to 10 users and that does allow me to proceed. ``` INFO | jvm 1 | 2017/05/11 10:16:44 | 2017-05-11 10:16:44,026 INFO [org.gluu.oxtrust.action.PersonImportAction] Found 10 persons in input Excel file INFO | jvm 1 | 2017/05/11 10:16:44 | 2017-05-11 10:16:44,061 INFO [org.gluu.oxtrust.action.PersonImportAction] Prepared 10 persons for creation ``` Any suggestions on how to diagnose what the issue is? Are there any restrictions on field length for the columns? Thanks

By Mohib Zico Account Admin 11 May 2017 at 5 a.m. CDT

Mohib Zico gravatar
Hi Simon, >> External sheet record for Biff 7 not supported I think I would concentrate on this 'Biff 7' thing. What's that? Some name from your spreadsheet?

By Simon Devlin user 11 May 2017 at 6:41 a.m. CDT

Simon Devlin gravatar
Biff is the file format of the xls file, but that said, it's a simple .xls with only a single tab in it. [http://stackoverflow.com/questions/514847/how-to-identify-whether-an-excel-file-conforms-to-excel-95-or-excel-97-specifica](http://stackoverflow.com/questions/514847/how-to-identify-whether-an-excel-file-conforms-to-excel-95-or-excel-97-specifica) 7 means that the file conforms to Excel 95, rather than the much newer xlsl format. With 10 rows the import validates correctly so I doubt it is a version issue (although that's clearly a guess) I did manage to turn on debug, and there's a debug message relating to /opt/tomcat/conf//gluuImportPerson.properties ``` /opt/tomcat/logs/wrapper.log.1:INFO | jvm 1 | 2017/05/11 11:09:57 | 2017-05-11 11:09:57,691 DEBUG [org.xdi.util.properties.FileConfiguration] Loading '/opt/tomcat/conf//gluuImportPerson.properties' configuration file from resources /opt/tomcat/logs/wrapper.log.1:INFO | jvm 1 | 2017/05/11 11:09:57 | 2017-05-11 11:09:57,719 DEBUG [org.apache.commons.configuration.PropertiesConfiguration] FileName set to /opt/tomcat/conf//gluuImportPerson.properties /opt/tomcat/logs/wrapper.log.1:INFO | jvm 1 | 2017/05/11 11:09:57 | 2017-05-11 11:09:57,719 DEBUG [org.apache.commons.configuration.ConfigurationUtils] ConfigurationUtils.locate(): base is null, name is /opt/tomcat/conf//gluuImportPerson.properties /opt/tomcat/logs/wrapper.log.1:INFO | jvm 1 | 2017/05/11 11:09:57 | 2017-05-11 11:09:57,720 DEBUG [org.apache.commons.configuration.DefaultFileSystem] Could not locate file /opt/tomcat/conf//gluuImportPerson.properties at null: no protocol: /opt/tomcat/conf//gluuImportPerson.properties /opt/tomcat/logs/wrapper.log.1:INFO | jvm 1 | 2017/05/11 11:09:57 | 2017-05-11 11:09:57,721 DEBUG [org.xdi.util.properties.FileConfiguration] Failed to load '/opt/tomcat/conf//gluuImportPerson.properties' configuration file from resources /opt/tomcat/logs/wrapper.log.1:INFO | jvm 1 | 2017/05/11 11:10:17 | 2017-05-11 11:10:17,313 INFO [org.gluu.oxtrust.action.PersonImportAction] Found 460 persons in input Excel file ``` The file is not present, but previous imports have worked.

By Mohib Zico Account Admin 17 May 2017 at 7:30 a.m. CDT

Mohib Zico gravatar
I'll try to reproduce your issue locally with 400+ user's entries.

By Simon Devlin user 18 May 2017 at 1:19 a.m. CDT

Simon Devlin gravatar
It's probably not worth bothering over. I ended up looking through the source code to see how you generate inums and wrote some code to create the LDAP entries directly from a CSV file. Advantage of this is I can set all the custom properties I need too. Seems to work fine :-) Thanks for your efforts

By Mohib Zico Account Admin 18 May 2017 at 1:29 a.m. CDT

Mohib Zico gravatar
Great. >> wrote some code to create the LDAP entries directly from a CSV file That's sound interesting.

By Mohib Zico Account Admin 22 May 2017 at 8:35 a.m. CDT

Mohib Zico gravatar
FYI... Tested this feature with 449 users. Operation completed successfully, attaching sample file I used. ``` 2017-05-22 13:31:51,470 INFO [qtp274064559-19] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:405) - Found 449 persons in input Excel file 2017-05-22 13:31:51,707 INFO [qtp274064559-19] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:282) - Prepared 449 persons for creation 2017-05-22 13:31:57,509 INFO [qtp274064559-17] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:137) - Attempting to add 449 persons 2017-05-22 13:32:11,008 INFO [qtp274064559-17] [org.gluu.oxtrust.action.PersonImportAction] (PersonImportAction.java:158) - All 449 persons added successfully ```