By: Hector Rodriguez user 06 Jul 2017 at 5:04 a.m. CDT

17 Responses
Hector Rodriguez gravatar
I've performed an upgrade using the scripts import30.py and export30.py to upgrade from version 3.0.1 to 3.0.2. The import script throws the following errors: ``` Error, entries missing! entry 215: ou=configuration,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu entry 344: ou=trustrelationships,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu ``` If I move on and restart gluu, I get an HTTP ERROR: 503 when accesing /identity. This error seems to be triggered by the following LDAP error: ``` org.gluu.site.ldap.persistence.exception.EntryPersistenceException: Failed to find entries with baseDN: ou=trustRelationships,inum=@!5409.5FE2.395B.B7F2!0002!A249.7637,ou=appliances,o=gluu, filter: (&(&(objectClass=top)(objectClass=gluuSAMLconfig))(&(gluuStatus=active))) ``` The thing is that a simple LDAP search such as: ``` ldapsearch -h localhost -p 1636 -Z -X -D "cn=directory manager,o=gluu" -w ******** -b "o=gluu" "objectclass=gluuSAMLconfig" ``` will succeed, returing the following entries: ``` dn: inum=@!2835.CFC4.E1FB.0890!0002!0E1C.6395!0006!A92D.411F,ou=trustRelationshi ps,inum=@!2835.CFC4.E1FB.0890!0002!0E1C.6395,ou=appliances,o=gluu description: Trust Relationship for the SP displayName: gluu SP on appliance gluuEntityId: 2835CFC4E1FB089000020E1C63950006A92D411F gluuSAMLspMetaDataFN: 2835CFC4E1FB089000020E1C63950006A92D411F-sp-metadata.xml gluuSAMLspMetaDataSourceType: file gluuStatus: active inum: @!2835.CFC4.E1FB.0890!0002!0E1C.6395!0006!A92D.411F objectClass: top objectClass: gluuSAMLconfig url: https://nextgeoss-sso.deimos-space.com dn: inum=@!5409.5FE2.395B.B7F2!0002!A249.7637!0006!0FCF.C8DB,ou=trustRelationshi ps,inum=@!5409.5FE2.395B.B7F2!0002!A249.7637,ou=appliances,o=gluu description: Trust Relationship for the SP displayName: gluu SP on appliance gluuEntityId: 54095FE2395BB7F20002A249763700060FCFC8DB gluuSAMLspMetaDataFN: 54095FE2395BB7F20002A249763700060FCFC8DB-sp-metadata.xml gluuSAMLspMetaDataSourceType: file gluuStatus: active inum: @!5409.5FE2.395B.B7F2!0002!A249.7637!0006!0FCF.C8DB objectClass: top objectClass: gluuSAMLconfig url: https://nextgeoss-sso.deimos-space.com ``` Given that the entries are already there in the LDAP instance, I don't understand why the identity service will throw such error. I'm guessing it could be something related to the formating of ldif files by the export script. Is there any fix or workaround for this issue?

By Mohib Zico staff 06 Jul 2017 at 5:08 a.m. CDT

Mohib Zico gravatar
>> I'm guessing it could be something related to the formating of ldif files by the export script. Yes, that's the only reason of this failure. >> ou=trustrelationships,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu >> ....!A92D.411F,ou=trustRelationshi ps,inum=@!2835.CFC4.E1FB.0890!0002!0E1C.6395,ou=appliances,o=gluu There is a Caps issue here in 'trustRelationship' and 'trustrelationship' >> ou=configuration,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu Can you please check if any other 'ou=configuration' entries imported in 3.0.2 or not?

By Hector Rodriguez user 06 Jul 2017 at 6:26 a.m. CDT

Hector Rodriguez gravatar
There is only one 'ou=configuration' entry: ldapsearch -h localhost -p 1636 -Z -X -D "cn=directory manager,o=gluu" -w ********* -b "o=gluu" "ou=configuration" ``` dn: ou=configuration,inum=@!2835.CFC4.E1FB.0890!0002!0E1C.6395,ou=appliances,o=gluu objectClass: top objectClass: organizationalUnit ou: configuration ```

By Michael Schwartz Account Admin 06 Jul 2017 at 9:03 a.m. CDT

Michael Schwartz gravatar
Assigning to Arun to check out.

By Arunmozhi P staff 12 Jul 2017 at 12:46 p.m. CDT

Arunmozhi P gravatar
I tried to reproduce the error multiple times, but I could not. I did not encounter the `Error, entries missing!` error even once. Can you confirm that this error is recurring?

By Hector Rodriguez user 13 Jul 2017 at 1:38 a.m. CDT

Hector Rodriguez gravatar
I've updated both scripts (import and export) and perfomed the same operations as explained in the first post, and still got the same results: ``` Error, entries missing! entry 198: ou=configuration,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu entry 327: ou=trustrelationships,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu ```

By Arunmozhi P staff 13 Jul 2017 at 11:18 a.m. CDT

Arunmozhi P gravatar
Is it possible to share the output of export24.py? I would like to go through the data and figure out the issue.

By Hector Rodriguez user 18 Jul 2017 at 2 a.m. CDT

Hector Rodriguez gravatar
The output is generated using the export30.py script as mentioned at the start of the issue. Please confirm that this is the output you're requesting.

By Arunmozhi P staff 20 Jul 2017 at 2:05 p.m. CDT

Arunmozhi P gravatar
Sorry for not being clear. I was wondering if it would be possible to share the data from "backup_30" folder created by the export30.py script.

By Hector Rodriguez user 21 Jul 2017 at 1:34 a.m. CDT

Hector Rodriguez gravatar
There you go, I've uploaded to Dropbox (link removed). I've modified the setup.properties file to remove admin credentials. The rest is basically the output of the export30.py script. Thank you for your time and patience.

By Arunmozhi P staff 21 Jul 2017 at 12:54 p.m. CDT

Arunmozhi P gravatar
Thank you for the data. I will go through it and upgrade the scripts if there are improvements and let let you know.

By Arunmozhi P staff 24 Jul 2017 at 1:01 p.m. CDT

Arunmozhi P gravatar
Hi, I tested the data you shared with me. Here are a few things I found: 1. You have run `export30.py` script twice. There were two sets of values in the `setup.properties` inside `backup_30` folder. I have updated the export30.py to ensure multiple runs will replace existing file. I removed the second set of values and did the migration. 2. During the data import step `./import30.py backup_30` I got the following output: ``` ------------------------------------------------------------ Gluu Server Community Edition Migration Tool ============================================================ Enter LDAP Server - 1.OpenLDAP, 2.OpenDJ [1]: 1 INFO Stopping Webapps oxAuth and Identity. INFO Stopping OpenLDAP Server. INFO Copying the Certificates. INFO Updating the CA Certs Keystore. INFO Copying the custom pages and assets of webapps. INFO Exporting LDAP data. INFO Processing the LDIF data. INFO Importing Processed LDAP data. ERROR 59762e4a <= str2entry: str2ad(geossUser): attribute type undefined slapadd: could not parse entry (line=9775) 59762e4a <= str2entry: str2ad(geossUser): attribute type undefined slapadd: could not parse entry (line=9812) 59762e4a <= str2entry: str2ad(geossUser): attribute type undefined slapadd: could not parse entry (line=9832) 59762e4a <= str2entry: str2ad(geossUser): attribute type undefined slapadd: could not parse entry (line=9915) 59762e4a <= str2entry: str2ad(geossUser): attribute type undefined slapadd: could not parse entry (line=9934) Error, entries missing! entry 199: inum=@!5409.5fe2.395b.b7f2!0001!0a42.ea33!0000!3d89.b0f1.c603.05a9,ou=people,o=@!5409.5fe2.395b.b7f2!0001!0a42.ea33,o=gluu entry 204: inum=@!5409.5fe2.395b.b7f2!0001!0a42.ea33!0000!367b.ff7d.1464.3d02,ou=people,o=@!5409.5fe2.395b.b7f2!0001!0a42.ea33,o=gluu entry 206: inum=@!5409.5fe2.395b.b7f2!0001!0a42.ea33!0000!eb04.20f5.e628.b974,ou=people,o=@!5409.5fe2.395b.b7f2!0001!0a42.ea33,o=gluu entry 209: inum=@!5409.5fe2.395b.b7f2!0001!0a42.ea33!0000!6702.3074.f9a5.c655,ou=people,o=@!5409.5fe2.395b.b7f2!0001!0a42.ea33,o=gluu INFO Fixing permissions for files. INFO Starting OpenLDAP Server. ============================================================ The migration is complete. Gluu Server needs to be restarted. # exit # service gluu-server-x.x.x restart ------------------------------------------------------------ ``` As I did not have the LDAP schema to support the `geossUser` attribute. I removed the attribute from the entries in `backup_30/ldif/people.ldif` and did the migration from scratch again. This time I didn't get any error and the import process ended successfully. So, I wasn't able to reproduce the errors you have reported. I recommend retrying the export and import process. And if you get any errors, copy the entire output (as above) from the scripts and share it, so I can figure out if something's wrong.

By Hector Rodriguez user 25 Jul 2017 at 3:11 a.m. CDT

Hector Rodriguez gravatar
I've retried the whole process, updating the migration scripts, trying an installation from scratch of the version 3.0.2 with the geossUser custom attribute schema added. [This is the backup folder generated.] link redacted The output contains the same 2 errors: ``` ------------------------------------------------------------ Gluu Server Community Edition Migration Tool ============================================================ Enter LDAP Server - 1.OpenLDAP, 2.OpenDJ [1]: 1 INFO Stopping Webapps oxAuth and Identity. INFO Stopping OpenLDAP Server. INFO Copying the Certificates. INFO Updating the CA Certs Keystore. INFO Copying the custom pages and assets of webapps. INFO Exporting LDAP data. INFO Processing the LDIF data. INFO Importing Processed LDAP data. ERROR Error, entries missing! entry 198: ou=configuration,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu entry 361: ou=trustrelationships,inum=@!5409.5fe2.395b.b7f2!0002!a249.7637,ou=appliances,o=gluu INFO Fixing permissions for files. INFO Starting OpenLDAP Server. ============================================================ The migration is complete. Gluu Server needs to be restarted. # exit # service gluu-server-x.x.x restart ------------------------------------------------------------ ``` I've performed the same procedure (with a clean installation) again but without the custom attribute to emulate all your steps and the result is the same.

By Hector Rodriguez user 25 Jul 2017 at 3:20 a.m. CDT

Hector Rodriguez gravatar
To add on the previous post: After the migration steps, when initialized, the oxTrust/Identity component generates the following WARN (SAML related?): [Trace](https://pastebin.com/FymBGcz5).

By Arunmozhi P staff 25 Jul 2017 at 11:39 a.m. CDT

Arunmozhi P gravatar
Hey, I retried again. Sorry, can't reproduce the issue. You have reported that the issue is in RHEL 6.5. Can you try the import process in a different and recent distro. Maybe Ubuntu 16 or CentOS 7?

By Hector Rodriguez user 27 Jul 2017 at 1:11 a.m. CDT

Hector Rodriguez gravatar
I can try the import process in a Centos7 distro as soon as I can. I'll report the results.

By Hector Rodriguez user 09 Aug 2017 at 3:08 a.m. CDT

Hector Rodriguez gravatar
So, this issue is closed, but I'll just post the results of the process performed on CentOS7: ``` ------------------------------------------------------------ Gluu Server Community Edition Migration Tool ============================================================ Enter LDAP Server - 1.OpenLDAP, 2.OpenDJ [1]: 1 INFO Stopping Webapps oxAuth and Identity. INFO Stopping OpenLDAP Server. INFO Copying the Certificates. INFO Updating the CA Certs Keystore. INFO Copying the custom pages and assets of webapps. INFO Exporting LDAP data. INFO Processing the LDIF data. INFO Importing Processed LDAP data. ERROR Error, entries missing! entry 1272: ou=configuration,inum=@!c38b.4ac1.f6cd.fc28!0002!0389.1710,ou=appliances,o=gluu entry 1453: ou=trustrelationships,inum=@!c38b.4ac1.f6cd.fc28!0002!0389.1710,ou=appliances,o=gluu INFO Fixing permissions for files. INFO Starting OpenLDAP Server. ============================================================ The migration is complete. Gluu Server needs to be restarted. # exit # service gluu-server-x.x.x restart ------------------------------------------------------------ ``` Looks like the problem is still there. This is a migration from a a fresh new installation on 3.0.2 to another.

By Arunmozhi P staff 09 Aug 2017 at 1:25 p.m. CDT

Arunmozhi P gravatar
Thank you for your report Hector. It helps a lot. We are prepping for another release. I will do my best to identify the case where this occurs and fix it. I really wish I could catch this happening, but I haven't seen this once in my runs. I am sure, there is some edge case reason for this. I don't know what it's yet.