By: Adam Westall user 26 May 2017 at 9:42 a.m. CDT

4 Responses
Adam Westall gravatar
Hi, I am currently migrating to 3.0.1 from 2.4.4, including a server move. I have followed the upgrade instructions however the import30.py script hangs at Processing the LDIF data. This is the view on the console: ``` ./import30.py backup_24 ------------------------------------------------------------ Gluu Server Community Edition Migration Tool ============================================================ Enter LDAP Server - 1.OpenLDAP, 2.OpenDJ [1]: 1 INFO Stopping Webapps oxAuth and Identity. INFO Stopping OpenLDAP Server. INFO Copying the Certificates. INFO Updating the CA Certs Keystore. INFO Copying the custom pages and assets of webapps. INFO Copying Shibboleth IDP files... INFO Converting Schema files of custom attributes. INFO Exporting LDAP data. INFO Processing the LDIF data. ``` And the log: ``` 2017-05-26 14:35:07,688 INFO root Stopping Webapps oxAuth and Identity. 2017-05-26 14:35:07,688 DEBUG root Running command : /sbin/service oxauth stop 2017-05-26 14:35:08,726 DEBUG root Stopping Jetty: OK 2017-05-26 14:35:08,726 DEBUG root Running command : /sbin/service oxauth status 2017-05-26 14:35:08,757 DEBUG root Jetty NOT running START_INI = /opt/gluu/jetty/oxauth/start.ini START_D = /opt/gluu/jetty/oxauth/start.d JETTY_HOME = /opt/jetty JETTY_BASE = /opt/gluu/jetty/oxauth JETTY_CONF = /opt/jetty/etc/jetty.conf JETTY_PID = /var/run/oxauth.pid JETTY_START = /opt/jetty/start.jar JETTY_LOGS = /opt/gluu/jetty/oxauth/logs JETTY_STATE = /opt/gluu/jetty/oxauth/oxauth.state CLASSPATH = JAVA = /opt/jre/bin/java JAVA_OPTIONS = -server -Xms256m -Xmx1418m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dpython.home=/opt/jython -Dcatalina.base=/opt/gluu/jetty/oxauth -Djetty.logging.dir=/opt/gluu/jetty/oxauth/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/oxauth -Djava.io.tmpdir=/opt/jetty-9.3/temp JETTY_ARGS = jetty.http.host=localhost jetty.http.port=8081 jetty.state=/opt/gluu/jetty/oxauth/oxauth.state jetty-logging.xml jetty-started.xml RUN_CMD = /opt/jre/bin/java -server -Xms256m -Xmx1418m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dpython.home=/opt/jython -Dcatalina.base=/opt/gluu/jetty/oxauth -Djetty.logging.dir=/opt/gluu/jetty/oxauth/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/oxauth -Djava.io.tmpdir=/opt/jetty-9.3/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8081 jetty.state=/opt/gluu/jetty/oxauth/oxauth.state jetty-logging.xml jetty-started.xml 2017-05-26 14:35:08,758 DEBUG root Running command : /sbin/service identity stop 2017-05-26 14:35:09,794 DEBUG root Stopping Jetty: OK 2017-05-26 14:35:09,794 DEBUG root Running command : /sbin/service identity status 2017-05-26 14:35:09,826 DEBUG root Jetty NOT running START_INI = /opt/gluu/jetty/identity/start.ini START_D = /opt/gluu/jetty/identity/start.d JETTY_HOME = /opt/jetty JETTY_BASE = /opt/gluu/jetty/identity JETTY_CONF = /opt/jetty/etc/jetty.conf JETTY_PID = /var/run/identity.pid JETTY_START = /opt/jetty/start.jar JETTY_LOGS = /opt/gluu/jetty/identity/logs JETTY_STATE = /opt/gluu/jetty/identity/identity.state CLASSPATH = JAVA = /opt/jre/bin/java JAVA_OPTIONS = -server -Xms256m -Xmx945m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dcatalina.base=/opt/gluu/jetty/identity -Dpython.home=/opt/jython -Dorg.eclipse.jetty.server.Request.maxFormContentSize=50000000 -Djetty.logging.dir=/opt/gluu/jetty/identity/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/identity -Djava.io.tmpdir=/opt/jetty-9.3/temp JETTY_ARGS = jetty.http.host=localhost jetty.http.port=8082 jetty.state=/opt/gluu/jetty/identity/identity.state jetty-logging.xml jetty-started.xml RUN_CMD = /opt/jre/bin/java -server -Xms256m -Xmx945m -XX:+DisableExplicitGC -Dgluu.base=/etc/gluu -Dcatalina.base=/opt/gluu/jetty/identity -Dpython.home=/opt/jython -Dorg.eclipse.jetty.server.Request.maxFormContentSize=50000000 -Djetty.logging.dir=/opt/gluu/jetty/identity/logs -Djetty.home=/opt/jetty -Djetty.base=/opt/gluu/jetty/identity -Djava.io.tmpdir=/opt/jetty-9.3/temp -jar /opt/jetty/start.jar jetty.http.host=localhost jetty.http.port=8082 jetty.state=/opt/gluu/jetty/identity/identity.state jetty-logging.xml jetty-started.xml 2017-05-26 14:35:09,826 INFO root Stopping OpenLDAP Server. 2017-05-26 14:35:09,826 DEBUG root Running command : /sbin/service solserver stop 2017-05-26 14:35:10,880 DEBUG root Stopping solserver (via systemctl): [ OK ] 2017-05-26 14:35:10,880 DEBUG root Running command : /sbin/service solserver status 2017-05-26 14:35:10,902 DEBUG root Symas OpenLDAP LDAP services slapd is not running 2017-05-26 14:35:10,902 INFO root Copying the Certificates. 2017-05-26 14:35:10,903 DEBUG root copying /etc/certs/idp-encryption.crt 2017-05-26 14:35:10,903 DEBUG root copying /etc/certs/idp-signing.crt 2017-05-26 14:35:10,903 DEBUG root copying /etc/certs/shibIDP.key.orig 2017-05-26 14:35:10,903 DEBUG root copying /etc/certs/asimba.csr 2017-05-26 14:35:10,904 DEBUG root copying /etc/certs/httpd.key.orig 2017-05-26 14:35:10,904 DEBUG root copying /etc/certs/oxauth-keys.json 2017-05-26 14:35:10,904 DEBUG root copying /etc/certs/httpd.key.old 2017-05-26 14:35:10,904 DEBUG root copying /etc/certs/oxauth-keys.jks 2017-05-26 14:35:10,905 DEBUG root copying /etc/certs/httpd.key 2017-05-26 14:35:10,905 DEBUG root copying /etc/certs/idp-encryption.csr 2017-05-26 14:35:10,905 DEBUG root copying /etc/certs/asimba.key 2017-05-26 14:35:10,905 DEBUG root copying /etc/certs/cacerts 2017-05-26 14:35:10,905 DEBUG root copying /etc/certs/idp-signing.csr 2017-05-26 14:35:10,905 DEBUG root copying /etc/certs/shibIDP.csr 2017-05-26 14:35:10,906 DEBUG root copying /etc/certs/cert_creds.json 2017-05-26 14:35:10,906 DEBUG root copying /etc/certs/shibIDP.key 2017-05-26 14:35:10,906 DEBUG root copying /etc/certs/idp-signing.key.orig 2017-05-26 14:35:10,906 DEBUG root copying /etc/certs/shibIDP.jks 2017-05-26 14:35:10,906 DEBUG root copying /etc/certs/asimbaIDP.jks 2017-05-26 14:35:10,907 DEBUG root copying /etc/certs/idp-signing.key 2017-05-26 14:35:10,907 DEBUG root copying /etc/certs/httpd.crt 2017-05-26 14:35:10,907 DEBUG root copying /etc/certs/shibIDP.pkcs12 2017-05-26 14:35:10,907 DEBUG root copying /etc/certs/asimba.crt 2017-05-26 14:35:10,907 DEBUG root copying /etc/certs/asimba.key.orig 2017-05-26 14:35:10,908 DEBUG root copying /etc/certs/asimba.pkcs12 2017-05-26 14:35:10,908 DEBUG root copying /etc/certs/duo_creds.json 2017-05-26 14:35:10,908 DEBUG root copying /etc/certs/httpd.csr.old 2017-05-26 14:35:10,908 DEBUG root copying /etc/certs/gplus_client_secrets.json 2017-05-26 14:35:10,908 DEBUG root copying /etc/certs/scim-rs.jks 2017-05-26 14:35:10,909 DEBUG root copying /etc/certs/idp-encryption.key.orig 2017-05-26 14:35:10,909 DEBUG root copying /etc/certs/idp-encryption.key 2017-05-26 14:35:10,909 DEBUG root copying /etc/certs/httpd.der 2017-05-26 14:35:10,909 DEBUG root copying /etc/certs/httpd.crt.old 2017-05-26 14:35:10,909 DEBUG root copying /etc/certs/shibIDP.crt 2017-05-26 14:35:10,909 DEBUG root copying /etc/certs/super_gluu_creds.json 2017-05-26 14:35:10,910 DEBUG root copying /etc/certs/opendj.crt 2017-05-26 14:35:10,910 INFO root Updating the CA Certs Keystore. 2017-05-26 14:35:10,910 DEBUG root Running command : hostname 2017-05-26 14:35:10,912 DEBUG root secure.redacted 2017-05-26 14:35:10,912 DEBUG root Deleting new secure.redacted_httpd 2017-05-26 14:35:10,912 DEBUG root Running command : /opt/jre/bin/keytool -delete -alias secure.redacted_httpd -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:11,143 DEBUG root 2017-05-26 14:35:11,144 DEBUG root Delete operation success. 2017-05-26 14:35:11,144 DEBUG root Importing old secure.redacted_httpd 2017-05-26 14:35:11,144 DEBUG root Running command : /opt/jre/bin/keytool -import -trustcacerts -file backup_24/etc/certs/httpd.crt -alias secure.redacted_httpd -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:11,388 DEBUG root 2017-05-26 14:35:11,388 DEBUG root Certificate import success. 2017-05-26 14:35:11,388 DEBUG root Deleting new secure.redacted_idp-signing 2017-05-26 14:35:11,388 DEBUG root Running command : /opt/jre/bin/keytool -delete -alias secure.redacted_idp-signing -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:11,626 DEBUG root 2017-05-26 14:35:11,627 DEBUG root Delete operation success. 2017-05-26 14:35:11,627 DEBUG root Importing old secure.redacted_idp-signing 2017-05-26 14:35:11,627 DEBUG root Running command : /opt/jre/bin/keytool -import -trustcacerts -file backup_24/etc/certs/idp-signing.crt -alias secure.redacted_idp-signing -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:11,869 DEBUG root 2017-05-26 14:35:11,870 DEBUG root Certificate import success. 2017-05-26 14:35:11,870 DEBUG root Deleting new secure.redacted_idp-encryption 2017-05-26 14:35:11,870 DEBUG root Running command : /opt/jre/bin/keytool -delete -alias secure.redacted_idp-encryption -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:12,115 DEBUG root 2017-05-26 14:35:12,115 DEBUG root Delete operation success. 2017-05-26 14:35:12,115 DEBUG root Importing old secure.redacted_idp-encryption 2017-05-26 14:35:12,116 DEBUG root Running command : /opt/jre/bin/keytool -import -trustcacerts -file backup_24/etc/certs/idp-encryption.crt -alias secure.redacted_idp-encryption -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:12,358 DEBUG root 2017-05-26 14:35:12,358 DEBUG root Certificate import success. 2017-05-26 14:35:12,359 DEBUG root Missing file: backup_24/etc/certs/shibidp.crt 2017-05-26 14:35:12,359 DEBUG root Deleting new secure.redacted_asimba 2017-05-26 14:35:12,359 DEBUG root Running command : /opt/jre/bin/keytool -delete -alias secure.redacted_asimba -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:12,631 DEBUG root 2017-05-26 14:35:12,631 DEBUG root Delete operation success. 2017-05-26 14:35:12,631 DEBUG root Importing old secure.redacted_asimba 2017-05-26 14:35:12,631 DEBUG root Running command : /opt/jre/bin/keytool -import -trustcacerts -file backup_24/etc/certs/asimba.crt -alias secure.redacted_asimba -keystore /opt/jre/jre/lib/security/cacerts -storepass changeit -noprompt 2017-05-26 14:35:12,883 DEBUG root 2017-05-26 14:35:12,883 DEBUG root Certificate import success. 2017-05-26 14:35:12,883 DEBUG root Missing file: backup_24/etc/certs/openldap.crt 2017-05-26 14:35:12,884 INFO root Copying the custom pages and assets of webapps. 2017-05-26 14:35:12,884 INFO root Copying Shibboleth IDP files... 2017-05-26 14:35:12,885 INFO root Converting Schema files of custom attributes. 2017-05-26 14:35:12,886 INFO root Exporting LDAP data. 2017-05-26 14:35:12,886 DEBUG root Running command : /opt/symas/bin/slapcat -f /opt/symas/etc/openldap/slapd.conf -l /root/migration/current.ldif 2017-05-26 14:35:12,903 DEBUG root 2017-05-26 14:35:12,903 DEBUG root 2017-05-26 14:35:12,903 INFO root Processing the LDIF data. 2017-05-26 14:35:23,330 DEBUG root Keeping old value for gluuScimEnabled 2017-05-26 14:35:23,330 DEBUG root Keeping old value for gluuIpAddress 2017-05-26 14:35:24,329 DEBUG root Keeping old value for description 2017-05-26 14:35:24,329 DEBUG root Keeping old value for displayName 2017-05-26 14:35:25,831 DEBUG root Keeping old value for oxAuthClientSecret 2017-05-26 14:35:25,918 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,007 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,040 DEBUG root Keeping old value for displayName 2017-05-26 14:35:26,040 DEBUG root Keeping old value for description 2017-05-26 14:35:26,040 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,100 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,134 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,167 DEBUG root Keeping old value for description 2017-05-26 14:35:26,167 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,235 DEBUG root Keeping old value for description 2017-05-26 14:35:26,326 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,361 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,393 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,427 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,462 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,496 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,530 DEBUG root Keeping old value for description 2017-05-26 14:35:26,530 DEBUG root Keep multiple old values for oxModuleProperty 2017-05-26 14:35:26,563 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,596 DEBUG root Keeping old value for displayName 2017-05-26 14:35:26,596 DEBUG root Keeping old value for description 2017-05-26 14:35:26,597 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,597 DEBUG root Keep multiple old values for oxConfigurationProperty 2017-05-26 14:35:26,632 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,693 DEBUG root Keeping old value for oxScript 2017-05-26 14:35:26,986 DEBUG root Keeping old value for oxFaviconImage ``` Any ideas what could be causing it to stop? It's been over an hour in that state. I must also note that CPU usage for one core has staid at 100% the whole time for python. Thanks

By Aliaksandr Samuseu staff 26 May 2017 at 9:43 a.m. CDT

Aliaksandr Samuseu gravatar
Hi, Adam. How much user entries do you have in your old instance? It was reported before that it may take a while in case there are a lot of existing users to migrate.

By Adam Westall user 26 May 2017 at 9:48 a.m. CDT

Adam Westall gravatar
Hi Aliaksandr, There is currently just over 30000 users. Would you have any idea of the timescale to import with that number? Thanks, Adam

By Aliaksandr Samuseu staff 26 May 2017 at 10:02 a.m. CDT

Aliaksandr Samuseu gravatar
>There is currently just over 30000 users. That's a lot of users, indeed. >Would you have any idea of the timescale to import with that number? In my earlier test with ~5k users it took around 40 mins to complete. I guess we need to do some performance testing of the latest version of migration script. We'll get back to you after some results are acquired.

By Adam Westall user 26 May 2017 at 10:05 a.m. CDT

Adam Westall gravatar
Thanks :)