We've tried to upgrade a few times over the years and always failed for one reason or another, so I'm finally making a ticket. Our goal is to transition from 3.1.2 to a more recent version. Ideally to a docker container as well, but that can wait (unless it's easily done here). When following [the 3.1.8 upgrade guide](https://gluu.org/docs/gluu-server/3.1.8/upgrade/) on a fresh Ubuntu 16.04 VM with all OS updates installed we see the following: The first issue we run into is the importer throws an error trying to copy a /var/ox folder from backup_3031. A bit strange but manually making a backup_3031/var/ox folder let's the import process run correctly. Of note, our custom pages don't actually come through. Import log is [here](https://gist.githubusercontent.com/thatguyyoulove/b18143cae53df30ce983f0f3e4875c8d/raw/3e60595a00edee0715334412c0a869e3a06dbc3b/import3031.log). We use custom SSL certs that aren't named httpd.crt/.key so we go make that change in our https_gluu.conf to match our cert file names like in the original 3.1.2 install. [Updated https_gluu.conf with cert names changed is here.](https://gist.githubusercontent.com/thatguyyoulove/4d02a3f8d1b64bc7a0bb3453994e70e4/raw/6a860f7fff9af0891698b3d59fe085f6411cad43/https_gluu.conf) After restarting the container we see that the oxauth login pages opens correctly, however, any identity page fails with a Not Found error. We verify that the service has a status of started and view the logs, but nothing sticks out. Example failures: * Problem accessing /identity/authentication/authcode. Reason: Not Found * Problem accessing /identity/register. Reason: Not Found Identity logs: [oxtrust.log](https://gist.githubusercontent.com/thatguyyoulove/5a9a7ff2c441eb003967ebde615b85dd/raw/8bd4b2eb7b1706efb9fb0f3ea482c0a39c58f635/oxtrust.log) [oxtrust_persistence.log](https://gist.githubusercontent.com/thatguyyoulove/0d327f26db1b24ad88875de06a5930a7/raw/def59ac54f35537372017bf79e10549e9fcaadb5/oxtrust_persistence.log) [oxtrust_cleaner.log](https://gist.githubusercontent.com/thatguyyoulove/36263e914bc4f88a49815265c83bf117/raw/fc3c11313042addc03fca02e0c8e8bb4afd3dec4/oxtrust_cleaner.log) [oxtrust_script.log](https://gist.githubusercontent.com/thatguyyoulove/b064955e0476aab8d07f005d80ee3b42/raw/39fe3d2c8a336ba021bff8c7be35b157dd8dcf3f/oxtrust_script.log) Manually restarting identity from within the container doesn't help. Not really sure where to go from there. Since our goal was to go to 4.1 or so, I decided to try just running the 4.0 upgrade script as well (from a cloned VM). Unfortunately that fails pretty much instantly (same result if run from the 3.1.8 upgraded version above). Error: ``` Ready to upgrade Gluu Server. Start now (y|N) y This upgrade replaces all the default Gluu Server scripts WITH SCRIPTS FROM 4.0 and removes other custom scripts. (This will replace any customization you may have made to these default script entries) Do you want to continue? (Y|n) Y 2020-06-23 18:34:01 URL:https://raw.githubusercontent.com/GluuFederation/community-edition-setup/master/pylib/cbm.py [4993/4993] -> "/root/upg40/setup/pylib/cbm.py" [1] Collecting properties Traceback (most recent call last): File "update.py", line 1963, in <module> setup_porperties = generate_properties(True) File "/root/upg40/setup/pylib/generate_properties.py", line 334, in generate_properties inumOrg = str(result[1][1]['o'][0]) KeyError: 1 ``` [Full output from 4.0 upgrade here](https://gist.githubusercontent.com/thatguyyoulove/3e80b6f72223457bd80133600f40e40b/raw/56dc22635d66c80ffdc40568b03685f3b2108609/upgrade40) Installing 3.1.2 and importing on another VM seems to work, with the exception of our custom pages not coming over, but attempts to upgrade that result in the same errors when using the 4.0 upgrade script. When trying the in-place 3.1.8 script a different error occurs when it attempts to stop OpenDJ (we are on OpenLDAP). ``` Stopping OpenDJ Traceback (most recent call last): File "/opt/upd/3.1.8-upg/bin/update.py", line 1813, in <module> updaterObj.updateJava() File "/opt/upd/3.1.8-upg/bin/update.py", line 1700, in updateJava self.stop_opendj() File "/opt/upd/3.1.8-upg/bin/update.py", line 549, in stop_opendj self.run(cmd) File "/opt/upd/3.1.8-upg/bin/update.py", line 298, in run p = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE, cwd=cwd, env=env) File "/usr/lib/python2.7/subprocess.py", line 711, in __init__ errread, errwrite) File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory ``` [Full output here](https://gist.githubusercontent.com/thatguyyoulove/9d46c1d04188c377e6ceb2676bbbaad5/raw/1ba7d18582838f89179194dfc07b17e3c9cdc085/3.1.8-upgrade.txt) We seem to be at a bit of a loss. Any help would be very appreciated as Gluu is the identity provider for all of our software.