By: Hernan Quevedo user 09 Oct 2017 at 10:52 a.m. CDT

53 Responses
Hernan Quevedo gravatar
I need to setup a two node cluster for Gluu, just for replication in the case of a failover, and I'm following [enter link description here](https://gluu.org/docs/ce/3.0.2/installation-guide/cluster/). Since I don't need an HA environment per se, just replication of user data, I don't need a NGIX load-balancer. Thus, can I ignore everything related to Ngix in the guide?

By Aliaksandr Samuseu staff 10 Oct 2017 at 10:31 a.m. CDT

Aliaksandr Samuseu gravatar
Hi, Herman. As each Gluu instance seems to retain Apache it uses as a frontend web server in this cluster setup, and if you just need to have a reserve server, it should do. You could try for yourself, and share your experience here. We may guide your further in case of any issues.

By Chris Blanton user 10 Oct 2017 at 11:03 a.m. CDT

Chris Blanton gravatar
Hernan, A couple things: Do you mean failover, as in you don't lose Gluu services when one server goes down? Say you used node1.server.com and node2.server.com as your Gluu servers. You'll hit issues with certificates, as they are tied to the domain name/host name you use when running set-up. You might be able to create multi-domain certificates, and configure everything manually, but I've not tested this approach. You're bound to hit issues. Using one single proxy server (e.g. NGINX) to route traffic helps to bypass this issue. Now if you just mean having a secondary LDAP server replicate your primary server, say for the circumstance of full drive failure on the primary, setting up replication of LDAP is easy. You can do Multi-Master or Master-Slave depending on your requirements. In the docs is Multi-Master and you only need to follow steps 4-6, I believe, to get OpenLDAP to replicate.

By Hernan Quevedo user 11 Oct 2017 at 9:11 a.m. CDT

Hernan Quevedo gravatar
Thanks you. What we need is exactly that, a secondary LDAP server replica, which captures every user creation or user modification done in the primary, so in the case of a primary node failure, the app can point to this secondary server, being able to authenticate existing users as well as manage them (create, update). I believe this can be done by installing primary and secondary Gluu server certificates. Another question is: the LDAP replication works in both ways? Say, I create a user in primary, it replicates online to the secondary; primary fails, secondary is now primary, user is created in secondary, this new user, when the primary is up again, gets synchronized from secondary?

By Chris Blanton user 11 Oct 2017 at 10:18 a.m. CDT

Chris Blanton gravatar
Hernan, Failover, like you're describing, would require that the front-end/app of your service would point to both servers and choose which one to assign itself to, in case one isn't responding. Again, this is handled by NGINX in our scenario, but maybe you could do some work to handle a scenario as such. I honestly wouldn't recommend it unless you have a lot of time for research and troubleshooting. As far as replication with n-way Multi-Master Replication (MMR), the OpenLDAP servers are configured to be able to accept reads/writes and replicate each other based on time-stamps, so in your scenario, it would come online and add all changes not in it's own log of entries.

By Hernan Quevedo user 11 Oct 2017 at 3:18 p.m. CDT

Hernan Quevedo gravatar
All right... I'm meditating about this. For the time being, 2 questions: in the https://gluu.org/docs/ce/3.0.2/installation-guide/cluster/ the LDAP password is the same as the one asked in the installation of Gluu, right? And, last, any particular hardware recommendation for the NGIX balancer?

By Chris Blanton user 11 Oct 2017 at 3:23 p.m. CDT

Chris Blanton gravatar
Hernan, To the first question, yes. It's the same password. Username admin. As far as hardware recommendation for NGINX, it's extremely lightweight. I've used it some really low settings in Digital Ocean. I think 512 MB and 2 CPU's. I'm sure it could go lower. It doesn't do anything resource intensive.

By Hernan Quevedo user 12 Oct 2017 at 10:42 a.m. CDT

Hernan Quevedo gravatar
Thank you. I'm having these two errors running the create_slapd_conf.py and create_slapd_conf_26.py scripts: -Traceback (most recent call last): File "create_slapd_conf.py", line 69, in <module> conf_file_name = '{}.conf'.format(ldp['fqn_hostname'].replace('.','_')) ValueError: zero length field name in format - /tmp/cluster-mgr/manual_install/slapd_conf_script Traceback (most recent call last): File "create_slapd_conf_26.py", line 26, in <module> if config.get(ldp, 'enable').lower() in ('yes', 'true', 'on', '1'): File "/usr/lib64/python2.6/ConfigParser.py", line 565, in get raise NoOptionError(option, section) ConfigParser.NoOptionError: No option 'enable' in section: 'nginx' fqn_hostname has a value, /etc/hosts has the nginx IP, and I think pretty much everything is ready. I thought the replace function was not encountering the dot, I changed it, but still same error. I'm gonna debug it to see if something comes up.

By Chris Blanton user 12 Oct 2017 at 11:22 a.m. CDT

Chris Blanton gravatar
ConfigParser.NoOptionError: No option 'enable' in section: 'nginx' Did you add `enable = Yes` to the NGINX portion of the sync_repl.cfg file? That shouldn't be there

By Hernan Quevedo user 12 Oct 2017 at 11:25 a.m. CDT

Hernan Quevedo gravatar
No, I didn't. But that error happens only with python2.6 script, the other error is the weird one.

By Chris Blanton user 12 Oct 2017 at 11:37 a.m. CDT

Chris Blanton gravatar
Which version of python is on your machine?

By Hernan Quevedo user 12 Oct 2017 at 12:07 p.m. CDT

Hernan Quevedo gravatar
It was the version; made the conf files on my local windows machine, and now I'm trying to make the the synchronization work, in addition to setting up Nginx. Thanks, I'll be updating this.

By Hernan Quevedo user 12 Oct 2017 at 12:38 p.m. CDT

Hernan Quevedo gravatar
Well, I have everything set up, the only thing is that /var/log/openldap/ldap.log hasn't moved, so most likely synchronization is not working. The other thing is that all the servers in the cluster conf, namely, nginx, gluuserver1 and gluuserver2 are on AWS, so probably some connectivity issues could arise. Now, for the app to test this, I have to point to the nginx, right? The app config file states that scimAPI is located on https://server1_FQDN/identity/seam/resource/restv1 and oxauth on https://server1_FQDN/oxauth/seam/resource/restv1/oxauth/token So, changing to NGINX, it would be http://nginx_FQDN/identity/seam/resource/restv1, right?

By Chris Blanton user 12 Oct 2017 at 12:45 p.m. CDT

Chris Blanton gravatar
Hernan, Were you able to still access the Gluu UI @ yourNGINXserver.com? That will be a test to see if everything is configured properly.

By Hernan Quevedo user 12 Oct 2017 at 1:03 p.m. CDT

Hernan Quevedo gravatar
It does show me the usual certificate problem message of browsers, but stays indefinitely waiting for a response. Oxauth.log on principal server says: "Failed to lookup entry --> null" and " Failed to load appliance entry from Ldap org.gluu.site.ldap.persistence.exception.EntryPersistenceException: Failed to find entry: inum=@!48F1.FF1E.5119.7351!0002!6E33.F796,ou=appliances,o=gluu"

By Chris Blanton user 12 Oct 2017 at 1:41 p.m. CDT

Chris Blanton gravatar
Did you modify your /etc/hosts file?

By Hernan Quevedo user 12 Oct 2017 at 2:55 p.m. CDT

Hernan Quevedo gravatar
I think I'm missing something in that third step but also on the configuration of LDAP: stopping gluu says it failed to stop OpenLDAP and starting gluu says: Unable to find configuration in LDAP, try to load configuration from file system... Failed to load configuration from /etc/gluu/conf/oxtrust-config.json java.io.FileNotFoundException: File '/etc/gluu/conf/oxtrust-config.json' does not exist

By Hernan Quevedo user 13 Oct 2017 at 11:56 a.m. CDT

Hernan Quevedo gravatar
I opted for start all over again, and I'm kinda stuck on the last instrucion of the second step: gluu-console.pub Cannot find any files or folders names like that. The third step is definitely unclear for me, specially, the second instruction.

By Chris Blanton user 13 Oct 2017 at 12:19 p.m. CDT

Chris Blanton gravatar
Hernan, If you can still start and log-in to Gluu without importing the key, you should be fine. As far as the 3rd step, after you unpack the gluu.gz file and login to the Gluu server chroot, run this command: `vi /etc/hosts` The ip address here will be the ip address of the first server you installed on. You want to change that ip address to the current server's ip address. Before: https://imgur.com/a/HzUnn After: https://imgur.com/a/7hYEi

By Hernan Quevedo user 13 Oct 2017 at 2:32 p.m. CDT

Hernan Quevedo gravatar
Ok, I'll do that, is just that on the page it says c3.gluu.info, Nginx is mentioned, so it sort of threw me off.

By Chris Blanton user 13 Oct 2017 at 3:06 p.m. CDT

Chris Blanton gravatar
Ah yes, in this instance c3 is my NGINX server. I'm going to rework the docs to make them more understandable.

By Hernan Quevedo user 13 Oct 2017 at 3:22 p.m. CDT

Hernan Quevedo gravatar
Thank you. What if service solserver force-reload cannot start the service again? Just says "FAILED"? On one server.

By Chris Blanton user 13 Oct 2017 at 3:24 p.m. CDT

Chris Blanton gravatar
For CentOS just use service solserver restart.

By Hernan Quevedo user 13 Oct 2017 at 3:25 p.m. CDT

Hernan Quevedo gravatar
What logs can I see, it won't start.

By Chris Blanton user 13 Oct 2017 at 3:27 p.m. CDT

Chris Blanton gravatar
What won't start?

By Hernan Quevedo user 13 Oct 2017 at 3:30 p.m. CDT

Hernan Quevedo gravatar
Symas OpenLDAP LDAP services, on the principal server.

By Chris Blanton user 13 Oct 2017 at 3:33 p.m. CDT

Chris Blanton gravatar
This should give you some log: systemctl status solserver.service Also this: journalctl -xe

By Hernan Quevedo user 13 Oct 2017 at 3:39 p.m. CDT

Hernan Quevedo gravatar
What if none of those commands is recognized? I'm Centos 6.8

By Chris Blanton user 13 Oct 2017 at 3:45 p.m. CDT

Chris Blanton gravatar
Okay try service solserver force-reload -d -1 The `debug level 1` will spout out everything happening with solserver/slapd

By Chris Blanton user 13 Oct 2017 at 3:57 p.m. CDT

Chris Blanton gravatar
Make sure it exists on server 1 as well. All the servers need an accesslog database for keeping track of entries between the two.

By Hernan Quevedo user 13 Oct 2017 at 4:12 p.m. CDT

Hernan Quevedo gravatar
Ok, it started. That -d -1 was great. Now, how do I know is working? This log /var/log/openldap/ldap.log is not moving, but connectivity is working. start -d -1 gives a lot of hexadecimal info, and I suppose is working.

By Chris Blanton user 13 Oct 2017 at 4:17 p.m. CDT

Chris Blanton gravatar
If the log is giving a bunch of information and doesn't break, it means it's still running. So now just run `service solserver force-reload` without the debug info.

By Hernan Quevedo user 13 Oct 2017 at 4:31 p.m. CDT

Hernan Quevedo gravatar
Got it. Now, if accessing to nginx gives me too many redirects, what did I screw up in?

By Chris Blanton user 13 Oct 2017 at 4:32 p.m. CDT

Chris Blanton gravatar
Inside Gluu go to: /opt/gluu/jetty/identity/logs/oxtrust.log This should show you an error if something's wrong. Check it on both servers.

By Chris Blanton user 13 Oct 2017 at 4:33 p.m. CDT

Chris Blanton gravatar
Also make sure to scroll through all the log until you find the last timestamp that correlates with when you tried to connect. The stacktrace for the logs sometimes reprints the same errors as it attempts to connect multiple times and there might be an issue above that's causing an error below.

By Hernan Quevedo user 13 Oct 2017 at 4:38 p.m. CDT

Hernan Quevedo gravatar
All right, it says - Failed to load configuration from LDAP - Failed to lookup entry --> null - Failed to persist entry: inum=@!48F1.FF1E.5119.7351!0002!6E33.F796,ou=appliances,o=gluu When I go to nginx, it redirects me to server 1, and there it starts to loop around until too many redirects message shows up.

By Chris Blanton user 16 Oct 2017 at 3:47 p.m. CDT

Chris Blanton gravatar
Hernan, I'm going to create a video here soon of the process. I'll keep you updated.

By Hernan Quevedo user 17 Oct 2017 at 9:36 a.m. CDT

Hernan Quevedo gravatar
Chris, thank you so much. I'll look forward to it.

By Hernan Quevedo user 19 Oct 2017 at 10:11 p.m. CDT

Hernan Quevedo gravatar
Hi. I'm still stuck at the part where Nginx serves everything: I followed the guide but somehow the nginx part never works. I managed to sync nodes and the app can manage users through Gluu, but the admin web console is now a message: ``` HTTP ERROR: 503 Problem accessing /identity/. Reason: Service Unavailable Powered by Jetty:// 9.3.15.v20161220 ``` I'm pointing the browser to https://nginxfqdn, but all I get is that message above. Thanks.

By Hernan Quevedo user 19 Oct 2017 at 11:35 p.m. CDT

Hernan Quevedo gravatar
I restarted the nodes and now it works fine, except for this new error message when creating new users: ``` (UmaValidationService.java:85) - Get request for another AM: 'backend'. Expected: 'ngnix_fqdn' CreateRptWS.java:102) - Exception happened javax.ws.rs.WebApplicationException: null ``` The only part where I find 'backend' is during the Nginx setup for balancing, updating nginx.conf

By Hernan Quevedo user 20 Oct 2017 at 11:10 a.m. CDT

Hernan Quevedo gravatar
Does this bug is the one mentioned [here](https://github.com/GluuFederation/oxAuth/issues/472) ? Do I have to use the lastest version then?

By Chris Blanton user 20 Oct 2017 at 11:18 a.m. CDT

Chris Blanton gravatar
Is 'backend' replacing your actual server name or is the error returning 'backend' and 'ngnix_fqdn'?

By Hernan Quevedo user 20 Oct 2017 at 11:26 a.m. CDT

Hernan Quevedo gravatar
The error is returning 'backend' and 'ngnix_fqdn'. And no, there is no 'backend' server. Currently, I have the 2-node cluster and the Nginx balancer; web console and authentication works fine, new users does not. I have a JUnit test client, that takes the domain, which is the nginxfqdn, the umaMetaDataUrl and the umaAatClientId, with the correct scim-rp.jks, and searches some inum. This client works until Nginx comes in: jboss.resteasy says: Error status 400 bad request, and the scimclient: Failed to get RPT token In other words, in my Windows local development machine, when I change the hosts file so nginx_fqdn points to any of the nodes IPs, it works; when it points to the balancer, it fails with that message.

By Hernan Quevedo user 23 Oct 2017 at 9:48 a.m. CDT

Hernan Quevedo gravatar
Since this string is only seen in nginx.conf, maybe the nginx is not working properly when it comes to creating new users, oxauth, and send back the string 'backend'; nothing else seems probable.

By Chris Blanton user 23 Oct 2017 at 9:50 a.m. CDT

Chris Blanton gravatar
Can you send me your nginx.conf file?

By Hernan Quevedo user 23 Oct 2017 at 9:59 a.m. CDT

Hernan Quevedo gravatar
The hostnames are provided by AWS. ``` events { worker_connections 768; } http { upstream backend_id { ip_hash; server ip-10-0-0-75:443; server ip-10-0-0-41:443; } upstream backend { server ip-10-0-0-75:443; server ip-10-0-0-41:443; } server { listen 80; server_name ip-10-0-0-213; return 301 https://ip-10-0-0-213$request_uri; } server { listen 443; server_name ip-10-0-0-213; ssl on; ssl_certificate /etc/nginx/ssl/httpd.crt; ssl_certificate_key /etc/nginx/ssl/httpd.key; location ~ ^(/)$ { proxy_pass https://backend; } location /.well-known { proxy_pass https://backend/.well-known; } location /oxauth { proxy_pass https://backend/oxauth; } location /identity { proxy_pass https://backend_id/identity; } } } ```

By Hernan Quevedo user 23 Oct 2017 at 4:22 p.m. CDT

Hernan Quevedo gravatar
Is it normal that ldap process on 1636 is up with a different user, e.g., chrony in one node, and with tomcat user in other?

By Hernan Quevedo user 24 Oct 2017 at 3 p.m. CDT

Hernan Quevedo gravatar
There is definitely something wrong with that nginx.conf file, the cluster is working, but the proxy passes are not.

By Chris Blanton user 24 Oct 2017 at 3:06 p.m. CDT

Chris Blanton gravatar
Hernan, Can you access the Gluu UI by going to the FQDN of this ip: ``` server_name ip-10-0-0-213; ``` If so the proxy should be working as expected. You also might need to add some additional routing to the proxy. Make the end of your nginx.conf look like this: ``` location ~ ^(/)$ { proxy_pass https://backend; } location /.well-known { proxy_pass https://backend_id/.well-known; } location /oxauth { proxy_pass https://backend/oxauth; } location /idp { proxy_pass https://backend/idp; } location /identity { proxy_pass https://backend_id/identity; } location /cas { proxy_pass https://backend/cas; } location /asimba { proxy_pass https://backend/asimba; } location /passport { proxy_pass https://backend/passport; } } } ```

By Hernan Quevedo user 24 Oct 2017 at 3:36 p.m. CDT

Hernan Quevedo gravatar
Yes, the Gluu UI can be accessed by the Nginx fqdn, and authenticate users, but only the part where I have to create them, is throwing the error: ``` UmaValidationService.java:85) - Get request for another AM: 'backend' . Expected: 'ip-10-0-0-213' 2017-10-24 20:35:08,253 ERROR [qtp242131142-17] [xdi.oxauth.uma.ws.rs.CreateRptWS] (CreateRptWS.java:102) - Exception happened javax.ws.rs.WebApplicationException: null ``` This request ``` /oxauth/seam/resource/restv1/requester/rpt? ``` Is giving a bad request on Nginx logs, related to the 'backend' error on oxauth.log.

By Hernan Quevedo user 25 Oct 2017 at 2:30 p.m. CDT

Hernan Quevedo gravatar
It seems like this is a bug in 3.0.2, with no solution nor plans to write one. So, if I install Gluu 3.1.1, does Redis is really necessary? The documentation is incomplete and does not say where to install it, how or why exaclty. And what about the connections details by requesting parties? Do they change much compared to 3.0.2? What I mean by this is, for instance, my app connects to Gluu through a clientId, then authorizes itself with Gluu with a umaAatClientId, includes JKS file and a secret, and that's how it creates users and updates them. Will this change for 3.1.1?

By Chris Blanton user 25 Oct 2017 at 5:47 p.m. CDT

Chris Blanton gravatar
Redis caching is required for 3.1.x because LDAP caching is no longer used, for performance requirements. You can install it anywhere. On one of the Gluu servers (outside the chroot, preferably) or another external server.

By Chris Blanton user 25 Oct 2017 at 5:48 p.m. CDT

Chris Blanton gravatar
Hernan, For your other question, please open another ticket.

By Hernan Quevedo user 26 Oct 2017 at 12:26 a.m. CDT

Hernan Quevedo gravatar
Finally I got it: documentation for Nginx configuration file, step 7, Gluu replication, must be modified in the part where it says 'backend', setting the actual Nginx FQDN instead. For some reason, at least on Nginx 1.12.1, petitions to the backend Gluu included the literal 'backend', but the Gluu expects the one given as hostname during setup.py. Putting the Nginx FQDN solved the issue, and now I can use the cluster properly and test it, but left me a nuisance as to why it did authenticate clients but was unable to manage users and create new resources. Thanks for the good support.