## Expected behaviour When we scale up[ opendj-repl statefulset ](https://github.com/GluuFederation/gluu-docker/blob/3.1.6/examples/kubernetes/gke/ldap/opendj-repl.yaml#L19), we are expecting all the opendj-repl pods are in sync. ## Actual behaviour We are experiencing opendj replication issue. Here is the opedj-repl logs as below ``` INFO - 2020-05-13 18:40:39,522 - Enabling OpenDJ replication of o=site between opendj-init-0.opendj.gluu.svc.cluster.local:1636 and opendj-repl-0.opendj.gluu.svc.cluster.local:1636. WARNING - 2020-05-13 18:40:43,325 - There are no base DNs available to enable replication between the two servers. INFO - 2020-05-13 18:40:43,325 - Initializing OpenDJ replication of o=site between opendj-init-0.opendj.gluu.svc.cluster.local:1636 and opendj-repl-0.opendj.gluu.svc.cluster.local:1636. WARNING - 2020-05-13 18:40:47,375 - Error during the initialization with contents from server opendj-init-0.opendj.gluu.svc.cluster.local:4444. Last log details: [13/May/2020:18:40:46 +0000] severity="NOTICE" msgCount=0 msgID=org.opends.messages.backend-413 message="Initialize From Replica task quicksetup-initialize23 started execution". Task state: STOPPED_BY_ERROR. Check the error logs of opendj-init-0.opendj.gluu.svc.cluster.local:4444 for more information. See /tmp/opendj-replication-7475846152827263655.log for a detailed log of this operation. INFO - 2020-05-13 18:40:47,375 - Enabling OpenDJ replication of o=metric between opendj-init-0.opendj.gluu.svc.cluster.local:1636 and opendj-repl-0.opendj.gluu.svc.cluster.local:1636. ``` ## More logs After investigating deeper, we narraw down the problem into dsreplication initialization failure with error "LDAP: error code 34 - No domain matches the provided base DN 'o=gluu'" But when we list-backends we can see the base DN exists. Please refer below logs ``` (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ kubectl get pod -l app=opendj -o custom-columns=POD:.metadata.name,CONTAINER:.spec.containers[0].name,IMAGE:.spec.containers[0].image POD CONTAINER IMAGE opendj-init-0 opendj gluufederation/opendj:3.1.6_02 opendj-repl-0 opendj gluufederation/opendj:3.1.6_02 opendj-repl-1 opendj gluufederation/opendj:3.1.6_02 opendj-repl-2 opendj gluufederation/opendj:3.1.6_02 opendj-repl-3 opendj gluufederation/opendj:3.1.6_02 (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ k exec -it opendj-repl-1 -- /opt/opendj/bin/dsreplication status --trustAll --no-prompt --adminUID admin --adminPassword ******* -h opendj-repl-1.opendj.gluu.svc.cluster.local Suffix DN : Server : Entries : Replication enabled : DS ID : RS ID : RS Port (1) : M.C. (2) : A.O.M.C. (3) : Security (4) ----------:--------------------------------------------------:---------:---------------------:-------:-------:-------------:----------:--------------:------------- o=gluu : opendj-init-0.opendj.gluu.svc.cluster.local:4444 : 71738 : true : 32004 : 26287 : 8989 : 0 : : true o=gluu : opendj-repl-0.opendj.gluu.svc.cluster.local:4444 : 71151 : true : 15981 : 22303 : 8989 : 0 : : true o=gluu : opendj-repl-1.opendj.gluu.svc.cluster.local:4444 : : true : 21571 : 17984 : 8989 : : : true o=gluu : opendj-repl-2.opendj.gluu.svc.cluster.local:4444 : 71431 : true : 5118 : 20283 : 8989 : 0 : : true o=gluu : opendj-repl-3.opendj.gluu.svc.cluster.local:4444 : 71151 : true : 16266 : 22653 : 8989 : 0 : : true o=metric : opendj-init-0.opendj.gluu.svc.cluster.local:4444 : 6 : true : 20832 : 26287 : 8989 : 1 : : true o=metric : opendj-repl-0.opendj.gluu.svc.cluster.local:4444 : 6 : true : 32103 : 22303 : 8989 : 0 : : true o=metric : opendj-repl-1.opendj.gluu.svc.cluster.local:4444 : : true : 14903 : 17984 : 8989 : : : true o=metric : opendj-repl-2.opendj.gluu.svc.cluster.local:4444 : 6 : true : 16058 : 20283 : 8989 : 0 : : true o=metric : opendj-repl-3.opendj.gluu.svc.cluster.local:4444 : 6 : true : 30738 : 22653 : 8989 : 0 : : true o=site : opendj-init-0.opendj.gluu.svc.cluster.local:4444 : 82857 : true : 24478 : 26287 : 8989 : 4 : : true o=site : opendj-repl-0.opendj.gluu.svc.cluster.local:4444 : 82857 : true : 18341 : 22303 : 8989 : 0 : : true o=site : opendj-repl-1.opendj.gluu.svc.cluster.local:4444 : : true : 20383 : 17984 : 8989 : : : true o=site : opendj-repl-2.opendj.gluu.svc.cluster.local:4444 : 82857 : true : 14243 : 20283 : 8989 : 0 : : true o=site : opendj-repl-3.opendj.gluu.svc.cluster.local:4444 : 82857 : true : 3750 : 22653 : 8989 : 4 : : true [1] The port used to communicate between the servers whose contents are being replicated. [2] The number of changes that are still missing on this server (and that have been applied to at least one of the other servers). [3] Age of oldest missing change: the date on which the oldest change that has not arrived on this server was generated. [4] Whether the replication communication through the replication port is encrypted or not. (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ kubectl exec -it opendj-init-0 -- /opt/opendj/bin/list-backends Backend ID : Base DN ---------------:------------------ adminRoot : cn=admin data ads-truststore : cn=ads-truststore backup : cn=backups config : cn=config metric : o=metric monitor : cn=monitor schema : cn=schema site : o=site tasks : cn=tasks userRoot : o=gluu (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ kubectl exec -it opendj-repl-0 -- /opt/opendj/bin/list-backends Backend ID : Base DN ---------------:------------------ adminRoot : cn=admin data ads-truststore : cn=ads-truststore backup : cn=backups config : cn=config metric : o=metric monitor : cn=monitor schema : cn=schema site : o=site tasks : cn=tasks userRoot : o=gluu (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ kubectl exec -it opendj-repl-1 -- /opt/opendj/bin/list-backends Backend ID : Base DN ---------------:------------------ adminRoot : cn=admin data ads-truststore : cn=ads-truststore backup : cn=backups config : cn=config metric : o=metric monitor : cn=monitor schema : cn=schema site : o=site tasks : cn=tasks userRoot : o=gluu (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ kubectl exec -it opendj-repl-2 -- /opt/opendj/bin/list-backends Backend ID : Base DN ---------------:------------------ adminRoot : cn=admin data ads-truststore : cn=ads-truststore backup : cn=backups config : cn=config metric : o=metric monitor : cn=monitor schema : cn=schema site : o=site tasks : cn=tasks userRoot : o=gluu (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ kubectl exec -it opendj-repl-1 -- /opt/opendj/bin/dsreplication initialize --adminUID admin --adminPassword ******** --baseDN o=gluu --hostSource opendj-init-0.opendj.gluu.svc.cluster.local --portSource 4444 --hostDestination opendj-repl-1.opendj.gluu.svc.cluster.local --portDestination 4444 --trustAll --no-prompt Initializing base DN o=gluu with the contents from opendj-init-0.opendj.gluu.svc.cluster.local:4444: Error launching initialization with contents from server opendj-init-0.opendj.gluu.svc.cluster.local:4444. Details: javax.naming.InvalidNameException: ds-task-id=quicksetup-initialize1,cn=Scheduled Tasks,cn=Tasks: [LDAP: error code 34 - No domain matches the provided base DN 'o=gluu']; remaining name 'ds-task-id=quicksetup-initialize1,cn=Scheduled Tasks,cn=Tasks' See /tmp/opendj-replication-3648714173922655504.log for a detailed log of this operation. command terminated with exit code 12 (⎈ |my-cluster:gluu)aucllt0027:~ mookkiahm$ kubectl exec -it opendj-repl-1 -- cat /tmp/opendj-replication-3648714173922655504.log [14/05/2020:04:23:41 +0000] category=org.opends seq=0 severity=INFO msg=Application launched May 14, 2020 4:23:41 AM GMT [14/05/2020:04:23:44 +0000] category=QUICKSETUP seq=1 severity=INFO msg=QuickSetup application launched May 14, 2020 4:23:44 AM GMT [14/05/2020:04:23:44 +0000] category=QUICKSETUP seq=2 severity=SEVERE msg=Error creating task {ds-task-initialize-replica-server-id=ds-task-initialize-replica-server-id: 32004, ds-task-initialize-domain-dn=ds-task-initialize-domain-dn: o=gluu, objectclass=objectclass: top, ds-task, ds-task-initialize-from-remote-replica, ds-task-class-name=ds-task-class-name: org.opends.server.tasks.InitializeTask, ds-task-id=ds-task-id: quicksetup-initialize1} [14/05/2020:04:23:44 +0000] category=TOOLS seq=3 severity=SEVERE msg=Complete error stack: exception=ReplicationCliException: Error launching initialization with contents from server opendj-init-0.opendj.gluu.svc.cluster.local:4444. Details: javax.naming.InvalidNameException: ds-task-id=quicksetup-initialize1,cn=Scheduled Tasks,cn=Tasks: [LDAP: error code 34 - No domain matches the provided base DN 'o=gluu']; remaining name 'ds-task-id=quicksetup-initialize1,cn=Scheduled Tasks,cn=Tasks' (ReplicationCliMain.java:7164 ReplicationCliMain.java:4287 ReplicationCliMain.java:1933 ReplicationCliMain.java:538 ReplicationCliMain.java:477 ReplicationCliMain.java:354 ReplicationCliMain.java:306) ``` Please help me understand what it means - No domain matches the provided base DN 'o=gluu' and how come only one pod result into this problem?