Hi, Mulk. Csync is a bit problematic tool. As your distro is CentOS 7.2, I take it you built it from sources, is it correct? What version of it did you build? Is it the latest one (2.x)? Regards, Alex.

Hi Alex, I didn't build it from sources. I followed the steps provided by Gluu https://gluu.org/docs/cluster/csync-installation/#centos-7x but definitely had very hard time making things work.

>I didn't build it from sources The part of the guide you are referring to implies you build it from sources. At the time of writting there wasn't a ready package of csync for CentOS 7. Do you mean it's now added? Could you run this `# csync2 --version` and share its output?

well I was not sure what those steps were for...i just followed them. Here is the output of the command you asked me to run: ``` [root@sdggluu ~]# csync2 --version csync2: invalid option -- '-' csync2 2.0 - cluster synchronization tool, 2nd generation Copyright (C) 2004 - 2013 LINBIT Information Technologies GmbH http://www.linbit.com See also: http://git.linbit.com/csync2.git/?a=blob;f=AUTHORS Version: 2.0-rc1 ```

Seems it's one of the recent versions, that's good. Could you also share output of this: `# cat /etc/xinetd.d/csync2` (within container) ?

Sure..here it is: ``` [root@sdggluu logs]# cat /etc/xinetd.d/csync2 # default: off # description: csync2 service csync2 { flags = REUSE socket_type = stream wait = no user = root group = root server = /install/community-edition-setup/building_csync/csync2-2.0/csync2 server_args = -i -N SDGVMLAB019.iaglab.sdgc.com port = 30865 type = UNLISTED #log_on_failure += USERID disable = no # only_from = 192.168.199.3 192.168.199.4 } ``` Please note that I had created this file in this directory manually.

> Please note that I had created this file in this directory manually. Yes, it's supposed to be created manually. I believe you are experiencing a known issue with csync2, it's mentioned in our guide, too. You need to add key `-l` to your xinetd confing file, i.e. it should be like this: `server_args = -i -l -N SDGVMLAB019.iaglab.sdgc.com` After that I would have removed csync's databases it created when you tried to run it before on both nodes, then would have repeated file system scan and resolved all initial conflicts manually too, before proceeding to automating it through cron (all required steps are here in our manual cluster guide) Feel free to ask if anything still will be unclear to you.

Sorry, Mulk, please ignore my remark on csync's location (I removed it from my original post), I've confused it with another issue. Your csync is placed within container, as it should.

Hi, Mulk. Do we need to keep this ticket open? Does csync still give you any troubles?

Hi Alex. I followed steps you mentioned and this is the final message that I got: ``` Peer> rs\0016\000\000\010\000\000\000\000\020\37777777717\37777777651\37777777617\37777777646Y\37777777715 p\37777777773s\37777777646\033\37777777721\37777777742\r\37777777723\37777777617\016\37777777727: Found diff in sig at -32:-0 Got 32 bytes, 0 bytes left .. File has been checked successfully (difference found). File is different on peer (rsync sig). Peer> OK (cmd_finished).\n While syncing file /opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt: response from peer(/opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt): sdgvmlab020.iaglab.sdgc.com [1] <- OK (cmd_finished). Local> PATCH OzOGnoGsD45ORsISPRK0ZBXa.oqLXyIVqRSWHBjy1TvDNcC6.JEa3Ps8smGjVj65 /opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt\n Peer> File is also marked dirty here!\n While syncing file /opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt: response from peer(/opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt): sdgvmlab020.iaglab.sdgc.com [10] <- File is also marked dirty here! Match (+): /opt/idp/ssl on /opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt Match (+): /opt/idp/ssl on /opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt File stays in dirty state. Try again later... Local> BYE\n Peer> OK (cu_later).\n response from peer(<no file>): sdgvmlab020.iaglab.sdgc.com [6] <- OK (cu_later). SQL: SELECT command, logfile FROM action GROUP BY command, logfile SQL Query finished. SQL: COMMIT Connection closed. Finished with 8 errors. ``` I don't know where these entire logs are getting logged. If you can share me the location, I can share that..

Hi, Mulk. What command did you use when you got this log fragment?

sorry, looks like i ran the command that was there in the guide to use when setup.last.properties method failed. I cleaned up the database file, restarted xinetd then ran following commands: ``` csync2 -cvvv -N SDGVMLAB019.iaglab.sdgc.com csync2 -xv ``` below are the output: ``` [root@sdggluu csync2]# csync2 -cvvv -N sdgvmlab019.iaglab.sdgc.com Config-File: /usr/local/etc/csync2/csync2.cfg My hostname is sdgvmlab019.iaglab.sdgc.com. Database-File: sqlite3:///usr/local/var/lib/csync2/sdgvmlab019.iaglab.sdgc.com.db3 Opening shared library libsqlite3.so Reading symbols from shared library libsqlite3.so SQL: SELECT count(*) from file Upgrading database schema to version 0. SQL: CREATE TABLE file ( filename, checktxt, UNIQUE ( filename ) ON CONFLICT REPLACE) SQL: CREATE TABLE dirty ( filename, forced, myname, peername, UNIQUE ( filename, peername ) ON CONFLICT IGNORE) SQL: CREATE TABLE hint ( filename, recursive, UNIQUE ( filename, recursive ) ON CONFLICT IGNORE) SQL: CREATE TABLE action ( filename, command, logfile, UNIQUE ( filename, command ) ON CONFLICT IGNORE) SQL: CREATE TABLE x509_cert ( peername, certdata, UNIQUE ( peername ) ON CONFLICT IGNORE) SQL: SELECT filename, recursive FROM hint SQL Query finished. SQL: SELECT command, logfile FROM action GROUP BY command, logfile SQL Query finished. Connection closed. Finished with 0 errors. [root@sdggluu csync2]# csync2 -xv Marking file as dirty: /opt/idp/ssl Marking file as dirty: /opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt Marking file as dirty: /opt/idp/ssl/FE074E3801117F1D0002158AC89F0006679BD6B4-sp-metadata.crt Marking file as dirty: /opt/idp/metadata Marking file as dirty: /opt/idp/metadata/FE074E3801117F1D0002158AC89F0006B90FE54B-sp-metadata.xml Marking file as dirty: /opt/idp/metadata/FE074E3801117F1D0002158AC89F0006679BD6B4-sp-metadata.xml Marking file as dirty: /opt/idp/metadata/FE074E3801117F1D0002158AC89F00065BCB9545-sp-metadata.xml Marking file as dirty: /opt/idp/metadata/FE074E3801117F1D00014942AC64-idp-metadata.xml Marking file as dirty: /opt/idp/conf Marking file as dirty: /opt/idp/conf/service.xml Marking file as dirty: /opt/idp/conf/relying-party.xml Marking file as dirty: /opt/idp/conf/login.config Marking file as dirty: /opt/idp/conf/logging.xml.bkp Marking file as dirty: /opt/idp/conf/logging.xml Marking file as dirty: /opt/idp/conf/internal.xml Marking file as dirty: /opt/idp/conf/handler.xml Marking file as dirty: /opt/idp/conf/attribute-resolver.xml Marking file as dirty: /opt/idp/conf/attribute-filter.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/web.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/tomcat-users.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2 Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/sp Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/sp/shibboleth2.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/sp/attribute-map.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/sp-metadata.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/xmldsig-core-schema.xsd Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/xml.xsd Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/xenc-schema.xsd Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/saml-schema-metadata-2.0.xsd Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/saml-schema-assertion-2.0.xsd Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/datatypes.dtd Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/XMLSchema.dtd Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/relying-party.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/login.config.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/idp-metadata.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/handler.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/attribute-resolver.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/attribute-filter.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/ShibbolethSSOProfileConfiguration.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML2SSOProfileConfiguration.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML2AttributeQueryProfileConfiguration.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML2ArtifactResolutionProfileConfiguration.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML1AttributeQueryProfileConfiguration.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML1ArtifactResolutionProfileConfiguration.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/validationFilter.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/signatureValidationFilter.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/requiredValidUntilFilter.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/entityRoleWhiteListFilter.xml.vm Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/server.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/salt Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/python Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/python/python.txt Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/python/duo_web.py Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/oxTrustLogRotationConfiguration.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/ox-ldap.properties Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/logging.properties Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/gluuTomcatWrapper.conf Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/context.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/catalina.properties Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/catalina.policy Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/asimba-selector.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/Catalina Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/Catalina/localhost Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/Catalina/localhost/oxauth.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/Catalina/localhost/idp.xml Marking file as dirty: /opt/apache-tomcat-7.0.65/conf/Catalina/localhost/identity.xml Marking file as dirty: /etc/csync2/csync2.cfg Connecting to host sdgvmlab020.iaglab.sdgc.com (PLAIN) ... Bound to 10.10.10.149:0 as sdgvmlab019.iaglab.sdgc.com. Connect to 10.10.10.120:30865 (sdgvmlab020.iaglab.sdgc.com). Updating /etc/csync2/csync2.cfg on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/Catalina on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/Catalina/localhost on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/Catalina/localhost/identity.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/Catalina/localhost/idp.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/Catalina/localhost/oxauth.xml on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Updating /opt/apache-tomcat-7.0.65/conf/asimba-selector.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/catalina.policy on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/catalina.properties on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/context.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/gluuTomcatWrapper.conf on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/logging.properties on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/ox-ldap.properties on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/oxTrustLogRotationConfiguration.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/python on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/python/duo_web.py on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/python/python.txt on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/salt on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/server.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2 on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/entityRoleWhiteListFilter.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/requiredValidUntilFilter.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/signatureValidationFilter.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/MetadataFilter/validationFilter.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML1ArtifactResolutionProfileConfiguration.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML1AttributeQueryProfileConfiguration.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML2ArtifactResolutionProfileConfiguration.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML2AttributeQueryProfileConfiguration.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/SAML2SSOProfileConfiguration.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/ProfileConfiguration/ShibbolethSSOProfileConfiguration.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/attribute-filter.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/attribute-resolver.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/handler.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/idp-metadata.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/login.config.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/relying-party.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/XMLSchema.dtd on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/datatypes.dtd on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/saml-schema-assertion-2.0.xsd on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/saml-schema-metadata-2.0.xsd on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/xenc-schema.xsd on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/xml.xsd on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/schema/xmldsig-core-schema.xsd on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/idp/sp-metadata.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/sp on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/sp/attribute-map.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/shibboleth2/sp/shibboleth2.xml.vm on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/tomcat-users.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/apache-tomcat-7.0.65/conf/web.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/conf on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/conf/attribute-filter.xml on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Updating /opt/idp/conf/attribute-resolver.xml on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Updating /opt/idp/conf/handler.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/conf/internal.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/conf/logging.xml on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Updating /opt/idp/conf/logging.xml.bkp on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/conf/login.config on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/conf/relying-party.xml on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Updating /opt/idp/conf/service.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/metadata on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/metadata/FE074E3801117F1D00014942AC64-idp-metadata.xml on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Updating /opt/idp/metadata/FE074E3801117F1D0002158AC89F00065BCB9545-sp-metadata.xml on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/metadata/FE074E3801117F1D0002158AC89F0006679BD6B4-sp-metadata.xml on sdgvmlab020.iaglab.sdgc.com ... Updating /opt/idp/metadata/FE074E3801117F1D0002158AC89F0006B90FE54B-sp-metadata.xml on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Updating /opt/idp/ssl on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/ssl/FE074E3801117F1D0002158AC89F0006679BD6B4-sp-metadata.crt on sdgvmlab020.iaglab.sdgc.com ... File is already up to date on peer. Updating /opt/idp/ssl/FE074E3801117F1D0002158AC89F00068272584C-sp-metadata.crt on sdgvmlab020.iaglab.sdgc.com ... File stays in dirty state. Try again later... Running '/sbin/service tomcat restart' ... Connection closed. Finished with 8 errors. ```

Have you added this `-l` argument to your csync's xinetd configuration, on both nodes? If you did, did you also restarted xinetd on both of them? Could you also share your csync's main configuration file?

Yes. I added -l argument on both nodes as you suggested, removed the database files on both, restarted xinetd on both..and then ran the above 2 commands that i mentioned in my previous post ONLY on 019 VM. Here are the files from both nodes: ``` From 019: [root@sdggluu xinetd.d]# more csync2 # default: off # description: csync2 service csync2 { flags = REUSE socket_type = stream wait = no user = root group = root server = /install/community-edition-setup/building_csync/csync2-2.0/csync2 server_args = -i -l -N SDGVMLAB019.iaglab.sdgc.com port = 30865 type = UNLISTED #log_on_failure += USERID disable = no # only_from = 192.168.199.3 192.168.199.4 } From 020: [root@sdggluu ~]# more /etc/xinetd.d/csync2 # default: off # description: csync2 service csync2 { flags = REUSE socket_type = stream wait = no user = root group = root server = /install/community-edition-setup/building_csync/csync2-2.0/csync2 server_args = -i -l -N SDGVMLAB020.iaglab.sdgc.com port = 30865 type = UNLISTED #log_on_failure += USERID disable = no # only_from = 192.168.199.3 192.168.199.4 } ```

I am also sharing csync2.cfg files from both nodes: ``` From 019 [root@sdggluu csync2]# more /usr/local/etc/csync2/csync2.cfg nossl * *; group cluster_group { host SDGVMLAB019.iaglab.sdgc.com; host SDGVMLAB020.iaglab.sdgc.com; key /etc/csync2/csync2.key; include /etc/csync2/csync2.cfg; include /opt/idp/conf; include /opt/idp/metadata; include /opt/idp/ssl; include /opt/apache-tomcat-7.0.65/conf; exclude *~ .*; action { logfile "/var/log/csync2_action.log"; do-local; } action { pattern /opt/apache-tomcat-7.0.65/conf/*; exec "/sbin/service tomcat restart"; logfile "/var/log/csync2_action.log"; do-local; } backup-directory /var/backups/csync2; backup-generations 3; # auto younger; } ``` And from 020: ``` [root@sdggluu ~]# more /usr/local/etc/csync2/csync2.cfg nossl * *; group cluster_group { host SDGVMLAB019.iaglab.sdgc.com; host SDGVMLAB020.iaglab.sdgc.com; key /etc/csync2/csync2.key; include /etc/csync2/csync2.cfg; include /opt/idp/conf; include /opt/idp/metadata; include /opt/idp/ssl; include /opt/apache-tomcat-7.0.65/conf; exclude *~ .*; action { logfile "/var/log/csync2_action.log"; do-local; } action { pattern /opt/apache-tomcat-7.0.65/conf/*; exec "/sbin/service tomcat restart"; logfile "/var/log/csync2_action.log"; do-local; } backup-directory /var/backups/csync2; backup-generations 3; # auto younger; } ``` Also..cron jobs were removed too as per your suggestion.

Thanks, Mulk. First, I would suggest to use `-N` key with explicit specification of the node's name every time you run the command. The more it's predictable the better. You also should use key `-r`. In general, it's safer to stay as close as possible to guidance provided in our article, unless you are totally sure it's okay to do something differently. You have `auto younger;` commented out. And if I got you right, you just run these 2 commands: `csync2 -cvvv ...` and `csync2 -xv`. But have you also tried to run `csync2 -frvvv ...`(note the `-f` key), and then `csync2 -xv` again after that (it's also mentioned in the guide)? Without `auto ...` directive csync won't resolve conflicts automatically, and need you first to mark files that must win the conflict during next run. Please try next steps on your current setup: 1. On the 1st node, do `# csync2 -crvvv -N sdgvmlab019.iaglab.sdgc.com /` 2. On the 2nd node, do `# csync2 -crvvv -N sdgvmlab020.iaglab.sdgc.com /` 3. On the 1st node again, do `# csync2 -frvvv sdgvmlab019.iaglab.sdgc.com /` 4. On the 1st node, do `# csync2 -xrvvv -N sdgvmlab019.iaglab.sdgc.com`

I followed all the 4 steps exactly the way you suggested and i saw no errors this time. So can you confirm what would be the commands I would need to use in future now?

> i saw no errors this time Most likely, it now works as expected. You could test by creating some file under `/opt/tomcat/conf/` directory on the 1st node, doing `csync2 -xrvvv` + `csync2 -frvv` + `csync2 -xrvvv` sequence there again, then checking whether it will appear on the 2nd node. Then try to remove this file on 2nd node, do `csync2 -xrvvv` + `csync2 -frvv` + `csync2 -xrvvv` sequence there this time (don't forget to change node name in `-N` argument), and check whether it will be removed on the 1st node. > So can you confirm what would be the commands I would need to use in future now? Yes, these are correct command to do an initial sync if you disabled `auto ...` feature by commenting it out in the configuration file. We recommend to do the initial sync this way as it's easier to spot any problems right away. After you've made sure it works, you should uncomment `auto younger` directive on both nodes, and automate everything with cron jobs (examples of them are provided in the guide; you may need to change path to executable and log files, and will need to change hostnames, though).

I would also double-check that `/opt/apache-tomcat-7.0.65/conf` is indeed the path to Tomcat's directory in your package, it may change over time, and many months has passed since that guide was composed. You'll need to update the path in your config files then.

I followed the steps provided and it appears its working. In my first attempt to sync on node 2 after deleting the file, i did get few dirty errors but after retrying it was gone. Thanks a lot for your help. Please close this case.

Thanks for confirming it works, Mulk. Closing now.