Admin basics

1. Starting up

To start the SVN MultiSite Plus replicator, follow these steps:

  1. Open a terminal window on the server and login with suitable file permissions.
  2. Run the svn-multisite service, located in the /etc/init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 svn-multisite -> /opt/svn-multisite-plus/bin/svn-multisite
    
  3. Run the start script:
    [root@localhost init.d]#  ./svn-multisite start
    20130520-164811 (24088) [INFO]: Starting WANdisco MultiSite Plus 20130520-164811 (24088) [INFO]: Started replicator (24100) 20130520-164811 (24088) [INFO]: Started ui (24110) 20130520-164811 (24088) [INFO]: Number of errors: 0 20130520-164811 (24088) [INFO]: Number of warnings: 0
  4. The two components of SVN MultiSite Plus, the replicator and the UI will start up. Read more about the svn-multisite init.d script

2. Shutting down

To shutdown:

  1. Open a terminal window on the server and login with suitable file permissions.
  2. Run the svn-multisite service, located in the init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 svn-multisite -> /opt/svn-multisite-plus/bin/svn-multisite
  3. Run the stop script, i.e.:
    [wandisco@ip-10-0-100-7 bin]$  ./svn-multisite stop
    
    20130520-165704 (24767) [INFO]: Stopping WANdisco MultiSite Plus
    20130520-165704 (24767) [INFO]: Request received to shut down replicator
    20130520-165704 (24767) [INFO]: replicator processes ended
    20130520-165704 (24767) [INFO]: Request received to shut down ui
    20130520-165704 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 1)...
    20130520-165707 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 2)...
    20130520-165710 (24767) [INFO]: ui processes ended
    20130520-165710 (24767) [INFO]: Number of errors: 0
    20130520-165710 (24767) [INFO]: Number of warnings: 0
    
  4. Both the replicator and the UI processes will shut down. Read more about the svn-multisite init.d script

3. Using the init.d script

The 'start-up' script for persistent running of SVN MultiSite Plus can be found in the /etc/init.d folder. Run the script with the help command to list the available commands:

[root@localhost init.d]# ./svn-multisite help
usage: ./svn-multisite-plus (start|stop|restart|force-reload|status|uistart|uistop|repstart|repstop)

start         Start SVN MultiSite Plus services
stop          Stop SVN MultiSite Plus services
restart       Restart SVN MultiSite Plus services
force-reload  Restart SVN MultiSite Plus services
status        Show the status of SVN Multisite Plus services
uistart       Start the SVN MultiSite Plus User Interface
uistop        Stop the SVN MultiSite Plus User Interface
repstart      Start the SVN Multisite Plus Replicator
repstop       Stop the SVN Multisite Plus Replicator

4. Changing the admin console password

You can change SVN MultiSite Plus's login password at any time by following this procedure:

  1. Login to the MultiSite admin console.
    Schedule for you

    Login.

  2. Click on the Settings tab.
    Schedule for you

    Settings.

  3. At the top of the settings screen is the password change form. Enter the current password, along with a new password.
    Schedule for you

    Changed password

  4. Click the SAVE button to store the new password. You can be sure that the new password has been accepted if you see a notification message appear at the bottom of the screen.
    Schedule for you

    Growl!

** Alert! **Changing Username
It's currently not possible to change the Administration username. In order to change the username you would need to re-install SVN MultiSite Plus.

5. Updating your license.key file

Follow this procedure if you ever need to change your product license. You would need to do this if, for example, you needed to increase the number of SVN users or the number of replication nodes.

  1. Login to your server's command line, navigate to the properties directory: /opt/wandisco/svn-multisite-plus/replicator/properties and rename the license.key to license.20130625.
    i.e.
    total 16
    -rw-r--r-- 1 wandisco wandisco 1183 Dec  5 15:58 application.properties
    -rw-r--r-- 1 wandisco wandisco  512 Dec  5 15:05 license.key <-
    -rw-r--r-- 1 wandisco wandisco  630 Dec 17 15:43 logger.properties
    -rw-r--r-- 1 wandisco wandisco  747 Dec  4 10:31 svnok.catalog
    
  2. Get your new license.key and drop it into the /opt/svn-multisite-plus/replicator/properties directory.
  3. Restart the replicator by running the SVN MultiSite Plus script with the following argument:
    /etc/init.d/svn-multisite-plus restart
    This will trigger an SVN MultiSite Plus replicator restart, which will force SVN MultiSite Plus to pick up the new license file and apply any changes to permitted usage.

tipIf you don't restart
If you follow the above instructions but don't do the restart SVN MultiSite Plus will continue to run with the old license until it performs a daily license validation (which runs at midnight). Providing that your new license key file is valid and has been put in the right place then SVN MultiSite Plus will then update its license properties without the need to restart.

If you run into problems, check the replicator logs (/opt/svn-multisite-plus/replicator/logs ) for more information.

PANIC: License is invalid com.wandisco.fsfs.licensing.LicenseException: Failed to load filepath>

6. Update a node's properties

In the System Data section of the Settings tab there's a bank of editable properties that can be quickly updated by re-entering, saving and allowing the SVN MultiSite replicator to restart - although this may cause brief disruption to users whose in-flight commits will fail.

Cha cha changes...

Node properties that you can change -- subject to a restart of the replicator.

Node Name
- This is the human-readable form of the node's ID. Unlike the Node ID it is possible to change the value of Node Name and reuse it (after it has been removed from the replication network). That is, you can't have two nodes with the same name, but you can now reuse a previously removed node name.
Location Longitude
The Node's geographical location is no longer recorded during installation. Instead you enter the details here.
Location Latitude
Along with Longitude, this value places the node on the internal map and helps the application determined the local time for the node based on the timezone in which it falls.
Hostname / IP Address
The hostname or underlying IP address can be updated.
DConE Port
The TCP port used for DConE agreement traffic - not to be confused with the Content Distribution port which carries the payload repository data.

After entering a new value, click the Save button. A growl message will appear to confirm that the change is being replicated - this will result in a restart of the replicator which may cause brief disruption to SVN users.

Other property changes

To change the Content Delivery port

content.port.<Node id>=<new port>

  • Once the file is in place, run the following command (on all the nodes except the one you have changed):
     java -jar svn-ms-replicator-updateinetaddress.jar -c <path to application.properties>
    

  • Go back to the node with the updated properties and Restart MultiSite.

  • You should login to the updated node and check its System Data (at the bottom of the Settings) tab. You should do some test commits to ensure that replication continues successfully.
  • Node Content Distribution Timeouts

    There are two configurable properties that you can modify as part of fine-tuning an SVN MultiSite Plus deployment. They are provided to allow you to balance best possible performance against the tolerance of a poor WAN connectivity. Both properties are contained within the application properties file, by default located here: /opt/wandisco/svn-multisite-plus/replicator/properties/application.properties.


    socket.timeout
    socket.timeout=90000
    The socket.timeout is an amount of time in milliseconds that the local node will wait for the connection to be established before throwing an exception - therefore signalling that it failed to connect within that timeout. Default value is 15 minutes (90,000 milliseconds).

    ** Alert! **Not less than 10 minutes!
    DO NOT set socket.timeout to less than 10 minutes (60,000 milliseconds) or you may encounter problems.


    content.pull.timeout
    content.pull.timeout=300000
    The content pull timeout sets how long the Content Distribution system will wait for new content to be pulled fully over from a remote node. The default value is 5 minutes (300,000 milliseconds). This default is set on the assumption that there are no problems with the deployment's WAN connectivity.
    Increasing the timeout

    Increasing the value may help if poor connectivity is resulting in the replicator repeatedly giving up on content distribution that would have eventually transferred had it been given enough time, i.e. not as a result of a slow network rather than something that has caused a permanent error.

    Decreasing the timeout

    Decreasing the value is not generally recommended. Doing so is not intended as a method for boosting performance - although this may occur in some situations. We recommend that you don't drop the timeout value below 5000 (5 seconds) without consulting with our support team.

    7. Setting up data monitoring

    The Monitoring Data tool monitors the disk usage of SVN MultiSite Plus's database directory, providing a basic level of protection against SVN MultiSite Plus consuming all disk space. The tool also lets you set up your own monitors for user-selected resources.

    ** Alert! **Monitoring Data - not intended as a final word in system protection
    Monitoring Data is no substitute for dedicated, system-wide monitoring tools. Instead, it is intended to be a 'last stand' against possible disk space exhaustion that could lead to data loss or corruption.

    Read our Recommendations for system-wide monitoring tools.

    Default settings

    Resource Monitor Tool

    Click the "View" link to go to a monitor's settings.


    By default MultiSite's database directory (/opt/wandisco/svn-multisite-plus/replicator/database) is monitored - this is the location of MultiSite's prevayler database where all data and transactions files for replication are stored.


    This built-in monitor runs on all nodes. Any additional monitors that you set up will monitor on a per-node basis. Monitors are not replicated so a monitor set up on one node is not applied to any other node.

    Additional monitors

    As well as SVN MultiSite Plus's own database folder, there are a number of other directories that could in certain circumstances grow very large and potentially consume all available file space.

    MultiSite directories that it may be worth monitoring:

    Other directories that should be monitored:

    For most deployments all these directories will reside on the same file system, so that our default monitor would catch if any of them were consuming the available space. However, there are two scenarios where we'd recommend that you set up your own monitor for the content directory:

    1) You wish to set a higher trigger amount than the default monitor (1GiB for warning, 0.09GiB for emergency shutdown).
    2) You have placed the content directory on a different filesystem with its own capacity that wouldn't be tracked by the default monitor.

    In either case you should follow up the setting up of a monitor with a corresponding email notification that will be sent if some or all of your monitor's trigger conditions are met.

    Create additional resource monitors using the following procedure:

    1. Login to the Administrator user interface.

    2. Click the "SETTINGS" link on the top menu bar.

    3. Monitoring Data is situated below the Administrator Settings. Enter the full path to the resource that you wish to monitor. For example, you might wish to monitor the replicator logs: "/opt/svn-multisite-plus/replicator/logs". Enter the path and click "Add".
      Resource Monitor Tool

      Add resource path



    4. The new resource monitor will appear as a new box - it will display "No records found", indicating that it doesn't yet have any monitoring rules set. Click its corresponding "Configure" link.
      Resource Monitor Tool

      Configure



    5. The screen will update to show the Resource Monitoring screen for your selected resource.
      Resource Monitor Tool

      Settings


    6. File Path:
      The full path for your selected resource
      Monitor Identity:
      The unique string that will identify the monitor
      Edit Condition and Event List
      Lists current resource monitors, initially this will state "No records found"

      Add Conditional and Event to List

      Storage amount entry field
      Enter an amount of disk space in Gigabytes. e.g. 0.2 would be equal to 200 Megabytes of storage.
      Select an Event from the dropdown:
      SEVERE
      - will initiate a shutdown of SVN MultiSite Plus and will also write a message to the log and the "SEVERE" logging level. See "When a Shut down is triggered" for more information.
      WARNING
      - will write a message to the log and the "WARNING" level of severity.
      DEBUG
      - will write a message to the log and the "DEBUG" level of severity.
      INFO
      - will write a message to the log and the "INFO" level of severity.
    7. When you have added all the trigger points and events that you require for the resource, click "Update". You can then navigate away - Click on "Resource Monitoring" on the breadcrumb trail to return to the settings screen.

    When a Shutdown is triggered

    If the disk space available to a monitored resource is less than the value you have for a "Severe" event then the event is logged and MultiSite's replicator will shut down after a set interval of 10 minutes. You can configure the interval in application.properties file:

    /opt/wandisco/svn-multisite-plus/replicator/properties/application.properties
    
    resourcemonitor.period.min=10L
    value in minutes!

    ** Alert! **Edits to property files require a replicator restart
    Any change that you make to the application.properties file will require that you restart SVN MultiSite Plus's replicator.

    Once shut down all SVN repositories will become unavailable to users, you should immediately take action to make more disk space available, the replicator can be restarted using SVN MultiSite Plus's service as soon as the resource that triggered the shutdown has enough available disk space not to shut down again.

    ** Alert! **Overriding the forced shutdown
    In the event that you can't start a node in order to resolve the cause of a forced shutdown -- in an absent minded moment you might create a data monitor that triggers a severe log message if there's less disk space than the disk's actual capacity. You'd be stuck because it wouldn't be possible to free up space -- short of swapping for a bigger disk.

    There's a method you can use to unlock the forced shutdown.

    1. Login to the locked node using a terminal.
    2. Navigate to the properties folder, by default you'll find this here:
      /opt/wandisco/svn-multisite-plus/replicator/properties/application.properties
    3. Create a backup, then edit the file, changing the line
      monitor.ignore.severe=false
      to say
      monitor.ignore.severe=true
      Save the change to the file.
    4. Restart the replicator (see Starting up). During the restart the replicator will now ignore the severe warning (which are still written to the log file) allowing you to delete the offending monitor.

      You can't use this procedure to override the default monitor, its emergency shutdown limit of <100MiB will ALWAYS shut down the replicator.

    8. Setting up email notifications

    The email notification is a rules-based system to deliver alerts based on user-defined templates over one or more channels to destinations based on triggers that are activated by arbitrary system events. Put simply, email notification sends out emails when something happens within the SVN MultiSite Plus environment. The message content, trigger rules and destinations are all user-definable.

    Email Notifications

    Automated alert emails

    8.1 Set up a Gateway

    The Gateway section stores your email (SMTP) server details. You can set up multiple gateways to ensure that the loss of the server doesn't prevent alert notifications from being delivered.

    1. Log into the admin UI, then click the Settings tab.

    2. Click on the Gateway section of the Notifications area.
      Email Notifications

      Add Gateway


    3. Enter your email gateway's settings:
      Email Notifications

      Enter settings


    4. IP/Hostname of SMTP Server:
      your email server's address.
      SMTP Server Port
      The port assigned for SMTP traffic (Port 25 etc).
      Encryption Type
      Indicate your server's encryption type - None, SSL (Secure Socket Layer) or TLS (Transport Layer Security). SSL is a commonly used. For tips on setting up suitable keystore and truststore files see Setting up SSL Key pair.

      ** Alert! **keystores?
      If you're not familiar with the finer points of setting up SSL keystores and truststores it is recommended that you read the following article: Using Java Keytool to manage keystores.

      Authentication Required
      Indicate whether you need a username and password to connect to the server - requires either "true" or "false".
      User Name
      If authentication is required, enter the authentication username here.
      Password
      If authentication is required, enter the authentication password here.
      Sender Address
      Provide an email address that your notifications will appear to come from. If you want to be able to receive replies from notifications you'll need to make sure this is a valid and monitored address.
      Number of Tries Before Failing
      Set the number of attempts SVN MultiSite Plus makes in order to send out notifications.
      Interval Between Tries (Seconds)
      Set the time (in seconds) between your server's attempts to send notifications.

    5. Click on the "+Add" button. Your gateway will appear in the table.
      You can add any number of gateways. SVN Multisite Plus will exhaust the "Number of Tries Before Failing" for each registered gateway before moving on down the list to the next.

      You can use the Test button to verify that your entered details will connect to a mail gateway server.

    8.2 Set up a Destination

    The destinations section stores the email address for your notification recipients.

    1. Click on the + on the Destinations line.

    2. Enter an email address for a notification recipient. Click the + Add link.
      Email Notifications

      Notification


    3. The destination will appear in a table. Click the Edit or Remove links to change the address or remove it from the system.

    8.3 Set up a Template

    The template section is used to store email messages. You can create any number of templates, each with its own notification message, triggered by one of a number of trigger scenarios that are set up in the Rule section.

    1. Click on the + on the Template line.

    2. Enter a Template Subject line, this will be the subject of the notification email.

    3. Enter some Body Text, this will be the message that is sent out when the notification is triggered. The message has a 1024 character limit, you can track the available number of characters at the bottom of the text box.
      Email Notifications

      Notification



    4. When the message has been entered, click the + Add link to save the message template.

    Available variables

    When writing email notification templates, you can insert variables into the template that will be interpolated when the notification is delivered. The following two variables are available for ALL event types:

    For the event types "Disk Monitor Info", "Disk Monitor Severe" and "Disk Monitor Warning", the following additional variable is available:

    For the event types "Deploy Repository Succeeded" and "Deploy Repository Failed", the following additional variables are available:

    8.4 Set up a Rule

    The Rule section is used to define which system event should trigger a notification, what message template should be used and which recipients should be sent the notification.

    ** Alert! **Known issue
    It's currently not possible to edit notification rules that you create. This issue will be addressed in a later release. For now, use the simple workaround of deleting then recreating rules that you want to change.

    1. Click on the + on the Rule line.

    2. Choose an Event from the Event drop-down list:
      Email Notifications

      Rules


    3. Event:

      Any Repository Global Read-Only Event
      In case of any repository entering a global read-only mode.
      Global Read-Only Due to Admin Action
      In case of any repository entering a global read-only mode as a result of administrator interaction through the admin UI.
      Disk Monitor Info
      Disk Storage has dropped below the Info level. This will trigger if any data monitor message is written to the logs at the "INFO" level.
      Disk Monitor Warning
      Disk Storage has dropped below the Warning level.This will trigger if any data monitor message is written to the logs. For more information about disk warning messages, see the Setting up data monitoring section.
      Disk Monitor Severe
      Disk Storage has hit the Severe level. This will trigger if any "Severe" level data monitor message is written to the logs. At this level, SVN MultiSite Plus will have shutdown to ensure that disk space exhaustion doesn't corrupt your system and potentially your SVN repositories. For more information about disk warning messages, see the Setting up data monitoring section.

      Deploy Repository Failed
      A repository added to SVN MultiSite Plus has failed to deploy, in which case the repository will not be replicated.
      Deploy Repository Succeeded
      A repository added to SVN MultiSite Plus has successfully deployed. Such an event might be sent to a mail group received by SVN users, telling them that their repository is now accessible.
      Global Read-Only Due to Consistency Check Failure
      In case of any repository entering a global read-only mode as a result of failing a consistency check with its replicas.
      Generic file replication error occurred
      An error occured with the Generic Replication script.

    4. Choose a Template from the drop-down list. These are the templates that you have already set up under the Templates section.

    5. Choose destinations for your notification from the available destination email addresses. You can make multiple selections so that a message is sent to more than one recipient address.

    6. Click on the + Add link to save your rule.
      Email Notifications

      Rule set


    9. Backing up SVN MultiSite Plus data

    It's possible to back up SVN MultiSite Plus's own database in case you need to quickly restore a node.

    ** Alert! **Only MultiSite Settings are backed-up
    This procedure backs up SVN MultiSite Plus's internal Prevayler database, it doesn't touch your SVN repository data or any other system files (such as Apache configuration, authz files etc.) that you should also be backing up.

      Create a backup of the current installation by invoking the following API call:
      curl --user <username>:<password> -X POST http://[node_ip_address]:8082/dcone/backup
      This will create a backup folder in [INSTALL-DIR]multisite-plus/replicator/db/backup/X.X.X_DConE_Backup directory.

    Back up while shut down

    (run from within /replicator):
    java -cp ./fsfsrestore.jar com.wandisco.fsfs.backup.FsfsBackup -c ./properties/application.properties

    Use this to back up the current state of all prevaylers when SVN MultiSite Plus is shut down - you don't therefore need to start the replicator in order to create a backup of the database.

    10. Restore SVN MultiSite Plus Data

    The restore functionality is no longer supported since the product upgrade functionality is handled using the installer.

    11. Manage Access to SVN MultiSite Plus

    SVN MultiSite Plus supports three different mechanisms for managing access to its admin UI:

    Internally Managed Users

    It is possible to set up multiple administrator accounts for accessing the SVN MultiSite Plus admin console. Accounts can be set up from within the admin UI (via the Security tab). These users are then able to login to any node's admin UI by providing their username and password.

    The folloing section will explain how you set up multiple accounts, set up managing LDAP authorities and export/import the resulting data.

    Adding additional users

    1. Login to the Admin UI using an existing admin account.
      Acc

      Login

    2. Click on the Security tab, then click on the Add User button
    3. Acc

      SVN MultiSite Plus - Add User

    4. Enter details for the new administrator, then click the Add User button situated at the end of the entry bar.
      Acc

      SVN MultiSite Plus - Click Add User to save their details.

    5. You'll see a growl message confirming that the user has been added. You'll see them listed on the Internally Managed Users after clicking the Reload button (or refershing your browser session).
      Acc

      SVN MultiSite Plus - New user appears

    Removing or editing user details

    You can modify any user details by clicking their corresponding Edit button on the Internally Managed Users table

      Acc

      SVN MultiSite Plus - Remove or Edit users

    LDAP Authorities

    SVN MultiSite Plus supports the use of LDAP authorities for managing admin loging accounts. See our brief Guide to LDAP

    When connecting SVN MultiSite Plus to available LDAP authorities it is possible to classify the authority as "Local" i.e. specific to the node in question or not - in which case the authority details will be replicated to the other nodes within the replication network.

    It's possible to run multiple LDAP authorities that are of mixed type, i.e. using some local authorities along with other authorities that are shared by all nodes. When multiple authorities are used, it's possible to set what order they are checked for users.

    The standard settings are supported for each configured LDAP authority: URL, search base and filter and bind user credentials. Note that the bind user's password cannot be one-way encrypted using a hash function because it must be sent to the LDAP server in plain text, so for this reason the bind user should be a low privilege user with just enough permissions to search the directory for the user being authenticated. Anonymous binding is permitted for those LDAP servers that support anonymous binding.

    Add Authority

    Use the Add Authority feature to add one or more LDAP authorities, either local to the node or connected via WAN. Locally LDAP services are treated as having presedence. When Internally managed users are enabled they are first checked when authenticating users - see Admin Account Precedence

    Procedure for adding an authority:

    1. Login to the admin UI, click on the Security tab.

    2. Click on Add Authority.
      Add auth

      Add Authority


    3. The Authority entry form will appear. Enter the following details:
      Add auth

      Add Authority

      URL
      Enter your authorities URL. You need to include the protocol ldap:// or ldaps://
      Bind User DN
      Enter a LDAP admin user account that will be used to query the authority
      Search Base
      Enter the Base DN, that is the location of users that you wish to retrieve.
      Search Filter
      Optionally add A query filter that will select users based on relevant LDAP attributes. For more information about query filter syntax, consult the documentation for your LDAP server.
      Is Local?
      Tick this checkbox if you want the authority to only apply to the current node and not be replicated to other nodes (which is otherwise done by default).

    4. Click the Add Authority. This will save the authority settings that you have just entered. You can click the Test button to verify that the details will successfully connect to the authority without yet adding the authority.

    5. When running with multiple authorities, you should determine the order by which MultiSite polls the authorities. Use the +- symbols at the end of each authority entry to push it up (+) or down (-) the list.
      order auth

      Order authorities

    Edit Authority

    Modify an existing authorities settings:

    1. Login to the admin UI, click on the Security tab.

    2. Click the edit link on the line that corresponds with the authority that you wish to edit.
      order auth

      Edit authorities link

    3. Update the settings in the popup box, then click Save.
      order auth

      Edit authorities box


    Kerberos Security

    This section covers the basic requirements for integrating SVN MultiSite Plus with your existing Kerberos systems. The procedure requires the following:

    ** Alert! **Time, ladies and gentlemen, please.
    Ensure that time synchronization and DNS are functioning correctly on all nodes before configuring Kerberos. A time difference between a client and the master Kerberos server that exceeds the Kerberos setting (5 mins default) will automatically cause auth failure.

    Configuration

    This procedure assumes that you have already set up your DNS service and master Key Distribution Center.


    1. On each node, add the service principal:
      # kadmin -p root/admin -q "addprinc -randkey HTTP/node1.example.com"
      # kadmin -p root/admin -q "ktadd -k /opt/krb5.keytab HTTP/node1.example.com"
      # chmod 777 /opt/krb5.keytab    
      

    2. Each node should have installed the add-on JCE Java 6 or Java 7 Unlimited Strength Jurisdiction Policy Files". These can be downloaded from Oracle, subject to your local import rules concerning encryption technology. Once downloaded, extract to the the Java security library, i.e.
      $JAVA_HOME/lib/security/

    3. At this point you can install SVN MultiSite Plus on each node. If that's already done, then configure the Kerberos settings under the Security tab.
      kerb

      Edit Kerberos box

      Serivce Principal:
      This unique name for an instance of a service, such as HTTP/node1.example.com
      Keytab Location:
      This is the location of the keytab, a file containing pairs of Kerberos principals and encrypted keys (often derived from the Kerberos password). It's used for logging into Kerberos without being prompted for a password.
      Kerberos Config Location:
      The krb5.conf file contains Kerberos configuration information, including the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm and for Kerberos applications, and mappings of hostnames onto Kerberos realms. Normally, you should install your krb5.conf file in the directory /etc. i.e. /etc/krb5.conf
      Save the settings. Log out. Return to the node in your browser, this time you should login automatically (in this as user sally@EXAMPLE.COM.)

    See Security Reference: Kerberos settings.
    See Configure browsers for Kerberos authentication.

    Nodes

    12. Adding a Node

    To replicate SVN repository data between sites, you first tie the nodes together in the form of a replication network, this process starts with the adding (connecting) of nodes in a process we call induction.

    You can also remove a node.

    ** Alert! **Unique Node Names
    You can't reuse Node IDs. If you have removed a node, you can't create a replacement that uses the old name. The replication network maintains a record of the old node and will block it from reintroduction.

    1. Login to the SVN MultiSite admin console of the new node that you are connecting to your existing servers.
    2. Click on the Nodes tab.
    3. Click on the Connect to node button.
      Acc

      Connect to Node

    4. Enter the details of an existing node. You can get these details from the Setting tab of the existing node.
      Acc

      Enter the details from an existing, connected node.


      Node ID
      This is the name that you gave the Node during installation. If you log into the node in question you can see the Node ID on the title of any screen that you view, it also appears in the logged in message: "Welcome to MultiSite, admin. You are connected to <NODE ID>"
      Location ID
      A unique string this created for each node as unique identifier. You can get this from the node's Settings tab:
      Identifier string for a node

      System Data table, found on the Settings tab.


      Hostname / IP Address
      The IP Address of the node's server.
      DConE Port
      The TCP port that the node uses for DConE, which handles agreement traffic. The default is 6444 See Reserved Ports.
    5. Click on the SEND CONNECTION REQUEST button. The new node will appear on the active list of Sites.
      Should a problem occur you may find that the new node gets stuck in a 'pending' state. If this happens see If Induction fails.

    13. Removing a Node

    The removal of a node from the SVN MultiSite Plus replication group is useful if you will no longer be replicating repository data to its location and wish to tidy up your replication group settings.

    ** Alert! **No ties allowed
    The option to remove a node should only appear if it is not currently a member of a replication group. You may need to remove and recreate replication groups in order make it eligible for removal.

    Known issue:
    NOTE: If a node is inducted but not in a replication group then it is possible (from that node) to remove other inducted nodes that are in a replication group. There's currently an issue in that a node isn't aware of the membership of replication groups of which it is not itself a member. This means that it is possible to remove a node that is a member of a replication group, if done from another node that doesn't have knowledge of the replication group.

    Until we block this capability you should do a manual check of any nodes that you plan to remove to make absolutely sure that it is not a member of a replication group.

    ** Alert! **Once removed a node can't come back
    Take care when removing nodes. In order to ensure that replication network is kept in sync, removed nodes are barred from being re-inducted. The only way that you can bring back a node is to perform a reinstallation of SVN MultiSite Plus using a new Node ID.

    14. Stopping nodes

    It's possible to bring all nodes to a stop through the use of a single button click (providing all associated repositories are replicating/writable).

    ** Alert! **A stop can't be synchronized if associated repositories are Local Read-only
    Before starting a Sync Stop All, make sure that none of your nodes have repositories in a Local Read-only state.

    Here's how:

    1. Log into the admin UI and click on the Nodes tab.

    2. Click the Sync Stop All button.
      Acc

      Stop all nodes.


      You'll get a 'growl' message confirming the stop has been triggered. You'll see the results on refreshing your browser session.
      Acc

      Stopped!


    3. On the Node table all nodes will show as Stopped. In this state it's possible to perform maintenance or repairs without risking your replication getting out-of-sync.
      Acc

      Node removed.

    4. The Sync Stop All button has changed to Start all, however, it is possible to start selected nodes by logging in to the admin console of each node that you want to start. Use the Start Node link that appears in the Action column of the nodes table.
    ** Alert! **Important!
    Bringing your replication to a global stop is not a trivial business. We strongly recommend that you take the time to watch and confirm that all nodes report as stopped. If you suspect that one or more nodes are not going to stop you should investigate immediately:

    On the dashboard in the Replicator tasks widget if a repository has gone local Read-only before or during your 'SyncStop all' the stop will fail without any specific error message, you'll just observe that the nodes aren't stopped.

    In the tasks widget you might get:
    Aborted tasksType PREPARE_COORDINATE_STOP_TASK_TYPE
    Delete Task
    Originating Node: Ld5UYU
    tasksPropertyTASK_ABORTING_NODE: Ld5UYU
    tasksPropertyTASK_ABORT_REASON: One or more replicas is already stopped.
    The replica was: [[[Ld5UYU][bf0c6395-77b6-11e3-9990-0a1eeced110e]]]
    
    The thing you would look for is the message:
    Aborted tasksType PREPARE_COORDINATE_STOP_TASK_TYPE
    in the replicator.log file you might also see the following error type:
    "DiscardTaskProposal <task id etc> message: One or more replicas is already stopped."
    

    15. Starting nodes

    1. If all nodes have been brought to a stop, click the Start All button to start them replicating again.
      Acc

      Stopped!


    2. After a browser refresh, all nodes will now show as running.

    Replication Groups

    16. Adding a Replication Group

    Use the procedure to add a new Replication Group. You need to add a new replication group when you need to replicate between a new combination of sites - i.e. sites that are not currently replicating in an existing group. If you are, instead, looking to replicate a new repository between existing sites, it's possible to add a new repository to those sites. In this case see Add a new repository.


    1. Log in to the SVN MultiSite browser-based user interface. Click on the REPLICATION GROUPS tab, then click on the CREATE REPLICATION GROUP button.
      Create the repo group

      Creating a replication group.


    2. Enter a name for the group in the Replication Group Name field, then click on the drop-down selector on the Add Sites field. Select the sites that you want to replicate between.
      Identifier string for a node

      replication group details.


      ** Alert! **Replication Ground Rules
      - A node can belong to any number of replication groups.
      - A repository can only be part of a single active replication group at any particular time.
              - It's possible to change membership on the fly, moving a repository between replication groups with minimal fuss.

    3. Click on each node label to set its node type.
      Identifier string for a node

      Click on node labels to change their type.


      ** Alert! **Advice on creating effective replication groups
      For an understanding of some of the ground rules for replication you should read the section Creating resilient Replication Groups.

      Nodes are automatically added to a group as "Active Voters". To understand the differences between the different types of nodes, read Guide to node types

    4. Once all sites are in place and their settings adjusted to your needs, click CREATE REPLICATION GROUP.
      CREATE REPLICATION GROUP

      Create Replication Group.


    5. Newly created replication groups will appear on the Replication Group tab, but only on the admin UI of nodes that are themselves members of the new group.
      GROUPGLOBAL

      The new replication group now appears - if you are logged into one of its constituent nodes.


    17. Deleting replication groups

    It's possible to remove replication groups from SVN MultiSite Plus, although only if they they have been emptied of repositories. Run through the following procedure as an example.

    1. We have identified that replication group "VineyanRepos" is to be removed from SVN MultiSite Plus. We can see that it has a single repository associated with it. Click on the View to see which one.
      Deleting Replication Group

      View


    2. On the Replication Group configuration screen we can see that Repo5 is associated with the group. We can see that currently the Delete Replication Group (VinyardRepos) is disabled. You can follow the link to the repositories page to remove the association.
      Deleting Replication Group

      Repositories


    3. On the repositories screen, click on the associated repository, in this example it's Repo5, then click on the EDIT button.
      Deleting Replication Group

      Select and Edit


    4. On the Edit Repository box, use the Replication Group drop-down to move the repository to a different Replication Group. Then click SAVE.
      Deleting Replication Group

      Edit


    5. Repeat this process until there are no more repositories assoicated with the Replication Group that you wish to delete. In this example VinyardRepos only had a single repository, so it is now empty, and can be deleted. Click on View, then on Configure.
      Deleting Replication Group

      Move it


    6. Now that Replication Group VinardRepos is effectively empty of replication payload the Delete link is enabled. Click on the link Delete Replication Group (VinyardRepos) to remove the replication group, taking note that there's no undo - although no data is removed when a replication group is deleted, it should be easy enough to recreate a group if necessary.
      Deleting Replication Group

      Click the Delete link button



    7. A growl will appear confirming that the replication group has been deleted.
      Deleting Replication Group

      Deleting the replication group


    18. Adding a node to a replication group

    ** Alert! **Don't add a node during a period of high replication load
    When adding nodes to a replication group that already contains three or more nodes, ensure that there isn't currently a large number of commits being replicated.

            Adding a node during a period of high traffic (heavy level of commits) going to the repositories may cause the process to stall.

    It's possible to add additional nodes to an existing replication group, so that there's minimal disruption to users. Here's the procedure:

    1. Login to a node, click on the REPLICATION GROUPS tab. Go to the replication group to which you will add a new node, click on its VIEW.
      Add node

      Replication Groups


    2. The replication group screen will appear. Click Add Nodes.
      Add Site

      View the group settings


    3. tipWhy the Add Nodes button is disabled?
      The Add Nodes button may be greyed out if the current replication group configuration won't support the addition of a new voter node.
      Add node
      It is also possible that a configuration that is scheduled in the future may block the addition of a new node. Check the schedule if you think that you should otherwise be able to add a new node to the replication group.

    4. Select the node that you wish to add to the replication group.
      Add node

      Select


    5. When there are no further nodes to add to the group, click on the Add Nodes button.
      Add node
    6. At this stage in the process we're ready to select a Helper node from which we'll synch repository data to the new node - select a Helper Node.
      Add node

      Helper node


    7. Heed the warning about not closing the browser or logging out during this process otherwise you'll need to perform a more lengthy repair procedure. Click the Start Sync button.
      Add node

      start sync


    8. You've reached the stage in the process where you need to manually synchronize the repositories from the helper node (which is places temporarily offline for users until this process is finished. See our guide: 30 Synchronizing repositories using rsync.

      The process lets you do a complete sync or select specific repositories that you wish to sync. Assuming that you have synced all repositories you would click Complete All. The helper node is then released from the process, allowing it to catch up with any transactions it missed while taking part in the procedure.
      Add node

      complete all


    9. A growl message will appear copnfirming that the new node has been added to the replication group.
      Add node

      new node!


    10. Returning to the Replication Group screen, you can see the new node count.
      Add node

      Adding new node complete!


    19. Removing a Node from a Replication Group

    It's possible to remove a node from a replication group. This functionality is required if the developers at one of your nodes are no longer going to contribute to the repositories handled by a replication group. Removing a node from a replication group will halt further updates to its repository replicas.

    tipRemove stray repositories
    In the event that you remove a node from a replication group, you should delete its copy of the repositories managed by the replication group. Having an out-of-date stray copy could result in confusion/users working from old data.

    You will not be allowed to remove a node that is currently assigned as the "Managing Node". In order to remove the managing node, go to the Configure Schedule page and assign a different node as a Managing Node.


    1. Login to the admin console of one of your nodes. The node will need to be the member of the relevant Replication Group, otherwise it won't appear on the tab. Click on the Replication Groups tab.
      On the Replication Groups tab, click on the View button that corresponds with the Replication Group from which you plan to remove a node. Rmove Node from group 01

      Login and go to REPLICATION GROUPS

    2. Click on the node that you plan to remove from the group. Providing that the removal of the node doesn't invalidate the remaining configuration you will see a Remove node from replication group link. Click the link.
      Remove Node from group 03

      Remove!

    3. A dialog will open which asks you to confirm the removal of the selected node from the Replication Group. Click Remove. Rmove Node from group 04

      Remove. Really!

    4. A growl message will confirm that the removal is in progress. You many need to click the Reload button to ensure that the action has been completed on all nodes. Rmove Node from group 04

      Reload to confirm the updated state.

    5. The node will now be removed from the Replication Group. On the Replication Groups panel you should now see that the constituent number of nodes has reduced by one. Rmove Node from group 06

      Less one member node

    20. Scheduling node changes - follow the sun

    You can schedule the member nodes of a replication group to change type according to when and where it is most beneficial to have active voters. To understand why you may want to change your nodes read about Node Types


    Schedule node type changes via the public API

    Instead of manually setting up schedules through a node's UI you can do it programmatically through calls to the public API.
    See Public API ScheduledNodeAPIDTOList element and scheduledNodeAPIDTOList Datatype

    Use the following API call

    http://<ip>:8082/public-api/replicationgroup/{repgroupID}/schedule
    
    e.g.
    http://10.0.100.135:8082/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule
    
    PUT with ReplicationGroupAPIDTO XML as body:

    To make Node N3 a tie-breaker 'T' FROM 10:00 - 16:00 (GMT) every day of the week with Node N1 as tie-breaker 'T' afterwards:

    tipTimes are always in UTC (GMT)
    When viewed on a node times are shifted to the local timezone although internally they are always recorded in UTC.

    Example curl command:

    Make a text file containing ReplicationgroupAPIDTO XML (as above) called schedule.xml

    curl -u username:password -X PUT -d @schedule.xml http://[IP]:[PORT]/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule

    Sample 'schedule.xml' file

    <ReplicationGroupAPIDTO>
           <replicationGroupName>global</replicationGroupName>
         <replicationGroupIdentity>97913c04-bbad-11e2-877a-028e03094f8d</replicationGroupIdentity>
           <scheduledNodes>
               <dayOfWeek>1</dayOfWeek>
               <hourOfDay>14</hourOfDay>
               <schedulednode>
                   <nodeIdentity>N1</nodeIdentity>
                   <locationIdentity>c0e486a0-bbab-11e2-863b-028e03094f8e</locationIdentity>
                   <isLocal>true</isLocal>
                   <isUp>true</isUp>
                   <lastStatusChange>0</lastStatusChange>
                   <role>AV</role>
               </schedulednode>
               <schedulednode>
                   <nodeIdentity>N3</nodeIdentity>
                   <locationIdentity>5480f515-bbad-11e2-8301-028e03094f8c</locationIdentity>
                   <isLocal>false</isLocal>
                   <isUp>true</isUp>
                   <lastStatusChange>0</lastStatusChange>
                   <role>T</role>
               </schedulednode>
               <schedulednode>
                   <nodeIdentity>N2</nodeIdentity>
                   <locationIdentity>478c766f-bbad-11e2-877a-028e03094f8d</locationIdentity>
                   <isLocal>false</isLocal>
                   <isUp>true</isUp>
                   <lastStatusChange>0</lastStatusChange>
                   <role>AV</role>
               </schedulednode>
          

    Download the full sample schedule.xml file.

    1. Login to a node, click on the REPLICATION GROUPS tab. Click on the VIEW link for the replication group that you wish to make a schedule.
      Schedule for you

      Scheduling is done through replication group settings.


    2. The replication group's pop-up window will open, showing the member nodes together, along with their current (scheduled) roles. Click the CONFIGURE button.
      Schedule for you

      Configure.

      tip" Membership views are what is scheduled not necessarily what is currently active
      The roles and membership displayed in the popup is based upon the agreed schedule, it's the setup that should be in place if everything is running smoothly. It is always possible that it doesn't accurately represent the state of the replication group, due to a delay in processing on a node, or if something has caused a process to hang. This should not be a cause for concern but it's important to be aware that the displayed membership is an approximation based on the information currently available to the local node.


    3. The replication groups configuration screen will appear. You may notice that to the left a Role Schedule is noted. By default this will show as DISABLED. Click on the Configure Schedule button, in the right-hand corner.
      Schedule for you

      Role Schedule: Disabled (for now).


    4. The Schedule screen will appear. The main feature of the screen is a table that lists all the nodes in the replication group, set against a generic day (midnight to midnight) that is divided into hourly blocks. Each hourly block is color-coded to indicate the specific node's type.

      In the image below NodeSanFrancisco is coded as blue which indicates that it is set as a Passive Voter. The hourly blocks associated with NodeChengdu are Magenta, indicating that it is set as a pure voter. The blocks for NodeParis are colored yellow, indicating that this node is set as an Active Voter.


      Schedule for you

      Vanilla Scheduling - no changes to type over time.



    5. To make a change to the schedule, click on a block. It doesn't matter which block you select as the New Scheduled Configuration form will let you modify any hours for any available node.
      Schedule for you

      New Schedule Form.


        Frequency
        Select from the available frequency patterns: Daily, Weekly, Monday-Friday or Saturday to Sunday.
        From
        The starting hour for the new schedule, e.g. 00 for the start of the day.
        To
        The hour at which the scheduled changes end, e.g. 24 would effectively end the scheduled change at midnight.
        Node list
        The member nodes are listed, in graphical form, colour coded to their type.
    6. Click on the node icon to change its type.

      In this example NodeSanFransisco is changed to a Tie-breaking Passive Voter, then NodeAuckland is changed into a Tie-breaker.

      Schedule for you

      Swapping roles.

      When all node changes have been made, click on the SAVE button to continue, or the CANCEL button if you change your mind.

    7. The schedule view will now change to show the changes that you make. You must click the Save Schedule button for the changes to be applied.
      With all necessary changes made, you need to review the change to the schedule table and then click SAVE SCHEDULE button.
    8. ** Alert! **Changing role of the managing node
      It's currently not possible to change the role of the node that is assigned as the managing node.
      If you need to change a node's role, first make a different node the manager. This restriction was intended stop the managing node from being given a non-active role. Not only would this stop the node from managing schedule changes, it would make it impossible to move the managing node status to another node.

      In a future release we may be able to make it possible to change the managing node's role to another compatible role, e.g. from Active Voter to Active.

    Repositories

    21. Adding Repositories

    You can add additional repositories for replication through the admin UI. The repository first needs be present on all the nodes that will be part of the corresponding replication group. So the repository copies need to be introduced to the replication system in an identical state.

    1. Ensure that the repository that you are going to replicate is copied to each node that is a member of the replication group that will be responsible for the repository. The repository copies must be in an identical state before you add them into SVN MultiSite Plus.

    2. Log in to the SVN MultiSite Plus admin UI. Click on the Repositories tab.
      Add Repository 01

      Login.


    3. Click on the Add button and enter the repository information:

      Add Repository 02

      Enter repository details.

      Repo name
      A name that you want SVN MultiSite Plus to use when referring to the new repository.

      tipKnown issue: duplicate repository names allowed
      It's currently possible to add multiple repositories with the same name (they'll need different paths though). Ensure that you don't use the same name for multiple repositories, this is for obvious reasons a bad practice and will be prevented in future releases.

      FS Path
      The absolute file system path for the repository. This should be identical on each node.
      Replication Group
      The replication group under which this repository will be managed. Select from available groups or go and create a new replication group.
      Global Read-Only
      This check-box lets you add the repository in a locked-down state that won't allow SVN users to commit changes to the repository. This feature is useful for putting a repository into maintenance mode where the copies might otherwise get out-of-sync.
      When all entry fields have been filled, click on the Add Repo button. You'll then see a 'growl' message confirming that the repository has been deployed and that you should reload your browser session to confirm that the repository has been added.

    4. Recheck the repositories table to see that the new repository has been added and has a "Replicating" status.
      Add Repository 03

      Replicating.

    tipRepository stuck in Pending state
    If a repository that you added gets stuck in the deploying state - you'll see this on the Dashboard, in the Replicator Tasks window - you can cancel the deployment and try adding the repository again. To cancel a deployment, go to the Replicator Tasks window and click on the Cancel Task link.

    ** Alert! **svnadmin pack support
    It's not currently possible to run the svnadmin pack command when running SVN MultiSite Plus. Support for this command is currently being added to FSFSWD and should be available in the near future.

    22. Removing Repositories

    It's possible to remove repositories from SVN MultiSite Plus. Follow this quick procedure.

    1. Login to the admin console of one of your nodes. The node will need to be the member of a replication group in which the repository is replicated, otherwise it won't appear on the tab. Click on the Repositories tab to see it.
      Rmove Repository 01

      Login.

    2. On the Repositories tab, click on the line that corresponds with the repository that you want to remove.
      Schedule for you

      Repositories.

    3. Once a repository has been highlighted (in yellow), the REMOVE button will become available. Click it. Schedule for you

      Remove.


      A dialog box will appear entitled "Remove repository from replication group". It will confirm that removing a repository from a replication group will stop any changes that are made to it from being replicated. However, no repository data is removed.

    23. Editing a repository

    It's possible to edit a repository's properties after they have been set up in SVN MultiSite Plus. Follow this quick procedure.

    1. Login to the admin console of one of your nodes. The node will need to be the member of a replication group in which the repository is replicated, otherwise it won't appear on the tab. Click on the Repositories tab to see it.
      Rmove Repository 01

      Login.


    2. On the Repositories tab, click on the line that corresponds with the repository that you want to edit. Then click the Edit button.

      Schedule for you

      Repositories.


    3. You will now see the edit window appear.
      Schedule for you

      Edit Repository.

      Local Read-only
      Changes the Read-only setting, enable or disable the repository Local Read-only setting. When enabled, the repository will not be writable, either for local users or for the replication system (that would push changes made to the repository on other nodes). However, changes that come from the other nodes are stored away to be played out as soon as the read-only state is removed.
      Global Read-only
      Changes the Read-only setting, enable or disable the repository Global Read-only setting. When enabled, the repository will not be writable either locally or globally. This is used to lock a repository from any changes.
      Replication Group
      Use the drop-down selector to change the replication group to which the repository belongs.

    24. Repository synchronized stop

    The Repository Synchronized Stop is used to stop replication between repository replicas, it can be performed on a per-repository basis or on a replication group basis (where replication will be stopped for all associated repositories). To bring some or all nodes to a stop, use the Sync Stop All command found on the Nodes tab.
    Repository Stops are synchronized between nodes using a 'stop' proposal to which all nodes need to agree. So that while not all nodes will come to a stop at the same time they do all stop at the same point.

    1. Login to a node's browser-based UI and click on the Repositories tab. Click on the repository that you wish to stop replicating.
      tip"
    2. With the repository selected, click the Sync Stop button. A growl message will appear to confirm that a synchronized stop has been requested. Note that the process may not be completed immediately, especially if there are large proposals transferring over a WAN link.
      tip"
    3. On refreshing the screen you will see that a successfully sync stopped repository will have a status of Stopped and will be Local RO (Locally Read-only) at all nodes.

    25. Repository synchronized start

    Restarting replication after performing a Synchronized Stop requires that the stopped replication be started in a synchronized manner.

    1. Click on a stopped repository and click on the Sync Start button.
      tip"
    2. The repository will stop being Local Read-only on all nodes and will restart replicating again.

    26. Logs

    SVN MultiSite Plus has a number of points where SVN and replication events are logged:

    Admin UI: Growl messages
    The growl messages provide immediate feedback in response to a user's interactions with the Admin UI. Growls are triggered only by local events and will only display on the node (and in the individual browser session) in which the event was triggered.
    Schedule for you
    Growl messages appear in the top right-hand corner of the screen and will persist for a brief period (15 seconds in most cases) or until the screen is refreshed or changed.

    tipAlways check the dashboard
    If you are troubleshooting a problem we strongly recommend that you check the Dashboard's Replicator Tasks panel as well as the log files. While we added the gowl messaging as way giving administrators an immediate alert for events as they happen, they are not intended to be used as the main method of tracking failures or important system events.

    Dashboard: Replicator Tasks
    Events that are more complex and are not bound by user interactions may appear on the Dashboard's Replicator Tasks. Tasks may consist of a simple statement or (with a click on the Task name) a multi-line report. Schedule for you
    Application Logs
    Read more about Application logs
    Replication Logs
    Read more about Replication logs

    SVN MultiSite Plus has two sets of logs, one set is used for application, the other logs replication activity:

    Application Logs

    /opt/wandisco/svn-multisite-plus/

    The general logs are chiefly produced by the watchdog process and contain messaging that is mostly related to getting SVN MultiSite Plus started up and running. replicator -- logging the startup etc of the replicator ui -- startup/everything to do with the UI, inc in-use logging. lightweight.
    -rw-r--r-- 1 wandisco wandisco   88 Jan 15 16:53 multisite.log
    -rw-r--r-- 1 wandisco wandisco  220 Jan 15 16:53 replicator.20140115-165324.log
    -rw-r--r-- 1 wandisco wandisco 4082 Jan 15 16:53 ui.20140115-164517.log 
    -rw-r--r-- 1 wandisco wandisco 1902 Jan 15 16:53 watchdog.log
    
    multisite.log
    Basic events that relate to the starting up/shutting down of SVN MultiSite Plus.
    e.g.
            2014-01-15 16:45:17: [3442] Starting ui
            2014-01-15 16:53:24: [3571] Starting replicator
            
    replicator.yyymmdd-hhmmss.log
    Events relating to the start up and shutdown of the replicator, and also logging. This log never includes information about the actual operation of the replicator, for that you need to see the fsfswd.x.log files located in the replicator's own logs directory (see below)
    watchdog.log
    Logs the running of the watchdog process which monitors and maintains the running of the SVN MultiSite processes.

    Replicator Logs

    The logging system has been implemented using Simple Logging Facade for Java (SLF4J) over the log4J Java-based logging library. This change from java.util.logging has brought some benefits:

    This change lets us collate data into specific package-based logs, such has a security log, application log, DConE messages etc.

    Logging behavior is mostly set from the log4j properties file. /svn-multisite-plus/replicator/properites/log4j.properties

    # Direct log messages to a file
    log4j.appender.file=com.wandisco.vcs.logging.VCSRollingFileAppender
    log4j.appender.file.File=fsfswd.log
    log4j.appender.file.MaxFileSize=100MB
    log4j.appender.file.MaxBackupIndex=10
    log4j.appender.file.layout=org.apache.log4j.PatternLayout
    log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
    log4j.appender.file.append=true
    
    # Root logger option
    log4j.rootLogger=INFO, file
    

    This configuration controls how log files are created and managed.