Admin basics

1. Starting up

To start the SVN MultiSite Plus replicator, follow these steps:

  1. Open a terminal window on the server and login with suitable file permissions.
  2. Run the svn-multisite service, located in the init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 svn-multisite -> /opt/svn-multisite-plus/bin/svn-multisite
  3. Run the start script:
    [root@localhost init.d]#  ./svn-multisite start
    20130520-164811 (24088) [INFO]: Starting WANdisco MultiSite Plus 20130520-164811 (24088) [INFO]: Started replicator (24100) 20130520-164811 (24088) [INFO]: Started ui (24110) 20130520-164811 (24088) [INFO]: Number of errors: 0 20130520-164811 (24088) [INFO]: Number of warnings: 0
  4. The two components of SVN MultiSite Plus; the replicator and the UI will start up. Read more about the svn-multisite init.d script

2. Shutting down

To shutdown:

  1. Open a terminal window on the server and login with suitable file permissions.
  2. Run the svn-multisite service, located in the init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 svn-multisite -> /opt/svn-multisite-plus/bin/svn-multisite
  3. Run the stop script, i.e.:
    [wandisco@ip-10-0-100-7 bin]$  ./svn-multisite stop
    20130520-165704 (24767) [INFO]: Stopping WANdisco MultiSite Plus
    20130520-165704 (24767) [INFO]: Request received to shut down replicator
    20130520-165704 (24767) [INFO]: replicator processes ended
    20130520-165704 (24767) [INFO]: Request received to shut down ui
    20130520-165704 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 1)...
    20130520-165707 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 2)...
    20130520-165710 (24767) [INFO]: ui processes ended
    20130520-165710 (24767) [INFO]: Number of errors: 0
    20130520-165710 (24767) [INFO]: Number of warnings: 0
  4. Both the replicator and the UI processes will shut down. Read more about the svn-multisite init.d script

3. Using the init.d script

The 'start-up' script for persistent running of SVN MultiSite can be found in the /etc/init.d folder. Run the script with the help command to list the available commands:

[root@localhost init.d]# ./svn-multisite help
Usage: svn-multisite {start|stop|status|uistart|uistop|repstart|repstart}

  start      Start WANdisco Multisite service
  stop       Stop WANdisco Multisite service
  status     Display running WANdisco Multisite services
  uistart    Start WANdisco Multisite UI service
  uistop     Stop WANdisco Multisite UI service
  repstart   Start WANdisco Multisite Replicator service
  repstop    Stop WANdisco Multisite Replicator service    

4. Changing the admin console password

You can change SVN MultiSite Plus's login password at any time by following this procedure:

  1. Login to the MultiSite admin console.
    Schedule for you


  2. Click on the Settings tab.
    Schedule for you


  3. At the top of the settings screen is the password change form. Enter the current password, along with a new password.
    Schedule for you

    Changed password

  4. Click the SAVE button store the new password. You can be sure that the new password has been accepted if you see a growl message appear at the bottom of sf.
    Schedule for you


** Alert! **Changing Username
It's currently not possible to change the Administration username. In order to change the username you would need to re-install SVN MultiSite Plus

5. Updating your license.key file

Follow this procedure if you ever need to change your product license. You would need to do this if, for example, you needed to increase the number of subversion users or the number of replication nodes.

  1. Login to your server's command line, navigate to the replicator directory (by default: /opt/svn-multisite-plus/replicator ) and rename the license.key to license.20130625.
    total 1120
    drwxr-xr-x 2 root root   4096 Jun 19 17:21 content
    drwxr-xr-x 5 root root   4096 Jun 19 17:21 database
    drwxrwxr-x 4 root root   4096 Jun 19 17:13 docs
    drwxrwxr-x 2 root root   4096 Jun 19 17:13 lib
    -rw-r--r-- 1 root root    256 Jun 19 17:18 license.key <-
    drwxrwxr-x 3 root root   4096 Jun 25 14:06 logs
    drwxrwxr-x 2 root root   4096 Jun 19 17:13 properties
    -rwxrwxr-x 1 root root 369538 Jun 18 16:37 svn-ms-replicator-fsfsrestore.jar
    -rwxrwxr-x 1 root root 369526 Jun 18 16:37 svn-ms-replicator.jar
    -rwxrwxr-x 1 root root 369535 Jun 18 16:37 svn-ms-replicator-updateinetaddress.jar
  2. Get your new license.key and drop it into the /opt/svn-multisite-plus/replicator directory.
  3. Restart the replicator by running the SVN MultiSite script with the following argument:
    perl /etc/init.d/svn-multisite repstart
    This will trigger an SVN MultiSite replicator restart, which will force SVN MultiSite to pickup the new license file and apply any changes to permitted usage.
If you run into problems, check the replicator logs (/opt/svn-multisite-plus/replicator/logs ) for more information.
PANIC: License is invalid com.wandisco.fsfs.licensing.LicenseException: Failed to load filepath>

6. Update a node's properties

It's possible to update a node's IP address/Hostname or assigned ports (for either DConE or content delivery). Use the following procedure:

  1. On the node that you want to change, open up the file. you can find this file here:
    You need to change either the of these lines in the file:
    After updating any or all of these properties, save the file.

  2. On each of the other nodes create a "payload" properties file that contains the changed information. Create the file in the "replicator" subdirectory, call it "update.properites" and make sure that it contains the relevant lines:

    To change the IP Address (or hostname):
    event.hostname.<Node id>=<new IP or hostname>
    content.hostname.<Node id>=<new IP or hostname>

    ** Alert! **Case sensitive!
    The Node ID is case sensitive, you must make sure that you match the nodes actual case as the system won't currently warn you if you get it wrong.

    To change the DConE port:
    event.port.<Node id>=<new port>
    To change the Content Delivery port:
    content.port.<Node id>=<new port>

  3. Once the file is in place, run the following command (on all the nodes except the one you have changed):
     java -jar svn-ms-replicator-updateinetaddress.jar -c <path to>

  4. Go back to the node with the updated properties and Restart MultiSite.

  5. You should login to the updated node and check its System Data (at the bottom of the Settings tab. You should do some test commits to ensure that replication continues successfully.

7. Setting up data monitoring

The Monitoring Data tool monitors the disk usage of SVN MultiSite's database directory, providing a basic level of protection against SVN MultiSite consuming all disk space. The tool also lets you set up your own monitors for user-selected resources.

** Alert! **Monitoring Data - not intended as a final word in system protection
Monitoring Data is no substitute for dedicated, system-wide monitoring tools. Instead, it is intended to be a 'last stand' against possible disk space exhaustion that could lead to data loss or corruption.

Default settings

Resource Monitor Tool

Click the "View" link to go to a monitor's settings.

By default MultiSite's database directory (/opt/wandisco/svn-multisite-plus/replicator/database) is monitored - this is the location of MultiSite's prevayler database where all data and transactions files for replication are stored.

Resource Monitor Tool

There is built-in monitor for /opt/wandisco/svn-multisite-plus/replicator/database which can not be deleted.

This built-in monitor runs on all nodes. Any additional monitors that you set up will monitor on a per-node basis. Monitors are not replicated so a monitor set up on one node is not applied to any other node.

Set up additional monitors

Create additional resource monitors using the following procedure:

  1. Login to the Administrator user interface.

  2. Click the "SETTINGS" link on the top menu bar.

  3. Monitoring Data is situated below the Administrator Settings. Enter the full path to the resource that you wish to monitor. For example, you might wish to monitor the replicator logs: "/opt/svn-multisite-plus/replicator/logs". Enter the path and click "Add".
    Resource Monitor Tool

    Add resource path

  4. The new resource monitor will appear as a new box - it will display "No records found", indicating that it doesn't yet have any monitoring rules set. Click its corresponding "Configure" link.
    Resource Monitor Tool


  5. The screen will update to show the Resource Monitoring screen for your selected resource.
    Resource Monitor Tool


  6. File Path:
    The full path for your selected resource
    Monitor Indentity:
    The unique string that will identify the monitor
    Edit Condition and Event List
    Lists current resource monitors, initially this will state "No records found"

    Add Condtional and Event to List

    Storage amount entry field
    Enter an amount of disk space in Gigabytes. e.g. 0.2 would be equal to 200 Megabytes of storage.
    Select an Event from the dropdown:
    - will initiate a shutdown of SVN MultiSite Plus and will also write a message to the log and the "SEVERE" logging level. See "When a Shut down is triggered" for more information.
    - will write a message to the log and the "WARNING" level of severity.
    - will write a message to the log and the "DEBUG" level of severity.
    - will write a message to the log and the "INFO" level of severity.
  7. When you have added all the trigger points and events that you require for the resource, click "Update". You can then navigate away - Click on "Resource Monitoring" on the breadcrumb trail to return to the settings screen.

When a Shutdown is triggered

If the disk space available to a monitored resource is less than the value you have for a "Severe" event then the event is logged and MultiSite's replicator will shut down after a set interval of 10 minutes. You can configure the interval in file:

value in minutes!

** Alert! **Edits to property files require a replicator restart
Any change that you make to the file will require that you restart SVN MultiSite Plus's replicator.

Once shut down all Subversion repositories will become unavailable to users, you should immediately take action to make more disk space available, the replicator can be restarted using SVN MultiSite Plus's service as soon as the resource that triggered the shutdown has enough available disk space not to shut down again.

Monitor Administration

Monitors that you set up are stored within an XML file within the application's properties directory:


In the event that this file can't be found, a substitute file is created in the temporary file space, "". The default resource will then be applied:

db folder (/opt/wandisco/svn-multisite-plus/replicator/database)

8. Setting up email notifications

The email notification is a rules-based system to deliver alerts based on user-defined templates over one or more channels to destinations based on triggers that are activated by arbitrary system events. Put simply, email notification sends out emails when something happens within the SVN MultiSite Plus environment. Everything, the message content, trigger rules and destinations are all user-definable.

Email Notifications

Automated alert emails

8.1 Set up a Gateway

The Gateway section stores your email (SMTP) server details. You can set up multiple gateways to ensure that the loss of the server doesn't prevent alert notifications for being delivered.

  1. Log into the admin UI, then click the Settings tab.

  2. Click on the Gateway section of the Notifications area.
    Email Notifications

    Add Gateway

  3. Enter your email gateway's settings:
    Email Notifications

    Enter settings

  4. IP/Hostname of SMTP Server:
    your email server's address.
    SMTP Server Port
    The port assigned for SMTP traffic (Port 25 etc).
    Encryption Type
    Indicate your server's encryption type - None, SSL (Secure Socket Layer) or TLS (Transport Layer Security).
    Authentication Required
    Indicate whether you need a username and password to connect to the server - requires either "true" or "false".
    User Name
    If authentication is required, enter the authentication username here.
    If authentication is required, enter the authentication password here.
    From Address
    Provide an email address that your notifications will appear to come from. If you want to be able to receive replies from notifications you'll need to make sure this is a valid and monitored address.
    Number of Tries Before Failing
    Set the number of attempts SVN MultiSite Plus makes in order to send out notifications.
    Interval Between Tries (Seconds)
    Set the time (in seconds) between your server's attempts to send notifications.

  5. Click on the "+Add" link. Your gateway will apear in a table.
    You can add any number of gateways. SVN Multisite Plus will exhaust the "Number of Tries Before Failing" for each registered gateway before moving on down the list to the next.

8.2 Set up a Destination

The destinations section stores the email address for your notification recipients.

  1. Click on the + on the Destinations line.

  2. Enter an email address for a notification recipient. Click the + Add link.
    Email Notifications


  3. The destination will appear in a table. Click the Edit or Remove links to change the address or remove it from the system.

8.3 Set up a Template

The template section is used to store email messages. You can create any number of templates, each with its own notification message, triggered by one of a number of trigger scenarios that are set up in the Rule section.

  1. Click on the + on the Template line.

  2. Enter a Template Subject line, this will be the subject of the notification email.

  3. Enter some Body Text, this will be the message that is sent out when the notification is triggered. The message has a 1024 character limit, you can track the available number of characters at the bottom of the text box.
    Email Notifications


  4. When the message has been entered, click the + Add link to save the message template.

8.4 Set up a Rule

The Rule section is used to define which system event should trigger a notification, what message template should be used and which recients should be sent the notification.

  1. Click on the + on the Rule line.

  2. Choose an Event from the Event drop-down list: Email Notifications


  3. Event:

    Disk Monitor Warning
    Disk Storage has dropped below the Warning level.This will trigger if any data monitor message is written to the logs. For more information about disk warning messages, see the Setting up data monitoring section.
    Disk Monitor Severe
    Disk Storage has hit the Severe level. This will trigger if any "Severe" level data monitor message is written to the logs. At this level, SVN MultiSite Plus will have shutdown to ensure that disk space exhaustion doesn't corrupt your system and potentially your SVN repositories. For more information about disk warning messages, see the Setting up data monitoring section.

    Deploy Repository Failed
    A repository added to SVN MultiSite Plus has failed to deploy, in which case the repository will not be replicated.
    Deploy Repository Succeeded
    A repository added to SVN MultiSite Plus has successfully deployed. Such an event might be sent to a mail group received by SVN users, telling them that their repository is now accessible.
    Disk Monitor Info
    Disk Storage has dropped below the Info level. This will trigger if any data monitor message is written to the logs at the "INFO" level.

  4. Choose a Template from the drop-down list. These are the templates that you have already set up under the Templates section.

  5. Choose destinations for your notification from the available destination email addresses. You can make multiple selections so that a message is sent to more than one recipient address.

  6. Click on the + Add link to save your rule.
    Email Notifications

    Rule set

9. Backing up SVN MultiSite Plus data

It's possible to back up SVN MultiSite's own database in case you need to quickly restore a node.

** Alert! **Only MultiSite Settings are backed-up
This procedure backs up SVN MultiSite's internal Prevayler database, it doesn't touch your SVN repository data or any other system files (such as Apache configuration, authz files etc.) that you should also be backing up.

    Create a backup of the current installation by invoking the following API call:
    curl --user <username>:<password> -X POST http://[node_ip_address]:8082/dcone/backup
    This will create a backup folder in [INSTALL-DIR]multisite-plus/replicator/db/backup/X.X.X_DConE_Backup directory.

Back up while shut down

(run from within /replicator):
java -cp ./fsfsrestore.jar com.wandisco.fsfs.backup.FsfsBackup -c ./properties/

Use this to back up the current state of all pervailers when SVN MultiSite Plus is shut down - you don't therefore need to start the replicator in order to create a backup of the database.

Back up selected prevailers

(run from within /replicator):
java -cp ./fsfsrestore.jar com.wandisco.fsfs.backup.FsfsClear -c ./properties/

This class clears selected prevaylers only, when the replicator is shut down. It does not clear all database instances, only those that are being restored during restore process.

10. Restore SVN MultiSite Data

Use the following procedure to restore SVN MultiSite Plus settings after reinstalling and starting a node:

  1. Shutdown the node
  2. Run the following jar file:
    java -jar fsfsrestore.jar path/to/ path/to/back-up-folder

    the first argument, path/to/ is

    path/to/backup by default is

    FsfsBackup.class path/to/
  3. Restart the node


11. Adding a Node

To replicate Subversion repository data between sites, you first first tie the nodes together in the form of a replication network, this process starts with the adding (connecting) of nodes in a process we call induction.

You can also remove a node.

** Alert! **Unique Node Names
You can't reuse NodeIDs. If you have removed a node, you can't create a replacement that uses the old name. The replication network maintains a record of the old node and will block it from reintroduction.

12. Removing a Node

The removal of a node from the SVN MultiSite Plus replication network is useful if you will no longer be replicating repository data to its location and wish to tidy up your replication network settings.

** Alert! **No ties allowed
The option to remove a node should only appear if it is not currently a member of a replication group. You may need to remove and recreate replication groups in order make it eligable for removal.

Known issue:
NOTE: If a node is inducted but not in a replication group then it is possible (from that node) to remove other inducted nodes that are in a replication group. There's currently an issue in that a node isn't aware of the membership of replication groups of which it is not itself a member. This means that it is possible to remove a node that is a member of a replication group, if done from another node that doesn't have knowledge of the replication group.

Until we block this capability you should do a manual check of any nodes that you plan to remove to make absolutely sure that it is not a member of a replication group.

** Alert! **Once removed a node can't come back
Take care when removing nodes. In order to ensure that replication network is kept in sync, removed nodes are barred from being re-inducted. The only way that you can bring back a node is to perform a reinstallation of SVN MultiSite Plus using a new Node ID.

13. Stopping nodes

It's possible to bring all nodes to a stop through the use of a single button click (providing all associated repositories are replicating/writable).

** Alert! **A stop can't be synchronized if associated repositories are Local Read-only
Before starting a Sync Stop All, make sure that none of your nodes have repositories in a Local Read-only state.

Here's how:

  1. Log into the admin UI and click on the Nodes tab.

  2. Click the Sync Stop All button.

    Stop all nodes.

    You'll get a 'growl' message confirming the stop has been triggered. You'll see the results on refreshing your browser session.


  3. On the Node table all nodes will show as Stopped. In this state it's possible to perform maintenance or repairs without risking your replication getting out-of-sync.

    Node removed.

  4. The Sync Stop All button has changed to Start all, however, it is possible to start selected nodes by logging in to the admin console of each node that you want to start. Use the Start Node link that appears in the Action column of the nodes table.

14. Starting nodes

  1. If all nodes have been brought to a stop, click the Start All button to start them replicating again.


  2. After a browser refresh, all nodes will now show as running.

Replication Groups

15. Adding a Replication Group

Use the procedure to add a new Replication Group. You need to add a new replication group when you need to replicate between a new combination of sites - i.e. sites that are not currently replicating in an existing group. If you are, instead, looking to replicate a new repository between existing sites, it's possible to add new a new reposity to those sites. In this case see Add a new repository.

  1. Log in to the Subversion MultiSite browser-based user interface. Click on the REPLICATION GROUPS tab, then click on the CREATE REPLICATION GROUP button.
    Create the repo group

    Creating a replication group.

  2. Enter a name for the group in the Group Name field, then click on the drop-down selector on the Add Sites field. Select the sites that you want to replicate between.
    Indentifer string for a node

    replication group details.

    ** Alert! **Replication Ground Rules
    - A node can belong to any number of replication groups
    - A repository can only be part of a single Active replication group at any particular time.
            - It's possible to change membership on the fly, moving a repository between replication groups with minimal fuss.

  3. Click on each node label to set its node type.
    Indentifer string for a node

    Click on node labels to change their type.

    ** Alert! **Advice on creating effective replication groups
    For an understanding of some of the ground rules for replication you should read the section Creating resilient Replication Groups.

    Nodes are automatically added to a group as "Active Voters". To understand the differences between the different types of nodes, read Guide to node types

  4. Once all sites are in place and their settings adjusted to your needs, click CREATE REPLICATION GROUP.

    Create Replication Group.

  5. Newly created replication groups will appear on the Replication Group tab, but only at the node that are themselves members of the new group.

    The new replication group now appears - if you are logged into one of its constituent sites.

16. Deleting replication groups

It's possible to remove replication groups from SVN MultiSite Plus, although only if they they have been emptied of repositories. Run through the following procedure as an example.

  1. We have identified that replication group "VinyardRepos" is to be removed from SVN MultiSite Plus. We can see that it has a single repository associated with it. Click on the View to see which one.
    Deleting Replication Group


  2. Click on Configure.
    Deleting Replication Group


  3. On the Replication Group configuration screen we can see that Repo5 is associated with the group. We can see that currently the Delete Replication Group (VinyardRepos) is disable. You can follow the link to the repositories page to remove the association.
    Deleting Replication Group


  4. On the repositories screen, click on the assoicated repository, in this example it's Repo5, then click on the EDIT button.
    Deleting Replication Group

    Select and Edit

  5. On the Edit Repository box, use the Replication Group drop-down to move the repository to a different Replication Group. Then click SAVE.
    Deleting Replication Group


  6. Repeat this process until there are no more repositories assoicated with the Replication Group that you wish to delete. In this example VinyardRepos only had a single repository, so it is now empty, and can be deleted. Click on the View, then on the Configure.
    Deleting Replication Group

    Move it

  7. Now that Replication Group VinardRepos is effectly empty of replication payload the Delete link is enabled. Click on the link Delete Replication Group (VinyardRepos) to remove the replication group, taking note that there's no undo - although no data is removed when a replication group is deleted, it should be easy enough to recreate a group if necessary.
    Deleting Replication Group

    Click the Delete link button

  8. A growl will appear confirming that the replication group has been deleted.
    Deleting Replication Group

    Deleting the replication group

17. Adding a node to a replication group

** Alert! **Don't add a site during a period of high replication load
When adding sites to a replication group that already contains three or more sites, ensure that there isn't currently a large number of commits being replicated.

        Adding a site during a period of high traffic (heavy level of commits) going to the repositories may cause the "adding node" process to stall.

It's possible to add additional sites to an existing replication group, so that there's minimal disruption to users. Here's the procedure:

  1. Login to a site, click on the REPLICATION GROUPS tab. Go to the replication group to which you will add a new site, click on its VIEW.
    Add Site

    Replication Groups

  2. The replication group pop-up will appear. Click CONFIGURE.
    Add Site

    Edit the replication group

  3. On the replication group configuration screen you can see the existing sites. Click the ADD SITES button. Click on one or more of the available sites to add them. It is possible to bring in a brand new site that isn't currently connected by clicking on CONNECT NEW SITE, this will take you to the SITES tab.
    Add Site

    Add a site

  4. Any new node(s) that you add will now appear on the screen as a blue icon tab. You'll be presented with the following information:
    Adding Nodes
    You are about to add "NodeAuckland" to the "ReplicationGroupGladius" replication group.
    Please read through the following steps before you continue:
    1. Click ADD NODES and choose an existing Node as a helper.
    2. Copy the repositories replicated in the group from the helper to the new node. During this period these repositories will be read-only on the helper node.
    3. Once a repository is in place, selecting it and clicking COMPLETE SELECTED will take it out of read-only on the helper and the new member nodes.
    WARNING: Do not close your browser or log out during this process. If you do you'll need to complete the sync process for each individual repository via the REPAIR option on the repositories screen.
    Click ADD NODES to confirm your selection and proceed with adding them.
    Add Site

    Adding Nodes process

  5. Select one of the existing sites to be the 'helper' from the CHOOSE HELPER NODE drop-down selector. This helper site will be used to supply copies of the relevant repositories to the new site. During the helper phase the relevant repositories will be temporarily read-only for their local users.
    Add Site

    Helper chosen

  6. Click START SYNC to begin your copying process. Once you click the button the helper node will's relevant repositories will bcome read-only to local users.

    ** Alert! **READ-ONLY ALERT
    Starting the Sync may cause a brief disruption to users on the helper node. You may wish to alert them or complete the work at a time where the disruption will be minimal.

    Add Site

    Start Sync

  7. The helper site is now ready for you to start copying data across. You can manually copy the repository files or you can use rsync - see our guide: 11. Synchronizing repositories using rsync.
    Add Site

    Complete Sync

    ** Alert! **Always copy, never create
    Manually copy or rsync repositories, even if they're empty. If you instead use svnadmin create the resulting repository will have a universally unique identifier (UUID) that will not match with the other replicas, potentially causing problems.

  8. Once the relevant repositories have been copied to the new site, and the copies have been checked to ensure they're up-to-date, it's time finish the process by clicking COMPLETE ALL.
  9. The new site will now appear as a member of the replication group. It is added as a non-voter by default, to change its type, click on its icon and change it to a Voter or Tie-breaker .
    Add Site


18. Removing a Node from a Replication Group

It's possible to remove a node from a replication group. This functionality is required if the developers at one of your sites are no longer going to contribute to the repositories handled by a replication group. Removing a node from a replication group will halt further updates to its repository replicas.

tip"Remove stray repositories
In the event that you remove a node from a replication group, you should delete its copy of the repositories managed by the replication group. Having an out-of-date stray copy could result in confusion/users working from old data.

  1. Login to the admin console of one of your sites. The site will need to be the member ofthe relevant Replication Group, otherwise it won't appear on the tab. Click on the Replication Groups tab.
    Rmove Node from group 01

    Login and go to REPLICATION GROUPS

  2. On the Replication Groups tab, click on the View button that corresponds with the Replication Group from which you plan to remove a node.
    Rmove Node from group 02


  3. Click on the node that you plan to remove from the group. Providing that the removal of the node doesn't invalidate the remaining configuration you will see a Remove node from replication group link. Click the link.
    Rmove Node from group 03


    tip"Remove stray repositories
    You will not be allowed to remove a node that is currently assigned as the "Managing Node". In order to remove the managing node, go to the Configure Schedule page and assign a different node as a Managing Node.

  4. A dialog will open which asks you to confirm the removal of the selected node from the Replication Group. Click Remove. Rmove Node from group 04

    Remove. Really!

  5. You many need to click the Reload button to ensure that the action has been completed on all nodes. Rmove Node from group 04

    Reload to confirm the updated state.

  6. The node will now be removed from the Replication Group. On the Replication Groups panel you should now see that the constituent number of nodes has reduced by one. Rmove Node from group 06

    Less one member node

19. Scheduling node changes - follow the sun

You can schedule the member sites of a replication group to change type according to when and where it is most beneficial to have active voters. To understand why you may want to change your sites, read about Node Types

** Alert! **Schedule node type changes via the public API
Instead of manually setting up schedules through a node's UI you can do it programmatically through calls to the public API.
See Public API ScheduledNodeAPIDTOList element and scheduledNodeAPIDTOList Datatype

Use the following API call

PUT with ReplicationGroupAPIDTO XML as body:

To make Node N3 a tie-breaker 'T' FROM 10:00 - 16:00 (GMT) every day of the week with Node N1 as tie-breaker 'T' afterwards: (N.B Times are in GMT -4 hours so 14 = 10:00 GMT)

Example curl command:

Make a text file containing ReplicationgroupAPIDTO XML (as above) called schedule.xml

curl -u username:password -X PUT -d @schedule.xml http://[IP]:[PORT]/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule

Sample 'schedule.xml' file


Download the full sample Schedule.xml file.

  1. Login to a node, click on the REPLICATION GROUPS tab. Click on the VIEW link for the replication group that you wish to make a schedule.
    Schedule for you

    Scheduling is done through replication group settings.

  2. The replication group's pop-up window will open, showing the member groups together, along with their current roles. Click the CONFIGURE button.
    Schedule for you


  3. The replication groups configuration screen will appear. You may notice that to the left a Role Schedule is noted. By default this will show as DISABLED. Click on the Configure Schedule button, in the right-hand corner.
    Schedule for you

    Role Schedule: Disabled (for now).

  4. The Schedule screen will appear. The main feature of the screen is a table that lists all the nodes in the replication group, set against a generic day (midnight to midnight) that is divided into hourly blocks. Each hourly block is color-coded to indicate the specific node's type.

    In the image below NodeSanFransisco is coded as blue which indicates that it is set as a Passive Voter. The hourly blocks associated with NodeChengdu are Magenta, indicating that it is set as a pure voter. The blocks for NodeParis are colored yellow, indicating that this node is set as an Active Voter.

    Schedule for you

    Vanilla Scheduling - no changes to type over time.

  5. To make a change to the schedule, click on a block. It doesn't matter which block you select as the New Scheduled Configuration form will let you modify any hours for any available node.
    Schedule for you

    New Schedule Form.

      Select from the available frequency patterns: Daily, Weekly, Monday-Friday or Saturday to Sunday.
      The starting hour for the new schedule, e.g. 00 for the start of the day.
      The hour at which the scheduled changes end, e.g. 24 would effectively end the scheduled change at midnight.
      Node list
      The member nodes are listed, in graphical form, colour coded to their type.
  6. Click on the node icon to change its type.

    In this example NodeSanFransisco is changed to a Tie-breaking Passive Voter, then NodeAuckland is changed into a Tie-breaker.

    Schedule for you

    Swapping roles.

    When all node changes have been made, click on the SAVE button to continue, or the CANCEL button if you change your mind.

  7. The schedule view will now change to show the changes that you make. You must click the Save Schedule button for the changes to be applied.
    Schedule for you

    Swapping roles.

    With all necessary changes made, you need to review the change to the schedule table and then click SAVE SCHEDULE button.


20. Adding Repositories

You can add additional repositories for replication through the admin UI. The repository first needs be present on all the nodes that will be part of the corresponding replication group. So the repository copies need to be introduced to the replication system in an identical state.

  1. Ensure that the repository that you are going to replicate is copied to each node that is a member of the replication group that will be responsible for the repository. The repository copies must be in an identical state before you add them into SVN MultiSite Plus.

  2. Log in to the SVN MultiSite Plus admin UI. Click on the Repositories tab.
    Add Repository 01


  3. Click on the Add button and enter the repository information:

    Add Repository 02

    Enter repository details.

    Repo name
    A name that you want SVN MultiSite Plus to use when referring to the new repository.
    FS Path
    The absolute file system path for the repository. This should be identical on each node.
    Replication Group
    The replication group under which this repository will be managed. Select from available groups or go and create a new replication group.
    Global Read-Only
    This check-box lets you add the repository in a locked-down state that won't allow SVN users to commit changes to the repository. This feature is useful for putting a repository into maintenance mode where the copies might otherwise get out-of-sync.
    When all entry fields have been filled, click on the Add Repo button. You'll then see a 'growl' message confirming that the repository has been deployed and that you should reload your browser session to confirm that the repository has been added.

  4. Recheck the repositories table to see that the new repository has been added and has a "Replicating" status.
    Add Repository 03


tip"Repository stuck in Pending state
If a repository that you added gets stuck in the deploying state - you'll see this on the Dashboard, in the Replicator Tasks window - you can cancel the deployment and try adding the repository again. To cancel a deployment, go to the Replicator Tasks window and click on the Cancel Task link.

21. Removing Repositories

It's possible to remove repositories from SVN MultiSite Plus. Follow this quick procedure.

  1. Login to the admin console of one of your sites. The site will need to be the member of a replication group in which the repository is replicated, otherwise it won't appear on the tab. Click on the Repositories tab to see it.
    Rmove Repository 01


  2. On the Repositories tab, click on the line that corresponds with the repository that you want to remove.
    Schedule for you


  3. Once a repository has been highlighted (Shown as a blue line), the REMOVE button will become available. Click it. Schedule for you


    A dialog box will appear entitled "Remove repository from replication group". It will confirm that removing a repository from a replication group will stop any changes that are made to it from being replicated. However, no repository data is removed.

22. Editing a repository

It's possible to Edit a repository properites after they have been set up in SVN MultiSite Plus. Follow this quick procedure.

  1. Login to the admin console of one of your sites. The node will need to be the member of a replication group in which the repository is replicated, otherwise it won't appear on the tab. Click on the Repositories tab to see it.
    Rmove Repository 01


  2. On the Repositories tab, click on the line that corresponds with the repository that you want to remove. Then Click the Edit Button.

    Schedule for you


  3. You will now see the edit window appear.
    Schedule for you

    Edit Repository.

    Local Read-only
    Changes the Read-only setting, enable or disable the repository Local Read-only setting. When enabled, the repository will not be writable to local users (or SVN Clients for that matter). However, changes that are made to the replica at other sites will be applied through replication.
    Global Read-only
    Changes the Read-only setting, enable or disable the repository Global Read-only setting. When enabled, the repository will not be writable either locally or globally. This is used to lock a repository from any changes.
    Replication Group
    Use the drop-down selector to change the replication group to which the repository belongs.

    tip"Important: Known problem concerning the moving of repositories between replication groups.
    There's a problem that occurs if you move a repository to a replication group that contains one or more sites from an earlier replication group, in which the repository was previously replicated.

    Currently, logic persists from the earlier replication group that will cause commits to the newly added repository to fail, and for commits to the repository replicas on other member sites to not be replicated.

    We are addressing the problem as a matter of urgency, until a fix is in place do not move repositories into replication groups that contains sites that have previously handled replication of the repository.

23. Repository synchronized stop

The Repository Synchronized Stop is used to stop replication between repository replicas, it can be performed on a per-repository basis or on a replication group basis (where replication will be stopped for all associated repositories). To bring some or all nodes to a stop, use the Sync Stop All command found on the Nodes tab.
Repository Stops are synchronized between nodes using a 'stop' proposal to which all nodes need to agree. So that while not all nodes will come to a stop at the same time they do all stop at the same point.

  1. Login to a node's browser-based UI and click on the Repositories tab. Click on the repository that you wish to stop replicating.
  2. With the repository selected, click the Sync Stop button. A growl message will appear to confirm that a synchronized stop has been requested. Note that the process may not be completed immediately, especially if there are large proposals transfering over a WAN link.
  3. On refreshing the screen you will see that a successfully sync stopped repository will have a status of Stopped and will be Local RO (Locally Read-only) at all nodes.

24. Repository synchronized start

Restarting replication after performing a Synchronized Stop requires that the stopped replication be started in a synchronized manner.

  1. Click on a stopped repository and click on the Sync Start button.
  2. The repository will stop being Local Read-only on all nodes and will restart replicating again.

25. Logs

SVN MultiSite Plus has two sets of logs, one set is used for application, the other logs replication activity:

Application Logs


These logs are used for troubleshooting problems with MultiSite's running:

drwxrwxr-x 2 wandisco wandisco 4096 Dec  5 14:20 logs

	-rw-rw-r-- 1 wandisco wandisco 32647705 Dec  6 09:50 replicator.log.20121205-142035
	-rw-rw-r-- 1 wandisco wandisco   775889 Dec  6 15:00 replicator.log.20121206-101206
	-rw-rw-r-- 1 wandisco wandisco     1666 Dec  5 13:21 setup.20121205-132141
	-rw-rw-r-- 1 wandisco wandisco     1674 Dec  5 13:53 setup.20121205-135328
	-rw-rw-r-- 1 wandisco wandisco      157 Dec  5 14:20 setup.20121205-142034
	-rw-rw-r-- 1 wandisco wandisco     1979 Dec  5 14:20 startup.20121205-142035
	-rw-rw-r-- 1 wandisco wandisco     1980 Dec  6 10:12 startup.20121206-101206
	-rw-rw-r-- 1 wandisco wandisco       94 Dec  6 09:57 stop.20121206-095714
	-rw-rw-r-- 1 wandisco wandisco     5136 Dec  6 10:26 ui.launch.log

Replication Logs


	-rw-rw-r--. 1 wandisco wandisco  371282 Jan 23 10:00 fsfswd.0
	-rw-rw-r--. 1 wandisco wandisco       0 Jan 18 10:40 fsfswd.0.lck
	-rw-rw-r--. 1 wandisco wandisco 1054726 Jan 23 06:29 fsfswd.1
	-rw-rw-r--. 1 wandisco wandisco 1055024 Jan 17 11:38 fsfswd.10
	-rw-rw-r--. 1 wandisco wandisco 1049842 Jan 17 11:38 fsfswd.11
	-rw-rw-r--. 1 wandisco wandisco 1050660 Jan 17 11:38 fsfswd.12
	-rw-rw-r--. 1 wandisco wandisco 1049618 Jan 17 11:37 fsfswd.13
	-rw-rw-r--. 1 wandisco wandisco 1049228 Jan 17 11:37 fsfswd.14

The fsfswd.xx logs record all actions taken by the replicator and a good starting point in the investigation of replication problems.

drwxrwxr-x. 2 wandisco wandisco    4096 Jan 17 11:43 stats

	-rw-rw-r--. 1 wandisco wandisco  22 Jan 17 11:43 111f4016-a23a-49d6-9769-ced78a250539
	-rw-rw-r--. 1 wandisco wandisco  22 Jan 17 11:43 420aa491-1bbb-4a6c-9cff-e8b60e911dd9

The Stats files are where the repository statistics used by Dashboard widgets are stored. These files are not intended to be readible by users.

drwxrwxr-x. 2 wandisco wandisco   12288 Jan 23 09:40 thread-dump

        -rw-rw-r--. 1 wandisco wandisco 55395 Jan 17 06:34 thread-dump-2013-01-17.06-34-00
        -rw-rw-r--. 1 wandisco wandisco 79150 Jan 17 07:34 thread-dump-2013-01-17.07-34-01    
        -rw-rw-r--. 1 wandisco wandisco 79162 Jan 17 08:34 thread-dump-2013-01-17.08-34-01

The thread-dump directory contains thread dumps that the replicator periodically writes to help with troubleshooting.

Logging Levels

Message level indicating a serious failure.
A message level indicating a potential problem.
Interesting runtime events (startup/shutdown). Expect these to be immediately visible on a console, so be conservative and keep to a minimum.
Details of configuration messages.
Provides a standard level of trace information.
Provides a more detailed level of trace information.
Provides a boggling level of trace information for troubleshooting hard to identify problems.

Changing the Loggoing Level

You can change the current logging level by editing the logger properties file  install-dir>/svn-multisite-plus/replicator/properties/ You can see a sample file.

26. Consistency Check

The consistency Check gives you a quick and easy check whether a selected repository remains in the same state across the nodes of a replication group. Follow these steps to check on consistency:

tip"Limits of the Consistency Checker
The Consistency Check will tell you the last common revision shared between repository replicas. Given the dynamic nature of a replication group it's possible that there will be in-flight proposals in the system that have not yet been agreed upon at all nodes. For this reason it isn't possible for a consistency check to be completely authorative.

  1. Login to a site, click on the REPOSITORIES tab.
    Schedule for you

    Go to the repository

  2. Click on one of the listed repositories. This will active the above line of buttons.
    Schedule for you

    Consistency Check is done on a per node basis

  3. Click on the Consistency check. A growl message "Invokling consistency check on repository <Repository Name>" will appear. Schedule for you

    Consistency check in action

    tip"Known Issue: Don't run a consistency check if the repository has been removed from one of the nodes.
    There's currently a problem with running a consistency check on a repository if the replica on one or more or more nodes has been deleted. In this situation a "Highest Common Revision" task will appear on the dashboard and will remain permanently in a 'pending' state. Until we resolve this problem you shouldn't run the consistency checker on a repository if it has been removed from the file system of any of your nodes.

  4. Click on the DASHBOARD tab. The results of the consistency check will appear on the Replicator Tasks widget - you'll need to select All Tasks, instead of the default Pending Tasks. Schedule for you

    Repository replicas need to be identical - are they?

  5. Originating Node:
    The site from which the check was requested
    Lowest Revision Checked:
    The oldest revision compared across all sites
    Number of Revisions Checked
    Total number of revisions checked
    Repository is Consistent
    The result of the check as a 'true' or 'false' statement
    Highest revision Checked
    The youngest revision compared across all sites
    Repository Being Checked
    The name of the repository that has had its consistency checked

Log results

It's also possible to check the results of a consistency check by viewing the replicator's log file (fsfswd.##). See Logs

27. Copying repositories

This section provides advice on getting your repository data distributed prior to starting replication.

Subversion installations must have:

These are things are a recap of the installation checklist, you must have them in place in order for replication to run effectively:

Copying Existing Repositories

It's simple enough to make a copy of a small repository and transfer it to each of your sites. However, remember that any changes made to the original repository will invalidate your copies unless you perform a syncronzation prior to starting replication.

If a repository needs to remain available to users during the proccess, you should briefly halt access, in order to make a copy. The copy can then be transferred to each site. Then, when you are ready to begin replication, you need use rsync to update each of your replicas. Fore more information about rsync, see Synchronizing repositories using rysnc.

New Repositories

If you are creating brand new repositories, don't create them at each site, instead create the repository once, then rsync it to the other sites. You need to do this to ensure that each replica has the same UUID.

If you do create repositories at each site instead of using rysnc, you can use Subversion's UUID command to get them all matching:

You can confirm the UUID of a repository using the svnlook uuid command:

[root@ip-10-0-100-6 Subversion]# svnlook uuid Repo0

You can use the Set UUID command to ensure that a new repository that you've created has a UUID that matches with the other replicas:

$ svnadmin setuuid /opt/Subversion/Repo0 67d41b33-3c7c-4ba0-8af1-119dbb0d42ba

28. Repair an out-of-sync repository

There are a number of situations where a repository may be corrupted or lose sync with its other copies -- this could be the result of file/permission changes on the server. In such an event the node on which this copy is situated will stop replicating data for that repository (other repositories will be unaffected and should continue to replicate.) SVN MultiSite Plus has a repair tool that can be used to quickly get the repository repaired and replicating again.

  1. Login to a site, click on the REPOSITORIES tab. A repository that is out-of-sync will be flagged as Local RO (Read-only) which signifies that other replica may continue to update. Note that the Status for Repo2 is marked as "Stopped" instead of "Replicating". Click on the Repair button.
    repair 1

    Out of sync

  2. The Repair Repository window will open. This runs through a three step procedure. First, select a 'helper' from the nodes that remain in replication. It may be worth while doing a test before you choose the helper to ensure that its copy of the repository is in fact the latest version. Once selected, click the Start Repair Process button. This will briefly take the selected node offline, to ensure that changes don't occur to the repository while you conduct the repair. At this point you need to login handle the repair manually.
    repair 1

    Start the repair!

  3. Use the good copy of the repository on the helper node, overwriting the out-of-date/corrupted copy. We recommend using rsync for this task. There's more about using rsync in the next chapter.
    [root@localhost repos]#  rsync -rvlHtogpc /opt/repos/repo2/ root@
    The authenticity of host ' (' can't be established.
    RSA key fingerprint is 9a:07:b2:bb:b6:85:fa:93:41:f0:01:d0:de:8f:e1:5d.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added '' (RSA) to the list of known hosts.
    root@'s password:
    sending incremental file list
    sent 1589074 bytes  received 701 bytes  167344.74 bytes/sec
    total size is 1585973  speedup is 1.00
    [root@localhost repos]#

  4. From a terminal window, delete the repository's cache file.
    This step is not essential and could result in the repository becoming slightly larger, however it removes the risk that the repaired repository will not match with the cache file.

  5. Restart Apache. This will free up file handlers that are holding the rep-cache.db file open as well as clearing any in-memory cache data that could point to refernces that don't exist in the repaired repository.

  6. Once the repository is updated you should click Complete Repair Process and check that the fixed repository now matches the version on your helper node.
  7. Looking back at the REPOSITORIES tab you'll now see that the problem repository is once again replicating.
    repair 3

    Back in sync

  8. 29. Synchronizing repositories using rysnc

    If for any reason repositories are corrupted or unable to automatically catch up it's usually possible to use rsync to get them back into sync.

    tip"Before you resync
    To rsync you'll need to copy data from a repository replica that is up-to-date, before you do so it's good practice to perform an SVN verify on the repository to be absolutely sure of it's integrity.

    Run the following Subversion command on the 'helper' site:
    svnadmin verify <Repository-path>

    From the site with the up-to-date repository, type the following commands:

    rsync -rvlHtogpc /path/to/local/repo remoteHost:/path/to/remote/repo

    For example:
    rsync -rvlHtogpc /Subversion/Repo root@

    Then follow up with an additional rsync that will ensure that contents of the locks directory are identical (by deleting locks that are not present on the originating server)

    rsync -rvlHtogpc --delete /path/to/repo/db/locks <Repository Name> remoteHost:/path/to/repo/db

    For example:
    rsync -rvlHtogpc --delete /Subversion/Repo/db/locks root@

    You can read a more detailed step-by-step guide to using rsync in the Knowledge Base article Reset and rsync Subversion repositories.

    30. Recover from the loss of a site

    It's possible for SVN MultiSite Plus to recover from the brief outage of a member site, which should be able to resync once it is reconnected. The crucial requirement for MultiSite's continued operation is that agreement over transaction ordering must be able to continue. Votes must be cast and those votes must always result in an agreement - no situation must arise where the votes are evenly split between voters.

    If after the loss of a site, a replication group can no longer form agreements then replication is halted. If the lost site was a voter, and there aren't enough remaining voters to form an agreement, then either the lost site must be repaired/reconnected, or the replication group must undergo emergency reconfiguration.

    Emergency Reconfiguration (EMR) is a final option for recovery

    The emergency reconfiguration process can't be undone, and it represents a big shakeup of your replication system. Only complete an emergency reconfiguration if the lost site can not be repaired or reconnected in an acceptable amount of time.

    ** Alert! **Gone but not forgotten
    After a lost node has been removed and a replication group reconfigured, the lost node should not be allowed to come back online. Whilst the DConE replication engine will not be phased by a rogue node,however, its presense could result in confusion as local developers mistakenly commit changes to a repository copy that will no longer receive updates from the other replicas. You should ensure that you perform a cleanup after completing an emergency reconfiguration.

    ** Alert! **Last Node Standing
    Any replication group which has its membership reduced to one node will continue to exist after the emergency reconfiguration as a non-replicating group. Once you have set up a replacement node you should be able to add it back to the group to restart replication.

    Emergency Reconfiguration

    So, having confirmed that an emergency reconfiguration is required, follow this procedure:

    1. Verify the details of the site that is now declared 'lost'. Login to the administrator user interface of one of the remaining sites and view the Sites tab. The missing site will show a status of Disconnected. Emergency Reconfiguration 1

    2. Select the lost node by ticking its corresponding checkbox and then click the Reconfigure button.
      Emergency Reconfiguration 2

    3. The Emergency Reconfiguration screen will appear. Check and confirm that you have selected the correct site, then click on the Start Reconfiguration button. Emergency Reconfiguration 3

    4. A warning will appear, asking you to confirm that you are ready to start the process, and that once started the process can not be cancelled. Click CONFIRM if you are ready to proceed.
      Emergency Reconfiguration 4

    5. The Reconfiguration process will now run, creating new, replacement replication groups, activating them, then removing the old groups. The process is finished when all items are listed as Complete. You'll finally be asked to confirm the removal of the lost node, click on the Remove Node(s) button.
      Emergency Reconfiguration 5

      ** Alert! **How Reconfiguration Works
      The emergency reconfiguration process seeks to recreate functional replication groups using the remaining member sites. In siutations where a replication group only contained two sites, including the lost site, then a reconfiguration is not possible, in this scenario a new replication group will need to be created once a replacement site has been inducted.

    6. You'll still see the removed node on the nodes table, it will be flagged as "Removed". You won't be able to reinduct this node, you will need to install SVN MultiSite Plus using a new NodeID.
      Emergency Reconfiguration 6
      Finally, you should check the state of the reformed replication groups to ensure that they'll still perform according to your organization's requirements.

    31. Restore replication on a problem node.

    It's possible that a problem with a single transaction can result in a repository being put into a read-only mode. Causing the replication of this repository at just one node. If possible, use the following procedure to get replication started again:

    1. The first sign that a transaction has not been able to complete on a node is when the repository is placed in a protective read-only state. This is done to ensure that it will remain in a condition in which it can be recover and catch up. On the Repositories tab you'll see the repository is now flagged as locally read-only.
      Repositories - local read-only

      Repository Repo01 is flagged as local read-only

      Providing there are still enough nodes to reach agreement, repository changes at the other nodes can continue to be made.
    2. At the problem site, you would now need to indentify the cause of the problem. Check SVN MultiSite's logs as well as the logs generated for SVN users who are trying to commit changes to the problem repository. It may be possible to quickly fix the causeo f the problem, such as a permission problem that has prevented a file to be written to on the node.
    3. When the problem has been fixed you can go to the Repositories tab and edit the read-only repository. Remove the Local RO (Read-only) tick. The node will then attempt to catch up and get the repository back into sync with its other replicas.

    32. Running Talkback

    Talkback is a bash script that is provided in your SVN MultiSite Plus installation for use in the event that you need to talk to the WANdisco support team.

    Run Talkback using the following procedure:

    1. Login to the server with admin privileges. Navigate to the SVN MultiSite Plus's binary directory:
    2. Run talkback.
      [root@localhost bin]# ./talkback
    3. You'll need to provide some information during the run:
            ===================== INFO ========================
            The talkback agent will capture relevant configuration
            and log files to help WANdisco diagnose the problem
            you may be encountering.
      Is the replicator currently running [Y/n]: y
      Please enter replicator admin username: adminUIusername  
      Please enter replicator admin password: thepasswordhere
      retrieving details for repository "Repo1"
      retrieving details for repository "Repo3"
      retrieving details for repository "Repo4"
      retrieving details for repository "repo2"
      retrieving details for node "NodeSanFransisco"
      retrieving details for node "NodeAuckland"
      retrieving details for node "NodeParis"
      Please enter your WANdisco support FTP username (leave empty to skip auto-upload process):
      Skipping auto-FTP upload
       Please upload the file:
       to WANdisco support with a description of the issue.
       Note: do not email the talkback files, only upload them
       via ftp or attach them via the web ticket user interface.

      Note that you'll need to talk to Support about setting up access to WANdisco's Support FTP space.

      ** Alert! **Don't send talkback files via email
      If you're not using our secure FTP you can upload your talkback output files to our support website. Just attach them to your case. Read our Knowledgebase article about How to raise a support case.

      Talkback Output Example