LATEST VERSION: The documentation for the latest version of Subversion Multisite: Subversion MultiSite 4.2

1. Introduction

Welcome to the Admin Guide for WANdisco's Subversion MultiSite. This is where you'll find all the information you'll need for planning a deployment, installing and managing the software.

If you can't find what you're looking for check out our online Knowledgebase which contains updates and extra documentation. Should you need further help and want to alert us to an error, please contact us by raising a case on our support website.

We'll be using terms like "node" and "replication group" without explaining what we mean. Check out the Glossary for an explanation of these and other WANdisco terms.

For a better understanding about how MultiSite and WANdisco's replication technology works read the Subversion MultiSite Technical Guide.

1.2 Feedback

If you find an error, or if you think that something needs improving we'd like to hear about it. Tell us about it by raising a case on our support website or emailing docs@wandisco.com.

1.3 Key

Throughout this document we'll highlight things using the following types of information box.

** Alert! **Alert
We use alerts to highlight something we think is important for you to know.

tipCaution
The stop symbol is used when we want to caution you against doing something.

tipTip
Tips are principles or practices that you'll benefit from knowing or using.

tipKnowledgebase
The KB symbol indicates that you can find out more about the subject in our online Knowledgebase.

1.4 Release Notes

Check out the latest release information on the Release Notes page.

2. Deployment guide

The deployment guide will help you prepare your development environment for the implementation of WANdisco's Subversion MultiSite.

2.1 Skills requirement

This section details the knowledge and technical requirements for deployment and operation of the WANdisco software, you should ensure that each of these sections is addressed before you begin the deployment.

Technical Skill Requirements

 System administration
  • Unix or Windows operating system installation
  • Disk Management
  • Memory monitoring and management
  • Command Line administration and manually editing configuration files

 Apache administration
  • Familiarity with Apache web server architecture
  • Management of httpd.conf / Apache2 configuration file management settings
  • WebDAV protocol
  • Start/stop/restart administration
  • User authentication options
  • Log set up and viewing

 Networking
  • IP Address assignation
  • TCP/IP ports and Firewall setup

 Subversion
  • Familiarity with Subversion administration (svnadmin) to create repository Command line user commands (svn)
  • Repository creation, and repository and/or file-system copying and synchronization.
  • Familiarity with WANdisco's replication architecture
  • Understanding of the installation procedure relevant to the OS in use
  • Concept of node types and Replication groups
  • Quorum configuration options

If you're not confident about handling any of these tasks, you can request a supported Installation from WANdisco.

A single administrator can manage all the systems running MultiSite, although it's a good idea to have someone at each site who is familar with the MultiSite Basics.

2.2 Deployment overview

As with any software implementation you should deploy WANdisco Subversion MultiSite following a well defined plan. Doing so will help keep things under control, understood, and any potential problems will be spotted and fixed before you go into production.
We recommend that any deployment plan includes the following steps:

1. Deployment Planning: Identifying the requirements, people and skills needed for deployment and operation. Also covers agreeing schedule and milestones, highlighting any assumptions, constraints, dependencies and risks to a successful deployment.

2. Deployment Preparation: Preparation and identification of server specifications, locations, node configuration, repository set-up, replication architecture, server and software configurations.

3. Testing Phase: Activities related to installation in an initial installation and testing in a non production environment, executing test cases, and verifying deployment readiness.

4. Production Deployment: Activities related to the installation, configuration, testing, and deployment in the production environment.

5. Post-deployment Operations and Maintenance: Post-deployment activities including environment monitoring, system maintenance, training and in-life technical support.

2.3 System requirements

This section covers everything you need to know when preparing your Subversion servers for replication. You should view this information as a set of guidelines, not as a fixed set of requirements that are followed slavishly.

2.3.1 Hardware Recommendations

Hardware Sizing Guidelines
Size #Users Repository Size (GB) CPU speed (GHz) #CPU #Cores RAM (GB) HDD (GB)
Small 100 25 2 1 2-4 8-16 100
Medium 500 100 2 2 4 16-32 250
Large 1000 500 2.66 4 4 32-64 750
Enterprise 5000 1000 2.66 4 4-6 128 1500

Storage tips

Processor tips

2.3.2 Setup requirements

MultiSite Servers must have:

This is a summary of the requirements, you'll need to run through the more detailed Deployment Checklist.

Subversion installations must have:

tip Tip
Run Subversion and MultiSite on the same server to gain the following benefits:
  • The installer can package a copy of your repositories when setting up nodes
  • Subversion MultiSite can manage the Subversion password file
  • Support for pre-replication hooks

2.3.3 Creating copies of your repository

Each MultiSite server must begin with an exact replica of the Subversion repository. The method you use to copy repositories depends on how big they are.

Small Repositories - Less than 1GB (Gigabyte)

If a repository is relatively small, it can be bundled up with the MultiSite install files when you come to activate additional MultiSite nodes. However, this option is only practical for relatively small repositories.

Large Repositories - 1GB (Gigabyte) or greater

For repositories that are over 1GB in size, it's no longer practical to bundle them during the installation of MultiSite software, instead you should use the following process:

This process ensures that the Subversion repositories are exact copies, and does so with minimal downtime for Subversion users.

2.3.4 Replicating multiple repositories

Subversion MultSite is capable of replicating any number of Subversion repositories, using the Apache SVNPath or SVNParentPath directives. For more information, see the chapter Setting up Apache.
Read more about adding repositories.

2.3.4.1 SVN Location (SVNPath) Directive

The SVNPath directive is used to tell Apache where your Subversion repository is located. You can have multiple Location Directives, each pointing to a different Subversion repository.

  <Location /repo>
    DAV svn
    SVNPath /home/svn/repo1
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
		
	
  <Location /xyz>
    DAV svn
    SVNPath /home/projects/repositories/xyz
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
    

Example httpd.conf syntax for using SVNPath.

2.3.4.2 SVNParentPath Directive

The ParentPath directive is used if you have multiple repositories that reside in the same directory. Instead of directing to a specific repository directly, it tells Apache to match to the parent directory, under which there may be several repositories.

e.g. subversion-domain.com/svn/repo1 and subversion-domain/svn/repo2 would map to repositories at /home/svn/repo1 and /home/svn/repo2.

    <Location /svn>
    DAV svn
    SVNParentPath /home/svn/
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
    

Example httpd.conf syntax for using SVNParentPath.

** Alert! **Alert
A repository can belong to only one replication group at a time.
Each replication group that a node belongs to will require its own SVNParentPath.


tipKnowledgebase
There's an example use of SVNParentPath featured in our Knowledgebase - Using SVNParentPath with multiple replication groups

2.3.5 Using Authz for authorization

Authz is the Apache module that gives you control over user authorization. It takes user information from the password file and associates access permissions.

tipTip
We recommend using the Authz along with Access Control to fully control access to your Subversion repositories.

** Alert! **Alert
If you move from Subversion MultiSite bundled with Access Control to the baseline MultiSite (that doesn't feature Access Control), you will need to manually remove the Authz directive from the Apache config file as well as the Authz file.

Setting up Authz

tipKnowledgebase
Read more about how to use Authz - Authz basics

2.3.6 MultiSite and Subversion password files

The Subversion server on each and every node must have the details of all users across the replication group. The easiest way to keep user details consistant is the allow MultiSite to take control of the Subversion password files, so that any user created in MultiSite's admin console is automatically added to Subversion servers across the replication group. Using this Access Control feature greatly simplifies the management of Subversion users.

You can choose to have Subversion MultiSite control the Subversion password during installation, where you identify the Subversion password file's location. The installer then incorporates it into the replication group.

tipTip
If you have a lot of Subversion users, you can bulk import them into Subversion MultiSite.

2.3.7 Quorum recommendations

The Quorum refers to the strategy used for handling agreement between the nodes in a replication group. You can read more about the different types of Quourum in the glossary. This section gives you a quick summary of the benefits of each quourum type, and in what setup it should be used.

Setup Type Quorum Recommendation
MultiSite
Singleton Response or Majority Response quorum, balancing performance versus availability.

MultiSite with Sub Groups
Majority Response quorum. If you select Singleton Response quorum, the distinguished node represents a single point of failure.

Stand-Alone Group at One Location
Have a group of at least three nodes, which automatically handles the failure of any single replicator node and its subsequent recovery.

With a two node group, select Singleton or Majority Response quorum, and the second node must be the distinguished node.

2.3.8 Firewalls and Anti-virus software

The unexpected blocking of ports can be a problem when setting up Subversion MultiSite. Before you start the installation you should determine if any of the nodes in your replication group will be accessing a firewall. You must ensure that any firewalls are configured so that the port numbers (6444 etc) you specify during installation are not blocked or filtered.

If you have a AV software/virus scanner running on your network, you must configure it to not filter traffic on the ports you specify during installation.

2.3.9 Configuring Apache

You can give Subversion MultiSite exclusive use of Apache or share Apache with other applications/locations.

basic apache

Example of a simple configuration without MultiSite.

Without Subversion MultiSite, Subversion clients simply connect to Subversion on port 80.

basic apache

Example setup that includes Subversion MultiSite.

Subversion MultiSite sits between the Subversion clients and Subversion server, acting as a web proxy. Subversion clients connect on the usual port 80, whilst Apache is set to listen on port 8080.

2.3.9.1 Changing Subversion Listen port (Unix)

The listen port is set in Apache's config file - httpd.conf. There's more information about the Listen directive in the Binding chapter of the Apache documentation.

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 8080

Snippet of the httpd.conf file.

With this configuration, Apache server listens on port 8080 instead of default port 80.

2.3.9.2 Sharing Apache, and Using HTTPS

You can choose to share Apache with other locations or applications, instead of dedicating it to Subversion. This illustration shows how:

apache shared

Using HTTPS or sharing Apache with other applications/locations.

In the above example all processes are running on the same machine. Apache is running on port 80, Apache's webDAV module is running on port 8181. WANdisco's Subversion MultiSite is listening on port 8080, forwarding requests to Apache's WebDAV module on port 8181. Some clients are connecting via SSL (HTTPS) with their connection running on port 443.

tipKnowledgebase
You can use this configuration if you enable a proxy pass.

Default port settings required during the installation of Subversion MultiSite:

In the Apache config httpd.conf file, specify the following parameters:

#
# Define Apache port and pass anything that matches location /svnrepos to
# WANdisco SVN Replicator
#
NameVirtualHost *:80
<VirtualHost *:80>
ProxyPass /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPassReverse /svnrepos http://127.0.0.1:8080/svnrepos
</VirtualHost>

Listen 8181
NameVirtualHost *:8181

<VirtualHost *:8181>
<Location /svnrepos>
AllowOverride None
Order allow,deny
Allow from 127.0.0.1
DAV svn
SVNParentPath /tmp/dav
AuthType Basic
AuthName wandisco
AuthUserFile /etc/httpd/conf/htpasswd
Require valid-user
</Location>
</VirtualHost>
# For the SSL option
Listen 443
<VirtualHost *:443>
ProxyPass /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPass /!svn http://127.0.0.1:8080/svnrepos/!svn
ProxyPassReverse /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPassReverse /!svn http://127.0.0.1:8080/svnrepos/!svn
SSLEngine on
SSLCertificateFile /etc/httpd/conf/ssl.crt/server.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/server.key
SSLCACertificateFile /etc/httpd/conf/ssl.crt/ca-bundle.crt
</VirtualHost>

2.3.9.3 Using HTTP with Apache

This section applies to any Apache configuration.

** Alert! **Alert
In order to make a Subversion repository function in a distributed environment, Subversion MultiSite requires exactly the same Apache/Subversion setup at all the nodes.

Even without Subversion MultiSite, Apache benefits from some tweaking in order to work effectively with Subversion.

2.3.9.4 Apache and SVNKit

There are potential problems with connection pooling when using Apache and SVNKit. Adding Subversion MultiSite to the mix doesn't change these issues, though you may need to revisit them after installing MultiSite.

We recommend using JavaHL with Eclipse IDE, which does not use connection pooling, and thereby eliminates the problem.

2.3.9.5 SVNKit and Connection Pooling

SVNKit uses connection pooling. For a given client, SVNKit opens two connections and keeps them open for later use. On a system with a heavy load this can hit performance. An open connection consumes an Apache worker thread, with lots of clients and pooling going on, Apache may run out.

Apache provides some tuning parameters to optimize connection pooling, while still releasing unused connections. The tunable parameters are Timeout, KeepAliveTimeout, MaxKeepAliveRequests, and KeepAlive. For more details refer to the Apache configuration documentation.

2.3.9.6 Optimize Your Configuration

Apache has two timeout configurations: Timeout and KeepAliveTimeout. In general, the Timeout value should be lower than the value for KeepAliveTimeout.

tipCaution
Setting KeepAlive to false is not recommended. If you set KeepAlive to false, a client's transactions create an enormous overhead establishing and destroying connections.


2.3.9.7 Apache Logging

Apache's logs are dumped to a single file, which can cause problem when the log file grows so big that it needs to be truncated. When this happens, Apache service is interrupted while the file is restarted. On Linux/Unix servers there's the option of piping logs to another process. For more information see our Knowledge Base article - Piping Apache logs

2.4 Deployment Checklist

Though you may have referred to the Deployment Checklist during your evaluation we strongly recommend that you revisit the checklist and confirm that your system still meets all requirements.

Though you may have referred to the Deployment Checklist prior to an evaluation of Subversion MultiSite we strongly recommend that you revisit the checklist and confirm that you're system still meets all requirements.

 System setup

 Operating Systems

We've tested the following operating systems:
  • Fedora (32 or 64 bit): 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
  • Red Hat Linux Enterprise Server (32 or 64 bit): 4, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 6, 6.1, 6.2
  • Sun Solaris (32 or 64 bit): 9, 10, 11
  • Linux: Linux kernel 2.6 or higher
  • CentOS: 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 6, 6.1, 6.2
  • Windows Server: (32 or 64 bit) 2003, 2008

  • In principle, any operating system that can support a Java environment, Apache and Subversion.

 Subversion server

Version 1.4 and above (we've tested up to version 1.7.1).
Run the Apache Portable Runtime that matches your Subversion version.

Subversion MultiSite Version 4.1 is required in order to use Subversion 1.7's HTTPv2 implementation. Read our Knowledge base article on a workaround that lets you Run MultiSite 4.0 with Subversion 1.7.

tipCertified Subversion Binaries
are now available from WANdisco. Providing the latest builds, without the risks associated with Open Source distribution.

 Subversion client

Any that are compatible with local Subversion servers.


 Triggers
  • Pre-commit triggers are not recommended. Use pre-replication hooks instead.
  • Any pre-replication hooks must be deterministic,
    i.e. have the same exact behavior and outcome at every node. Post-commit triggers can be tested at only one node.

 System memory

Ensure RAM and swapping containers are at least four times larger than the largest Subversion file.
Minimum recommended: 2 GB RAM; 4 GB swapping container


 Disk space

Subversion: Match to projects and issues.
MultiSite Transaction Journal: Equivalent of seven days of changes.

To estimate your disk requirements, you need to quantify some elements of your deployment:
  • overall size of all of your SVN repositories.
  • frequency of commits in your environment.
  • types of files being modified - text,binaries (SVN clients only send deltas for text).
  • number and size of files being changed.
  • rate that new files are being added to the repository.
Checkouts: You need sufficient disk space to handle large, in-transit checkouts which may get buffered to a tmp directory beneath the replicator installation until that checkout has been completed. The required space can be calculated using the following guideline:

Recommended storage = Number of clients checking out files(N) x average checkout sizes (Kilobytes)


 File descriptor limit

Ensure hard and soft limits are set to 64000 or higher. Check with the ulimit or limit command.


 Journaling file system

Replicator logs should be on a journaling file system, for example, ext3 on Linux or VXFS from Veritas.

    ** Alert! **Alert
    NTFS is a journaling file system. ext4 is also a journaling file system, although its use of deferred writes makes it incompatible with Subversion MultiSite.


 Max. User Process Limit

At least three times the number of Subversion users.

 Java

Install JDK 1.6. We recommend using Oracle JDK 1.6.

** Alert! **Alert
Important: We strongly discourage upgrading to Java 7 as problems have been reported with HotSpot optimization, which is enabled by default. The problem occurs in Java 6, although only if Hotspot optimization is enabled manually. In which case, we'd recommend that you disable it.



 Perl

Install version 5.6.1 or later. For Access Control: Perl::DBI module for Audit Reports other than Users and Groups.

** Alert! **Alert
Windows users: Use the 32-bit edition of ActivePerl version 5.8, even if you are running a 64-bit Windows server.


 Network settings

 Reserved ports

MultiSite needs a dedicated port for its replication (DconENet) and HTTP (for the admin console) protocols. We recommend having a port available in case you have to copy (rsync) the repository from one node to another.


 Firewall or AV software

If your network has a firewall, ensure that traffic is not blocked on the reserved ports noted above. Configure any AV software on your system so that it doesn't block or filter traffic on the reserved ports.


 VPN
Set up IPsec tunnel, and ensure WAN connectivity.

 VPN persistent connections

Ensure that your VPN doesn't reset persistent connections for Subversion MultiSite.


 Bandwidth

Put your WAN through realistic load testing before going into production. You can then identify and fix potential problems before they the impact productivity.


 DNS setup

Use IP addresses instead of DNS hostnames, this ensures that DNS problems won't hinder performance. If you are required to use hostnames, test your DNS servers performance and availability prior to going into production.

 Apache setup

 Apache version

All nodes must have the same version, 2.2.3 or above.


 Apache modules version

All nodes have the same version of mod_dav and mod_dav_svn.


 mod_deflate.c for SVN_DAV

DO NOT USE mod_deflate.


 Location URI

Ensure the Apache config file on all nodes have the same location URI for Subversion repository access.


 Valid user for write methods

Ensure that all WebDAV methods require authentication for SVN-DAV protocol.


 Use port 80 for WANdisco

Standard Port 80 avoids confusion, change default Apache port if using 80.


 Apache server port

Use non-standard Apache server port to avoid conflict with replicator port.
See Dedicated Apache - Changing Subversion Port On


 File permissions in svnroot

See this article (http://www.reallylongword.org/articles/svn/)

 MultiSite setup

 Quorum

Default is singleton, providing performance over availability. Read more about Quorum options.


 Rotating quorum schedule

Ensure the distinguished node is in the active time zone.


 Disk space for recovery journal

Provision large disk for svn-replicator/systemdb, at least number of commits within a two to four hour window

 Batch processing

If there are any batch processes that interact with WANdisco, turn the deferred-writes to false.
 Access Control

 Audit reports

For reports other than Users and Groups, you'll need to use a database such as MySQL, and php.


 mod_authz_svn

Optional but recommended. See Access Control - Using the Authz Module. Module may be bundled with your version of Apache.

3. Installation

The installation guide runs through setting up Subversion MultiSite for the first time. If you're already using an earlier version of MultiSite, refer to the Upgrade section.

3.1 Installation overview

Before we begin, here is a recap of the installation process:

3.2 Before you start

Some things you should bear in mind before starting the installation:

3.3 Starting the installation

Ensure that you install the MultiSite files to a directory called svn-replicator and that you preserve the following subdirectory structure:

Directory Contents
/audit Contains the audit trail logs for Access Control.
/bin Contains scripts like svnreplicator and shutdown
/config Contains prefs.xml which contains MultiSite configuration information. The license file needs to be place here before you can run MultiSite.
/lib Contains the Java archives (jar) files required to run the MultiSite.
/logs Stores logs and other temporary files. MultiSite's main log file is named SVNProxyServer-prefs.log.0
/systemdb Contains the system database with its transaction journal. Warning: Deleting or modifying files from systemdb will likely corrupt your installation.
.

If using selective replication you need seperate instances of MultiSite installed, but each situated in a directory called svn-replicator, for example:

<replicator01>/svn-replicator
<replicator02>/svn-replicator
<replicator03>/svn-replicator 

Selective replication is used when you need to replicate only specific repositories. You can find further information in 1.23 Selective Replication.

4. Complete setup through a browser

5. Set up using the Silent Installer

The Silent installer scripts the WANdisco setup process, avoiding the need for a systems administrator. All the relevant data required for setup, which is currently captured via the GUI installer can be provided in an XML file. This XML file will then be used by setup instead of starting the normal GUI application. If all of the data provided is valid, setup will exit as normal and the replicator will start.

The process for the silent installer is as follows:

Silent installer completes successfully

The replicator will be started automatically and the setup_Settings.xml file will be archived as installed_Settings_Date-Time.xml.

It's possible to reuse the saved xml file by renaming it back to setup_Settings.xml, although you will need to edit the file to re-enter passwords, as these are stripped out at the end of a successful setup.

Silent installer fails

The setup process will exit and the /logs/WebProxy log file will contain any setup errors.

Silent installer technical description

There's a detailed explanation of the Silent Installer's XML file in the Technical Guide -
Silent Installer Document Type Definition.

6. Login to the Admin Console for the first time

7. Copying the installation files manually

Follow these instructions if you were unable or chose not to use the automated packaging option Covered in section 7.

MultiSite creates a [node name].zip file for each of the nodes in a replication group. The first node is installed, so perform the following for all new nodes in the group:

8. Post Installation Configuration

There are some changes you may need to make before starting to use Subversion through MultiSite. These changes are made through the admin console. For more information see the admin Console guide

8.1 Using pre-commit hooks?

If you are going to be using pre-commit hooks, these become WANdisco pre-replication hooks. Read Setting up hooks
in the User Guide.



User Guide

1. Replicator Management

1.1 Adding a node

To add a node to an existing replication group, you have to create a new replication group that contains the new node and then activate the new replication group. See Creating a New Replication Group.

Determine a plan for copying the repository to the new node. See 2.3.3 Creating copies of your repository.

The new replication group is activated with all nodes.

1.2 Removing a node

It's not possible to remove a node from an active replication group, instead a new replication group is created excluding the node that you want to remove:

tipAlert
Only one replication group can be active at a time.

1.3 Creating a new replication group

You can create as many replication groups as you like, although only one replication group can be active at a time.
You can't delete the active replication group, first you need to create a new replication group and make it the active group.
See Deleting replication groups

tipTip
Create replication groups from the node that will be the distinguished node in the new replication group.

tipAlert
Only one replication group can be active at a time.

1.4 Deleting a replication group

Follow this procedure to delete a replication group. Remember, you can only delete a replication group that is inactive.

1.5 Adding a repository

1.6 Removing a repository

1.7 Moving a repository to another replication group

To move a repository from one replication group to another:

1.8 Setting MultiSite to start up on system boot

Subversion MultiSite comes with sample scripts debiansvnreplicator-init.d-sample and svnreplicator-init.d-sample which can be found in the svn-replicator/bin directory. A copy of the script debiansvnreplicator-init.d-sample (written for Debian, though also compatible with Ubuntu) is listed here:

tipTip
The script features a status check that confirms the following:
  • Script run with necessary permissions (root)
  • Confirms correct version of script is being run (for O.S.)
  • Perl installation
  • JAVA environment variable
  • user is valid
  • locates the replicator binary
  • shutdown script
  • config file
  • replicator running
#!/bin/bash
#
# description: Ubuntu Linux dist compatible init.d script 
#              for starting/stopping WANdisco svnreplicator daemon
#
# Copyright (c) 2006-2016 WANdisco,Inc. Pleasanton,CA,USA
# All rights reserved.
#
# Author: WANdisco Support Staff <support@wandisco.com>
#
############################################################
# MODIFY THE FOLLOWING VARIBLES TO MATCH YOUR ENVIRONMENT

export JAVA_HOME=/usr/lib/jvm/java-6-sun
WD_USER="root"
WD_INSTALL_DIR="/home/user/svnrep/svn-replicator"
# (WD_INSTALL_DIR _MUST_ end with "/svn-replicator" or errors in installation can occur.

############################################################
# DO NOT MODIFY THESE VARIABLES

# Source function library.
. /lib/lsb/init-functions

CURRENT_OS=`uname -sv`
OS_DEFINER="/etc/debian_version"
PERL_LOC="/usr/bin"
WD_AGENT_NAME="svnreplicator"
WD_SHUTDOWN="${WD_INSTALL_DIR}/bin/shutdown"
WD_AGENT_BIN="${WD_INSTALL_DIR}/bin/${WD_AGENT_NAME}"
WD_AGENT_CONF="${WD_INSTALL_DIR}/config/prefs.xml"

# Maximum number of file descriptors.
MAXFDS="65000"

############################################################

# Startup procedure.
start() {

# Checks to see if the replicator is running.
REPRUN=`ps ax | grep "${WD_AGENT_NAME}" | grep -v grep | wc -l` 
RETVAL=$?

if [ ${REPRUN} -gt 0 ]
then
  echo "The replicator is already running."
  echo "------------------------------"
else
  echo "Starting ${WD_AGENT_BIN}:"
  # Don't produce core files.
  ulimit -S -c 0 >/dev/null 2>&1
  # Set maximum number of file descriptors.
  ulimit  -n ${MAXFDS} >/dev/null 2>&1
  start-stop-daemon --start --quiet --chuid ${WD_USER} --exec ${WD_AGENT_BIN} > /dev/null 2>&1

  # Checks the last transaction and returns a value depending on success or failure.
  RETVAL=$?

  # Sleep to prevent errors in restarts.
  sleep 4

  if [ ${RETVAL} -eq 0 ] 
  then
    echo "Status Code: $RETVAL: The ${WD_AGENT_NAME} started successfully."
    echo "------------------------------"
  fi

  if [ ${RETVAL} -eq 2 ] 
  then
    echo "Status Code: $RETVAL: The ${WD_AGENT_NAME} failed to start successfully."
    status
  fi
fi
}

############################################################

# Shutdown procedure.
stop() {


REPRUN=`ps ax | grep "${WD_AGENT_NAME}" | grep -v grep | wc -l` 
RETVAL=$?

# Checks to see if the replicator is running. If it is, script will shut it down, otherwise
a message will be printed.
if [ ${REPRUN} -gt 0 ]
then
  echo "Shutting down ${WD_AGENT_NAME}"
  start-stop-daemon --start --chuid ${WD_USER} --exec ${WD_SHUTDOWN} > /dev/null 2>&1

  # Checks the last transaction and returns a value depending on success or failure.
  RETVAL=$?
  
  # Sleep to prevent errors in restarts.
  sleep 4

  if [ ${RETVAL} -eq 0 ] 
  then
    echo "Status Code: $RETVAL: The ${WD_AGENT_NAME} shutdown successfully."
    echo "------------------------------"
  fi

  if [ ${RETVAL} -eq 2 ]
  then
    echo "Status Code: $RETVAL: The ${WD_AGENT_NAME} failed to shut down successfully."
    status
  fi
else
  echo "The replicator is currently not running."
  echo "------------------------------"
fi

}

############################################################

# Replicator status checker.
status() {

echo "------------------------------"

# Checks for to make sure that the person running the script is root.

if [ `id -u` = 0 ]
then
  echo "[OK] This script is being run by root."
else
  echo "[FAIL] This script is not being run by root. Please switch the user to root
  to avoid any potential errors."
fi

echo "------------------------------"

echo "The current operating system is ${CURRENT_OS}."
# Checks the system info see if Ubuntu is installed.
if test -f "${OS_DEFINER}"
then
  echo "[OK] The current operating system is valid to run this script."
else
  echo "[FAIL] You are not running on this script on a valid operating system."
fi

echo "------------------------------"

echo "Checking ${PERL_LOC} for Perl binary."
# Checks to see if Perl is installed.
if test -f "${PERL_LOC}/perl"
then
  echo "[OK] Perl is installed on this machine."
else
  echo "[FAIL] Could not find Perl on this machine."
fi

echo "------------------------------"

echo "The Java environment variable is currently set to: ${JAVA_HOME}"
# Tests for the JAVA_HOME variable.
if test -f "${JAVA_HOME}/bin/java"
then
  echo "[OK] The JAVA_HOME variable is set correctly."
else
  echo "[FAIL] The JAVA_HOME variable is not set correctly."
fi

echo "------------------------------"

echo "The user is currently set to: ${WD_USER}"
# Tests to see if a valid user exists.
id "${WD_USER}" >/dev/null 2>&1
RETVAL=$?
if [ ${RETVAL} -eq 0 ]
then
  echo "[OK] This is a valid user."
else
  echo "[FAIL] This is not a valid user."
fi

echo "------------------------------"

# Returns the base directory of the {WD_INSTALL_DIR}.
BASE_INS_DIR=`basename ${WD_INSTALL_DIR}`

echo "The install directory is currently set to: ${WD_INSTALL_DIR}"
# Checks to see if the WD_INSTALL_DIR ends with "/svn-replicator"
if `[ "${BASE_INS_DIR}" == "svn-replicator" ]`
then
  echo "[OK] The replicator is installed to a suitable directory."
else
  echo "[FAIL] Please rename the replicator install directory to /svn-replicator to
  avoid installation errors."
fi

echo "------------------------------"

echo "The replicator binary is currently set to: ${WD_AGENT_BIN}"
# Checks to see if the replicator binary exists in the specified directory.
if test -f "${WD_AGENT_BIN}"
then
  echo "[OK] The replicator script was found successfully."
else
  echo "[FAIL] The replicator script was not found."
fi

echo "------------------------------"

echo "The shutdown script is currently set to: ${WD_SHUTDOWN}"
# Checks to see if the config file exists.
if test -f "${WD_SHUTDOWN}"
then
  echo "[OK] The shutdown script was found successfully."
else
  echo "[FAIL] The shutdown script was not found."
fi

echo "------------------------------"

echo "The config file is currently set to: ${WD_AGENT_CONF}"
# Checks to see if the config file is readable.
if test -r "${WD_AGENT_CONF}"
then
  echo "[OK] The config file could be read."
else
  echo "[FAIL] The config file could not be read."
fi

echo "------------------------------"

# Checks to see if the replicator is running.
if ps ax | grep -v grep | grep ${WD_AGENT_BIN} > /dev/null
then
  # Returns the process ID of the replicator and checks the relevent file for the file descriptors.
  PROC_ID=`pidof -sx "${WD_AGENT_BIN}"`
  FDS_LOC=`grep -i "Max open files *${MAXFDS}" /proc/${PROC_ID}/limits`
  RETVAL=$?

  echo "The attempted max number of set file descriptors is ${MAXFDS}."
  # Checks the /proc/${PROC_ID}/limits file for the file descriptors.
  if [ ${RETVAL} -eq 0 ]
  then
    echo "[OK] The file descriptors were set successfully."
    echo "------------------------------"
  else
    echo "[FAIL] The file descriptors were not set successfully."
    echo "------------------------------"
  fi
  echo "[OK] The replicator is currently running."

else
  echo "[FAIL] The replicator is currently not running."
fi

echo "------------------------------"

}

############################################################

# Restart procedure.
restart() {

  stop
  sleep 4
  start

}

############################################################

case "$1" in
start)
  start
  ;;
stop)
  stop
  ;;
status)
  status
  ;;
restart|reload)
  restart
  ;;
*)

echo $"Usage: $0 {start|stop|status|restart}"
exit 1

esac

1.9 Changing the Subversion replicated client port

To change the Subversion replication port that clients use to connect to repositories:

1.10 Shutting down MultiSite

You can shut down a replicator at any time, either from the node's admin console (see the Admin Console Guide 5.9 Shut down node), or through running the shutdown script ./shutdown in the svn-replicator/bin/ directory.

2. Setting up the replicator for production

2.1 Setting up MultiSite as a Windows service

We offer a Perl plug-in that lets you run MultiSite as a Windows service through the Windows Service Control Manager.

To set up MultiSite as a Windows service you need:

Follow these steps on all the nodes where you want to run your WANdisco product as a Windows service:

2.2 Setting up MultiSite as a Linux/Unix service

Your installation provides the following scripts:

Apply these steps on all nodes.

  1. Make sure that your MultiSite installation is in a directory called svn-replicator.
  2. Copy the relevant script to /etc/init.d/svnreplicator.
  3. Make the script executable to the root user.
  4. Edit the script to set the correct values for the following variables:
    export JAVA_HOME=/usr/java/default
    WD_USER="root"
    WD_INSTALL_DIR="/opt/wandisco/svn-replicator"
        
    where:
    • JAVA_HOME is the location of the Oracle JDK installation.
    • WD_USER is the user that will run the Java svnreplicator process.
    • WD_INSTALL_DIR is the location of the MultiSite installation.
  5. Activate the scripts to start on boot and stop on shutdown per the OS-provided utilities (chkconfig, update-rc.d).

2.3 Logs

The service creates a winservice.log, located in the svn-replicator\logs directory.

2.4 Changing the quorum type

To change the quorum type create a new replication group and then specify the new quorum type.
See Creating a New Replication Group>.

2.5 Toggling the quorum check

Subversion MultiSite verifies if a network quorum is reachable when a write command is submitted. If the quorum is unreachable, by default, the write command is aborted and the following message appears on the Subversion client console:

Check the Network connectivity, failed to reach a
minimum quorum of nodes. Aborting the svn write operation.

To turn off the quorum check, edit the prefs file (svn-replicator/config/prefs.xml). Set the parameter, AlwaysVerifyQuorum to false in the file. For instance,

<SVNProxy>
<AlwaysVerifyQuorum>false</AlwaysVerifyQuorum>
....
</SVNProxy>

If the check is turned off and quorum is un-reachable, the write transaction will be applied to the WANdisco Subversion MultiSite's transaction journal and stay in a pending state till network connectivity and quorum is restored.

** Alert! **Alert
With singleton quorum, if the current node is also the distinguished node, the quorum check will always succeed irrespective of network connectivity to other nodes.

2.6 About Watchdog mode

By default, Subversion MultiSite starts in watchdog mode. Whenever the replicator goes down, the watchdog mode restarts it. In watchdog mode, the replication process automatically disassociates from the terminal and becomes a daemon process, so you should not try running it in the background (with &).

Watchdog mode is not supported in Windows, use Windows Cygwin.

You can turn off watchdog by typing:

-nowdog

If MultiSite is unable to start up, for example if it terminates several times in quick secession, watchdog starts the the node in read-only mode.

$ ./bin/svnreplicator -h

Usage:

svnreplicator [-v] [-verbose] [-nowdog] [-pause time] [-email email-address]

-v
Print the svnreplicator version
-verbose
Verbose console messages go to STDOUT/STDERR instead of logs/console.txt
-nowdog
Turn off watchdog mode. WANdisco will not restart automatically if it terminates. Use this option for testing.
-pause
Time in seconds that the watchdog pauses for, before restarting service. Defaults to 0 seconds.
-email
Specify an email address to send an alert to, whenever the Watchdog restarts or shuts down WANdisco. WANdisco generates an email per local replicator activity. If the email settings were not set up during installation, see Email Settings, described in Using the Admin Console.

Use the -email option to generate email alerts whenever MultiSite restarts. For instance:
$ svn-replicator/bin/svnreplicator -pause 5 -email "admin@blueandgold.com, scmuser@blueandgold.com"

** Alert! **Alert
In order to have the replicator automatically started on system reboots, see Setting Replicator to Start Up on System Boot.

2.7 Using SSL

Subversion MultiSite can be set up to use SSL encryption. First you should run through the following steps after extracting the Subversion MultiSite files (but before running setup).

SSLv3 is support (though not enforced). If your browser setting has SSLv3 disabled, you will get a handshake error message. If it has both SSLv3 and TLS enabled, then, depending on the browser, it will try to switch from TLS to SSLv3 during the handshake. If you receive a handshake error message in your browser, make sure that TLS is disabled and only SSLv3 is enabled. All current browsers support SSLv3.

2.8 Setting the server key

In the keystore, the server certificate is associated with a key. By default, we look for a key named server to validate the certificate. If you use a key for the server with a different name, enter this in the SSL settings.

2.8.1 SSL Troubleshooting

A complete debug of the SSL logging will be required to diagnose the problems. To capture the debugging, run the java process with:

'-Djavax.net.debug=all' flag.

To enable the logging of SSL implemented layer, turn the logging to FINEST for 'com.wandisco.platform.net' package.

2.9 Using SSL for both LDAP and emails

If you're specifying secure email (using a truststore) and LDAP authentication over SSL (using a truststore), the same truststore must be used for both sets of certificates. If different truststores are used then the LDAP truststore will overwrite the email truststore and secure emails will stop working.

3. Changing prefs.xml settings

Subversion MultiSite's settings are stored within a prevayler database. However, some settings (for all nodes in the replication group) are stored within a preference file (prefs.xml) which is located in the <WANDISCO/svn-replicator/config directory.

** Alert! **Alert
If you make changes that affect more than one node, you must change each node's specific file. If your change only effects just one node, you can change just that node's prefs.xml file.

3.1.1 Changing prefs.xml file on a single node

3.1.2 Changing All Nodes' prefs.xml Files

3.2 Performing a Synchronized Stop

** Alert! ** Alert
Previously, all nodes in a replication group needed to be available in order to perform a synchronized stop. Now it's possible to synchronize stop a subgroup of nodes which is useful if you need to bring your replication group to a stop even if you don't have all nodes up and working. However,         this action should only be used as a final resort and will require additional work to ensure that sync is maintained.

All Sync stops make Subversion read-only. You'll need to warn your Subversion users of some down-time.

Selective Synchronized Stops

If you wish to perform a synchronized stop on a subgroup of nodes, untick the node(s) that you do not wish to bring to a stop and click Continue.
Synchronized stop 01

You'll be shown a warning box:

"Are you sure you want to perform a synchronized stop on a subgroup?

If this is preparation for a reset, THE EXCLUDED NODES MUST BE RESET AS WELL and will then require
an rsync of every repository before they are restarted."

Synchronized stop 01

** Alert! ** Alert
The purpose of a synchronized stop is to ensure that all repositories are maintained in exactly the same state, that one node doesn't process transactions that are not completed on another. Any node that is not stopped in sync with the rest of the replication group is more than likely going to fall out of sync, causing replication to stop.

All nodes that are not included in the synchronized stop must be reset to purge all queued traffic and all their repositories must be manually synchronized (using rsync) with the rest of the replication group.

3.3 Resuming from a Synchronized Stop

3.4 Verifying that the replicator is working

The quickest way to check replication is to add a new user through the Admin Console of one of your nodes. Then, jump to another node and confirm that new user has appears there too.

Another way to check replication is to verify there are commit transactions posted to the log file
svn-replicator/logs/SVNProxyServer-prefs0.log. More about log files

3.5 Changing the Distinguished Node

3.5.1 What distinguishes the Distinguished node?

The distinguished node has greater voting power within the quorum. In a singleton quorum the distinguished node decides transaction ordering and keeps the other nodes in sync.

In a singleton qourum, Subversion users who operate from the distinguished node will see the best performance because their local replicator never needs to wait for agreement to be reached on a distant node.

In a singleton qourum it's advantageous to change the distinguished node to correspond with the site where the most repository changes are taking place, i.e. operating within business hours. You can manually change the distinguished node at any time. You can also automate the process by setting up a Rotation Schedule.

The distinguished node is selected from the replication group, it can be any node.

** Alert! **Alert
Changing the Distinguished Node requires a Unanimous Quorum where agreement must be reached between all nodes.
See the Troubleshooting guide if one or more of your nodes become unreachable.

In the majority quorum the distinguished node acts as a tie breaker when there are an even number of nodes. For example, with a 4 node setup, in order to achieve a majority you would technically need 3 nodes. With the distinguished node's slightly heavier weighted vote, you can achieve a majority with only 2 nodes as long as one of the two is the distinguished node.

3.5.2 Manually changing the distinguished node

3.5.3 Scheduled rotation of the distinguished node

3.6 Using Subversion hooks for sending e-mails

Many administrators like to set up Subversion backend hooks that fire whenever a Subversion user commits a set of file changes. With a master/slave Subversion server setup, e-mails can be initiated once when the post-commit trigger fires.

You can set up Subversion hooks to send emails when Subversion users commit file changes. However, with MultiSite, these emails will get duplicated, with a potential for spamming with notification emails.

To avoid this, set up one node as an "e-mail hub" by enabling the post-commit trigger to fire from a single node within the replication group.

Alternatively, you could use the time of day to fire the e-mail alerts from a specific node. For example, you could modify the post-commit trigger to send e-mails from India during 9:00 a.m. to 5:00 p.m. IST, and from the US during 9:00 a.m. to 5:00 p.m. PST.

tipCaution
If using asymmetrical e-mail hooks ensure that you don't disable the pre-commit trigger. That may cause a Subversion commit transaction to abort at some nodes but commit at other nodes causing a loss of sync.

tipTip
Configure email setup to avoid long blockages or delays. The default SMTP service on node should be adequate. We recommend that you set up a local e-mail hub or a local SMTP agent/server. The local SMTP server should preferably be on the same host as the Subversion server. It should be set up to forward/relay e-mails to the organization-wide SMTP server. This ensures the e-mail hooks are a lot faster and just need to enqueue the e-mails to the local SMTP server. Test that your emails are delivered before going into production.

3.7 Changing the admin console username or password

The default login username for the Admin Console is admin, and the password is user-defined during installation. That way, all nodes initially have the same login and password.

You can set up different login credentials for each node, however, you should ensure that all administrators who manage nodes are able to login to all nodes.

tipTip
We recommend that you keep admin authorization simple, and have the same login and password for all nodes. This will greatly simplify managing replication.

To change the login at a particular node, enter the following in prefs.xml.

<Security>
 <Admin>
   <user>newlogin</user>
  </Admin>
</Security>

3.7.1 Changing the admin password

3.8 Setting Up Hooks

A hook is a script that gets triggered by a specified repository event, such as creation of a new revision or the modification of an unversioned property. Read more about Hook Scripts.

Hook How to Integrate with WANdisco
start-commit Standard Subversion implementation.
Must be present at all nodes, and must either execute identically at all nodes or fail identically at all nodes.
pre-commit This becomes the pre-replication hook.
See the following Replication Hook section.
post-commit Standard Subversion implementation.
Must be present at only one node.
pre-revprop-change Standard Subversion implementation.
Must be present at all nodes, and must either execute identically at all nodes or fail identically at all nodes.
post-revprop-change Standard Subversion implementation.
Must be present at only one node.

3.8.1 Pre-Replication Hook

To use pre-replication hooks, Subversion MultiSite needs to be set with the version of Subversion you are using. Go to the Proxy tab, select SVN Settings, and select the Yes radio button for Use Pre-Replication Hooks. specify the Subversion Server Version. See Subversion Server Version in Using the Admin Console.

Use a pre-replication hook 01

The SVN DAV pre-commit hook becomes the pre-replication hook. Subversion does not execute it, instead MultiSite invokes it before forming a proposal. As per the SVN DAV specification, if the hook succeeds, nothing is communicated back to the client. The handling of the command proceeds normally. If the hook fails, stderr is packaged as an XML response to the client. In response, the client typically deletes the activity; i.e., cleans up the temporary files, etc., on the server side.

3.8.2 Configuration

Modify your prefs.xml file at all nodes to contain the following tags. See Changing a prefs.xml File. Below is a sample configuration of a pre-commit pre-replication hook.

tipTip
Make sure the hook is not installed in the repository's hooks directory, since you don't want the SVN server to find it.

<Hooks>
 <enabled>true</enabled>
  <list>
    <hook name="pre-commit">
     <command>C:/cygwin/home/user/bin/pre-commit.bat</command>
     <captureExitCode>true</captureExitCode>
    </hook>
  </list>
</Hooks>

3.8.3 Repository-Specific Hooks

If you want different hooks to act on different repositories, you can do so. Create a script (for example, pre-replication.sh or .bat) that contains a case statement that calls each repository-specific hook by passing the repository name as a parameter. Change the prefs.xml to point to that script.

<Hooks>
 <enabled>true</enabled>
  <list>
    <hook name="pre-commit">
     <command>C:/cygwin/home/user/bin/pre-replication.bat</command>
     <captureExitCode>true</captureExitCode>
    </hook>
  </list>
</Hooks>

3.9 Selective Replication

By default, MultiSite replicates all SVNROOTS associated with a Subversion repository. However, you can specify a set of SVNROOTS that you do not want to replicate. Use Excludes for SVNROOTS in the prefs.xml file to identify the repositories you do not wish to have replicated. For example:

<SVNProxy>
....
<ExcludeRepositories>/exclude,/dir0</ExcludeRepositories>
....
</SVNProxy>

If MultiSite can't find the entry, it's included in the replication. There are no wildcard (regular) expressions. For Apache and SVNPath, the syntax must match what is listed in the Location directive. For Apache and SVNParentPath, the syntax must match what is listed in the Location directive and Path to the repository.

All the included roots go through the same agreement manager. For example, say you have three repositories.

/repos/rep1
/repos/rep2
/repos/rep3

For SVNPath, the Location directive is

<Location/rep1>
  SVNPath /repos/rep1
</Location/rep1>

<Location/rep2>
  SVNPath /repos/rep2
</Location/rep2>

<Location/rep3>
  SVNPath /repos/rep3
</Location/rep3>

For SVNParentPath, the Location directive is

<Location/repository>
  SVNParentPath /repos
</Location/repository>

To exclude the first repository, rep1, for SVNPath, you would exclude rep1: for SVNParentPath, you would exclude /repository/rep1.

You must name a repository after the / character. Subversion does not support a location named with just the / character.

4. Updating Apache or Subversion in WANdisco Deployment

This procedure requires that your MultiSite deployment, including Subversion servers, are offline during the time it takes to upgrade at each node. Please plan this procedure accordingly.

5. Upgrading Subversion MultiSite

Use these instructions to upgrade your Subversion MultiSite to a later build or version, rather than those found in The Deployment Guide: 3. Installation section, which are intended for a first time installation.

The correct method for upgrading Subversion MultiSite depends on which versions you are moving between. Use the guide below to ensure that you upgrade with the right method.

Upgrading from Subversion MultiSite 3.6 or earlier - See our Knowledge base articles on Installation and Upgrade

tipUpgrade rule of thumb

When looking to upgrade Subversion MultiSite, apply the following rule:

Upgrading within a version, e.g. 4.0 build x --> 4.0 build y , use the Upgrade with Script

Upgrading between versions. e.g. 3.7 --> 4.0 will require you to Upgrade from backup

5.1 Upgrade from backup

Use this procedure if you are upgrading between product versions (3.7 to 4.0 etc.) rather than builds (4.0 build x to 4.0 build y etc.). To upgrade between builds, see Upgrade with script.

Summary of the Upgrade procedure:

tipInstalling in order to perform an upgrade using the backup.xml file?
Don't enable Authz during the installation - wait until the import is completed, then enable Authz from the Subversion Settings screen.
Enabling Authz during the installation can greatly impact the performance of the backup.xml import.

Stop MultiSite by performing a Synchronized Stop

 

5.1.1. Open the Admin Console, on the left-hand menu click Stop Proxy.

nothing goes to plan

5.1.2. Check that radio button for "Synchronized Stop of all proxies". MultiSite will now wait for any remaining transactions to complete, after which all nodes will stop listening, stopping replication.

Backup your settings from the admin console

5.1.3. Open the admin console and click on the System tab.

5.1.4. Click on Export Settings. Confirm the filename and location, then click on Backup Settings. All settings and user data will be exported in a backup xml file.

Next you need to Shutdown all nodes, then rename the svn-replicator directory to backup-svn-replicator.

Extract and run the setup file

5.1.5. Extract the installation files to the same location as your previous install, before it was renamed.

5.1.6. Copy the license evaluation key file from your old installation to the svn-replicator/config directory in your new installation.

5.1.7. Go to /svn-replicator/bin/, then start the installation with the following command:

perl setup

Respond to the Yes / No prompt relating to java memory settings.

upgrade01


5.1.8. The setup will now start up the browser based installer. Open your browser and go to address shown at the end of the setup. This will be the IP of your installation machine, on the WANdisco port (6444 by default).

upgrade with backup

Complete the upgrade through a browser.

5.1.9. From the Welcome screen, click Continue.

upgrade with backup


5.1.10. Once you've read the WANdisco End User License Agreement, click I Agree to continue with the upgrade.

upgrade with backup


5.1.11. Enter a password for the default Admin Console account, the username for this account is now admin (for earlier versions it was "root"). Click Next to continue.

upgrade with backup


5.1.12. The next two screens explain how MultiSite will act as a proxy between Subversion clients and the Subversion server. Click Next to continue.

upgrade with backup


5.1.13. By default, MultiSite will listen on port 80, while Subversion will listen on port 8080. The benefit of this setup is that Subversion end-user don't need to make any changes.

upgrade with backup


5.1.14. You'll confirm the proxy settings. These will be populated with the default settings noted in the previous screens.

Node Name: The name that MultiSite will use for this, the first node. The name cannot contain spaces, but can be changed later from the Admin Console.

Node IP: The node's IP.

Bind Host: By default, all network interfaces bind to the node. Select "Bind advertised host only" to limit to the node's IP address (stated in the field, above).

Client Port: By default this is 80, allowing Subversion users to continue without making change to their client setup.

Admin Console Port: 6444 by default. Should you need to use a different port for accessing the Admin Console, restart the setup (step 3.1.3) using the following command:

perl setup <preferred port for Admin Console access>

Reserved Ports: MultiSite will reserve a block of 10 ports, you can either leave the default ports or enter ports that you prefer.

Once you've finished making any changes to the proxy settings, click Next to continue.

upgrade with backup


5.1.15. Setup will check the Apache config file for settings that might cause problems for MultiSite.

If the httpd.conf file isn't found, enter its path into the Configuration File entry box, then click Reload Configuration.

Look out for warning boxes for where setup finds a problem. You'll need to manually edit the httpd.conf file, then click on Reload Configuration to have setup check your changes.

User: Owner of the file.

Group: The group in which the owner belongs.

KeepAlive: Setup will look to see that the Keep-Alive directive is set to On.

KeepAliveRequests: Setup will look for 0, which indicates that no maximum limit will be set for connection requests.

KeepAliveTimeout: Set very high (500,000 seconds) to ensure connections don't timeout.

Listening IP: For a node with multiple IPs, this will indicate the IP used for listening.

Listening Port: The default listening port is 8080.

Override Listen directive with a virtual host? Tick if you are using a virtual host.

upgrade with backup


5.1.16. Setup now allows you to modify your Subversion settings. Watch for alerts that confirm the port and path that MultiSite will associate with Subversion.

SVN Settings

Use Pre-Replication Hooks: Select whether you are using Pre-Replication Hooks.


SVN and WANdisco are on the same server: Confirm if MultiSite and Subversion are running on the same machine, this is a requirement for MultiSite to manage your Subversion password file.

Subversion Server Port: 8080 by default.

SVN Executable:This is the fully qualified path to the Subversion executable. Setup will try to fill this in automatically, otherwise type it into the entry box.

Use authz-based access control Click on the box if you are using Authz access control. See the following note about why you may wish to hold off selecting this option until later.

tipInstalling in order to perform an upgrade using the backup.xml file?
Don't enable Authz during the installation - wait until the import is completed, then enable Authz from the Subversion Settings screen.
Enabling Authz during the installation can greatly impact the performance of the backup.xml import.

At the bottom of the screen is a table that confirms the DAV Location and password control for your repositories. You can click edit to make changes. You can add additional repositories by clicking Add Repository. To continue setup, click Next.

upgrade with backup

5.1.17. You can enter email settings so that MultiSite can send alerts.

SMTP Authentication: If you select No, you'll need to provide your account.
Username and Password in the following entry boxes.
Use SSL/TLS: Choose yes if you wish to send emails over a secure connection.
Host: Enter the address of your mail server.
Port: Enter the SMTP port, 25 by default.
Send Admin Notification To: The email address to where notifications will be sent.

You don't need to provide email settings, in which case, click Skip to continue. Otherwise, click Next.

upgrade with backup

 

5.1.20. You have now completed setup. You can go back and make changes or click Complete installation with these settings to save them and restart MultiSite.

upgrade with backup


5.1.18. You will next see a Completing Installation screen, which MultiSite restarts. When completed, you'll see the authentication screen for the Admin Console. Enter the username 'admin' and the password you entered during setup to enter.

upgrade with backup



Import your saved settings from the Admin Console

From the Admin Console, click on the System tab. Click on Import Settings. Enter the path to your backup file or folder (exported in step 5.1.5) or click on the magnifying glass to browser for the file. Click Import Settings.

Important All files for use with WANdisco MultiSite should use UTF-8 encoding (and/or single-byte encoding if there are no none-ASCII characters). Any other encoding eg. Unicode will not be parsed correctly.

upgrade with backup

You've now completed the upgrade.

Restoring your replication groups

5.1.19 Next, you'll need to recreate your replication group and apply other settings that don't get preserved (such as Consistency Check or Distinguished Node Rotation Schedules).

5.1.20 Start by redefining your other nodes, using the Add a node procedure.

5.1.22 With your nodes defined you should recreate your replication groups. Use the Create a new replication group procedure.

5.1.23 Finally, activate the replication group, if you select to use the packaging option "Activate Nodes via SSH?" then installation of your other nodes will be handled automatically.

5.1.24 Before resuming replication of your live repository data, perform some tests to make sure that replication is working properly.

5.2 Upgrade using the script

Use this procedure if you are upgrading between builds (4.0 build x to 4.0 build y etc.) of the same product version.

5.2.1 Before you begin

Things you must know before you make a start:

** Alert! **Alert
Some MultiSite settings will need to be manually applied after the update. Take note of the following settings so that you can re-enter them at the end of the upgrade:

** Alert! **Alert
Don't make changes to your setup (replication groups or nodes) during this procedure. If you plan to make changes, complete the upgrade first, then make changes once you have confirmed the upgrade has worked.

5.2.2 Upgrade steps

5.2.3 Perform a reset on all nodes

5.2.4 Create a backup in case a rollback is required

5.2.5 Run the upgrader script

** Alert! ** The upgrader script is only for moving to a later build of the same product version.
Do not use the upgrader script for moving from 3.7 to 4.0, or 4.0 to 4.1.
When upgrading between full versions, backup your user data, complete a fresh installation, then import your backed-up data.
         See Upgrade from backup.

Complete these steps on all nodes:

** Alert! ** Alert
Don't forget to repeat these steps for each node.

5.2.6 Using the Upgrader script

** Alert! ** The upgrader script is only for moving to a later build of the same product version.
Do not use the upgrader script for moving from 3.7 to 4.0, or 4.0 to 4.1.
When upgrading between full versions, backup your user data, complete a fresh installation, then import your backed-up data.
         See Upgrade from backup.

You can read more about how to use the upgrader script by running it with the -help flag.

$ java -jar svn-replicator.jar -help

** Alert! ** Alert
Valid parameters are:

-dprefs show dynamic preferences.
-nodes show node information.
-upgrade <path> upgrade an existing installation at the specified path.
-upgradeNoConfirm <path> upgrade an existing installation at the specified path WITHOUT PROMPTING FOR CONFIRMATION.
-help | -h show help

6. Authz Notifier Utility

The Authz Notification (authznotifier.jar) utility passes the location of the diff file containing the changes between the existing and newly updated Authz file to the replicators, along with the output of the md5sum tool on the existing Authz file.

6.1 Usage

The authznotifier.jar file is located in the svn-replicator/lib directory.

Authz notifiier (When not using SSL encryption) runs using the following command:

java -Dauthznotifier.nossl=true -jar authznotifier.jar </path/to/admin/password-file> </path/to/diff>
<md5-value> [-patchtool </path/to/patch>] [-md5sumtool </path/to/md5sum>]

example:
java -Dauthznotifier.nossl=true -jar authznotifier.jar /sec/pass.file /home/scm/diff.out "2889b653854f68872fbb6c771348f3d9 /home/scm/svn.auth"

Authz notifiier (When using SSL encryption) runs using the following command:

java -jar authznotifier.jar </path/to/admin/password-file> </path/to/diff>
<md5-value> [-patchtool </path/to/patch>] [-md5sumtool </path/to/md5sum>]

example:
java -jar authznotifier.jar /sec/pass.file /home/scm/diff.out "2889b653854f68872fbb6c771348f3d9 /home/scm/svn.auth"

Key

</path/to/admin/password-file>
The full path to the password file.
</path/to/diff>
The full path to the diff file. The diff file is generated by running the command:
              diff <original-authz-file> <new-authz-file> > diff.out
<md5-value>
The result of running md5sum on the full path to the original authZ file. e.g. md5sum /path/to/original-authz-file
<patchtool>
The full path to the tool that should be used to apply the patch.
Defaults to /usr/bin/patch
<md5sumtool>
Full path to the tool for generating new md5sum values, e.g. /usr/bin/md5sum Defaults to /usr/bin/md5sum

The password file should be in plain text and contain MultiSite's admin authentication details in the following format:

Username
Password

Or, if you use the default 'admin' username, then the password file can just contain the password.



To use a different port, invoke java with -Dauthznotifier.port=xxxx
For example:

java -Dauthznotifier.port=xxxx -jar authznotifier.jar /path/to/password-file
</path/to/diff> <md5-value> [-patchtool </path/to/patch>] [-md5sumtool </path/to/md5sum>]

6.2 Disabling LDAP for Admin Authentication

6.3 LDAP Admin Authentication URL syntax

The LDAP Admin login functionality lets you specify an LDAP group or subtree that contains users who can login as administrators. The format of the URL is:

ldap(s)://host:port/basedn?attribute?scope?filter
ldap
For regular LDAP, use the string ldap. For secure LDAP, use ldaps instead.
host:port
The name/port of the LDAP server.
basedn
The DN of the branch of the directory where all searches should start from. This would typically specify a group populated with admin users
attribute
The attribute to match user names against. If no attributes are provided, the default is to use uid. It's a good idea to choose an attribute that will be unique across all entries in the subtree you will be using.
scope
The scope of the search. Can be either one, sub or obj. The default is to use a scope of sub.

-one: Entities will be searched below the DN one level only - if an entity with a matching attribute=username is found then the user is considered an admin.
-sub: Entities will be searched below the DN throughout the entire subtree - if an entity with a matching attribute=username is found then the user is considered an admin.
-obj: A specific object is being identified by the DN - if this object is located AND a matching attribute=username is found then the user is considered an admin.
filter
A valid LDAP search filter. If not provided, defaults to (objectClass=*), which will search for all objects in the tree.

** Alert! **Once a user is deemed to be an admin based on the criteria above, authentication is carried out against the LDAP Authority system, so it is important to ensure a relevant LDAP authority has been defined.

6.4 LDAP Timeouts

MultiSite's pre-replication authentication operates with some timeout values which may need to be modified to match your environment.

ldapauth.timeout
Use this property to set the timeout for LDAP (in milliseconds), the default value is 15000 (15 seconds).
ldapauth.cachettl
Use this property to set the length of time that credentials are cached if they have been successfully authenticated against the LDAP service. The default is 600,000 (600 seconds).

To override these defaults, ensure that the following statements are present as SERVER_JVM_ARGS: statements in the
<WANdisco Installlation>/bin/reputils.pm file:

$SERVER_JVM_ARGS .= " -Dldapauth.timeout=<LDAP timeout> -Dldapauth.cachettl=<Cache expiry time>";

Both timeouts are in milliseconds

tipExample LDAP Timeout error as it appears in the log:

Mon Feb 28 15:27:24 GMT 2011 1298906844401 org.nirala.util.services.ldap.LDAPAuthenticator getLdapContext
        WARNING: [request-handler-1] LDAP response read timed out, timeout used:1ms.
        Mon Feb 28 15:27:24 GMT 2011 1298906844402 org.nirala.util.services.ldap.LDAPAuthenticator isUserAuthed
        WARNING: [request-handler-1] LDAP Authorization failed: null
        Mon Feb 28 15:27:24 GMT 2011 1298906844402 org.nirala.util.services.ldap.LDAPAuthenticator isUserAuthed
        INFO: [request-handler-1] LDAP Auth - User: test@wandisco.com Authenticated: false

Troubleshooting

1. Finding the Last Committed Transaction

Even though committed transactions are always in the same order for each node, the timing of the commits usually varies from node to node. So unless there are no Subversion users logged in, you probably are going to have variations per node for committed transactions.

Go to any node's Dashboard. Type:
http://<IP address>:<WANdisco port number>/dashboard2

You see all the nodes on the Dashboard to compare the listed transactions. Click on a Tx Id (Transaction ID) to view more details about the WebDAV command, along with it's transaction number.

2. Repair Repository

Repair Repository is a tool for making a repair of corrupted repositories. Previously, repairing a repository would require that all nodes in the replication group be brought to a stop, making all repositories read-only until the repair was completed.

2.1 Repository Repair Overview

This overview will help you understand the sequence of events that are involved in a typical repository repair, for a step-by-step explanation of how to repair a repository see the Repository Repair Procedure.


2.2 Repository Repair Procedure

This is a step-by-step guide to doing a repository repair. Before running through this procedure, it may be helpful to read through the overview which explains how the nodes and repositories change their state through the Repository Repair Overview.

WARNING: After starting a repository repair you Must Not navigate away from the repair browser session or open up a new session on the node. To ensure that the repair doesn't fail or stall, don't perform any other action on the node until the repair has been completed.

2.3 Manual overwrite of the corrupted repository

With this option, you will take an up-to-date copy of the repository from the helper node, once you have the copy, click "I have copied the repository". The helper node will be returned to duty, and will catch up with the remaining active nodes.

During a repair, don't close the admin console session, or navigate away from the screen. You shouldn't start a repair if another repair is underway on other nodes in your replication group.

We strongly recommend that you organise to have exclusive access to the admin consoles of your nodes during a repair, to ensure that no actions are triggered that might cause the repair to fail.

This section illustrates a possible method for creating a copy of the repository.

2.4 Manual rsync corrupted repository

With this repair method you'll use the rsync command to synchronize the out-of-date repository with the good copy on the helper node.

2.5 Disabling Access to Subversions

** Alert! **Alert
This makes Subversion read-only. You may need to warn you Subversion users.

Temporarily Disabling Subversion Access At Selected Nodes

You can stop transactions at one or more nodes. For a discussion of stopping one, but not all, nodes, see WANdisco is Listening

** Alert! **Alert
To stop replication you need to run a synchronized stop.

2.5.1 Disabling Subversion access (all nodes)

To stop all nodes at once, run a synchronized stop.

2.6 Emergency Reconfiguration of Quorum

Use this procedure as a last resort, if the distinguished node (in a singleton quorum) has become permanently unavailable. The procedure basically creates a new replication group that excludes the unreachable node.

If a quorum can't be formed, an alert will appear at the top of the admin console.
Follow the "Click here for more details" link.

Emergency Reconfiguration of Quorum 1

Use the emergency reconfiguration procedure only when the recovery of the distinguished node will take an unacceptable amount of time.

Using a Singleton Quorum - If the distinguished node is periodically reachable, instead, try using the Change Distinguished Node while its available.

Using a Majority Quorum - If the quorum is periodically reachable, instead, wait for the quorum to become available and recreate the replication group, using a singleton quorum, instead of using the emergency procedure. You can change back to the majority quorum after stabilising your nodes.

tipCaution
We don't recommend using this procedure because of the potential for data loss. If the distinguished node is unreachable by other nodes it could still be processing transactions on its own (with singleton quorum, the distinguished node can operate alone). Therefore, if the other nodes form a new quorum that excludes the distinguished node, any transactions made at the distinguished node since it became unreachable to the other nodes would be lost.

2.7 Using rsync to repair a repository using an up-to-date copy

This process can fix a corrupted or out-of-sync repository by restoring it to a replica of the repository that is known to be in a good state.

From the machine with the master repository, type the following commands:

rsync -rvlHtogpc /path/to/local/repositories/<Repository Name> remoteHost:/path/to/remote/repositories/

For example:
rsync -rvlHtogpc /Subversion/Repo/db/ root@172.7.2.33:/Subversion/Repo/

Then follow up with an additional rsync that will ensure that contents of the locks directory are indentical (by deleting locks that are not present on the originating server)

rsync -rvlHtogpc --delete /path/to/local/repo/db/locks/<Repository Name> remoteHost:/path/to/remote/repo/

For example:
rsync -rvlHtogpc --delete /Subversion/Repo/db/locks root@172.7.2.33:/Subversion/Repo/db/

tipKnowledgebase
You can read a more detailed step-by-step guide to using rsync in the Knowledge Base article Reset and rsync Subversion repositories.

How read-only repositories can lead to performance problems
If a transaction fails on one node, and is not bypassed, the repository is placed in a read-only mode to stop the failure from halting replication between the replicas on the other nodes. The subsequent transactions between the other replicas will stack up until the read-only replica is again writable. Garbage collection is held up and on the node with the read-only repository, all buckets subsequent to the failed transaction remain active and continue to eat into the Java heap. In a heavy traffic environment this can lead to Java heap overflows unless the repository is quickly recovered or taken offline.

tipKnowledgebase
For a detailed explanation of how Subversion MultiSite's Garbage Collection works, see the article Garbage Collection.

2.8 Automatic offline mode

A read-only repository will switch offline should the number of transactions exceed a threshold. This threshold is controlled from the Garbage Collection screen, found on the admin console's System tab.

Admin Console - Garbage Collection Settings

For an explanation of the Garbage Collection page, see the Admin Console Guide - 4.5 Garbage Collection.


The threshold is called the Auto Offline High Water Mark and by default it is set to 100,000 transactions. You can increase the number if there's sufficient Java heap space to handle large amounts of queued transactions. If JAVA resources are tight, you can reduce the threshold so that a read-only repository is taken offline sooner.

Automatic Offline Example

Here is an example of the Automatic offline mode in action.

Manual offline mode

A repository that is read-only can be placed offline through the admin console. The option to take a repository offline will coincide with the option to attempt a bypass of a failed transaction.

Admin Console - Offline Mode Button

From the Proxy tab, click on Repository Status which is the second link on the status menu.

3. Talkback script

When you do contact WANdisco with a problem, the first thing WANdisco support asks for is the talkback file. Run the file by typing at svn-replicator/bin
perl talkback

Type in the pathname to SVNROOT when prompted. The output looks like this:


Please open a ticket by visiting http://support.wandisco.com and upload the /talkback-<machine name>.zip, with a description of the issue.


Do not email the talkback files, only attach them via the web case interface.
The zip file is located at the root directory. Do not email the .zip file, just attach them to your case. Read our Knowledgebase article about How to raise a support case.

4. Connection Request Timeout Messages

Sometimes in the WANdisco logs, you see connection request timeout information messages logged. These are informational messages and should be ignored unless it is guaranteed that the connection can be established in xxx milli-seconds and happens often.

In normal operation of MultiSite, two connections are established between each of the replicated machines, MultiSite connection and a DFTP connection. These two connections were established when MultiSite started and are used when required. A keep-alive signal is periodically sent on the WANdisco port. There is no traffic on DFTP until a file transfer begins.

Some lesser routers in the path of the two end points will close an established connection if there is no traffic on the connection without notifying the end points. When end points sent data on this stale connection, they hang forever. To deal with these lesser routers, MultiSite does not keep the DFTP open in its connection pool forever. MultiSite establishes a DFTP connection from receiver to sender when a file transfer was required. This solved the problem dealing with lesser routers.

Some companies have a corporate policy that network connections can only be established in one direction. To deal with this scenario, the replicated machines establish a DFTP connection to other nodes periodically and tear them down if there is no traffic within a known interval. Once a connection is established, any side is free to use the connection regardless of which side initiated the connection. A connection in use is never torn down until it is available as a free connection. This is the current implementation.

It takes between 300 to 400 milli-seconds to establish a network connection even on a slow WAN. By default, MultiSite waits for 500 milli-seconds before giving up that a connection cannot be established to a peer machine and prints this informational message. What if the establishments of connection always take 501 milli-seconds. In this case, a connection is never established. To solve this problem, the timeout value is adjusted in 10% increments of the last timeout, starting at 500 milli-seconds, to a maximum of 10 seconds for each timed out connection.

Upon establishment of a successful connection, this timeout value is used for subsequent connection establishment unless an adjustment is required for failed attempts.

5. VPN, NAT, Firewall Timeouts

This section is useful if you are experiencing issues with slow commits on the non-distinguished node or if you have port-forwarding in your environment.

In a multi-site configuration, most nodes are connected through a WAN. Often, VPN and NAT devices are used to do IP translation and port forwarding. These devices need to maintain state in order to do the port forwarding on-the-fly. This state can grow if not cleaned out. Many devices simply reset the internal state after an inactivity timeout. For example, some Cisco NAT routers reset state after 7200 seconds or 2 hours.

The MultiSite replicator uses persistent TCP connections between the replicators. If these TCP connection are going through a NAT or port forwarding device, it's important to tune the VPN and/or the TCP stack at the replicator host machine. Many NAT devices have buggy implementation that resets the internal state without resetting the TCP connections.

In such a situation, the replicators may see a connection as established but no communication actually happens. The symptoms include a slow commit that is blocking WAN communication. You can run netstat -a | grep <DConEPort> to see if the TCP send queues are backing up. That, in conjunction with slow commits that appear to be hanging, or frozen, typically indicates the NAT is not gracefully resetting TCP connections.
You can further confirm this by using tcpdump or ethereal to check for excessive retransmissions on the DConENet connections. You could also look at your VPN/NAT device log to see if it reset any DConENet connections that appear to be in an ESTABLISHED state via the netstat -a command.

These are a few ways of addressing the issue :

6. A node is in read-only mode

7. Updating Apache or Subversion

This procedure requires that your MultiSite deployment, including Subversion servers, are offline during the time it takes to upgrade at each node. Please plan this procedure accordingly.

8. Missing License Key file

Subversion MultiSite depends on a license key file being present in the svn-replicator/config directory for each node. Please get a valid license from WANdisco and copy the file to the config directory. MultiSite won't start without the license file.

8.1 I'm getting a severe exception

I'm getting a SEVERE exception, and replicator is aborting the Subversion transaction and shutting down.

If you get a message in the logs/SVNProxy*.log file similar to
svn: Commit failed (details follow):
svn: File not found: transaction '10-d', path '/development/Hello.txt'

it means the replicator has detected an out of sync condition. Remember, the replicator continuously monitors your repository for any out of sync issues. If it detects this has occurred, it triggers an automatic shutdown to prevent further corruption.

This could happen if someone accidently committed directly to Subversion, bypassing the replicator, and ramped up the version in one node without giving the replicator any chance of replicating. This can be easily resolved by following the reset procedure outlined in I Directly Committed to Subversion, How Do I Rsync?.

Follow all precaution to avoid bypassing the replicator:

Admin console guide

Subversion MultiSite has a browser-based admin console for the management of replication groups, setting changes or viewing system logs.

Connect to the admin console through a web browser, using your server's hostname or IP address, along with the administration port (defaults to 6444).
e.g.

http://10.2.5.101:6444   (http://(Server IP):(Admin port)

1. Admin console key

key

The admin console screen consists of the following elements:
Top menu bar: includes the product license and any expiry date, along with links to documentation and support.

Section tabs: The admin consoles different sections are seperated on tabs. They consist of the following sections:

** Alert! **Alert
The layout and features available in admin console alter depending on your license agreement. So you may not see all the elements described in this guide.

Jump to node...: This drop-down link will open the admin console for one of the other nodes. You can identify which node you're viewing by checking the webpage's title.

Section menu: Appears on the left-hand side and contains links to features that are specific to the tab.

Product version line appears at the bottom of the screen and shows the product version and build, the version of Java installed and the operating system.

2. Login

On the authentication screen there are a number of links available, even if you don't login.

key

Quick Links

Proxy Status - Proxy Status Displays the node's status in the tab's main panel.
Transaction Status - Lets you check on the status of individual transactions.
Dashboard - Lists DAV traffic
Change User Password - Follow this link to change the password of a User, it's not used for changing the Admin password.

3. Security

The security tab handles all user related functionality.

tipTip
The Security tab features product capabilities that are available when the Access Control product has been bundled with Subversion MultiSite. The documentation for using Access Control can be found in the Access Control Admin Guide.

tipCaution
If you are using LDAP based authentication, do not make changes to users using the admin console user administration.

LDAP passwords
Use only alphanumeric characters in LDAP passwords. Due to a bug in sn Apache/mod_dav_svn, the use of non-alphanumeric characters in passwords could result in replication failure.

3.1 Role Administration

Roles are used to define the permissions available to different kinds of users.

Create Role

Create new roles and assign them privileges.

Admin Console - Create Role

Role Name: Enter a name for the new role.

Tick the checkboxes that correspond with the Subversion file permissions that you want to assign to the role.
The available permissions are:

Click Create Role to save the new role.

3.1.1 List Roles

View the roles that have already been set up, along with their corresponding permissions.

Admin Console - List Roles

Display all roles, including pre-defined and any new roles created using the Create Roles screen.

To delete a role, click on the corresponding checkbox and click Delete Selected.
Admin Console - List Roles

To Update a role, click on the Role Name and change the permissions by ticking and unticking the checkboxes. Then click Update Role.
Admin Console - List Roles


3.2 User Administration

Manage Subversion user accounts. If you have set Subversion MultiSite to manage the Subversion password file, then any user that you create in the admin console, will automatically be added to the Subversion password file.

3.2.1 Create User

Enter the details of the Subversion user, select a default role.
Admin Console - Create User

Create User
Enter the details of a new Subversion User.

tipTip
Usernames can contain any characters except for ~ (tilde), " (double quote) or : (colon). When importing users, it's possible to include a comma in a username by using an escape character, e.g. ,"Reninngton\, Jr.","Oscar"

Select the Default Role, which will decide which file permissions the user will have. e.g. With the Default Role set to "Developer" the user will have write,copy,delete permissions. When the User details are entered click Create User to save them.

3.2.2 List Users

All users entered in the system are listed here. Though users who were already present in the Subversion password file will not appear unless they are imported or entered into the admin console. The Last Recorded Access column shows the date and time that each user last accessed (or attempted to access) a Subversion repository.
Admin Console - List User

** Alert! **Alert
The finer details about how the Last Recorded Access is handled:
  • A Failed authentication is recorded with an access time.
  • Read transactions will update the user Last Recorded Access time on the local node only.
  • Write transactions (in a successfully executed proposal) will update the user Last Recorded Access time on all nodes.

Userid You can view and edit each user's details by clicking on that user's Userid entry.

Admin Console - Create Group

3.2.3 Import User

Admin Console - Import User

Import Users You can import an existing list of users. The import file must be a comma delimited text file, of the format; userid,lastName,firstName,role,email[,group1[,group2...groupN]]


Important All files for use with WANdisco MultiSite should use UTF-8 encoding (and/or single-byte encoding if there are no none-ASCII characters). Any other encoding eg. Unicode will not be parsed correctly.


3.2.4 Change Admin Password

Admin Console - Change Password

Change Admin Password You can change the Access Control Admin password with this screen.

Group Administration

Control the groups used for organising Subversion users.

3.2.5 Create Group

Admin Console - Create Group

Create Group
Create a new group.

Group Properties
Name: A name for the group
Description: A field for writing an explanation of what the group does.
Client IP Pattern to Allow: Provide an IP range pattern to determine membership of the group.
Manage User Membership: Select either Manually or via LDAP.
Manually handles the assignment of users to groups through the admin console.
Via LDAP allows an organisation's LDAP service to control which users are assigned to groups.
Add Users Add users to the group.
Remove Users Remove users from the group.
View Membership View the group's current members.

Admin Console - Create Group
Current LDAP Query: The LDAP query that identifies the group membership requirements
LDAP Authority: The LDAP service that is currenly being use for LDAP.
Group Filter: The filter used to identify the criteria for group membership.
Imported User Default Role When a user is added to the group, this is the role they are given by default.
Run on this node: Select whether the groups membership is managed via the LDAP settings on this node. Unselected, changes to Access Control will replicate from the node on which the LDAP control is running.
View Membership View the group's current members.
Regenerate Membership Use this to poll LDAP for group membership, this will recreate the group membership.
Access Control Rule: You can add access rules that group members will inherit. Click the Add allow or add deny to add a new rule line. You can then specify whether a rule is an allow or deny by using the dropdown.

Group Assignment
You can organise groups using the assignment feature. Selecting Root will place the new group at the top of the groups tree. You can also select another group, which will place the new group as a subgroup.

Click Create Group to save the settings.

3.2.6 List Groups

Admin Console - List Group

List Group Shows all the available groups.

You can Edit, Delete groups or List Users.

3.2.7 Assign Users

Admin Console - Assign User

Assign Users Allows you to assign users to groups. If a user is already in a group, his or her name does not appear in the list of available users.


3.2.8 Remove Users

Admin Console - Remove User

Remove Users Use this to remove users from a group.


3.2.9 Import Groups

Admin Console Group Import

Import Groups You can import a list of existing groups. The import file must be a comma delimited text file, of the format groupname,parentname[,description]. If there is no parent name, specify null.


Important All files for use with WANdisco MultiSite should use UTF-8 encoding (and/or single-byte encoding if there are no none-ASCII characters). Any other encoding eg. Unicode will not be parsed correctly.


3.3 ACL Administration

This menu can be toggled off. See Toggling the ACL Options. For a complete discussion on ACLs, See About Access Control Lists.

3.3.1 Create ACL

Admin Console - Create ACL

Create ACL Create more than one at a time, use List ACLs.

User / Group: Enter either a User or Group name.
Rule: Either Allow or Deny.
Privilege:Permitted file operations - list, read, delete, copy, admin. The 'admin' privilege permits all operations and is the equivalent of ticking all the other privileges.
Operate on: User or Group
IP Pattern: An IP address or a partial pattern (using regular expressions).
File/Dir Pattern: Either a file or a directory within the repository.

Click Create ACL to save the new list item.

3.3.2 List ACLs

Admin Console - List ACLs

List ACLs Lists all existing ACLs. You can create, edit or delete ACLs from this screen. Use the List ACLs screen if you need to create multiple ACLs. Click edit to make changes to an existing rule, delete to delete a rule.

tipTip
Access Control uses the concept of a valid Principal with adequate privileges to access a secured resource. Powerful Perl style regular expressions can be used wherever patterns are allowed. Principal can be any valid user, group or IP pattern for instance - engineering.* (note the dot) or 217\.[0-9]+ are all valid. By default the HEAD branch is specified but you can enter a regular expression just as well - release9\.0_.* for instance.

3.4 ACL Options

ACL Options

The Access Control Options screen is used to filter display of Role ACLs and Sibling Groups.

Show Role ACLs? Yes or No. "Yes" will enable the Access Control Lists screens, allowing fine-tuning of user and group controls. With "No" selected, the ACL Administration section of the menu will no longer appear. The only ACLs that apply will be those inherited from assigned groups.
Display Sibling Groups? Yes or No. With a "Yes", when creating or editing groups you will be shown a group entry for each role. This gives you finer control over how access control rules are applied.

4. System

The system tab provides access to logging and backup features.

4.1 Log Viewer

Admin Console - Log View

MultiSite log files can be viewed from here.

console.txt
Logs from the operating system concerning the status of MultiSite's running processes.
reset.log
Logs instances where MultiSite has been reset.
SVNProxyServer-prefs.log.(time-stamp)
The MultiSite log file contains most of the activities of the replicator.
user-import.log
Logs instances where users have been imported.
WebProxy.log.(time-stamp)
Logs errors and changes applicable to the MultiSite replicator.

tipKnowledgebase
There's a knowledgebase article about WANdisco Log Files.

user-import.txt
Logs each user import that has been completed on the node.

4.1.1 Limiting Log lines

Logs can be generated over multiple lines, which can make reading/performing greps difficult. It's possible to limit each log to a single line by using the following procedure:


  • 1.Set the following line in the svn-replicator/config/log.properties file:
    org.nirala.trace.WandiscoLoggingFileHandler.formatter = org.nirala.trace.SingleLineFormatter
    
  • 2. Replacing any other formatter for this handler.
  • 3. Apply the above change to all nodes.
  • 4. Perform a synchronized stop, then restart all nodes.

4.2 Disk Monitor

The disk monitor can be used to lessen the chance of a node running out of disk space, which would result in the loss of replication or data corruption.

SVN disk monitor
Use disk monitor: Tick the checkbox to have the disk monitor running.

Interval between checks (in milliseconds): 900000 (15 minutes) is the default interval time between checks.

Determine disk state by: Choose to measure disk usage either by percentage used or bytes free (in bytes, or append KB, MB or GB).

Warn Level: State the level of usage (either as a percentage or bytes free) at which point an alert email will be sent, and resent each time the disk monitor completes a check. So, if you set the Interval between checks to 900000 milliseconds, a warning email will be triggered every 15 minutes.

Critical Level: State the level of usage (either as a percentage or bytes free) at which the node will send an alert email and then shutdown. This will prevent the data on the node from being corrupted.

Click the Save Settings button to save any changes you make.

** Alert! **Alert
You MUST restart the node for any changes you make to take effect.


4.3 License Info

license

The License Info page contains the details of your WANdisco product license, useful for confirming the details of a number of license-specific features.
Company: Your company name.

Product: The product type. This is currently either svn (Subversion) or (CVS).

License Type: This indicated whether the license is either for evaluation or production.

Access Control:
Shows if Access Control has been enabled on the product. With Access Control enabled, the User's tab is renamed "Security" and is expanded to include many additional features for Access Control.

Number of Allowed Nodes: Number of replicator instances that have been licensed.

Licensed IP Addresses: The IP addresses of each node.

Number of Allowed Users: Total number of possible users that are covered under the license.

License Expires: The date on which the current license will expire.

Maintenance Start Date: The first date covered by the current maintenance contract.

Maintenance End Date: The final date covered by the current maintenance contract.


4.4 System Settings

Admin Console - Log View

The system settings screen provides a number of extra controls that allow you to fine-tune MultiSite for potential performance improvements.

Auto skip bypassable transactions: Tick the box to force the replicator to automatically skip bypassable transactions.

A transaction is bypassable so long as it fails on all nodes within the replication group. In spite of the failure, all repository replicas remain in sync, allowing replication to continue.

Max INFORM Messages: This is a an advanced tunable parameter that can limit network notification traffic in situations where a node is pulling in a large number of changes, such as while recovering from a brief network outage.

Default value: 1000,
For Unlimited INFORM Messages use the setting "-1".

The default value of 1000 is optimal for the majority of network configurations and file sizes, but can be tuned to make best use of higher or lower bandwidth networks handling varying file sizes.

Once the value is altered at one node, it is replicated to other nodes that are part of the replication group. We recommend that you consult with our support team before changing this setting.

HTML format failed pre-replication hook output: The labelling of the output of failed pre-prelication hook scripts can cause problems. Untick the checkbox to removed the extra formatting, which will make outputs less human-readible but will support scripts that were written around the previous output.

Use generic file replication: Use this checkbox to activate the RESTful API function that allows for generic file replication.

** Alert! **Alert
Turning on this feature allows you to replicate files within your file system, i.e. files outside of your repositories such as Authz or Apache configuration files. The feature should be activated with caution as it allows the admin console user to overwrite any file.


Automatically send problem repos offline on global transaction:
Tick this option to ensure that any repositories that are "read-only" are sidelined (taken offline) if a global transaction arrives, so that they don't risk halting the whole replication group. This setting is required because a global transaction needs to pass on every repository before it can execute. Unable to pass against a read-only repository, the transaction will get stuck until the read-only repository can catch up, blocking further repository traffic. Sending the read-only repository offline clears all pending proposals and allow replication of the other repositories to continue.

Limit commit size:
The commitance of a very large file can easily clog DFTP transfer between nodes, which will hold up all other traffic behind a single PUT operation. It's possible to mitigate against attempts to commit giant files using pre-replication hook scripts, however, hooks don't fire commits until the MERGE command is sent, which can be too late to stop the disruption.

Admin Console - Garbage Collection Settings

Instead, use this check box to set a maximum file size for a commit. When switched on, the default is 1,024MB (1GB). You can use the units: "KB", "MB" or "GB" to denote your filesize. Known issue: although a comma is shown in the default value, currently the entry doesn't support commas.

LDAP group update frequency: Time between the refreshing of LDAP group membership, specified using h, m or s (not case sensitive) to denote hours/minutes/seconds eg. "1h 30m 10s". An unmodified number will be interpreted as milliseconds. The default is 5m.

Default role to assign incoming users from REST: Users who are added via the RESTful API are automatically assigned a default role, selected from the dropdown list.

4.5 Garbage Collection

Admin Console - Garbage Collection Settings

The Garbage collection page lets you tune MultiSite's memory recycling system - the process by which objects that are no longer require get deleted in order to free up resources.

tipKnowledgebase
For a detailed explanation of how Subversion MultiSite's Garbage Collection works, see the article Garbage Collection.


Max buckets: The maximum number of buckets to clear in a single garbage collection; prevents the system being overwhelmed if a large number of transactions have built up (default 1000).

Auto Offline High Water Mark: The number of transactions between a problem transaction and the most recent one where the repository should be taken offline and all proposals against it discarded to prevent it occupying excessive heap space (default 100 000).

Auto Offline Low Water Mark: After auto-offlining is triggered, the difference between the oldest remaining problem transaction and the most recent transaction when auto-offlining will automatically cease (default 50 000).

For more information about Auto Offline settings, see the Troubleshooting section: 2.8 Offline mode.


4.6 Transaction Status

Admin Console - Trans Log

The Transaction Status page lets you query individual transactions to check on their status.
Transaction Number: The transaction's reference number.
Node Submitted From: The originating node of the transaction.


4.7 Log Level

Admin Console - Log Level

MultiSite uses one log, and the default level is info. The levels vary from severe, where you get only the most severe warnings, to finest, which logs every action.

Logger Name: References the logger object's name.
New Log Level: The following options are available:

Severe
message level indicating a serious failure.
Warning
message level indicating a potential problem.
Info
message level for informational messages.
Config
message level for static configuration messages
Fine
message level providing tracing information.
Finer
message level providing additional tracing information.
Finest
indicates a highly detailed tracing message.


4.8 Free Memory

Admin Console - Free Memory

This command frees the memory (GC stands for garbage collection) for the current node. The command occurs when you click on this menu selection. The display shows information on the command that was just performed.

Max mem used by JVM the maximum memory that JVM can use on the current node.
Free memory before GC the amount of free memory before you ran this command
Free memory after GC the amount of free memory after you ran this command
Memory freed the total amount of memory freed at the command's completion

4.9 Dashboard

Admin Console - Dashboard

Dashboard provides a list of replicated transactions.

By default, the Dashboard doesn't refresh automatically. Click Update to force a refresh, or set an automatic refresh by enterying a refresh rate, in seconds in the Auto Refresh Every box.

As soon as MultiSite receives a Subversion transaction request, that transaction joins the replication group's queue. There is one queue for the replication group. Pending transactions are not displayed, but are counted in the Total Transactions Pending box in the upper right.

When MultiSite finishes processing a transaction, its displayed in the TX Id field. The Replicator field shows the node from where the transaction originated.

e.g. 0.0.0.0:6444 means the local node originated the transaction.


4.10 Export Settings

Admin Console - Export Settings

Export Settings Allows you to export settings, including all users, for later importation into a WANdisco product.


4.11 Import Settings

Admin Console - Import Settings

Allows you to import settings, including all users.


Important All files for use with WANdisco MultiSite should use UTF-8 encoding (and/or single-byte encoding if there are no none-ASCII characters). Any other encoding eg. Unicode will not be parsed correctly.

5. Proxy

The Proxy tab handles proxy settings which alter the way that MultiSite works between Subversion and users.

5.1 Proxy Status

Admin Console - Proxy Settings

Proxy Status Displays the node's status in the tab's main panel.


5.2 Repository Status

Admin Console - Repo Status
Repository Status gives you the status of all repositories that are hosted on the node.

Show All: Click on Show All to display all hosted repositories.

Show Read-only: Click on Show Read-only to show only those hosted repositories that are in read-only mode.

Auto Refresh: The Repository Status screen will automatically refresh every xxx seconds. Click on Auto Refresh to turn this feature off.

Repository Search: Use this entry box to seach for a particular repository.


For each Subversion repository there is a table entry that includes the following information:

Admin Console - Proxy Settings
Tick the checkbox to select multiple repositories for the Retry Transactions command.

DAV Location:The directory used for access to the repository.

SVN Repository Path: The absolute path to the Subversion repository.

Scheduled TXs: Number of transactions that are scheduled for replication.

Completed TXs: Number of transactions that have been completed.

Bypassable TXs: Number of transactions that can be bypassed. These are transactions that have failed on the node, but can be ignored by the replicator because the transaction has failed on all nodes in the replication group. To force the replicator to bypass the transactions, click on Bypass. You can choose to have any bypassable transactions, bypassed automatically by using the Auto skip bypassable transactions checkbox, situated on the System Settings.

Retry TXs: Number of transactions that can be retried. If write access to the repository was briefly lost, the retry command allows you to replay the transactions and hopefully bring the repository back up-to-date.

Retry Count: Number of attempted retries that have been made.

Details: Click to view the dashboard for the corresponding node.

Bypass: Click to attempt to Bypass the transaction. This command will only be available if the transaction can be bypassed.

Retry: Click to retry any pending transactions.

Take offline: Click to sideline the repository by putting it into the offline mode. This tells the other nodes to press on, and not queue up subsequent proposals. Once a repository has been taken offline, it can never catch up and will require a Repository Repair.

Repair: Click to being a repository repair. There's a full explanation of how to complete a respository repair in Repair Repository.

5.3 Log Viewer

Admin Console - Log Viewer

Log Viewer You can view the logs, including the main log SVNProxyServer-prefs.log.0


5.4 SVN Settings

These settings control how Subversion is set up and include host and authorization details.

SVN Settings
Use Pre-Replication Hooks
Indicates whether pre-replication hooks are being used. (Default: No). A pre-replication hook is a script that gets triggered by a specified repository event, which runs before the event is replicated to the other nodes. If you are using pre-replication hooks, select Yes.

tipKnowledgebase
You must select the version of Subversion you are using, otherwise pre-replication hooks will not work. See Setting up hooks.

SVN and WANdisco are on the same server
There are signifcant benefits from running both Subversion and MultiSite on the same host, for example, MultSite can manage the Subversion password.
Subversion Host
The IP address of the Subversion Server. If Subversion is running locally you can put 'localhost'.
Subversion Server Port
The port on which Subversion is actually running (8080 by default).
SVN Executable
The fully qualified path to the svn executable. On Linux you can run "which svn" to determine the path to the file.
Replicate the location of the svn exectutable to all nodes
It's best to have your nodes set up in exactly the same way. However, if you need to run Subversion from different locations on each node, select Only set the location of the svn exectutable at this node.
Use authz-based access control?
If you intend to use Apache's Authz authorization, tick the check box.
Authorization File
If you have selected to use Authz, enter the path to the authorization file here.
Use LDAP for authorization
If your organisation runs an LDAP service, you can use it to handle Subversion authorization. Tick the check box to use LDAP.
Replicate LDAP Changes
If your LDAP settings are the same for all nodes then select Define LDAP Authorities and settings globally. If LDAP settings are different on each node then select Define LDAP Authorities and settings locally.
Edit LDAP Authorities
Click on the Edit LDAP Authorites link to view or modify the main LDAP settings.
Continue onto next LDAP authority on user not found:
If enabled, when a username is not found in a valid authority, authentication will continue onto the next valid authority (equivalent of "AuthzLDAPAuthoritative off" in Apache)
Continue onto next LDAP authority on user authentication failure:
If true, if a user is found in an LDAP authority, but fails authentication, authentication will continue onto the next valid authority (equivalent of "AuthLDAPBindAuthoritative off" in Apache)
LDAP Server Required TruststoreTruststore File
Select this to indicate that you are authenticating your nodes using a truststore.
Truststore File
Path to the file where SSL public keys are stored. This may also be the KeyStore file.
Truststore Password
The password for the truststore file.
Use LDAP for WANdisco Admin authentication
Tick the box if you want to manage access to the Subversion Multisite admin console using LDAP. If selected an additional button will appear to View Admin Users, once connected to your LDAP server you can use this button to review those users who have administrator permissions.
LDAP connection URL for WANdisco Admin users:
The LDAP connection URL in the format ldap(s)://host:port/basedn?attribute
Bind User (optional):
The Bind DN to use on connecting to admin LDAP (required if your LDAP server does not permit anonymous binding).
Bind Password (optional):
The password for the bind DN

Save Settings Click to save your settings.

Regenerate Authz File If an Authz file is deleted or corrupted, it's now possible to regenerate the file by clicking this button.

Edit LDAP Authorities
Clicking on the Edit LDAP Authorities link takes you to a separate window in which multiple LDAP authorities can be added.

Admin Console - LDAP Auth Entry

Order:
Assign a number for this LDAP authorities relative priority. Authentication will run through available LDAP authorities, going from lowest order number to highest, using the first server that matches the user's username against their regular expression.

Regular Expression The regular expression determining whether a user should be authenticated against this authority. The wildcard ".*" will ensure that all uses the authority.

URL: The LDAP URL to authenticate against in the format ldap(s)://host:port/basedn?attribute. Equivalent to AuthLDAPURL in Apache's MOD_AUTH_LDAP.
Bind User DN (Distinguished Name) (optional) A DN to bind with when accessing the given LDAP URL. Equivalent to AuthLDAPBindDN in Apache's MOD_AUTH_LDAP.
Bind Password (optional) The bind DN (Distinguished Name) password. Equivalent to AuthLDAPBindPassword in Apache's MOD_AUTH_LDAP.
Add Authority: The "Add Authority" button will add the LDAP details.

Repository Directory and DAV Location

You can Edit, Delete or Check the consistency of the Subversion Repository.

Add Location: Click to add an additional location to the directory in which you are managing repositories.

Edit Repository

Admin Console - Edit Repository
Admin Console - Edit Repository

Directory on File System: This is the fully qualified path to the Subversion repository. Not the URL used by clients. You can seach for the location by clicking on the magnifying glass button.
Manage Password File: Tick the checkbox to have MultiSite control your Subversion password file. You can only use this feature if both MuliSite and Subversion are located on the same server.
Username and Password: Username and password of the Subversion user account that will be used to browse the repository. The account will need read and write access to the whole repository.
DAV Location: the directory in which WebDAV is located.
Multiple SVN Repositories: Choose whether you are using SVNParentPath or SVNPath in the Apache location directive.

If you select Yes, SVNParentPath is used in the apache location directive, you'll be presented with a entry box that allows you to select and edit available Root Names.

Consistency Check

Provides a basic check that replication is working properly by comparing Subversion revision information between nodes. Although this is a fairly superficial test of consistency, it will more often than not pick up on problems.

Admin Console - Consistency check

The following parameters are available, allowing for more selective checks to be performed:

DAV sub path: Select the Repository by its DAV sub path.

From Revision: The most recent revision to take the consistency check from. Can be specified by revision number or "HEAD" to indicate the latest in the repository.

To Revision: The oldest revision to include in the consistency check.

Revision Increment: The increment to display the revisions in eg. display every 5th, 10th, 100th revision (1 to display all revisions in range, can be used to speed up result retrieval for large revision ranges).

# revisions: The maximum number of revisions to include in the consistency check.

Admin Console - Consistency check

When a consistency check confirms that replicas are consistent the screen will appear as above

Admin Console - Consistency check

When a consistency check finds that there's an inconsistency between replicas the screen will appear as above.

Schedule Consistency Check

Admin Console - Schedule Consistency check

Allows you to set an automatic consistency check once per day or once per hour.


5.5 Email Settings

Admin Console - Mail settings

Email Settings Email settings that Access Control uses to send status alerts.


5.6 Change Distinguished Node

Admin Console - Change DN

Allows you to change the node in your replication group that holds distinguished status.

Change the distinguished node immediately. Verify with Proxy Status.

Distinguished Node Rotation Schedule

Admin Console - Change DN

If you have a distinguished node rotation schedule in place, the distinguished node is changed immediately, but then the schedule rotates to the next node at the scheduled time of rotation.


5.7 Repair Repository

Admin Console - Stop Proxy

5.8 Stop Proxy

Admin Console - Stop Proxy

Stop Proxy Stops MultiSite from and prevents clients from writing to the repository.


5.9 Shut down node

Admin Console - Shut down node

Shut Down Node Shuts down MultiSite completely on the node.

To restart the node:
Go to svn-replicator/bin and type:

perl svnreplicator


Replication

Provides control over node and replication group settings

5.10 Nodes

Admin Console - Nodes

Lists the available nodes. You can edit or create a new node.


Create / Edit Node

Admin Console - Create Node

5.11 Replication Group

Admin Console - Create Replication Group

View or create new replication groups.

Admin Console - Create Replication Group

5.12 Replication Group History

Admin Console - Replication Group History

A table presents the current and any past active replication group activity.


6. Reports

The reports tab gives you acess to various reporting tools for use with the optional Access Control component of MultiSite.

6.1 Configure URI

Admin Console - Create Replication Group

Configure URI If you set up a web-based PHP report form, you enter its address here.


6.2 User Group Report

Admin Console - Create Replication Group

User Group Reports Generate User Group reports and view them with Log Viewer in the System and Proxy tabs.


6.3 Audit Reports

Admin Console - Create Replication Group

Access Control logs any Subversion user access, these logs are controlled through the Audit Reports tab.

Technical Guide

1. Introduction

This technical guide will help you understand the underlying technology and technical concepts that relate to WANdisco's Subversion MultiSite software.

We'll be using terms like "node" and "replication group" without explaining what we mean. Check out the Glossary for an explanation of these and other WANdisco terms.

2. MultiSite Overview

Subversion is designed to run as a central server to which multiple Subversion clients connect. WANdisco's replication technology makes it possible to have multiple active replicas of a Subversion repository that are in synch. The Subversion replicas can be anywhere on a WAN - distributed throughout a company's campus or throughout the world. WANdisco users experience the performance of a local Subversion repository, with the semantics of a single shared Subversion repository. We call this "active replication with one-copy-equivalence."

Replication ensures that each replica acts as a hot backup to every other replica. If a local server experiences a problem that takes it offline, only local users are disrupted, the rest of the replication group continues as normal.


Example Replication Group


SVN MiltiSite
This illustration shows a replication group with five Subversion severs.



WANdisco offers a High Availability solution based on Zeus Software that ensures no disruption in service. Even if a Subversion server failed, a Zeus cluster ensures that Subversion and repository data continue to be available.



SVN MS
Subversion MultiSite acts as a proxy between the Subversion Server and clients. An instance of the proxy runs at each replica. All the communication paths involved in the operation of WANdisco are illustrated in diagram above.

3. WANdisco MultiSite Concepts

All MultiSite nodes are synchronized at all times: each Subversion repository is a functional replica of the others. WANdisco replication technology is the concept of one repository, multiplied. Because there are multiple synchronized repositories, each replicated node is effectively a current hot backup, which makes disaster recovery easy to plan and implement.

The Subversion usernames and passwords on all repository hosts must match. This is required because MultiSite creates a peer-to-peer replication system. Any replica of the Subversion repository is accessible by every valid Subversion user. WANdisco offers the user the option of having MultiSite manage the Subversion password file.

4. How Replication Works

The sites in the replication group are continuously coordinating the Subversion write transactions users are making. The group establishes transaction ordering through the agreement of a quorum of replicas. When you install the first node, that node by default is the distinguished node with Singleton quorum. When you create the replication group that includes other nodes, you select the quorum type best suited to your configuration. For which quorum is best for particular situations, see Quorum Recommendations.

Singleton Quorum Singleton Response quorum - only one of the nodes in the membership decides on the transaction order. With Singleton Response quorum, the node that decides transaction ordering is called the distinguished node. The Singleton quorum offers the fastest response time for those users working at the distinguished node, because as soon as the distinguished node determines that a transaction can be processed in the correct order, the transaction is sent to Subversion. Any replicator except the distinguished node can go down, but the replication group continues. The replication group replays the missing transactions when that node rejoins the group. However, the Singleton quorum also represents a single point of failure, since replication halts if the distinguished node fails.

Majority Quorum Majority Response is another quorum option, whereby you specify that a majority of the sites must agree on transaction order before any transaction is committed. Having a majority quorum ensures that if one site goes down in a replication group, even the distinguished node, the other sites can continue uninterrupted, as long as a majority of the sites remain available. The replication group replays the missing transactions when that site rejoins the group.

In a majority quorum, the distinguished node?s role is that of a tie-breaker. For example, in a four node replication group, three sites make the quorum (three sites must agree about transaction ordering). If two nodes want one transaction first, and the other two want another transaction first, then the distinguished node gets a weighted vote. The group with the distinguished node determines the transaction ordering. With an even number of nodes with majority quorum, you can schedule the distinguished node to rotate to different nodes around the world, to "follow the sun".

Unanimous Quorum The last quorum option is unanimous response, which requires that all replicators must be reachable to accomplish transaction ordering.

5. Replication Example

Here is an overview of what occurs when a write transaction is received by any replicator in the replication group.

  1. The originating client sends the transaction to Subversion MultiSite, which passes it along through the replication group.
  2. Transaction data is successfully received by the quorum (the distinguished node for Singleton quorum, or a majority of sites for Majority quorum). The quorum assigns the transaction a Global Sequence Number (GSN).
  3. After receiving the transaction, each Subversion MultiSite passes the transaction data to its local Subversion server.
  4. Each local Subversion server processes the transaction.
  5. Subversion MultiSite waits for Subversion to complete the transaction. Subversion MultiSite only marks the transaction complete when Subversion returns a completion status. If for some reason replication goes down during this process (the replicator crashes, is stopped by an admin or the server it's running on shuts down), Subversion MultiSite does not mark the transaction as complete, and it gets reprocessed upon replication restart.

6. WANdisco is Listening

There is a field in the Admin Console that tells you whether WANdisco is accepting any incoming Subversion client requests. Replication still continues among the WANdisco sites, whether WANdisco is listening or not at one or more sites.

WANdisco is listening

You can turn the listening on and off through the Admin Console (through the Start Proxy and Stop Proxy commands). Issuing the Stop Proxy command on a site puts that Subversion server in read-only mode.

The following illustration shows Sites 2 and 5 are not listening. (An administrator executed the Stop Proxy command for those sites.) Replication continues, and Sites 2 and 5 are still receiving and processing replicated transactions originating from the other sites. However, Subversion users at Sites 2 and 5 cannot make any write transactions. Once an administrator issues the Start Proxy command for Sites 2 and 5, the local Subversion users can again issue Subversion commands.

SVN MiltiSite 3 not listening

Please follow your company guidelines in regards to notifying Subversion users of maintenance.

7. Synchronized Stop of All Sites

When an administrator issues a synchronized stop command, the Subversion servers stop accepting write commands from clients. Pending transactions are processed, but no new write transactions are accepted. Subversion users continue to have read access to the repository, but cannot perform write operations, such as commit or lock.

When an administrator issues a resume command, the WANdisco proxies restart and begin accepting write transactions.

SVN MultiSite synchro stop

8. Handling Node and Network Failure

8.1 Node Failure

For Singleton Response quorum: If a nodes goes down, replication continues at the remaining nodes, as long as the quorum* is reached, although users connecting to the downed node will be unable to write to repositories until the node is repaired (and the quorum reached).

In a Singleton Response quorum, the distinguished node must be available.

As soon as a node comes back up, the replication group catches up the node on its missing transactions, so that all nodes are again synchronized.

For Majority Response quorum: say you have a five node replication group. One or two nodes could go down, and replication would continue at the other nodes, as long as a majority of nodes remain up. The one or two downed nodes go into read-only mode. As soon as a node comes back up, the replication group catches up the node on its missing transactions, so that all nodes are again synchronized.

8.2 Per-repository scheduling

Subversion MultiSite 4.0/GA includes some changes to transaction scheduling and failure handling. In previous versions, a single scheduler handled all transactions on a node, so that in the event of a single repository becoming corrupted, and entering read-only mode, all the repositories hosted on the node would also become read-only.

Previous model
Transaction failures would result in the replicator restarting and retrying an arbitrary number of times. If the transaction still failed, a check would be made to see if the transaction was bypassable (this is so if it failed on all nodes). If the transaction is bypassable, replication would continue after a restart, if the transaction was not bypassable, the node would restart in read-only mode.

New model
Instead of a single scheduler per node, Subversion transactions now use a scheduler per svnroot, with AuthzProposal and UserPasswordProposal shared between all schedulers.

There are two key changes in behaviour, as a result of this change:

  1. 1. Failed transactions, or moving in or out of read-only mode no longer triggers the node to restart.
  2. 2. Failed transactions will now only interfere other transactions going through the same scheduler, and with global transactions. This allows the other repositories to continue to replicate, greatly increasing resilience in environments where large numbers of repositories are hosted on a single node.

8.3 Network Failure

If a network link goes down for one node, and outside connectivity is completely lost, there are two possible scenarios, depending on your quorum:

  • If you have Singleton quorum, and the distinguished node's network link goes down, the distinguished node alone can make progress. The Subversion users local to the distinguished node continue working uninterrupted, while users at other sites in the replication group can make only read operations (like up, co, log, etc.) working with stale data.
  • If you have Majority quorum, and one site's network link is lost, then users at that site can execute only read operations (like up, co, log, etc.) working with stale data. Providing that the remaining sites can still meet quorum (having a majority of sites responding), the other sites continue working uninterrupted

When connectivity is restored or the errored node is back online, the local node syncs up with the replication group automatically. First, the local node consults its local recovery journal (similar to a database redo log), and then, if necessary, attempts recovery from any of the quorum sites.

The recovery infrastructure and details of WANdisco fault-tolerance can be found at http://www.wandisco.com/get/?f=documentation/whitepapers/WANdisco_DConE_White_Paper.pdf.

9. LDAP Authentication

Authentication is handled by the MultiSite replicator performing an LDAP login attempt from the originating node, using the credentials supplied by the user. As users successfully authenticate, they are added to the replicator's user database and added to the (replicated) Apache password file with a known common password.

Apache is configured to use the password file containing common passwords and the WebDAV requests are populated with an authentication string from that password file. The remainder of the replication work flow remains unchanged. The key difference is that the authentication credentials used by Apache will not change over time and thus authenticated transactions can be executed on each replicated node.

tipCaution
If you are using LDAP based authentication, do not make changes to users using the admin console User Administration screen.

LDAP passwords
Use only alphanumeric characters in LDAP passwords. Due to a bug in sn Apache/mod_dav_svn, the use of non-alphanumeric characters in passwords could result in replication failure.

9.1 Requirements

  • Incoming requests must contain accessible username and password credentials.
  • The LDAP server must be available for authentication to succeed.
  • Only WANdisco is modifying the htpasswd file

9.2 LDAP data flow

WANdisco LDAP architecture

10. RESTful API


Some of the REST API commands have an associated commandline client to simplify execution, the clients can be found in the <svn-replicator>/lib directory.

The command line clients use a properties file to specify common properties; an example properties file, named restclient.properties is provided in the /svn-replicator/ config directory.

This file specifies the following attributes:

rest.client.host - the replicator hostname (required)
rest.client.port  - the replicator port (required)
rest.client.username  - the replicator login username (required)
rest.client.password  - the replicator login password (required)
rest.client.truststore  - the truststore file to use if SSL is enabled (optional)
rest.client.truststore_password  - the truststore password to use if SSL is enabled (optional)
rest.client.usessl  - specifies if SSL is enabled (optional, defaults to false)

tipAlert
The default port for the RESTful API is 8182.

10.1 Repositories

Overview: View existing repositories and add new repositories

Command-line client: svnrepo-client.jar

Usage: java -jar svnrepo-client.jar </path/to/properties/file> <command> [parameters]
e.g.

java -jar svnrepo-client.jar ../config/restclient.properties list

Commands:
list
addrepo <DAVLocation> </path/to/repository>
addroot <DAVLocation> <rootname> [-force]
-force = do not validate repository path on disk.


URLs:
http://<host>:<port>/wandisco-rest/repos
http://<host>:<port>/wandisco-rest/repos/<DAVLocation>

The end points use standard web security and will challenge for the admin username and password on invocations.

List all repos:
GET http://<host>:<port>/wandisco-rest/repos

List specific repo:
GET http://<host>:<port>/wandisco-rest/repos/<DAVLocation>

Add repository to SVNParentPath:
PUT http://<host>:<port>/wandisco-rest/repos/<DAVLocation>
form values must be specified as:
newroot=<child repository name> (parent path)
or
newpath=<repository name>


10.2 LDAP Authorities

Overview: View existing LDAP Authorities and add new LDAP Authorities

Commandline client: svnldap-client.jar
Usage: java -jar svnldap-client.jar </path/to/properties/file> <command> [parameters]
e.g.

java -jar svnldap-client.jar ../config/restclient.properties list
Commands:
list
add <auth-order> <auth-url> <auth-RE>
add <auth-order> <auth-url> <auth-RE> <bind-user> <bind-password>
delete <auth-order>
URLs:
http://<host>:<port>/wandisco-rest/ldapauths
http://<host>:<port>/wandisco-rest/ldapauths/<order>
The end points use standard web security and will challenge for the admin username and password on invocations.
List all LDAP authorities:
GET http://<host>:<port>/wandisco-rest/ldapauths
Delete an LDAP authority:
DELETE http://<host>:<port>/wandisco-rest/ldapauths/<order>
Add an LDAP authority:
PUT http://<host>:<port>/wandisco-rest/ldapauths/<order>
form values must be specified as:
authurl = <url>
authregex = <regex>
authsearchuser = <bind-user> (optional)
authsearchpass = <bind-user-pass> (optional)

10.3 Dynamic Preferences:

Overview: Add or overwrite non-replicated dynamic preferences.

Commandline client: dynpref-client.jar

Usage: java -jar dynpref-client.jar </path/to/properties/file> <command> [parameters]

e.g.

java -jar dynpref-client.jar ../config/restclient.properties <command>
Commands:
updatepref <encrypt> <pref-key> <pref-value>
URLs:
http://<host>:<port>/wandisco-rest/dynpref/<dynpref-key>

The end points use standard web security and will challenge for the admin username and password on invocations.

Add/Overwrite pref:

PUT http://<host>:<port>/wandisco-rest/dynpref/<dynpref-key>

form values must be specified as:
newvalue = <new-pref-value> encrypt = <true|false>

10.4 Distinguished Node

Overview: Display information about the current distinguished node.

No command line client available.
URLs:

http://<host>:<port>/wandisco-rest/dn

Get DN info:

GET http://<host>:<port>/wandisco-rest/dn
Optional Params: Will return only the specified value.
?name
?hostname
?ip
?port
?guid

Transaction Count

Overview: Display information about the current transaction count.

No command line client available.

URLs:

http://<host>:<port>/wandisco-rest/txns
shut
Get DN info:
GET http://<host>:<port>/wandisco-rest/txns

Replicator

Overview: Display information about the replicator and can shutdown the replicator.

No command line client available.

The end points use standard web security and will challenge for the admin username and password on invocations.
URLs:

http://<host>:<port>/wandisco-rest/replicator
Get Replicator status:
GET http://<host>:<port>/wandisco-rest/replicator
Shutdown replicator:
GET http://<host>:<port>/wandisco-rest/replicator?shutdown

Generic File Replication:

Overview: Replicate a file to all nodes.

tipKnowledgebase
A command line method for invoking generic file replication is currently being tested. If you would like to know more, Contact Support.


The end points use standard web security and will challenge for the admin username and password on invocations.
URLs:
http://<host>:<port>/wandisco-rest/filenotifier
Replicate File:
GET http://<host>:<port>/wandisco-rest/filenotifier?filelocation=
<source-file-location>&destlocation=<dest-file-location>
<dest-file-location> is optional

10.5 Rest Service Port

To specify the port that the REST server will listen on the following value must be passed into the JVM.

-Drest.server.port=xxxx

Where xxxx is the port number you wish to specify.

tipAlert
The default port for the RESTful API is 8182.

This can be configured at installation time by setting the environment variable WD_JVMARGS=-Drest.server.port=xxxx prior to running setup.

To configure this value after installation, open the svn-replicator/bin/reputils.pm file and edit each instance of the line:

$SERVER_JVM_ARGS = "-Xms128m -Xmx2048m -ea -server -Djava.net.preferIPv4Stack=true";

by adding the -Drest.server.port=xxxx to the end of the line. e.g.

$SERVER_JVM_ARGS = "-Xms128m -Xmx2048m -ea -server -Djava.net.preferIPv4Stack=true -Drest.server.port=xxxx";

11. Silent Installer Document Type Definition

To run the silent installer you require an XML file that is valid against the following setup.dtd:

<?xml version="1.0" encoding="UTF-8"?>
<!ELEMENT SetupSettings 
     (    
     adminPassword, adminUsername, nodeName, selectedNodeIP, apacheConfigFile?, bindHost,
     clientPort, adminConsolePort, apacheUser?, apacheGroup?, apacheKeepAlive?, apacheMaxKeepAliveRequest?,
     apacheKeepAliveTimeout?, apacheListeningIP?,apacheListeningPort?, usePreReplicationHooks,
     subversionServerVersion?, svnServerCoHosted, svnHost?, svnHostPort, svnExecutableLocation,
     ldapAuthenticationUsed, ldapConnectionUrl?, ldapTruststoreFile?, ldapTruststorePassword?,
     authzUsed, authzFile?, emailSMTPAuthRequired?,
     emailSettingsSkipped, emailUsername?, emailPassword?, useEmailSSL?, emailSSLTrustStoreFile?,
     emailSSLTrustStorePassword?,emailHost?, emailPort?, emailAdminAddress?, sslUsed, sslCertificateAlias?,
     sslKeyPassword?, sslKeyStoreFile?, sslKeyStorePassword?,
     sslDefaultTrustStoreUsed?, sslTrustStoreFile?, sslTrustStorePassword?, directories 
     ) >
 
<!ELEMENT adminPassword ( #PCDATA ) >
<!ELEMENT adminUsername ( #PCDATA ) >
<!ELEMENT nodeName ( #PCDATA ) >
<!ELEMENT selectedNodeIP ( #PCDATA ) >
<!ELEMENT apacheConfigFile ( #PCDATA ) >
<!ELEMENT bindHost ( #PCDATA ) >
<!ELEMENT clientPort ( #PCDATA ) >
<!ELEMENT adminConsolePort ( #PCDATA ) >
<!ELEMENT apacheUser ( #PCDATA ) >
<!ELEMENT apacheGroup ( #PCDATA ) >
<!ELEMENT apacheKeepAlive ( #PCDATA ) >
<!ELEMENT apacheMaxKeepAliveRequest ( #PCDATA ) >
<!ELEMENT apacheKeepAliveTimeout ( #PCDATA ) >
<!ELEMENT apacheListeningIP ( #PCDATA ) >
<!ELEMENT apacheListeningPort ( #PCDATA ) >
<!ELEMENT usePreReplicationHooks ( #PCDATA ) >
<!ELEMENT subversionServerVersion ( #PCDATA ) >
<!ELEMENT svnServerCoHosted ( #PCDATA ) >
<!ELEMENT svnHost ( #PCDATA ) >
<!ELEMENT svnHostPort ( #PCDATA ) >
<!ELEMENT svnExecutableLocation ( #PCDATA ) >
<!ELEMENT ldapAuthenticationUsed ( #PCDATA ) >
<!ELEMENT ldapConnectionUrl ( #PCDATA ) >
<!ELEMENT ldapTruststoreFile ( #PCDATA ) >
<!ELEMENT ldapTruststorePassword ( #PCDATA ) >
<!ELEMENT authzUsed ( #PCDATA ) >
<!ELEMENT authzFile ( #PCDATA ) >
<!ELEMENT emailSMTPAuthRequired ( #PCDATA ) >
<!ELEMENT emailSettingsSkipped ( #PCDATA ) >
<!ELEMENT emailUsername ( #PCDATA ) >
<!ELEMENT emailPassword ( #PCDATA ) >
<!ELEMENT useEmailSSL ( #PCDATA ) >
<!ELEMENT emailSSLTrustStoreFile ( #PCDATA ) >
<!ELEMENT emailSSLTrustStorePassword ( #PCDATA ) >
<!ELEMENT emailHost ( #PCDATA ) >
<!ELEMENT emailPort ( #PCDATA ) >
<!ELEMENT emailAdminAddress ( #PCDATA ) >
<!ELEMENT sslUsed ( #PCDATA ) >
<!ELEMENT sslCertificateAlias ( #PCDATA ) >
<!ELEMENT sslKeyPassword ( #PCDATA ) >
<!ELEMENT sslKeyStoreFile ( #PCDATA ) >
<!ELEMENT sslKeyStorePassword ( #PCDATA ) >
<!ELEMENT sslDefaultTrustStoreUsed ( #PCDATA ) >
<!ELEMENT sslTrustStoreFile ( #PCDATA ) >
<!ELEMENT sslTrustStorePassword ( #PCDATA ) >
<!ELEMENT directories ( directory+) >
<!ELEMENT directory 
     (    
     path, DAVLocation, managePasswordFile, passwordFile?, username?,
     password?, svnParentPathUsed, rootnames?
     ) >

<!ELEMENT path ( #PCDATA ) >
<!ELEMENT DAVLocation ( #PCDATA ) >
<!ELEMENT managePasswordFile ( #PCDATA ) >
<!ELEMENT passwordFile ( #PCDATA ) >
<!ELEMENT username ( #PCDATA ) >
<!ELEMENT password ( #PCDATA ) >
<!ELEMENT svnParentPathUsed ( #PCDATA ) >
<!ELEMENT rootnames ( #PCDATA ) >

The elements of this DTD are described in more detail below:

Element Name Requirement Description Where Persisted
adminPassword Required Once Administrator Password PasswordDB prevayler database
adminUsername Required Once Administrator Username PasswordDB prevayler database
nodeName Required Once Descriptive Name for Node NodeRegistry prevayler database
selectedNodeIP Required Once IP to use for this Node NodeRegistry prevayler database
apacheConfigFile Optional Location of your Apache configuration file Not persisted
bindHost Required Once IP Address to bind to NodeRegistry prevayler database and prefs.xml
clientPort Required Once Port clients will use to connect to SVN NodeRegistry prevayler database and pref.xml
adminConsolePort Required Once Port the WANdisco admin console will be available on NodeRegistry prevayler database
apacheUser Optional Apache User parsed from your apache configuration file Not persisted
apacheGroup Optional Apache Group parsed from your apache configuration file Not persisted
apacheKeepAlive Optional Not persisted
apacheMaxKeepAliveRequest Optional Not persisted
apacheKeepAliveTimeout Optional Not persisted
apacheListeningIP Optional IP your Apache is hosted on Not persisted
apacheListeningPort Required Once IP your apache is listening on Not persisted
usePreReplicationHooks Required Once Specify if you are using PreReplicationHooks Not persisted
subversionServerVersion Required Once The version of your Subversion server DynamicPreferences
svnServerCoHosted Required Once Once Specify if your subversion server is co hosted with WANdisco Not persisted
svnHost Required Once IP of you subversion server machine NodeRegistry prevayler database and prefs.xml
svnHostPort Required Once Port that SVN is listening on NodeRegistry prevayler database and prefs.xml
svnExecutableLocation Required Once Location of your svn.exe file DynamicPreferences
ldapAuthenticationUsed Required Once Are you using LDAP authentication Not Persisted
ldapConnectionUrl Optional URL used to connect to your LDAP server DynamicPreferences
ldapTruststoreFile Optional LDAP Truststore file location DynamicPreferences
ldapTruststorePassword Optional LDAP Truststore Password DynamicPreferences
authzUsed Required Once Are you using authz authorization Not Persisted
authzFile Optional Location of your authz file DynamicPreferences
emailSMTPAuthRequired Optional Does your email server require SMTP authentication? mailconfig.properties
emailSettingsSkipped Required Once Do you want to skip setting up your mail configuration Not Persisted
emailUsername Optional An email username to connect to your email server mailconfig.properties
emailPassword Optional The password for the email username specified mailconfig.properties
useEmailSSL Optional Does your email server use SSL/TLS mailconfig.properties
emailSSLTrustStoreFile Optional The location of your email SSL Truststore file mailconfig.properties
emailSSLTrustStorePassword Optional The password for your email SSL Truststore file mailconfig.properties
emailHost Optional The port used by your email service mailconfig.properties
emailAdminAddress Optional The Email address used by your admin user mailconfig.properties
sslUsed Required Once Do you require SSL connections DynamicPreferences
sslCertificateAlias Optional Your SSL certificate alias DynamicPreferences
sslKeyPassword Optional Your SSL Key password DynamicPreferences
sslKeyStoreFile Optional Your SSL Key Store file location DynamicPreferences
sslKeyStorePassword Optional Your SSL Key Store password DynamicPreferences
sslDefaultTrustStoreUsed Optional Do you want to use the default SSL Truststore DynamicPreferences
sslTrustStoreFile Optional Your SSL Trust store file location DynamicPreferences
sslKeyPassword Optional Your SSL Key password DynamicPreferences
sslKeyStoreFile Optional Your SSL Key Store file location DynamicPreferences
sslKeyStorePassword Optional Your SSL Key Store password DynamicPreferences
sslDefaultTrustStoreUsed Optional Do you want to use the default SSL Truststore DynamicPreferences
sslTrustStoreFile Optional Your SSL Trust store file location DynamicPreferences
sslTrustStorePassword Optional Your SSL Trust Store password DynamicPreferences
directories Required Once Your SSL Trust Store password DynamicPreferences
directory At Least Once An individual SVN repository directory
path Required Once The path to this SVN repository RepositoryDB prevayler database
DAVLocation Required Once The DAV location value for this repository RepositoryDB prevayler database
managePasswordFile Required Once Do you want WANdisco to manage the password file for this repository RepositoryDB prevayler database
passwordFile Optional The location of the svn password file for this repository RepositoryDB prevayler database
username Optional An SVN username that can be used to connect to this repository RepositoryDB prevayler database
password Optional An SVN username that can be used to connect to this repository RepositoryDB prevayler database
svnParentPathUsed Required Once Has SVN Parent Path been used for this repository RepositoryDB prevayler database
rootnames Optional The rootnames used for this repository (for multiple rootnames separate with a comma) RepositoryDB prevayler database