1. Introduction

Welcome to the Admin Guide for WANdisco's Subversion MultiSite. This is where you'll find all the information you'll need for planning a deployment, installing and managing the software.

If you can't find what you're looking for check out our online Knowledgebase which contains updates and extra documentation. Should you need further help and want to alert us to an error, please contact us by raising a case on our support website.

We'll be using terms like "node" and "replication group" without explaining what we mean. Check out the Glossary for an explanation of these and other WANdisco terms.

For a better understanding about how MultiSite and WANdisco's replication technology works read the Subversion MultiSite Technical Guide.

1.2 Feedback

If you find an error, or if you think that something needs improving we'd like to hear about it. Tell us about it by raising a case on our support website or emailing docs@wandisco.com.

1.3 Key

Throughout this document we'll highlight things using the following types of information box.

** Alert! **Alert
We use alerts to highlight something we think is important for you to know.

tipCaution
The stop symbol is used when we want to caution you against doing something.

tipTip
Tips are principles or practices that you'll benefit from knowing or using.

tipKnowledgebase
The KB symbol indicates that you can find out more about the subject in our online Knowledgebase.

2. Deployment guide

The deployment guide will help you prepare your development environment for the implementation of WANdisco's Subversion MultiSite.

2.1 Skills requirement

This section details the knowledge and technical requirements for deployment and operation of the WANdisco software, you should ensure that each of these sections is addressed before you begin the deployment.

Technical Skill Requirements

 System administration
  • Unix or Windows operating system installation
  • Disk Management
  • Memory monitoring and management
  • Command Line administration and manually editing configuration files

 Apache administration
  • Familiarity with Apache web server architecture
  • Management of httpd.conf / Apache2 configuration file management settings
  • WebDAV protocol
  • Start/stop/restart administration
  • User authentication options
  • Log set up and viewing

 Networking
  • IP Address assignation
  • TCP/IP ports and Firewall setup

 Subversion
  • Familiarity with Subversion administration (svnadmin) to create repository Command line user commands (svn)
  • Repository creation, and repository and/or file-system copying and synchronization.
  • Familiarity with WANdisco's replication architecture
  • Understanding of the installation procedure relevant to the OS in use
  • Concept of node types and Replication groups
  • Quorum configuration options

If you're not confident about handling any of these tasks, you can request a supported Installation from WANdisco.

A single administrator can manage all the systems running MultiSite, although it's a good idea to have someone at each site who is familiar with the MultiSite Basics.

2.2 Deployment overview

As with any software implementation you should deploy WANdisco Subversion MultiSite following a well defined plan. Doing so will help keep things under control, understood, and any potential problems will be spotted and fixed before you go into production.
We recommend that any deployment plan includes the following steps:

1. Pre-deployment Planning: Identifying the requirements, people and skills needed for deployment and operation. Also covers agreeing schedule and milestones, highlighting any assumptions, constraints, dependencies and risks to a successful deployment.

2. Deployment Preparation: Preparation and identification of server specifications, locations, node configuration, repository set-up, replication architecture, server and software configurations.

3. Testing Phase: Activities related to installation in an initial installation and testing in a non production environment, executing test cases, and verifying deployment readiness.

4. Production Deployment: Activities related to the installation, configuration, testing, and deployment in the production environment.

5. Post-deployment Operations and Maintenance: Post-deployment activities including environment monitoring, system maintenance, training and in-life technical support.

2.3 System requirements

This section covers everything you need to know when preparing your Subversion servers for replication. You should view this information as a set of guidelines, not as a fixed set of requirements that are followed slavishly.

2.3.1 Hardware Recommendations

Hardware Sizing Guidelines
Size #Users Repository Size (GB) CPU speed (GHz) #CPU #Cores RAM (GB) HDD (GB)
Small 100 25 2 1 2-4 8-16 100
Medium 500 100 2 2 4 16-32 250
Large 1000 500 2.66 4 4 32-64 750
Enterprise 5000 1000 2.66 4 4-6 128 1500

Storage tips

Processor tips

2.3.2 Setup requirements

MultiSite Servers must have:

This is a summary of the requirements, you'll need to run through the more detailed Deployment Checklist.

Subversion installations must have:

tip Tip
Run Subversion and MultiSite on the same server to gain the following benefits:
  • The installer can package a copy of your repositories when setting up nodes
  • Subversion MultiSite can manage the Subversion password file
  • Support for pre-replication hooks

2.3.3 Creating copies of your repository

Each MultiSite server must begin with an exact replica of the Subversion repository. The method you use to copy repositories depends on how big they are.

Small Repositories - Less than 1GB (Gigabyte)

If a repository is relatively small, it can be bundled up with the MultiSite install files when you come to activate additional MultiSite nodes. However, this option is only practical for relatively small repositories.

Large Repositories - 1GB (Gigabyte) or greater

For repositories that are over 1GB in size, it's no longer practical to bundle them during the installation of MultiSite software, instead you should use the following process:

This process ensures that the Subversion repositories are exact copies, and does so with minimal downtime for Subversion users.

2.3.4 Replicating multiple repositories

Subversion MultiSite is capable of replicating any number of Subversion repositories, using the Apache SVNPath or SVNParentPath directives. For more information, see the section Configuring Apache.
Read more about adding repositories.

2.3.4.1 SVN Location (SVNPath) Directive

The SVNPath directive is used to tell Apache where your Subversion repository is located. You can have multiple Location Directives, each pointing to a different Subversion repository.

  <Location /repo>
    DAV svn
    SVNPath /home/svn/repo1
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
		
	
  <Location /xyz>
    DAV svn
    SVNPath /home/projects/repositories/xyz
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
    

Example httpd.conf syntax for using SVNPath.

2.3.4.2 SVNParentPath Directive

The ParentPath directive is used if you have multiple repositories that reside in the same directory. Instead of directing to a specific repository directly, it tells Apache to match to the parent directory, under which there may be several repositories.

e.g. subversion-domain.com/svn/repo1 and subversion-domain/svn/repo2 would map to repositories at /home/svn/repo1 and /home/svn/repo2.

    <Location /svn>
    DAV svn
    SVNParentPath /home/svn/
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
    

Example httpd.conf syntax for using SVNParentPath.

** Alert! **Alert
A repository can belong to only one replication group at a time.
Each replication group that a node belongs to will require its own SVNParentPath.

2.3.5 Using Authz for authorization

Authz is the Apache module that gives you control over user authorization. It takes user information from the password file and associates access permissions.

tipTip
We recommend using the Authz along with Access Control to fully control access to your Subversion repositories.

** Alert! **Alert
If you move from Subversion MultiSite bundled with Access Control to the baseline MultiSite (that doesn't feature Access Control), you will need to manually remove the Authz directive from the Apache config file as well as the Authz file.

Setting up Authz

tipKnowledgebase
Read more about how to use Authz in Authz basics.

2.3.6 MultiSite and Subversion password files

The Subversion server on each and every node must have the details of all users across the replication group. The easiest way to keep user details consistent is the allow MultiSite to take control of the Subversion password files, so that any user created in MultiSite's admin console is automatically added to Subversion servers across the replication group. Using this Access Control feature greatly simplifies the management of Subversion users.

You can choose to have Subversion MultiSite control the Subversion password during installation, where you identify the Subversion password file's location. The installer then incorporates it into the replication group.

tipTip
If you have a lot of Subversion users, you can bulk import them into Subversion MultiSite.

2.3.7 Quorum recommendations

The Quorum refers to the strategy used for handling agreement between the nodes in a replication group. You can read more about the different types of quorum in the glossary. This section gives you a quick summary of the benefits of each quorum type, and in what setup it should be used.

Setup Type Quorum Recommendation
MultiSite
Singleton Response or Majority Response Quorum, balancing performance versus availability.

MultiSite with Sub Groups
Majority Response Quorum. If you select Singleton Response quorum, the distinguished node represents a single point of failure.

Stand-Alone Group at One Location
Have a group of at least three nodes, which automatically handles the failure of any single replicator node and its subsequent recovery.

With a two node group, select Singleton or Majority Response Quorum, and the second node must be the distinguished node.

2.3.8 Firewalls and Anti-virus software

The unexpected blocking of ports can be a problem when setting up Subversion MultiSite. Before you start the installation you should determine if any of the nodes in your replication group will be accessing a firewall. You must ensure that any firewalls are configured so that the port numbers (6444 etc) you specify during installation are not blocked or filtered.

If you have a AV software/virus scanner running on your network, you must configure it to not filter traffic on the ports you specify during installation.

2.3.9 Configuring Apache

You can give Subversion MultiSite exclusive use of Apache or share Apache with other applications/locations.

basic apache

Example of a simple configuration without MultiSite.

Without Subversion MultiSite, Subversion clients simply connect to Subversion on port 80.

basic apache

Example setup that includes Subversion MultiSite.

Subversion MultiSite sits between the Subversion clients and Subversion server, acting as a web proxy. Subversion clients connect on the usual port 80, whilst Apache is set to listen on port 8080.

2.3.9.1 Changing Subversion Listen port (Unix)

The listen port is set in Apache's config file - httpd.conf. There's more information about the Listen directive in the Binding chapter of the Apache documentation.

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 8080

Snippet of the httpd.conf file.

With this configuration, Apache server listens on port 8080 instead of default port 80.

2.3.9.2 Sharing Apache, and Using HTTPS

You can choose to share Apache with other locations or applications, instead of dedicating it to Subversion. This illustration shows how:

apache shared

Using HTTPS or sharing Apache with other applications/locations.

In the above example all processes are running on the same machine. Apache is running on port 80, Apache's webDAV module is running on port 8181. WANdisco's Subversion MultiSite is listening on port 8080, forwarding requests to Apache's WebDAV module on port 8181. Some clients are connecting via SSL (HTTPS) with their connection running on port 443.

tipKnowledgebase
You can use this configuration if you enable a proxy pass.

Default port settings required during the installation of Subversion MultiSite:

In the Apache config httpd.conf file, specify the following parameters:

#
# Define Apache port and pass anything that matches location /svnrepos to
# WANdisco SVN Replicator
#
NameVirtualHost *:80
<VirtualHost *:80>
ProxyPass /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPassReverse /svnrepos http://127.0.0.1:8080/svnrepos
</VirtualHost>

Listen 8181
NameVirtualHost *:8181

<VirtualHost *:8181>
<Location /svnrepos>
AllowOverride None
Order allow,deny
Allow from 127.0.0.1
DAV svn
SVNParentPath /tmp/dav
AuthType Basic
AuthName wandisco
AuthUserFile /etc/httpd/conf/htpasswd
Require valid-user
</Location>
</VirtualHost>
# For the SSL option
Listen 443
<VirtualHost *:443>
ProxyPass /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPass /!svn http://127.0.0.1:8080/svnrepos/!svn
ProxyPassReverse /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPassReverse /!svn http://127.0.0.1:8080/svnrepos/!svn
SSLEngine on
SSLCertificateFile /etc/httpd/conf/ssl.crt/server.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/server.key
SSLCACertificateFile /etc/httpd/conf/ssl.crt/ca-bundle.crt
</VirtualHost>

2.3.9.3 Using HTTP with Apache

This section applies to any Apache configuration.

** Alert! **Alert
In order to make a Subversion repository function in a distributed environment, Subversion MultiSite requires exactly the same Apache/Subversion setup at all the nodes.

Even without Subversion MultiSite, Apache benefits from some tweaking in order to work effectively with Subversion.

2.3.9.4 Apache and SVNKit

There are potential problems with connection pooling when using Apache and SVNKit. Adding Subversion MultiSite to the mix doesn't change these issues, though you may need to revisit them after installing MultiSite.

We recommend using JavaHL with Eclipse IDE, which does not use connection pooling, and thereby eliminates the problem.

2.3.9.5 SVNKit and Connection Pooling

SVNKit uses connection pooling. For a given client, SVNKit opens two connections and keeps them open for later use. On a system with a heavy load this can hit performance. An open connection consumes an Apache worker thread, with lots of clients and pooling going on, Apache may run out.

Apache provides some tuning parameters to optimize connection pooling, while still releasing unused connections. The tunable parameters are Timeout, KeepAliveTimeout, MaxKeepAliveRequests, and KeepAlive. For more details refer to the Apache configuration documentation.

2.3.9.6 Optimize Your Configuration

Apache has two timeout configurations: Timeout and KeepAliveTimeout. In general, the Timeout value should be lower than the value for KeepAliveTimeout.



tipCaution
Setting KeepAlive to false is not recommended. If you set KeepAlive to false, a client's transactions create an enormous overhead establishing and destroying connections.


2.3.9.7 Apache Logging

Apache's logs are dumped to a single file, which can cause problem when the log file grows so big that it needs to be truncated. When this happens, Apache service is interrupted while the file is restarted. On Linux/Unix servers there's the option of piping logs to another process. For more information see our Knowledgebase article, Piping Apache logs

2.4 Deployment Checklist

Though you may have referred to the Deployment Checklist during your evaluation we strongly recommend that you revisit the checklist and confirm that your system still meets all requirements.

Though you may have referred to the Deployment Checklist prior to an evaluation of Subversion MultiSite we strongly recommend that you revisit the checklist and confirm that you're system still meets all requirements.

 System setup

 Operating Systems

We've tested the following operating systems:
  • Fedora (32 or 64 bit): 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16
  • Red Hat Linux Enterprise (RHEL) Server (32 or 64 bit): 4, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 6, 6.1, 6.2, 6.3
  • Sun Solaris (32 or 64 bit): 9, 10, 11
  • Linux: Linux kernel 2.6 or higher
  • CentOS: 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 6, 6.1, 6.2, 6.3
  • Windows Server: (32 or 64 bit) 2003, 2008

  • In principle, any operating system that can support a Java environment, Apache and Subversion.

 Subversion server

Version 1.4 and above (we've tested up to version 1.7.1).
Run the Apache Portable Runtime that matches your Subversion version.

Subversion MultiSite is now compatible with Subversion 1.7's HTTPv2 implementation.
MultiSite 4.0 Users: Read our Knowledge base article on a workaround that lets you
Run MultiSite 4.0 with Subversion 1.7.


tipCertified Subversion Binaries
are now available from WANdisco. Providing the latest builds, without the risks associated with Open Source distribution.

 Subversion client

Any that are compatible with local Subversion servers.


 Hooks
  • Pre-commit hooks are not recommended. Use pre-replication hooks instead.
  • Any pre-replication hooks must be deterministic,
    i.e. have the same exact behavior and outcome at every node. Post-commit hooks can be tested at only one node.

 System memory

Ensure RAM and swapping containers are at least four times larger than the largest Subversion file.
Minimum recommended: 2 GB RAM; 4 GB swapping container


 Disk space

Subversion: Match to projects and issues.
MultiSite Transaction Journal: Equivalent of seven days of changes.

To estimate your disk requirements, you need to quantify some elements of your deployment:
  • overall size of all of your SVN repositories.
  • frequency of commits in your environment.
  • types of files being modified - text,binaries (SVN clients only send deltas for text).
  • number and size of files being changed.
  • rate that new files are being added to the repository.

** Alert! **Alert
Changes to the network layer introduced in 4.1 mean that MultiSite no longer buffers replicated data, removing the requirement for sufficient disk space for handling large concurrent checkouts for large numbers of users.


 File descriptor limit

Ensure hard and soft limits are set to 64000 or higher. Check with the ulimit or limit command.


 Journaling file system

Replicator logs should be on a journaling file system, for example, ext3 on Linux or VXFS from Veritas.

    ** Alert! **Alert
    NFS must not be used with Subversion MultiSite:Why you shouldn't use NFS.

    ext4 can be used as your journaling file system, although it must be configured appropriately. See Using Ext4 filesystem for journaling.

    tipAvoiding Data Loss
    This Knowledgebase article describes implementation strategies that militate against potential data loss as a result of power outages:         Data Loss and Linux


 Max. User Process Limit

At least three times the number of Subversion users.

 Java

Install JDK 6. We require that you use Oracle JDK 1.6.

** Alert! **Alert
Important
Java 7 is supported in the latest 4.2 releases (4.2 build 6801 and later). If you want to use Java 7, you need to upgrade.



 Perl

Install version 5.6.1 or later. For Access Control: Perl::DBI module for Audit Reports other than Users and Groups.

** Alert! **Alert
Windows users: Use the 32-bit edition of ActivePerl version 5.8, even if you are running a 64-bit Windows server.


 Network settings

 Reserved ports

When the installer confirms the admin console port (6444 by default), the next nine consecutive ports (by default 6445-6453) are now also reserved for various MultiSite network activities (REST, DCone, DFTP).
Not all these ports are currently assigned, some are reserved for possible future use.

We also recommend having a port available in case you have to copy (rsync) the repository from one node to another.


 Firewall or AV software

If your network has a firewall, ensure that traffic is not blocked on the reserved ports noted above. Configure any AV software on your system so that it doesn't block or filter traffic on the reserved ports.


 VPN
Set up IPsec tunnel, and ensure WAN connectivity.

 VPN persistent connections

Ensure that your VPN doesn't reset persistent connections for Subversion MultiSite.


 Bandwidth

Put your WAN through realistic load testing before going into production. You can then identify and fix potential problems before they the impact productivity.


 DNS setup

Use IP addresses instead of DNS hostnames, this ensures that DNS problems won't hinder performance. If you are required to use hostnames, test your DNS servers performance and availability prior to going into production.

 Apache setup

 Apache version

All nodes must have the same version, 2.2.3 or above.


 Apache modules version

All nodes have the same version of mod_dav and mod_svn_dav.


 mod_deflate.c for SVN_DAV

DO NOT USE mod_deflate.


 Location URI

Ensure the Apache config file on all nodes have the same location URI for Subversion repository access.


 Valid user for write methods

Ensure that all WebDAV methods require authentication for SVN-DAV protocol.


 Use port 80 for WANdisco

Standard Port 80 avoids confusion, change default Apache port if using 80.


 Apache server port

Use non-standard Apache server port to avoid conflict with replicator port.
See Dedicated Apache - Changing Subversion Port On.

 MultiSite setup

 Quorum

Default is singleton, providing performance over availability. Read more about Quorum options.


 Rotating quorum schedule

Ensure the distinguished node is in the active time zone.


 Disk space for recovery journal

Provision large disk for svn-replicator/systemdb, at least number of commits within a two to four hour window

 Batch processing

If there are any batch processes that interact with WANdisco, turn the deferred-writes to false.
 Access Control

 Audit reports

For reports other than Users and Groups, you'll need to use a database such as MySQL, and php.


 mod_authz_svn

Optional but recommended. See Access Control - Using the Authz Module. Module may be bundled with your version of Apache.

3. Installation

The installation guide runs through setting up Subversion MultiSite for the first time. If you're already using an earlier version of MultiSite, refer to the Upgrade section.

3.1 Installation overview

Before we begin, here is a recap of the installation process:

3.2 Before you start

Some things you should bear in mind before starting the installation:

3.3 Starting the installation

Ensure that you install the MultiSite files to a directory called svn-replicator and that you preserve the following subdirectory structure:

 svn-replicator directory contents

 /audit

Contains the audit trail logs for Access Control.


 /bin

Contains scripts like svnreplicator and shutdown


 /config

Contains prefs.xml which contains MultiSite configuration information. The license file needs to be place here before you can run MultiSite.


 /lib

Contains the Java archives (jar) files required to run the MultiSite.


 license.txt

The Subversion MultiSite End User License Agreement. This is the same license that appears during installation.


 /logs

Stores logs and other temporary files. MultiSite's main log file is named SVNProxyServer-prefs.log.0


 /systemdb

Contains the system database with its transaction journal. Warning: Deleting or modifying files from systemdb will likely corrupt your installation.


 /utils

The utilities directory contains a number of extra programs, such as the sample startup script, the perl script for running the Windows service and the LDAP Admin reset script.


 version.txt

Contains the installation's version number. This is useful for confirming the version number without having to run the installer.


 /webapp

Contains the code for Subversion MultiSite's web server.

If using selective replication you need separate instances of MultiSite installed, but each situated in a directory called svn-replicator, for example:

<replicator01>/svn-replicator
<replicator02>/svn-replicator
<replicator03>/svn-replicator 

Selective replication is used when you need to replicate only specific repositories. You can find further information in Selective Replication.

4. Complete setup through a browser

5. Login to the Admin Console for the first time

6. Copying the installation files manually

Follow these instructions if you were unable or chose not to use the automated packaging option Covered in section 7.

MultiSite creates a [node name].zip file for each of the nodes in a replication group. The first node is installed, so perform the following for all new nodes in the group:

7. Post Installation Configuration

There are some changes you may need to make before starting to use Subversion through MultiSite. These changes are made through the admin console. For more information see the Admin Console Guide.

7.1 Using pre-commit hooks?

If you are going to be using pre-commit hooks, these become WANdisco pre-replication hooks. Read Setting up hooks
in the User Guide.



3. User Guide

1. Replicator Management

1.1 Adding a node

To add a node to an existing replication group, you have to create a new replication group that contains the new node and then activate the new replication group. See Creating a New Replication Group.

Determine a plan for copying the repository to the new node. See Creating copies of your repository.

The new replication group is activated with all nodes.

1.2 Removing a node

It's not possible to remove a node from an active replication group, instead a new replication group is created excluding the node that you want to remove:

tipAlert
Only one replication group can be active at a time.

1.3 Creating a new replication group

You can create as many replication groups as you like, although only one replication group can be active at a time.
You can't delete the active replication group, first you need to create a new replication group and make it the active group.
See Deleting replication groups

tipTip
Create replication groups from the node that will be the distinguished node in the new replication group.

tipAlert
Only one replication group can be active at a time.

1.4 Deleting a replication group

Follow this procedure to delete a replication group. Remember, you can only delete a replication group that is inactive.

1.5 Adding a repository

1.6 Removing a repository

1.7 Moving a repository to another replication group

To move a repository from one replication group to another:

1.8 Setting MultiSite to start up on system boot

Subversion MultiSite comes with sample scripts debiansvnreplicator-init.d-sample and svnreplicator-init.d-sample which can be found in the /utils/init.d/<OS specific dir> directory.

View a copy of the script: debiansvnreplicator-init.d-sample
(Written for Debian, though also compatible with Ubuntu.)

1.9 Changing the Subversion replicated client port

To change the Subversion replication port that clients use to connect to repositories:

1.10 Shutting down MultiSite

You can shut down a replicator at any time, either from the node's admin console (see the Admin Console Guide 5.9 Shut down node), or through running the shutdown script ./shutdown in the svn-replicator/bin/ directory.

2. Setting up the replicator for production

This section covers a number of requirements for using Subversion MultiSite in a production environment.

2.1 Setting up MultiSite as a Windows service

To set up MultiSite as a Windows service you need:

Follow these steps on all the nodes where you want to run your WANdisco product as a Windows service:

2.2 Setting up MultiSite as a Linux/Unix service

Your installation provides the following scripts:

Apply these steps on all nodes.

  1. Make sure that your MultiSite installation is in a directory called svn-replicator.
  2. Copy the relevant script to /etc/init.d/svnreplicator.
  3. Make the script executable to the root user.
  4. Edit the script to set the correct values for the following variables:
    export JAVA_HOME=/usr/java/default
    WD_USER="root"
    WD_INSTALL_DIR="/opt/wandisco/svn-replicator"
        
    where:
    • JAVA_HOME is the location of the Oracle JDK installation.
    • WD_USER is the user that will run the Java svnreplicator process.
    • WD_INSTALL_DIR is the location of the MultiSite installation.
  5. Activate the scripts to start on boot and stop on shutdown per the OS-provided utilities (chkconfig, update-rc.d).

2.3 Logs

The service creates a winservice.log, located in the svn-replicator\logs directory.

2.4 Changing the quorum type

To change the quorum type create a new replication group and then specify the new quorum type.
See Creating a New Replication Group.

2.5 Toggling the quorum check

Subversion MultiSite verifies if a network quorum is reachable when a write command is submitted. If the quorum is unreachable, by default, the write command is aborted and the following message appears on the Subversion client console:

Check the Network connectivity, failed to reach a
minimum quorum of nodes. Aborting the svn write operation.

To turn off the quorum check, edit the prefs file (svn-replicator/config/prefs.xml). Set the parameter, AlwaysVerifyQuorum to false in the file. For instance,

<SVNProxy>
<AlwaysVerifyQuorum>false</AlwaysVerifyQuorum>
....
</SVNProxy>

If the check is turned off and quorum is un-reachable, the write transaction will be applied to the WANdisco Subversion MultiSite's transaction journal and stay in a pending state till network connectivity and quorum is restored.

** Alert! **Alert
With singleton quorum, if the current node is also the distinguished node, the quorum check will always succeed irrespective of network connectivity to other nodes.

2.6 About Watchdog mode

By default, Subversion MultiSite starts in watchdog mode. Whenever the replicator goes down, the watchdog mode restarts it. In watchdog mode, the replication process automatically disassociates from the terminal and becomes a daemon process, so you should not try running it in the background (with &).

Watchdog mode is not supported in Windows, use Windows Cygwin.

You can turn off watchdog by typing:

-nowdog

If MultiSite is unable to start up, for example if it terminates several times in quick secession, watchdog starts the the node in read-only mode.

$ ./bin/svnreplicator -h

Usage:

svnreplicator [-v] [-verbose] [-nowdog] [-pause time] [-email email-address]

-v
Print the svnreplicator version
-verbose
Verbose console messages go to STDOUT/STDERR instead of logs/console.txt
-nowdog
Turn off watchdog mode. WANdisco will not restart automatically if it terminates. Use this option for testing.
-pause
Time in seconds that the watchdog pauses for, before restarting service. Defaults to 0 seconds.
-email
Specify an email address to send an alert to, whenever the Watchdog restarts or shuts down WANdisco. WANdisco generates an email per local replicator activity. If the email settings were not set up during installation, see Email Settings, described in Using the Admin Console.

Use the -email option to generate email alerts whenever MultiSite restarts. For instance:
$ svn-replicator/bin/svnreplicator -pause 5 -email "admin@blueandgold.com, scmuser@blueandgold.com"

** Alert! **Alert
In order to have the replicator automatically started on system reboots, see Setting Replicator to Start Up on System Boot.

2.7 Using SSL

Subversion MultiSite can be set up to use SSL encryption. First you should run through the following steps after extracting the Subversion MultiSite files (but before running setup).

SSLv3 is support (though not enforced). If your browser setting has SSLv3 disabled, you will get a handshake error message. If it has both SSLv3 and TLS enabled, then, depending on the browser, it will try to switch from TLS to SSLv3 during the handshake. If you receive a handshake error message in your browser, make sure that TLS is disabled and only SSLv3 is enabled. All current browsers support SSLv3.

2.8 Setting the server key

In the keystore, the server certificate is associate with a key. By default, we look for a key named server to validate the certificate. If you use a key for the server with a different name, enter this in the SSL settings.

2.8.1 SSL Troubleshooting

A complete debug of the SSL logging will be required to diagnose the problems. To capture the debugging, run the java process with:

'-Djavax.net.debug=all' flag.

To enable the logging of SSL implemented layer, turn the logging to FINEST for 'com.wandisco.platform.net' package.

2.9 Using SSL for both LDAP and emails

If you're specifying secure email (using a truststore) and LDAP authentication over SSL (using a truststore), the same truststore must be used for both sets of certificates. If different truststores are used then the LDAP truststore will overwrite the email truststore and secure emails will stop working.

3. Changing prefs.xml settings

Subversion MultiSite's settings are stored within a prevayler database. However, some settings (for all nodes in the replication group) are stored within a preference file (prefs.xml) which is located in the <WANDISCO/svn-replicator/config directory.

** Alert! **Alert
If you make changes that affect more than one node, you must change each node's specific file. If your change only effects just one node, you can change just that node's prefs.xml file.

To change prefs.xml file on a single node:

To change all nodes' prefs.xml files:

3.1 Performing a Synchronized Stop

** Alert! ** Alert
Previously, all nodes in a replication group needed to be available in order to perform a synchronized stop. Now it's possible to synchronize stop a subgroup of nodes which is useful if you need to bring your replication group to a stop even if you don't have all nodes up and working. However,         this action should only be used as a final resort and will require additional work to ensure that sync is maintained.

All Sync stops make Subversion read-only. You'll need to warn your Subversion users of some down-time.

Selective Synchronized Stops

If you wish to perform a synchronized stop on a subgroup of nodes, untick the node(s) that you do not wish to bring to a stop and click Continue.
Synchronized stop 01

You'll be shown a warning box:

"Are you sure you want to perform a synchronized stop on a subgroup?

If this is preparation for a reset, THE EXCLUDED NODES MUST BE RESET AS WELL and will then require
an rsync of every repository before they are restarted."

Synchronized stop 01

** Alert! ** Alert
The purpose of a synchronized stop is to ensure that all repositories are maintained in exactly the same state, that one node doesn't process transactions that are not completed on another. Any node that is not stopped in sync with the rest of the replication group is more than likely going to fall out of sync, causing replication to stop.

All nodes that are not included in the synchronized stop must be reset to purge all queued traffic and all their repositories must be manually synchronized (using rsync) with the rest of the replication group.

3.2 Resuming from a Synchronized Stop

3.3 Verifying that the replicator is working

The quickest way to check replication is to add a new user through the Admin Console of one of your nodes. Then, jump to another node and confirm that new user has appears there too.

Another way to check replication is to verify there are commit transactions posted to the log file
svn-replicator/logs/SVNProxyServer-prefs0.log. See More about log files.

3.4 Changing the Distinguished Node

3.4.1 What distinguishes the Distinguished node?

The distinguished node has greater voting power within the quorum. In a singleton quorum the distinguished node decides transaction ordering and keeps the other nodes in sync.

In a singleton quorum, Subversion users who operate from the distinguished node will see the best performance because their local replicator never needs to wait for agreement to be reached on a distant node.

In a singleton quorum it's advantageous to change the distinguished node to correspond with the site where the most repository changes are taking place, i.e. operating within business hours. You can manually change the distinguished node at any time. You can also automate the process by setting up a Rotation Schedule.

The distinguished node is selected from the replication group, it can be any node.

** Alert! **Alert
Changing the Distinguished Node requires a Unanimous Quorum where agreement must be reached between all nodes.
See the Troubleshooting guide if one or more of your nodes become unreachable.

In the majority quorum the distinguished node acts as a tie breaker when there are an even number of nodes. For example, with a 4 node setup, in order to achieve a majority you would technically need 3 nodes. With the distinguished node's slightly heavier weighted vote, you can achieve a majority with only 2 nodes as long as one of the two is the distinguished node.

3.4.2 Manually changing the distinguished node

3.4.3 Scheduled rotation of the distinguished node

3.5 Using Subversion hooks for sending e-mails

Many administrators like to set up Subversion backend hooks that fire whenever a Subversion user commits a set of file changes. With a master/slave Subversion server setup, e-mails can be initiated once when the post-commit trigger fires.

You can set up Subversion hooks to send emails when Subversion users commit file changes. However, with MultiSite, these emails will get duplicated, with a potential for spamming with notification emails.

To avoid this, set up one node as an "e-mail hub" by enabling the post-commit trigger to fire from a single node within the replication group.

Alternatively, you could use the time of day to fire the e-mail alerts from a specific node. For example, you could modify the post-commit trigger to send e-mails from India during 9:00 a.m. to 5:00 p.m. IST, and from the US during 9:00 a.m. to 5:00 p.m. PST.

tipCaution
If using asymmetrical e-mail hooks ensure that you don't disable the pre-commit trigger. That may cause a Subversion commit transaction to abort at some nodes but commit at other nodes causing a loss of sync.

tipTip
Configure email setup to avoid long blockages or delays. The default SMTP service on node should be adequate. We recommend that you set up a local e-mail hub or a local SMTP agent/server. The local SMTP server should preferably be on the same host as the Subversion server. It should be set up to forward/relay e-mails to the organization-wide SMTP server. This ensures the e-mail hooks are a lot faster and just need to enqueue the e-mails to the local SMTP server. Test that your emails are delivered before going into production.

3.6 Changing the admin console username or password

The default login username for the Admin Console is admin, and the password is user-defined during installation. That way, all nodes initially have the same login and password.

You can set up different login credentials for each node, however, you should ensure that all administrators who manage nodes are able to login to all nodes.

tipTip
We recommend that you keep admin authorization simple, and have the same login and password for all nodes. This will greatly simplify managing replication.

To change the login at a particular node, enter the following in prefs.xml:

<Security>
 <Admin>
   <user>newlogin</user>
  </Admin>
</Security>

3.6.1 Changing the admin password

3.7 Setting Up Hooks

A hook is a script that gets triggered by a specified repository event, such as creation of a new revision or the modification of an unversioned property.

Hook How to Integrate with WANdisco
start-commit Standard Subversion implementation.
Must be present at all nodes, and must either execute identically at all nodes or fail identically at all nodes.
pre-commit This becomes the pre-replication hook.
See the following Replication Hook section.
post-commit Standard Subversion implementation.
Must be present at only one node.
pre-revprop-change Standard Subversion implementation.
Must be present at all nodes, and must either execute identically at all nodes or fail identically at all nodes.
post-revprop-change Standard Subversion implementation.
Must be present at only one node.

3.7.1 Pre-Replication Hook

To use pre-replication hooks, Subversion MultiSite needs to be set with the version of Subversion you are using. Go to the Proxy tab, select SVN Settings, and select the Yes radio button for Use Pre-Replication Hooks. Then specify the Subversion Server Version.

Use a pre-replication hook 01

How Pre-Replication Hooks Are Run

Before Subversion executes a pre-replication hook, MultiSite invokes it before forming a proposal. As per the SVN DAV specification, if the hook succeeds, nothing is communicated back to the client. The handling of the command proceeds normally. If the hook fails, stderr is packaged as an XML response to the client. The client will then typically delete the activity; i.e., clean up any temporary files on the server side.

3.7.2 Configuration

There are two methods for invoking hooks scripts: globally (across a replication group) via the Repository settings screen in the admin console, or on a per-repository basis ( see Repository-specific Hooks).

Global Hook Scripts
To invoke a hook script globally, go to the SVN Settings screen and click edit for the repository (see Admin Console Guide: SVN Settings). Within the repository settings screen entry fields for the three hook types that can be replicated.

Use a pre-replication hook 01

Pre-Commit Hook: Equivalent to normal pre-commit hook but it is automatically picked up and invoked in a replication safe manner.

Pre-lock Hook: Equivalent to normal pre-lock hook (invoked when a user attempts to lock a resource) but it is automatically picked up and invoked in a replication safe manner.

Pre-Unlock Hook: Equivalent to normal pre-unlock hook (invoked when a user attempts to destroy an exclusive lock) but it is automatically picked up and invoked in a replication safe manner.

** Alert! **Using ParentPath
when re-replication hooks are enabled within a ParentPath repository environment, they run for all child repositories, unless specific hooks are specified, for each child.
Use a pre-replication hook 01

3.7.3 Repository-Specific Hooks

Three different Pre-replication hooks are supported on a per-repository basis:


MultiSite will pick up that a hook script is aimed for pre-replication if its appended with "ms_".
e.g.
ms_pre-commit.pl 
ms_pre-lock.pl 
ms_pre-unlock.pl 
This renaming will stop Apache running the scripts, instead, MultiSite checks if the Use Pre-Replication Hooks is enabled in the SVN Settings - and if enabled, the above 'ms_' appended scripts will be run.

3.8 Selective Replication

By default, MultiSite replicates all SVNROOTS associated with a Subversion repository. However, you can specify a set of SVNROOTS that you do not want to replicate. Use Excludes for SVNROOTS in the prefs.xml file to identify the repositories you do not wish to have replicated. For example:

<SVNProxy>
....
<ExcludeRepositories>/exclude,/dir0</ExcludeRepositories>
....
</SVNProxy>

If MultiSite can't find the entry, it's included in the replication. There are no wildcard (regular) expressions. For Apache and SVNPath, the syntax must match what is listed in the Location directive. For Apache and SVNParentPath, the syntax must match what is listed in the Location directive and Path to the repository.

All the included roots go through the same agreement manager. For example, say you have three repositories.

/repos/rep1
/repos/rep2
/repos/rep3

For SVNPath, the Location directive is

<Location/rep1>
  SVNPath /repos/rep1
</Location/rep1>

<Location/rep2>
  SVNPath /repos/rep2
</Location/rep2>

<Location/rep3>
  SVNPath /repos/rep3
</Location/rep3>

For SVNParentPath, the Location directive is

<Location/repository>
  SVNParentPath /repos
</Location/repository>

To exclude the first repository, rep1, for SVNPath, you would exclude rep1: for SVNParentPath, you would exclude /repository/rep1.

You must name a repository after the / character. Subversion does not support a location named with just the / character.

4. Updating Apache or Subversion in WANdisco Deployment

This procedure requires that your MultiSite deployment, including Subversion servers, are offline during the time it takes to upgrade at each node. Please plan this procedure accordingly.

5. Upgrading Subversion MultiSite

Use these instructions to upgrade your Subversion MultiSite to a later build or version. The instructions in the Deployment Guide Installation section are for your first installation.

The correct method for upgrading Subversion MultiSite depends on which versions you are moving between. Use the guide below to ensure that you upgrade with the right method.

Upgrading from Subversion MultiSite 3.6 or earlier - See our Knowledge base articles on Installation and Upgrade

tipUpgrade rule of thumb

When looking to upgrade Subversion MultiSite, apply the following rule:

Upgrading within a version, e.g. 4.0 build x --> 4.0 build y , use the Upgrade from Script.

Upgrading between versions. e.g. 4.0 --> 4.1 will require you to Upgrade from backup

5.1 Upgrade from backup

Use this procedure if you are upgrading between versions rather than builds (4.0 to 4.1 rather than 4.1 build x to 4.1 build y etc).
To upgrade between builds, see Upgrade from script.

Upgrading from backup requires that MultiSite be installed from scratch. First you export your user settings so they can be imported into the new version of MultiSite.

tipInstalling in order to perform an upgrade using the backup.xml file?
Don't enable Authz during the installation - wait until the import is completed, then enable Authz from the Subversion Settings screen.
Enabling Authz during the installation can greatly impact the performance of the backup.xml import.

Stop MultiSite by performing a Synchronized Stop

 

5.1.1. Open the Admin Console, on the left-hand menu click Stop Proxy.

nothing goes to plan

5.1.2. Check that radio button for "Synchronized Stop of all proxies". MultiSite will now wait for any remaining transactions to complete, after which all nodes will stop listening, stopping replication.

Backup your settings from the Admin Console

5.1.3. Open the Admin Console and click on the System tab.

5.1.4. Click on Export Settings. Confirm the filename and location, then click on Backup Settings. All settings and user data will be exported in the resulting backup xml file.

Create a backup in case you need to back out of the upgrade

5.1.5. Shutdown and reset all nodes.

5.1.6. Move the svn-replicator directory to a backup location.

Extract and run the setup file

5.1.7. Decompress the new Subversion MultiSite archive file.

5.1.8. Copy the license evaluation key file to the svn-replicator/config directory.

5.1.9. Go to /svn-replicator/bin/, then start the installation with the following command:

perl setup

Respond to the Yes / No prompt relating to java memory settings.

5.1.10. The setup will now start up the browser based installer. Open your browser and go to address shown at the end of the setup. This will be the IP of your installation machine, on the WANdisco port (6444 by default).

Complete the upgrade through a browser.

5.1.11. From the Welcome screen, click Continue.

upgrade with backup


5.1.12. Once you've read the WANdisco End User License Agreement, click I Agree to continue with the upgrade.

upgrade with backup


5.1.13. Enter a password for the default Admin Console account, the username for this account is now admin (for earlier versions it was "root"). Click Next to continue.

upgrade with backup


5.1.14. The next two screens explain how MultiSite will act as a proxy between Subversion clients and the Subversion server. Click Next to continue.

upgrade with backup


5.1.15. By default, MultiSite will listen on port 80, while Subversion will listen on port 8080. The benefit of this setup is that Subversion end-user don't need to make any changes.

upgrade with backup


5.1.16. You'll confirm the proxy settings. These will be populated with the default settings noted in the previous screens.

Node Name: The name that MultiSite will use for this, the first node. The name cannot contain spaces, but can be changed later from the Admin Console.

Node IP: The node's IP.

Bind Host: By default, all network interfaces bind to the node. Select "Bind advertised host only" to limit to the node's IP address (stated in the field, above).

Client Port: By default this is 80, allowing Subversion users to continue without making change to their client setup.

Admin Console Port: 6444 by default. Should you need to use a different port for accessing the Admin Console, restart the setup (step 3.1.3) using the following command:

perl setup <preferred port for Admin Console access>

Reserved Ports: MultiSite will reserve a block of 10 ports, you can either leave the default ports or enter ports that you prefer.

Use SSL: Tick the checkbox if you've already got keystore/truststore settings in plac. See more about SSL Settings.

Once you've finished making any changes to the proxy settings, click Next to continue.

upgrade with backup


5.1.17. Setup will check the Apache config file for settings that might cause problems for MultiSite.

If the httpd.conf file isn't found, enter its path into the Configuration File entry box, then click Reload Configuration.

Look out for warning boxes for where setup finds a problem. You'll need to manually edit the httpd.conf file, then click on Reload Configuration to have setup check your changes.

User: Owner of the file.

Group: The group in which the owner belongs.

KeepAlive: Setup will look to see that the Keep-Alive directive is set to On.

KeepAliveRequests: Setup will look for 0, which indicates that no maximum limit will be set for connection requests.

KeepAliveTimeout: Set very high (500,000 seconds) to ensure connections don't timeout.

Listening IP: For a node with multiple IPs, this will indicate the IP used for listening.

Listening Port: The default listening port is 8080.
Override Listen directive with a virtual host? Tick to tell MultiSite that you are using a virtual host.

upgrade with backup


5.1.18. Setup now allows you to modify your Subversion settings. Watch for alerts that confirm the port and path that MultiSite will associate with Subversion.

SVN Settings

Use Pre-Replication Hooks: Select whether you are using Pre-Replication Hooks.


Subversion Host: Confirm the hostname of the server running Subversion. You'll benefit from making this the same location as MultiSite, e.g. 'localhost'.

Subversion Server Port: 8080 by default.

SVN Executable:This is the fully qualified path to the Subversion executable. Setup will try to fill this in automatically, otherwise type it into the entry box.

Use authz-based access control Click on the box if you are using Authz access control. See the following note about why you may wish to hold off selecting this option until later.

tipInstalling in order to perform an upgrade using the backup.xml file?
Don't enable Authz during the installation - wait until the import is completed, then enable Authz from the Subversion Settings screen.
Enabling Authz during the installation can greatly impact the performance of the backup.xml import.

At the bottom of the screen is a table that confirms the DAV Location and password control for your repositories. You can click edit to make changes. You can add additional repositories by clicking Add Repository. To continue setup, click Next.

upgrade with backup

5.1.19. You can enter email settings so that MultiSite can send alerts.

SMTP Authentication: If you select No, you'll need to provide your account.
Username and Password in the following entry boxes.
Use SSL/TLS: Choose yes if you wish to send emails over a secure connection.
Host: Enter the address of your mail server.
Port: Enter the SMTP port, 25 by default.
Send Admin Notification To: The email address to where notifications will be sent.

You don't need to provide email settings, in which case, click Skip to continue. Otherwise, click Next.

upgrade with backup

 

5.1.20. You have now completed setup. You can go back and make changes or click Complete installation with these settings to save them and restart MultiSite.

upgrade with backup


5.1.21. You will next see a Completing Installation screen, which MultiSite restarts. When completed, you'll see the authentication screen for the Admin Console. Enter the username 'admin' and the password you entered during setup to enter.

upgrade with backup



Import your saved settings

So you backed up your data, installed the latest version of Subversion MultiSite, it's now time to import your backed up data.

Use this procedure to restore user information, although any system settings will need to be manually applied