1. Introduction

Welcome to the Admin Guide for WANdisco's SVN MultiSite. This is where you'll find all the information you'll need for planning a deployment, installing and managing the software.

If you can't find what you're looking for check out our online Knowledgebase which contains updates and extra documentation. Should you need further help and want to alert us to an error, please contact us by raising a case on our support website.

We'll be using terms like "node" and "replication group" without explaining what we mean. Check out the Glossary for an explanation of these and other WANdisco terms.

For a better understanding about how MultiSite and WANdisco's replication technology works read the SVN MultiSite Technical Guide.

1.2 Feedback

If you find an error, or if you think that something needs improving we'd like to hear about it. Tell us about it by raising a case on our support website or emailing docs@wandisco.com.

1.3 Key

Throughout this document we'll highlight things using the following types of information box.

Alert
We use alerts to highlight something we think is important for you to know.

Caution
The stop symbol is used when we want to caution you against doing something.

Tip
Tips are principles or practices that you'll benefit from knowing or using.

Knowledgebase
The KB symbol indicates that you can find out more about the subject in our online Knowledgebase.

2. Deployment guide

The deployment guide will help you prepare your development environment for the implementation of WANdisco's SVN MultiSite.

2.1 Skills requirement

This section details the knowledge and technical requirements for deployment and operation of the WANdisco software, you should ensure that each of these sections is addressed before you begin the deployment.

If you're not confident about handling any of these tasks, you can request a supported Installation from WANdisco.

A single administrator can manage all the systems running MultiSite, although it's a good idea to have someone at each site who is familiar with the MultiSite Basics.

2.2 Deployment overview

As with any software implementation you should deploy WANdisco SVN MultiSite following a well defined plan. Doing so will help keep things under control, understood, and any potential problems will be spotted and fixed before you go into production.
We recommend that any deployment plan includes the following steps:

1. Pre-deployment Planning: Identifying the requirements, people and skills needed for deployment and operation. Also covers agreeing schedule and milestones, highlighting any assumptions, constraints, dependencies and risks to a successful deployment.

2. Deployment Preparation: Preparation and identification of server specifications, locations, node configuration, repository set-up, replication architecture, server and software configurations.

3. Testing Phase: Activities related to installation in an initial installation and testing in a non production environment, executing test cases, and verifying deployment readiness.

4. Production Deployment: Activities related to the installation, configuration, testing, and deployment in the production environment.

5. Post-deployment Operations and Maintenance: Post-deployment activities including environment monitoring, system maintenance, training and in-life technical support.

2.3 System requirements

This section covers everything you need to know when preparing your Subversion servers for replication. You should view this information as a set of guidelines, not as a fixed set of requirements that are followed slavishly.

2.3.1 Hardware Recommendations

Storage tips

• Use separate physical disks for Subversion and SVN MultiSite.
• Use the fastest possible disks for storage, disk IO is the critical path for improving repository responsiveness.
• The use of RAID-1 or RAID-2 solutions are recommended, although using RAID-0 is strongly discouraged.
Knowledgebase
For more information about calculating how much storage capacity is required, read the Knowledgebase article - Hardware Sizing Guide.

Processor tips

• A single 2GHz CPU should be considered a minimum.
• You should aim to have no more than 15 concurrent Subversion users per CPU. e.g. Quad CPU for 60 concurrent users.
• Processors with multiple cores will offer better performance, although each additional core will usually add about 50% more performance compared to adding a physical processor.

2.3.2 Setup requirements

MultiSite Servers must have:

This is a summary of the requirements, you'll need to run through the more detailed Deployment Checklist.

• the same operating system
• the same Apache major/minor version, and mod_dav and mod_svn_dav versions
• Java & Perl installed
• a browser with network access to all servers
• a command line compression utility
• a unique license key file. (These are always sent to you in an email)

Subversion installations must have:

• the same version of Subversion server
• matching file and directory level permissions on repositories
• exactly matched contents of the svnroot directories between servers (including the repository UUID)
• matching Subversion user/passwords

Tip
Run Subversion and MultiSite on the same server to gain the following benefits:

• The installer can package a copy of your repositories when setting up nodes
• SVN MultiSite can manage the Subversion password file
• Support for pre-replication hooks

2.3.3 Creating copies of your repository

Each MultiSite server must begin with an exact replica of the Subversion repository. The method you use to copy repositories depends on how big they are.

Small Repositories - Less than 1GB (Gigabyte)

If a repository is relatively small, it can be bundled up with the MultiSite install files when you come to activate additional MultiSite nodes. However, this option is only practical for relatively small repositories.

Large Repositories - 1GB (Gigabyte) or greater

For repositories that are over 1GB in size, it's no longer practical to bundle them during the installation of MultiSite software, instead you should use the following process:

• 1. Have the Subversion repository copied to each server that will be set up as a node. This can be done while production continues as we'll resync them later.

  Alert
  Ensure file ownerships and permissions are the same on all nodes.



• 2. Run through the MultiSite installation for your first node.


• 3. Shutdown SVN MultiSite. This put the repository in a read-only mode, so it's best to complete the remaining steps outside of your developers' working hours.


• 4. Use rsync to sychronize the live repository with the copy on all nodes.


• 5. Restart SVN MultiSite, go to the admin console and follow the procedure for Creating a replication group.

This process ensures that the Subversion repositories are exact copies, and does so with minimal downtime for Subversion users.

2.3.4 Replicating multiple repositories

SVN MultiSite is capable of replicating any number of Subversion repositories, using the Apache SVNPath or SVNParentPath directives. For more information, see the chapter Setting up Apache.
Read more about adding repositories.

2.3.4.1 SVN Location (SVNPath) Directive

The SVNPath directive is used to tell Apache where your Subversion repository is located. You can have multiple Location Directives, each pointing to a different Subversion repository.

  <Location /repo>
    DAV svn
    SVNPath /home/svn/repo1
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
		
	
  <Location /xyz>
    DAV svn
    SVNPath /home/projects/repositories/xyz
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
    

Example httpd.conf syntax for using SVNPath.

2.3.4.2 SVNParentPath Directive

The ParentPath directive is used if you have multiple repositories that reside in the same directory. Instead of directing to a specific repository directly, it tells Apache to match to the parent directory, under which there may be several repositories.

e.g. subversion-domain.com/svn/repo1 and subversion-domain/svn/repo2 would map to repositories at /home/svn/repo1 and /home/svn/repo2.

    <Location /svn>
    DAV svn
    SVNParentPath /home/svn/
    AuthType Basic
    AuthName wandisco
    AuthUserFile /home/scm/htpasswd
    Require valid-user  </Location>
    

Example httpd.conf syntax for using SVNParentPath.

Alert
A repository can belong to only one replication group at a time.
Each replication group that a node belongs to will require its own SVNParentPath.

Knowledgebase
There's an example use of SVNParentPath featured in our Knowledgebase - Using SVNParentPath with multiple replication groups

2.3.5 Using Authz for authorization

Authz is the Apache module that gives you control over user authorization. It takes user information from the password file and associates access permissions.

Tip
We recommend using the Authz along with Access Control to fully control access to your Subversion repositories.

Alert
If you move from SVN MultiSite bundled with Access Control to the baseline MultiSite (that doesn't feature Access Control), you will need to manually remove the Authz directive from the Apache config file as well as the Authz file.

Setting up Authz

• 1. Ensure that the Authz module (authz_default_module) is loaded into Apache. You can confirm the module is loaded if you run:

  apachectl -M 

  This will produce a list of loaded modules, you should be looking for the authz_svn_module (shared).


• 2. Enable Authz on each node by adding the following SVN location directive to the Apache config file, /etc/httpd/conf/httpd.conf
  

  <Location /repos> 
     DAV svn 
     SVNPath /home/wandisco/repository 
     AuthType Basic 
     AuthName WANdisco 
     AuthUserFile /home/wandisco/htpasswd 
     Require valid-user 
     SVNAutoversioning on 
     ###
     # Authz module for authorization configuration
     ###
     AuthzSVNAccessFile /home/wandisco/authz
     </Location>
  

• 3. If this is a live system, perform a synchronized stop to prevent write transactions.


• 4. The httpd service will need to be restarted at each node for the change to take affect.
  

  sudo /sbin/service httpd restart

  Create an empty authz file on each node:
  

  touch /home/wandisco/authz

• 5. Open SVN MultiSite's admin console, click the Proxy tab and select SVN Settings on the side menu.


• 6. Check the Use authz-based access control? checkbox at the bottom of the svn settings panel.
  
  
  
  
• 7. The authorization file input will appear. Enter its absolute path. e.g.

  /home/wandisco/authz

  Alert
  Make sure MultiSite has permission to read and write to the svn.auth file. Also, this is a replicated change that will affect all nodes.



• 8. The /home/wandisco/authz file should be automatically updated with your current users and groups, check the contents of this file to verify that it has been updated.
  

  In the file, You should add an admin user for each repository: e.g. for repository mapped to /svn
  add the following:

   [svn:/]
  <admin username> = rw

  or for a repo mapped to /myrepo:

  [myrepo:/]
    <admin username> = rw

  where <admin username> is the username that the user enters when adding a repository.



• 9. If you brought the replicator to a stop in step 3, now return to the Proxy tab and click the Resume button.

Knowledgebase
Read more about how to use Authz - Authz basics

2.3.6 MultiSite and Subversion password files

The Subversion server on each and every node must have the details of all users across the replication group. The easiest way to keep user details consistent is the allow MultiSite to take control of the Subversion password files, so that any user created in MultiSite's admin console is automatically added to Subversion servers across the replication group. Using this Access Control feature greatly simplifies the management of Subversion users.

You can choose to have SVN MultiSite control the Subversion password during installation, where you identify the Subversion password file's location. The installer then incorporates it into the replication group.

Tip
If you have a lot of Subversion users, you can bulk import them into SVN MultiSite.

2.3.7 Quorum recommendations

The quorum refers to the strategy used for handling agreement between the nodes in a replication group. You can read more about the different types of quorum in the glossary. This section gives you a quick summary of the benefits of each quorum type, and in what setup it should be used.

2.3.8 Firewalls and Anti-virus software

The unexpected blocking of ports can be a problem when setting up SVN MultiSite. Before you start the installation you should determine if any of the nodes in your replication group will be accessing a firewall. You must ensure that any firewalls are configured so that the port numbers (6444 etc) you specify during installation are not blocked or filtered.

If you have a AV software/virus scanner running on your network, you must configure it to not filter traffic on the ports you specify during installation.

2.3.9 Configuring Apache

You can give SVN MultiSite exclusive use of Apache or share Apache with other applications/locations.

Example of a simple configuration without MultiSite.

Without SVN MultiSite, Subversion clients simply connect to Subversion on port 80.

Example setup that includes SVN MultiSite.

SVN MultiSite sits between the Subversion clients and Subversion server, acting as a web proxy. Subversion clients connect on the usual port 80, whilst Apache is set to listen on port 8080.

2.3.9.1 Changing Subversion Listen port (Unix)

The listen port is set in Apache's config file - httpd.conf. There's more information about the Listen directive in the Binding chapter of the Apache documentation.

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, instead of the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 8080

Snippet of the httpd.conf file.

With this configuration, Apache server listens on port 8080 instead of default port 80.

2.3.9.2 Sharing Apache, and Using HTTPS

You can choose to share Apache with other locations or applications, instead of dedicating it to Subversion. This illustration shows how:

Using HTTPS or sharing Apache with other applications/locations.

In the above example all processes are running on the same machine. Apache is running on port 80, Apache's webDAV module is running on port 8181. WANdisco's SVN MultiSite is listening on port 8080, forwarding requests to Apache's WebDAV module on port 8181. Some clients are connecting via SSL (HTTPS) with their connection running on port 443.

Knowledgebase
You can use this configuration if you enable a proxy pass.

Default port settings required during the installation of SVN MultiSite:

• WANdisco replication/admin console port: 6444
• proxy (Listen) port: 8080
• Apache WebDAV port: 8181 on localhost

In the Apache config httpd.conf file, specify the following parameters:

#
# Define Apache port and pass anything that matches location /svnrepos to
# WANdisco SVN Replicator
#
NameVirtualHost *:80
<VirtualHost *:80>
ProxyPass /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPassReverse /svnrepos http://127.0.0.1:8080/svnrepos
</VirtualHost>

Listen 8181
NameVirtualHost *:8181

<VirtualHost *:8181>
<Location /svnrepos>
AllowOverride None
Order allow,deny
Allow from 127.0.0.1
DAV svn
SVNParentPath /tmp/dav
AuthType Basic
AuthName wandisco
AuthUserFile /etc/httpd/conf/htpasswd
Require valid-user
</Location>
</VirtualHost>

# For the SSL option
Listen 443
<VirtualHost *:443>
ProxyPass /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPass /!svn http://127.0.0.1:8080/svnrepos/!svn
ProxyPassReverse /svnrepos http://127.0.0.1:8080/svnrepos
ProxyPassReverse /!svn http://127.0.0.1:8080/svnrepos/!svn
SSLEngine on
SSLCertificateFile /etc/httpd/conf/ssl.crt/server.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/server.key
SSLCACertificateFile /etc/httpd/conf/ssl.crt/ca-bundle.crt
</VirtualHost>

2.3.9.3 Using HTTP with Apache

This section applies to any Apache configuration.

Alert
In order to make a Subversion repository function in a distributed environment, SVN MultiSite requires exactly the same Apache/Subversion setup at all the nodes.

Even without SVN MultiSite, Apache benefits from some tweaking in order to work effectively with Subversion.

• 1. Change Apache's connection keep-alive settings to allow long lived HTTP connections. Add this to the Apache configuration file conf/httpd.conf or included conf/extra/httpd-defaults.conf. For instance,

  # Timeout: The number of seconds before receives and sends time out.
  #
  Timeout 7200
  # KeepAlive: Whether or not to allow persistent connections (more than
  # one request per connection).
  #
  KeepAlive On
  # MaxKeepAliveRequests: The maximum number of requests to allow
  # during a persistent connection. Set to 0 to allow an unlimited amount.
  #
  MaxKeepAliveRequests 0
  #
  # KeepAliveTimeout: Number of seconds to wait for the next request from the
  # same client on the same connection.
  #
  KeepAliveTimeout 14400
  ...
  

  Example httpd.conf snippet showing recommended Apache timing configuration

• 2. Ensure the SVN DAV settings in Apache's configuration files are exactly the same at all nodes. The top level location URI prefix should be the same.

  Caution
  If you are using ProxyPass, avoid using a single '/' (slash) DAV location as this is not compatible with MultiSite and can cause replication to crash.

  We recommend copying the current httpd.conf file, and then changing the host:port settings. For instance, here is a conf file snippet with Apache virtual hosts (you do not have to use Apache virtual hosts, this is only an illustration):

  # Site A
  $ cat conf/extra/httpd-svn-dav.conf
  ...
  NameVirtualHost site-a:8181
  <VirtualHost site-a:8181>
  <Location /dir0>
  DAV svn
  SVNPath /home/site-a/svnroot
  AuthType Basic
  AuthName wandisco
  AuthUserFile /home/site-a/apache2/dist/conf/htpasswd
  Require valid-user
  </Location>
  </VirtualHost>
  ...
  # Site B
  $ cat conf/extra/httpd-svn-dav.conf
  ...
  NameVirtualHost site-b:9191
  <VirtualHost site-b:9191>
  <Location /dir0>
  DAV svn
  SVNPath /home/site-b/svnroot
  AuthType Basic
  AuthName wandisco
  AuthUserFile /home/site-b/apache2/dist/conf/htpasswd
  Require valid-user
  </Location>
  </VirtualHost>
  ...
  

  Example httpd.conf file illustrating the use of Apache virtual hosts

• 3. The Apache usernames and passwords should match at all nodes.

  Alert
  SVN MultiSite must have a valid username inside the HTTP authorization header to be passed for all DAV commands.

2.3.9.4 Apache and SVNKit

There are potential problems with connection pooling when using Apache and SVNKit. Adding SVN MultiSite to the mix doesn't change these issues, though you may need to revisit them after installing MultiSite.

We recommend using JavaHL with Eclipse IDE, which does not use connection pooling, and thereby eliminates the problem.

2.3.9.5 SVNKit and Connection Pooling

SVNKit uses connection pooling. For a given client, SVNKit opens two connections and keeps them open for later use. On a system with a heavy load this can hit performance. An open connection consumes an Apache worker thread, with lots of clients and pooling going on, Apache may run out.

Apache provides some tuning parameters to optimize connection pooling, while still releasing unused connections. The tunable parameters are Timeout, KeepAliveTimeout, MaxKeepAliveRequests, and KeepAlive. For more details refer to the Apache configuration documentation.

2.3.9.6 Optimize Your Configuration

Apache has two timeout configurations: Timeout and KeepAliveTimeout. In general, the Timeout value should be lower than the value for KeepAliveTimeout.

Caution
Setting KeepAlive to false is not recommended. If you set KeepAlive to false, a client's transactions create an enormous overhead establishing and destroying connections.

2.3.9.7 Apache Logging

Apache's logs are dumped to a single file, which can cause problem when the log file grows so big that it needs to be truncated. When this happens, Apache service is interrupted while the file is restarted. On Linux/Unix servers there's the option of piping logs to another process. For more information see our Knowledge Base article - Piping Apache logs

2.4 Deployment Checklist

Though you may have referred to the Deployment Checklist prior to an evaluation of SVN MultiSite we strongly recommend that you revisit the checklist and confirm that your system still meets all requirements.

System setup
Operating Systems	We've tested the following operating systems: Red Hat Linux Enterprise (RHEL) Server (32 or 64 bit): 4, 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 6, 6.1, 6.2, 6.3 Oracle Linux (Running Red Hat Compatible Kernel): Oracle 5 (v2.6.18), Oracle 6 (v2.6.32) Sun Solaris (32 or 64 bit): 9, 10, 11 SuSE Linux Enterprise Server: 11 Ubuntu: 10.04 & 12.04 Debian: 5 & 6 CentOS: 5.2, 5.3, 5.4, 5.5, 5.6, 5.7, 5.8, 6, 6.1, 6.2, 6.3 Windows Server: (32 or 64 bit) 2003, 2008 In principle, any operating system that can support a Java environment, Apache and Subversion.
Subversion server	Version 1.4 and above (we have successfully tested the recent 1.8 release). Run the Apache Portable Runtime that matches your Subversion version. SVN MultiSite is now compatible with Subversion 1.7's HTTPv2 implementation. MultiSite 4.0 Users: Read our Knowledge base article on a workaround that lets you Run MultiSite 4.0 with Subversion 1.7. Alert There is a known issue that results in incorrect error reporting during "disk full" scenarios running with Subversion 1.7. Read our knowledgebase article:         Incorrect error reporting when running SVN MultiSite 4.2 with Subversion 1.7 Certified Subversion Binaries are now available from WANdisco. Providing the latest builds, without the risks associated with Open Source distribution.
Subversion client	Any that are compatible with local Subversion servers.
Hooks	Pre-commit hooks are not recommended. Use pre-replication hooks instead. Any pre-replication hooks must be deterministic, i.e. have the same exact behavior and outcome at every node. Post-commit hooks can be tested at only one node.
System memory	Ensure RAM and swapping containers are at least four times larger than the largest Subversion file. Minimum recommended: 2 GB RAM; 4 GB swapping container
Disk space	Subversion: Match to projects and issues. MultiSite Transaction Journal: Equivalent of seven days of changes. To estimate your disk requirements, you need to quantify some elements of your deployment: overall size of all of your SVN repositories. frequency of commits in your environment. types of files being modified - text,binaries (SVN clients only send deltas for text). number and size of files being changed. rate that new files are being added to the repository. Alert Changes to the network layer introduced in 4.1 mean that MultiSite no longer buffers replicated data, removing the requirement for sufficient disk space for handling large concurrent checkouts for large numbers of users.
File descriptor limit	Ensure hard and soft limits are set to 64000 or higher. Check with the ulimit or limit command.
Journaling file system	Replicator logs should be on a journaling file system, for example, ext3 on Linux or VXFS from Veritas. Alert NFS must not be used with Subversion MuliteSite - Why you shouldn't use NFS. ext4 can be used as your journaling file system, although it must be configured appropriately. See Using Ext4 filesystem for journaling . Avoiding Data Loss We have an article in our Knowledgebase that looks at a number of implementation strategies that will militate against potential data loss as a result of power outages -         Data Loss and Linux
Max. User Process Limit	At least three times the number of Subversion users.
Java	Install JDK 6 (previously referred to as 1.6). We recommend using Oracle JDK 6. Alert Important: JDK 7 (AKA 1.7) is not compatible with SVN MultiSite. We're working to address the problems, but until they are resolved you should not use JDK 7 with SVN MultiSite.
Perl	Install version 5.6.1 or later. For Access Control: Perl::DBI module for Audit Reports other than Users and Groups. Alert Windows users: Use the 32-bit edition of ActivePerl version 5.8, even if you are running a 64-bit Windows server.

sudo a2dismod deflate

sudo service apache2 restart

    ls /etc/apache2/mods-enabled/

3. Installation

The installation guide runs through setting up SVN MultiSite for the first time. If you're already using an earlier version of MultiSite, refer to the Upgrade section.

3.1 Installation overview

Before we begin, here is a recap of the installation process:

• 1. Revisit the pre-deployment checklist. Taking extra time making sure you have everything set up and ready may save you from running into problems during the installation process. The main boxes you need to tick are:


• Subversion authenticating: Subversion installed, and using authentication.

  Alert
  Authentication Must be enabled (within the location directive of the Apache config file). Otherwise changes to the repository will not be replicated.



• JDK6: You need to have Java JDK 6 (previously referred to as 1.6) installed. We recommend using Oracle JDK 6.
  

  Alert
  Important: JDK 7 is not compatible with SVN MultiSite. Until we've addressed the compatibility problems you should not upgrade to JDK 7.



• Java Memory Settings: MultiSite will start with 128MB and sets a maximum of 2GB of memory when it starts up. Changing the default Java heap size.


• System resources: Ensure that your system is going to operate with a comfortable margin. Hardware Sizing Guide Knowledgebase article.


• 2. Ensure that your repository is copied into place on all nodes. See 2.3.3 Creating copies of your repository


• 3 Ensure that Apache is running and that the Apache config file has been updated to support MultiSite.


• 4. Download and copy the MultiSite files into place.


• 5. Run the setup, then complete the installation from a web browser.

3.2 Before you start

Some things you should bear in mind before starting the installation:

• Don't run the installer on each node The installation is only run once, for the first node. Subsequent nodes are created using MultiSite's procedure for "Adding a node".


• Once SVN MultiSite is installed and you have synchronized repositories, NEVER make a change to your repository that will bypass MultiSite's replicator. This will almost certainly cause replication to fail as the repositories will no longer match.


• On a Linux/Unix platform, MultiSite must be run as root, in order to use port 80.


• Once SVN MultiSite is installed, we strongly recommend that you set up the replicator as a service to ensure that it automatically restarts on a system reboot.

3.3 Starting the installation

• 1. Extract the setup file.
• 2. Save the svnrep.tar.gz archive to your Installation node.
• 3. Extract the setup files using a compression utility that supports the .gz (gzip) format.

Install to a writable directory
Ensure that you install the SVN MultiSite to a directory that is writable by the user running MultiSite on all nodes. If this is not the case, some operations will fail when the system attempts to make changes to MultiSite's files.

Don't alter SVN MultiSite's directory structure
Ensure that you install the SVN MultiSite files to a directory called svn-replicator and that you preserve the following subdirectory structure:

If using selective replication you need separate instances of MultiSite installed, but each situated in a directory called svn-replicator, for example:

<replicator01>/svn-replicator
<replicator02>/svn-replicator
<replicator03>/svn-replicator 

Selective replication is used when you need to replicate only specific repositories. You can find further information in 1.23 Selective Replication.

• 4. Place the license.key file in the svn-replicator/config directory.
• 5. Run the setup file. Setup is completed by running through steps in a browser-based wizard.
  
• 6. From the /bin directory, start setup with the following command:

  perl setup
  

  WANdisco products use port 6444 by default. If you're not able to use this port on your network, you can set a different port by starting setup using the command:

  perl setup <chosen port number>
  

• 7. Setup will begin by confirming how much memory needs to be allocated to JAVA. By default, MultiSite starts with 128MB and reserves 2048MB (2GB).

  =======================================================================
  WANdisco java processes will start with the following memory settings:
  
       -Xms128m -Xmx2048m -ea -server -Djava.net.preferIPv4Stack=true
  
  Change the environment variable WD_JVMARGS if you wish to start java differently
  WARNING: if the host does not meet these specified memory requirements, you will
  encounter problems starting the WANdisco processes.
  Continue,  Y or N ? [Y] :  y
  Mar 29, 2010 11:29:47 AM org.nirala.trace.Logger info
  INFO: Invoked from WANdisco installation at: /wandisco/svn-replicator
  [I] using specified port: 6444
  [I] Starting Subversion web installer
  

  You can change how memory is allocated by editing references to SERVER_JVM_ARGS in the bin/reputils.pm file.
  Read our knowledgebase article about Changing the default Java heap size.

  If you're happy with the allocation, type 'y' and then press return.

  Point a web browser to http://10.2.5.46:6444/ to configure the product.	
  

• 8. Setup will now start up the web interface. Open a browser and type in the given address. The default port is 6444. If you entered a different port (step3.1) then use this instead:

  http://<server IP>:<WANdisco port you selected when starting setup>

4. Complete setup through a browser

• 4.1 The welcome screen will appear in your browser. Follow the link to the pre-installation checklist to make a final check that everything is place for a successful installation. When you're ready, click continue.
  
  
• 4.2 Read through the WANdisco End User License, click I Agree to continue.
• 
  
• 4.3 The default username for getting into the admin console is admin. In the next step you need to enter a password for the admin account. Once you've chosen a password, click Next.
  
• 4.4 The next two screens explain how MultiSite will act as a proxy between Subversion clients and the Subversion server. Click Next to continue.
  
• 4.5 By default, MultiSite will listen on port 80, while Subversion will listen on port 8080. The benefit of this setup is that Subversion end-users don't need to make any changes. Click Next to continue.
  
• 4.6 You now can choose to enable SSL encryption . Tick the check box to open the SSL Settings screen, next.
  
  Click Next. If you tick the Configure SSL keystore check box, you'll be taken to the SSL Settings screen:
  
  
  Certificate Alias: The certificate name for the server.

  Key Password: Password for the KeyStore file.

  KeyStore Filename: Path to file where private keys are stored.

  KeyStore Password: Password for the KeyStore file.

  TrustStore Filename: Path to the file where public keys are stored. This may also be the KeyStore file.

  TrustStore Password: Password for the TrustStore file.

  The settings are populated with default values to aid setup, make sure these match your environment and security requirements before clicking Next.

  
  

  Tip
  You may wish to set up the SSL encryption after you've successfully set up MultiSite without SSL. If you experience problems with your SSL setup you'll then already know that the underlying replication is working.



• 4.7 Next, you'll confirm the proxy settings. These will be populated with the default port settings noted in the previous screens.
  
  
Node Name: The name that Access Control will use for sever on which it's installed. The name cannot contain spaces, but can be changed later using the Admin Console.
Node IP: The server's IP address.
Bind Host: By default, this uses the wildcard 0.0.0.0 IP that binds to all network interfaces on the node. Read about the benefits of using the wildcard IP

Client Port: By default this is 80, allowing Subversion users to continue without making change to their client setup.

(Linux/Unix) In order to use port 80, MultiSite must be run as root.


Admin Console Port: 6444 by default.

Reserved Ports: 6444-6453 by default

Use Manually Distributed Ports: Tick this check box to manually assign the range of ports instead of using the default range (6444-6453).

Admin Console Port: the port that will be used to access SVN MultiSite's admin console. The default is 6444.

WANdisco Port: The port that DConE (the WANdisco replication service) will run on. The default is 6444 (the same port that is used for the admin console).

DFTP Port: The direct file transfer protocol port is used for communication between nodes

Shutdown Port: Listener that is contacted by the shutdown and LDAP admin reset scripts (binds to localhost only).

Use SSL: Enable SSL to use encrypted communication between nodes. If SSL is enabled, then https: is also required for browsing to the web console.

Once you've finished making any changes to the proxy settings, click Next to continue.


4.8 Setup will now check the Apache config file for settings that might cause problems for MultiSite.
More about Apache configuration changes for MultiSite.








If the httpd.conf file isn't found, enter its path into the Configuration File entry box,
then click Reload Configuration.

Look out for warning boxes for where setup finds a problem - like this one:



If a problem is highlighted, you'll need to manually edit the httpd.conf file, then click on Reload Configuration to have setup check your changes.

User: Owner of the file.
Group: The group in which the owner belongs.
KeepAlive: Setup will look to see that the Keep-Alive directive is set to On.
KeepAliveRequests: Setup will look for 0, which indicates that no maximum limit will be set for connection requests.
KeepAliveTimeout: Set high to ensure connections don't timeout.
Listening IP: For a node with multiple IPs, this will indicate the IP used for listening.
Listening Port: The default listening port for Apache is 8080.

When you want to continue, click Next. To continue without the Apache config check, click Skip.

4.9 Setup now allows you to modify your Subversion settings. Watch for alerts that confirm the port and path that MultiSite will associate with the Subversion server.


Use Pre-Replication Hooks: Select Yes or No.

If you select Yes you must select the version of Subversion installed on your server. If you don't know the version as you run through the setup you can select "Unknown". However, once the installation is completed, you must confirm the version of Subversion you are using, then edit the option from the SVN Settings page.



Subversion Host: The host name for the server on which Subversion is running. You'll benefit from making this the same as the server running MultiSite, e.g. 'localhost.
Subversion Server Port: 8080 by default.
SVN Executable: This is the fully qualified path to the Subversion executable. Setup will try to fill this in automatically, otherwise type it into the entry box.
Use authz-based access control? Tick the box to have MultiSite use Authz authentication.

Authz and regular expression based Access Control rules
If you use Authz to manage user authorization through Apache you need to be aware that Access Control rules that use regular expressions to define resources are not supported by Authz so won't be written to the Authz file.

Selecting authz access will reveal an additional entry field for recording the location of the authz file.

For example, given the root /svnroot with the repositories /svnroot/one, /svnroot/two and /svnroot/three the value would be: one,two,three.

Repository Settings

At the bottom of the screen is a table that confirms the DAV Location and password control for your repositories. You can click Edit to make changes. You can add additional repositories by clicking Add Repository. To continue setup, click Next.

If you click Edit to change Repository you'll be able to edit the following settings:



Directory on Files System: Fully qualified path to the Subversion repository, not the URL used by clients to connect.
Manage Password File: Tick the box to allow your Subversion password file to be controlled by Access Control.
Password File: If you tick "Manage Password File", you'll see a browsable entry field for the password file's location.

Pre-Commit Hook: Equivalent to normal pre-commit hook but it is automatically picked up and invoked in a replication safe manner.

Pre-lock Hook: Equivalent to normal pre-lock hook (invoked when a user attempts to lock a resource) but it is automatically picked up and invoked in a replication safe manner.

Pre-Unlock Hook: Equivalent to normal pre-unlock hook (invoked when a user attempts to destroy an exclusive lock) but it is automatically picked up and invoked in a replication safe manner.

Authentication Realm: Associate an authentication realm with the repository. You should follow the guidance in the Apache documentation for the AuthName Directive.


Settings in Apache
DAV Location: You can specify the location of the DAV file.
Multiple SVN Repositories: click Yes if you are using SVNParentPath for multiple repositories, or No if using SVNPath.

If you select Yes, SVNParentPath is used in Apache location directive, you'll be shown a list of Root Names, and can add additional subdirectories by entering their names in the Add Root box.

For example, given the root /svnroot with the repositories /svnroot/one, /svnroot/two and /svnroot/three the value would be: one,two,three.

Click Update to apply your changes, or Cancel to return to the previous screen without making changes.

When you've finished making changes to the repository settings, click Update to return to the previous screen, then click Continue.



4.10 You can enter email settings so that MultiSite can send alerts when problems are detected.

Example email

The distinguished node sends alerts on behalf of the whole replication group. The node and repository that the alert to which the email applies are noted in the email subject.






SMTP Authentication: If you select No, you'll need to provide your account.
Username and Password in the following entry boxes.
Use SSL/TLS: Choose yes if you wish to send emails over a secure connection.
Host: Enter the address of your mail server.
Port: Enter the SMTP port, 25 by default.
Send Admin Notification To: The email address (You can only specify a single address, not a comma delimited list) to where notifications will be sent.

Email settings are optional. If you don't need alert emails, click Skip to continue. Otherwise, click Next.

4.11 The setup has finished gathering information, You can go back and make changes or click Complete installation with these settings to save them and complete the installation.



When you click Complete installation MultiSite will start running. You'll be prompted to reload the admin console using an SSL connection.

5. Login to the Admin Console for the first time

• 5.1 The browser will refresh to show the admin console login screen.
  
  To login for the first time, enter the username "admin" and the password you entered during setup (step 4.3).



• 5.2 When you first enter the admin console you'll be prompted to create a new user and other nodes.
  
  For more information see the guide to the admin console.

6. Copying the installation files manually

Follow these instructions if you were unable or chose not to use the automated packaging option Covered in section 7.

MultiSite creates a [node name].zip file for each of the nodes in a replication group. The first node is installed, so perform the following for all new nodes in the group:

• 6.1 Copy the [node name].zip to the new node
• 6.2 At the new node, extract the [node name].zip file to a temporary directory called 'wandisco'.
• 6.3 From a command prompt type:

  cd wandisco
  perl unpackage
  

• 6.4 Follow the directions provided by the unpackage script.

7. Post Installation Configuration

There are some changes you may need to make before starting to use Subversion through MultiSite. These changes are made through the admin console. For more information see the admin Console guide

7.1 Using pre-commit hooks?

If you are going to be using pre-commit hooks, these become WANdisco pre-replication hooks. Read Setting up hooks
in the User Guide.

3. User Guide

1. Replicator Management

1.1 Adding a node

To add a node to an existing replication group, you have to create a new replication group that contains the new node and then activate the new replication group. See Creating a New Replication Group.

Determine a plan for copying the repository to the new node. See 2.3.3 Creating copies of your repository.

Verify before replication
Before you replicate your repositories, run the svnadmin verify command to ensure that there are no corrupted revisions - Read about svnadmin verify

• 1. Open the admin console and Click Click here to create new nodes.
  

  Creating a new node.

  
  If this link doesn't appear, you can also click on the Proxy tab, then on the Nodes link on the menu, then click the Create Node button.
  
  
• 2. Enter the properties for the new node.
  

  Entering the properties for the new node.

  Enter the details of your new node:
  
  Node properties
  Name: A name to be used to refer to this node.
  Host Name: DNS name or IP address of the node.
  Bind Host: The IP bound to MultiSite. We suggest using 0.0.0.0. why we bind to 0.0.0.0.
  Port:Port number allocated to the node. The default port is 6444.

  Subversion Properties (for remote node)
  The Subversion Properties are populated with settings of the first node, which assumes that your new node will use the same proxy setup. Change them if required.

  SSH Properties
  SSH is used by MultiSite to automatically install on your additional nodes. You enter the details here for giving SSH access to the new node.

  Why don't I see SSH settings?
  You won't see SSH settings if you have created the current replication group using the manual approach, instead of selecting to "Activate the nodes automatically via SSH?" demonstrated here.

  SSH Port: Default port for SSH is 22. Assign if a different port if required.
  Authentication: By username/password or by Private Key File.
  

  Selecting Authentication with Username/ Password:

  SSH Username: The SSH authentication Username.
  SSH Password: The SSH authentication password.
  JAVA Home:The path to JAVA on the new node. This field is optional. If not specified here the value for the environmental variable JAVA_HOME will be used.
  

• 3. When all details have been entered, click the Create Node or Create and Test. The test confirms the SSH properties that will be used to push out the installation files to the new node, as well as the Java home directory.
  

  Successful test of the node's SSH properties.

The new replication group is activated with all nodes.

1.2 Removing a node

It's not possible to remove a node from an active replication group, instead a new replication group is created excluding the node that you want to remove:

• 1. Follow the procedure for creating a new replication group. The new replication group should contain all the nodes in your current replication group, except for the node(s) you want to remove.
• 2. Active the new replication group.
• 3. Once the old replication group is no longer active you can delete it, along with any nodes that are no longer members of the active replication group.
  

  Select any inactive replication group and click the "Delete Selection" button.

Alert
Only one replication group can be active at a time.

1.3 Creating a new replication group

You can create as many replication groups as you like, although only one replication group can be active at a time.
You can't delete the active replication group, first you need to create a new replication group and make it the active group.
See Deleting replication groups

Tip
Create replication groups from the node that will be the distinguished node in the new replication group.

Alert
Only one replication group can be active at a time.

• 1. From the Proxy tab, click Replication Groups. All defined replication groups are listed, including the active one. Click the Create Replication Group.
  

  Replication Groups screen, with one previously created replication group listed.



• 2. Name the new replication group and select the appropriate quorum. See more about quorum selection. Tick the nodes that you want to replicate within the group, then click Create Replication Group.
  

  Replication Groups screen basic settings.



• 3. You will see a summary of the new replication group. Check the details are correct, then click Activate.
  

  Replication group created - awaiting activation.



• 4. The next step will activate the new replication group. The current replication group will become inactive, any nodes that are not members of the new replication group will continue to be accessible to local users but will no longer replicate.
  

  Caution
  Activating a replication group will stop all nodes and temporarily prevent write transactions.
  Nodes that are retired from replication should be shut down.

  
  Activation Information
  The Activation Information determines how the replicas of the node will be transferred to the other servers.
  WANdisco Directory: confirms the installation directory, which is not editable as it must mirror the directory structure on the current node.
  
  Activate Nodes via SSH?
  Select Yes, Activate the nodes automatically via ssh to have the node's setup files copied to the new nodes via SSH.
  Select No. I will copy the archives manually. if you prefer to copy the files yourself.
  
  Packaging Options
  Delete Packaging Dir?: This option will appear if node package files already exist in your specified Packaging directory. You can select to automatically overwrite the exiting package files or provide a warning:
  
  Package Repository:
  Select Yes, Create an archive of the repositories to have the repositories automatically packaged.
  Select No. I will synchronize the repositories manually to transfer the repositories using a sync utility, such as rsync
  
  
  Set Owner / Group:
  Select Yes. Set the owner and/or group of the copied files to allow the owner and group to be automatically set during the archive creation.
  Select No. Do not set the owner and/or group of the copied files if you want to handle the assignment of owner and group yourself.
  

  Caution
  If you manually transfer your repositories, ensure that their owner and group are set appropriately, otherwise permission errors may occur during replication.

  
  Packaging Dir: The directory in which the package files will be transferred.
  
  Active Replication Group: A dropdown selector where you choose which replication group that you wish to make active.
  
  When all options have been made, click the Continue button.
  
  
  


• 5. In the next step, the nodes' SSH details are tested. Providing SSH connections can be made to each node click the Continue button.
  
  
  

  If the test fails, check your SSH settings, and that the node is able to accept connections on the SSH port (port 22 by default).

• 6. The package files are created. Once finished, the current node will automatically reset. The distinguished node will then be ready.
  


• 7. The replication group is now ready, click Deploy Node(s) to transfer the package files to each node.
  



• 8. The package files for each node will now be transferred. If you selected (in step 4.) to bundle up the repository file it will also get transferred.
  
  


• 9. The next screen appears if you selected to manually transfer your repositories. You should ensure that copies of the repository are in place on each node before clicking Continue.
  
  
  
• 10. Confirm that the repositories for each node have been copied into place or synched. If you choose yes, the node will start up with write access to the repository enabled.
  
  


• 11. The replication group has now been activated. Click on the Proxy Status link to check that all nodes are up and working.
  

1.4 Deleting a replication group

Follow this procedure to delete a replication group. Remember, you can only delete a replication group that is inactive.

• 1. Click on the Proxy tab, then click on Replication Groups on the menu.
• 2. From the list of replication group, tick the checkbox that corresponds with the replication group that you want to delete, then click Delete Selected.
  
• 3. You'll see a confirmation message that the group has been deleted.
  

1.5 Adding a repository

You can add any number of repositories to the replication group at any time, using the following procedure:

• 1. Ensure that the repository that you're adding can't be changed (in a read-only state) then copy it to each node in the replication group.


• 2. Add a reference to the new repository to the Apache config file on each node. The form of reference will depend on whether you are using SVNParentPath or SVNPath. See 2.3.4 Replicating multiple repositories.



• 3. Open the admin console using a browser. Click on the Proxy tab and then click SVN Settings from the menu.
  


• 4. Click on the Add Repository button, found at the bottom of the list of repositories.
  
  


• 5. Fill in the details of the new repository:
  
  Directory on File System: the fully qualified path to the Subversion root (in which repositories are stored).
  Manage Password File: checkbox to indicate whether SVN MultiSite will take control of the Subversion password file.
  
  Pre-commit hook Equivalent to normal pre-commit hook but it is invoked in a replication safe manner.
  Pre-Lock Hook Equivalent to normal pre-Lock hook but it is invoked in a replication safe manner. Pre-UnLock Hook Equivalent to normal pre-Unlock hook but it is invoked in a replication safe manner.
  Settings in Apache
  DAV Location: This is the path used to access the repository via Apache. For example, if your repository has a DAV location of 'zeppo', then the URL to reach the repository would be http://domain.com/zeppo.
  
  Tip
  The DAV location is set within the Apache config file, using the location directive. e.g.
  

  <Location /zeppo>
  
  AllowOverride None
  Order allow,deny
  Allow from 127.0.0.1
  DAV svn
  SVNParentPath /tmp/dav
  AuthType Basic
  AuthName wandisco
  AuthUserFile /etc/httpd/conf/htpasswd
  Require valid-user
  </Location>
  

  Multiple SVN Repositories Answer Yes if you're using SVNParentPath or No if you're using SVNPath.

1.6 Removing a repository

• 1. Perform a synchronized stop of the replication group.


• 2. If the repository was referenced in the Apache config file (using SVNPath) you should now delete the reference (the whole location directive), then restart Apache for the change to take effect. This step is unnecessary if you are using SVNParentPath.
• 3. Go to the admin console, click on the proxy tab. then click on the SVN Settings link on the menu.


• 4. In the Repository section, go to the line that corresponds with the repository that you are removing. Click the Delete icon (white cross in a red circle).
  
  


• 5. Restart the replication group. Go to the Proxy tab and click the Resume.
  
  

1.7 Moving a repository to another replication group

To move a repository from one replication group to another:

• 1. Remove the repository from the nodes of the current replication group using the Removing a repository procedure.
• 2. Add the repository to the nodes of the new replication group, using the Adding a repository procedure.

1.8 Setting MultiSite to start up on system boot

SVN MultiSite comes with sample scripts debiansvnreplicator-init.d-sample and svnreplicator-init.d-sample which can be found in the /utils/init.d/<OS specific dir> directory.

View a copy of the script: debiansvnreplicator-init.d-sample
(Written for Debian, though also compatible with Ubuntu.)

1.9 Changing the Subversion replicated client port

To change the Subversion replication port that clients use to connect to repositories:

• 1. Navigate to the admin console's Proxy tab.
  
  


• 2. Click on Nodes on the left side menu.
  
  


• 3. Select a node, then click the corresponding Edit Node link.
  
  


• 4.Ensure that Replicate Changes is set to Yes. Update the node definition at all nodes in this replication group..
  
  


• 5. Enter a new port number in the Subversion Replicated Client Port field.
  
  


• 6. Click the Update Node button at the bottom of the screen. A confirmation message will appear near top of the screen confirming that the change has been made.
  
  

1.10 Shutting down MultiSite

You can shut down a replicator at any time, either from the node's admin console (see the Admin Console Guide 5.9 Shut down node), or through running the shutdown script ./shutdown in the svn-replicator/bin/ directory.

2. Setting up the replicator for production

2.1 Setting up MultiSite as a Windows service

We offer a configurable Java launcher for Windows based on WinRun4J. Follow these instructions for setting it up:

• 1. Shutdown and reset all nodes.


• 2. Open a command prompt that has administrator priveleges.
  
  


• 3. Navigate to the svnreplicator/bin folder in the SVN MultiSite Installation directory.


• 4.Start the service by running the executable that matches your server's architecture i.e.
  
  (32Bit)

  MultiSiteWinService32.exe --WinRun4J:RegisterService

  (64 Bit)

  MultiSiteWinService64.exe --WinRun4J:RegisterService

  The service will start up as follows:

  Directory of C:\Program Files\svn-replicator\bin
  
  C:\Program Files\svn-replicator\bin>MultiSiteWinService64.exe --WinRun4J:RegisterService
  [info] Module Name: C:\Program Files\svn-replicator\bin\MultiSiteWinService64.exe
  [info] Module INI: C:\Program Files\svn-replicator\bin\MultiSiteWinService64.ini
  [info] Module Dir: C:\Program Files\svn-replicator\bin\
  [info] INI Dir: C:\Program Files\svn-replicator\bin\
  [info] Registering Service...
  
  C:\Program Files\svn-replicator\bin>
  



• 5. Verify that the service is running by checking the Windows Service Manager
  
  


• 6. You may need to provide account login details, especially if the server isn't run with full admin privileges. Click the Log On tab, then provide log on credentials for the required account. Once added, click Apply.
  
  


• 7. Right click on the service and select Start.

2.2 Toggling the quorum check

SVN MultiSite verifies if a network quorum is reachable when a write command is submitted. If the quorum is unreachable, by default, the write command is aborted and the following message appears on the Subversion client console:

Check the Network connectivity, failed to reach a
minimum quorum of nodes. Aborting the svn write operation.

To turn off the quorum check, edit the prefs file (svn-replicator/config/prefs.xml). Set the parameter, AlwaysVerifyQuorum to false in the file. For instance,

<SVNProxy>
<AlwaysVerifyQuorum>false</AlwaysVerifyQuorum>
....
</SVNProxy>

If the check is turned off and quorum is un-reachable, the write transaction will be applied to the WANdisco SVN MultiSite's transaction journal and stay in a pending state till network connectivity and quorum is restored.

Alert
With singleton quorum, if the current node is also the distinguished node, the quorum check will always succeed irrespective of network connectivity to other nodes.

2.3 About Watchdog mode

By default, SVN MultiSite starts in watchdog mode. Whenever the replicator goes down, the watchdog mode restarts it. In watchdog mode, the replication process automatically disassociates from the terminal and becomes a daemon process, so you should not try running it in the background (with &).

Watchdog mode is not supported in Windows, use Windows Cygwin.

You can turn off watchdog by typing:

-nowdog

If MultiSite is unable to start up, for example if it terminates several times in quick secession, watchdog starts the the node in read-only mode.

$ ./bin/svnreplicator -h

Usage:

svnreplicator [-v] [-verbose] [-nowdog] [-pause time] [-email email-address]

-v

Print the svnreplicator version

-verboseVerbose

console messages go to STDOUT/STDERR instead of logs/console.txt

-nowdogTurn off

watchdog mode. WANdisco will not restart automatically if it terminates. Use this option for testing.

-pauseTime

in seconds that the watchdog pauses for, before restarting service. Defaults to 0 seconds.

-emailSpecify

an email address to send an alert to, whenever the Watchdog restarts or shuts down WANdisco. WANdisco generates an email per local replicator activity. If the email settings were not set up during installation, see Email Settings, described in Using the Admin Console.

Use the -email option to generate email alerts whenever MultiSite restarts. For instance:
$ svn-replicator/bin/svnreplicator -pause 5 -email "admin@blueandgold.com, scmuser@blueandgold.com"

Alert
In order to have the replicator automatically started on system reboots, see Setting Replicator to Start Up on System Boot.

2.4 Using SSL

SVN MultiSite can be set up to use SSL encryption. First you should run through the following steps after extracting the SVN MultiSite files (but before running setup).

• 1. Open a terminal and navigate to <INSTALL_DIR>/svn-replicator/config.


• 2. Within /config make a new directory called ssl.
  

  -rw-rw-r-- 1 User User   512 2010-03-12 19:03 license.key
  -rw-r--r-- 1 User User  3327 2010-05-21 14:12 log.properties
  -rw-r--r-- 1 User User   579 2010-05-21 14:12 mailconfig.properties
  drwxr-xr-x 5 root root  4096 2010-06-18 12:39 membership
  drwxr-xr-x 2 root root  4096 2010-06-18 12:39 passwd
  drwxr-xr-x 3 root root  4096 2010-06-18 12:33 prefs
  -rw-r--r-- 1 User User  2005 2010-05-21 14:12 prefs-template-failover.xml
  -rw-r--r-- 1 User User  3047 2010-05-21 14:12 prefs-template.xml
  -rw-r--r-- 1 User User 92160 2010-06-11 14:26 reports.tar
  drwxr-xr-x 3 root root  4096 2010-06-18 12:39 scm
  drwxr-xr-x 6 root root  4096 2010-06-18 12:39 security
  -rw-r--r-- 1 User User    72 2010-05-21 14:12 svnerr.catalog
  -rw-r--r-- 1 User User   625 2010-05-21 14:12 svnokerr.catalog
  
  [User@Fed11-2 config]$ mkdir ssl



• 3. Go into the new directory.
  

  cd ssl



• 4. Copy your private key into the directory. If you don't already have keys set up you can use JAVA's keygen utility, using the command:
  

  keytool -genkey -keyalg RSA -keystore wandisco.ks -alias server -validity 3650 -storepass  <YOUR PASSWORD>

  Knowledgebase
  Read more about the Java keystore generation tool in the KB article - Using Java Keytool to manage keystores

  -genkey

  Switch for generating a key pair (a public key and associated private key). Wraps the public key into an X.509 v1 self-signed certificate, which is stored as a single-element certificate chain. This certificate chain and the private key are stored in a new keystore entry identified by alias.

  -keyalg RSA

  The key algorithm, in this case RSA is specified.

  wandisco.ks

  This is file name for your private key file that will be stored in the current directory.

  - alias server

  Assigns an alias "server" to the key pair. Aliases are case-insensitive.

  -validity 3650

  Validates the keypair for 3650 days (10 years). The default would be 3 months

  - storepass <YOUR PASSWORD>

  This provides the keystore with a password.

  If no password is specified on the command, you'll be prompted for it. Your entry will not be masked so you (and anyone else looking at your screen) will be able to see what you type.

  Most commands that interrogate or change the keystore will need to use the store password. Some commands may need to use the private key password. Passwords can be specified on the command line (using the -storepass and -keypass options).
  However, a password should not be specified on a command line or in a script unless it is for testing purposes, or you are on a secure system.
  

  The utility will prompt you for the following information

  What is your first and last name?  [Unknown]:  
  What is the name of your organizational unit?  [Unknown]:  
  What is the name of your organization?  [Unknown]:  
  What is the name of your City or Locality?  [Unknown]:  
  What is the name of your State or Province?  [Unknown]:  
  What is the two-letter country code for this unit?  [Unknown]:  
  Is CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown correct?  [no]:  yes
  
  
  Enter key password for <mykey>
  (RETURN if same as keystore password):
  

• 5. With the keystore now in place, the setup will automatically pick the file up if you tick the Use SSL checkbox, during the installation (of after installation - see 4.5 SSL Settings.
  
  Changes in these value require a restart. Any invalid value will restart the replicator and no DConE traffic will flow.

SSLv3 is support (though not enforced). If your browser setting has SSLv3 disabled, you will get a handshake error message. If it has both SSLv3 and TLS enabled, then, depending on the browser, it will try to switch from TLS to SSLv3 during the handshake. If you receive a handshake error message in your browser, make sure that TLS is disabled and only SSLv3 is enabled. All current browsers support SSLv3.

2.6 Setting the server key

In the keystore, the server certificate is associate with a key. By default, we look for a key named server to validate the certificate. If you use a key for the server with a different name, enter this in the SSL settings.

2.6.1 SSL Troubleshooting

A complete debug of the SSL logging will be required to diagnose the problems. To capture the debugging, run the java process with:

'-Djavax.net.debug=all' flag.

To enable the logging of SSL implemented layer, turn the logging to FINEST for 'com.wandisco.platform.net' package.

2.7 Using SSL for both LDAP and emails

If you're specifying secure email (using a truststore) and LDAP authentication over SSL (using a truststore), the same truststore must be used for both sets of certificates. If different truststores are used then the LDAP truststore will overwrite the email truststore and secure emails will stop working.

3. Changing prefs.xml settings

SVN MultiSite's settings are stored within a prevayler database. However, some settings (for all nodes in the replication group) are stored within a preference file (prefs.xml) which is located in the <WANDISCO/svn-replicator/config directory.

Alert
If you make changes that affect more than one node, you must change each node's specific file. If your change only effects just one node, you can change just that node's prefs.xml file.

3.1.1 Changing prefs.xml file on a single node

• 1. On the node on which you want to make the change, go to the Proxy tab and click Shut Down Node.
• 2. Make the change to the node's prefs.xml file and save the changes.
• 3. Restart the node by running the command:./svnreplicator
  The changes you made in the prefs.xml file are now in effect.

3.1.2 Changing All Nodes' prefs.xml Files

• 1. Perform a synchronized stop. See Performing a Synchronized Stop. That procedure includes resuming MultiSite.
• 2. Make the desired changes to each node's prefs.xml file.
• 3. Restart WANdisco at all servers. Click Resume.

3.2 Performing a Synchronized Stop

Alert
Previously, all nodes in a replication group needed to be available in order to perform a synchronized stop. Now it's possible to synchronize stop a subgroup of nodes which is useful if you need to bring your replication group to a stop even if you don't have all nodes up and working. However,         this action should only be used as a final resort and will require additional work to ensure that sync is maintained.

All Sync stops make Subversion read-only. You'll need to warn your Subversion users of some down-time.

• 1. Go to the Proxy tab on the Admin Console. On the side menu click Stop Proxy.
  



• 2. Select the Synchronized stop radio button and ensure that All node checkboxes are ticked, then click Continue. If one or more of your nodes is down, and can't be quickly recovered, you can deselect the problem node and synchronize stop only the remaining nodes. See Selective Synchronized Stops
  
  This procedure makes Subversion read-only. You'll need to warn your Subversion users of some down-time.
  
  On nodes included in the synchronized stop: All in-flight transactions will be completed, MultiSite stops listening. The Subversion servers go into read-only mode.



• 3. Check the status of the nodes by clicking Proxy Status Refresh the Proxy tab and click Resume.
  

Selective Synchronized Stops

If you wish to perform a synchronized stop on a subgroup of nodes, untick the node(s) that you do not wish to bring to a stop and click Continue.

You'll be shown a warning box:

"Are you sure you want to perform a synchronized stop on a subgroup?

If this is preparation for a reset, THE EXCLUDED NODES MUST BE RESET AS WELL and will then require
an rsync of every repository before they are restarted."

Alert
The purpose of a synchronized stop is to ensure that all repositories are maintained in exactly the same state, that one node doesn't process transactions that are not completed on another. Any node that is not stopped in sync with the rest of the replication group is more than likely going to fall out of sync, causing replication to stop.

All nodes that are not included in the synchronized stop must be reset to purge all queued traffic and all their repositories must be manually synchronized (using rsync) with the rest of the replication group.

3.3 Resuming from a Synchronized Stop

• 1. To restart replication click the Proxy tab and click Resume.
  


• 2. The page refreshes, and you can see the Listening field now shows yes.
  

3.4 Verifying that the replicator is working

The quickest way to check replication is to add a new user through the Admin Console of one of your nodes. Then, jump to another node and confirm that the new user also appears there.

Another way to check replication is to verify there are commit transactions posted to the log file
svn-replicator/logs/SVNProxyServer-prefs0.log. More about log files

3.5 Changing the Distinguished Node

3.5.1 What distinguishes the Distinguished node?

The distinguished node has greater voting power within the quorum. In a singleton quorum the distinguished node decides transaction ordering and keeps the other nodes in sync.

In a singleton quorum, Subversion users who operate from the distinguished node will see the best performance because their local replicator never needs to wait for agreement to be reached on a distant node.

In a singleton quorum it's advantageous to change the distinguished node to correspond with the site where the most repository changes are taking place, i.e. operating within business hours. You can manually change the distinguished node at any time. You can also automate the process by setting up a Rotation Schedule.

The distinguished node is selected from the replication group, it can be any node.

Alert
Changing the Distinguished Node requires a Unanimous Quorum where agreement must be reached between all nodes.
See the Troubleshooting guide if one or more of your nodes become unreachable.

In the majority quorum the distinguished node acts as a tie breaker when there are an even number of nodes. For example, with a 4 node setup, in order to achieve a majority you would technically need 3 nodes. With the distinguished node's slightly heavier weighted vote, you can achieve a majority with only 2 nodes as long as one of the two is the distinguished node.

3.5.2 Manually changing the distinguished node

• 1. Click on the Proxy tab and then click Change Distinguished Node on the menu.
  
  



• 2. Select the new distinguished node from the dropdown, then click Assign Selected Node.
  
  
  
• 3. The screen will refresh, check that your selection now appears as the current distinguished node.
  

3.5.3 Scheduled rotation of the distinguished node

• 1. Click on the Proxy tab and then click Change Distinguished Node on the menu.
  


• 2. Click on the Distinguished Node Rotation Schedule link.
  


• 3. Select the Controller Node. This node will transmit the rotation command, according to it's local time.
  
  For each node in the replication group you can assign a time at which it becomes the distinguished node. Assign a time to each available Node that you wish to be the distinguished. You can add or remove nodes using the 'plus' or 'cross' buttons. When you have set up a schedule click Update Rotation Schedule.
  
  You can remove a schedule by clicking Remove Rotation Schedule.

3.6 Using Subversion hooks for sending e-mails

Many administrators like to set up Subversion backend hooks that fire whenever a Subversion user commits a set of file changes. With a master/slave Subversion server setup, e-mails can be initiated once when the post-commit trigger fires.

You can set up Subversion hooks to send emails when Subversion users commit file changes. However, with MultiSite, these emails will get duplicated, with a potential for spamming with notification emails.

To avoid this, set up one node as an "e-mail hub" by enabling the post-commit trigger to fire from a single node within the replication group.

Alternatively, you could use the time of day to fire the e-mail alerts from a specific node. For example, you could modify the post-commit trigger to send e-mails from India during 9:00 a.m. to 5:00 p.m. IST, and from the US during 9:00 a.m. to 5:00 p.m. PST.

Caution
If using asymmetrical e-mail hooks ensure that you don't disable the pre-commit trigger. That may cause a Subversion commit transaction to abort at some nodes but commit at other nodes causing a loss of sync.

Tip
Configure email setup to avoid long blockages or delays. The default SMTP service on node should be adequate. We recommend that you set up a local e-mail hub or a local SMTP agent/server. The local SMTP server should preferably be on the same host as the Subversion server. It should be set up to forward/relay e-mails to the organization-wide SMTP server. This ensures the e-mail hooks are a lot faster and just need to enqueue the e-mails to the local SMTP server. Test that your emails are delivered before going into production.

3.7 Changing the admin console username or password

The default login username for the Admin Console is admin, and the password is user-defined during installation. That way, all nodes initially have the same login and password.

You can set up different login credentials for each node, however, you should ensure that all administrators who manage nodes are able to login to all nodes.

Tip
We recommend that you keep admin authorization simple, and have the same login and password for all nodes. This will greatly simplify managing replication.

To change the login at a particular node, enter the following in prefs.xml.

<Security>
 <Admin>
   <user>newlogin</user>
  </Admin>
</Security>

3.7.1 Changing the admin password

• 1. Click the Security tab then click Change Admin Password on the menu (under User Administration).
  

  This changes the password for this node only. The password does not get replicated.
  Ensure that all node administrators throughout the replication group are informed about password changes.



• 2. When the new password has been entered, click Change Password. The change will be confirmed with a admin password updated message.

3.8 Setting Up Hooks

A hook is a script that gets triggered by a specified repository event, such as creation of a new revision or the modification of an unversioned property.

3.8.1 Pre-Replication Hook

To use pre-replication hooks, SVN MultiSite needs to be set with the version of Subversion you are using. Go to the Proxy tab, select SVN Settings, and select the Yes radio button for Use Pre-Replication Hooks. Then specify the Subversion Server Version.

Before Subversion executes a pre-replication hook, MultiSite invokes it before forming a proposal. As per the SVN DAV specification, if the hook succeeds, nothing is communicated back to the client. The handling of the command proceeds normally. If the hook fails, stderr is packaged as an XML response to the client. The client will then typically delete the activity; i.e., clean up any temporary files on the server side.

3.8.2 Configuration

There are two methods for invoking hooks scripts, globally (across a replication group), via the Repository settings screen in the admin console, or on a per-repository basis (See 3.8.3 Repository-Specific Hooks).

Global Hook Scripts
To invoke a hook script globally, go to the SVN Settings screen and click edit for the repository (see Admin Console Guide: SVN Settings). Within the repository settings screen entry fields for the three hook types that can be replicated.



Pre-Commit Hook: Equivalent to normal pre-commit hook but it is automatically picked up and invoked in a replication safe manner. If field is left blank and hook exists in repository hooks directory called ms_pre-commit it will be executed.

Pre-lock Hook: Equivalent to normal pre-lock hook (invoked when a user attempts to lock a resource) but it is automatically picked up and invoked in a replication safe manner. If field is left blank and hook exists in repository hooks directory called ms_pre-lock it will be executed.

Pre-Unlock Hook: Equivalent to normal pre-unlock hook (invoked when a user attempts to destroy an exclusive lock) but it is automatically picked up and invoked in a replication safe manner. If field is left blank and hook exists in repository hooks directory called ms_pre-unlock it will be executed.

Using ParentPath
when re-replication hooks are enabled within a ParentPath repository environment, they run for all child repositories, unless specific hooks are specified, for each child.

3.8.3 Repository-Specific Hooks

Three different Pre-replication hooks are supported on a per-repository basis:

• pre-commit: Invoked before or after a commit is completed.
• pre-lock: Invoked when a user attempts to lock a path.
• pre-unlock: Invoked when a user attempts to destroy an exclusive lock.

ms_pre-commit.pl 
ms_pre-lock.pl 
ms_pre-unlock.pl 

3.9 Selective Replication

By default, MultiSite replicates all SVNROOTS associated with a Subversion repository. However, you can specify a set of SVNROOTS that you do not want to replicate. Use Excludes for SVNROOTS in the prefs.xml file to identify the repositories you do not wish to have replicated. For example:

<SVNProxy>
....
<ExcludeRepositories>/exclude,/dir0</ExcludeRepositories>
....
</SVNProxy>

If MultiSite can't find the entry, it's included in the replication. There are no wildcard (regular) expressions. For Apache and SVNPath, the syntax must match what is listed in the Location directive. For Apache and SVNParentPath, the syntax must match what is listed in the Location directive and Path to the repository.

All the included roots go through the same agreement manager. For example, say you have three repositories.

For SVNPath, the Location directive is

<Location/rep1>
  SVNPath /repos/rep1
</Location/rep1>

<Location/rep2>
  SVNPath /repos/rep2
</Location/rep2>

<Location/rep3>
  SVNPath /repos/rep3
</Location/rep3>

For SVNParentPath, the Location directive is

<Location/repository>
  SVNParentPath /repos
</Location/repository>

To exclude the first repository, rep1, for SVNPath, you would exclude rep1: for SVNParentPath, you would exclude /repository/rep1.

You must name a repository after the / character. Subversion does not support a location named with just the / character.

4. Updating Apache or Subversion in WANdisco Deployment

This procedure requires that your MultiSite deployment, including Subversion servers, are offline during the time it takes to upgrade at each node. Please plan this procedure accordingly.

• 1. Ensure MultiSite supports the version you want to upgrade to. Check on the pre-deployment checklist.
• 2. Test the new version of Apache or Subversion in a test environment.
• 3. Once you are ready to deploy the new version in your production environment, perform a synchronized stop. See Performing a Synchronized Stop.
• 4. Update each of the nodes.
• 5. Resume MultiSite.
• 6. Perform some tests at each node to confirm the update has completed successfully.

5. Upgrading SVN MultiSite

Use these instructions to upgrade your SVN MultiSite to a later build or version, rather than those found in The Deployment Guide: 3. Installation section, which are intended for a first time installation.

The correct method for upgrading Subversion MultiSite depends on which versions you are moving between. Use the guide below to ensure that you upgrade with the right method.

Upgrading from SVN MultiSite 3.6 or earlier - See our Knowledge base articles on Installation and Upgrade

Upgrade rule of thumb

When looking to upgrade SVN MultiSite, apply the following rule:

Upgrading within a version, e.g. 4.0 build x --> 4.0 build y , use the Upgrade with Script

Upgrading between versions. e.g. 4.0 --> 4.2 will require you to Upgrade from backup

Select your upgrade path:

Upgrade from SVN MultiSite 4.0

Use this procedure if you are upgrading from the 4.0 version of SVN MultiSite. Note that you may need to complete an upgrade to the latest version of 4.0 before you can attempt to upgrade to version 4.2

Upgrade from SVN MultiSite 4.1

Use this procedure if you are upgrading from SVN MultiSite version 4.1.

Upgrade from SVN MultiSite 4.2

Use this procedure if you are already running version 4.2 but need to upgrade to the latest build. This procedure is shorter as you can use an upgrader script that ships with SVN MultiSite instead of completing a fresh installation, then importing your data.

6. Authz Notifier Utility

The Authz Notification (authznotifier.jar) utility passes the location of the diff file containing the changes between the existing and newly updated Authz file to the replicators, along with the output of the md5sum tool on the existing Authz file.

6.1 Usage

Authz notifier (When not using SSL encryption) runs using the following command:

java -Dauthznotifier.nossl=true -jar authznotifier.jar </path/to/admin/password-file> </path/to/diff>
<md5-value> [-patchtool </path/to/patch>] [-md5sumtool </path/to/md5sum>]

example:
java -Dauthznotifier.nossl=true -jar authznotifier.jar /sec/pass.file /home/scm/diff.out "2889b653854f68872fbb6c771348f3d9 /home/scm/svn.auth"

Authz notifier (When using SSL encryption) runs using the following command:

java -jar authznotifier.jar </path/to/admin/password-file> </path/to/diff>
<md5-value> [-patchtool </path/to/patch>] [-md5sumtool </path/to/md5sum>]

example:
java -jar authznotifier.jar /sec/pass.file /home/scm/diff.out "2889b653854f68872fbb6c771348f3d9 /home/scm/svn.auth"

Key

path to password-file

The full path to the password file.

path to diff>

The full path to the diff file. The diff file is generated by running the command:
diff <original-authz-file> <new-authz-file> > diff.out

<md5-value>

The result of running md5sum on the full path to the original authZ file. e.g. md5sum /path/to/original-authz-file

<patchtool>

[OPTIONAL] The full path to an optional tool that would apply the patch.
Defaults to /usr/bin/patch

<md5sumtool>

[OPTIONAL] The full path to a tool for generating new md5sum values, e.g. /usr/bin/md5sum Defaults to /usr/bin/md5sum

MD5 Handling
To improve consistency and usability (particularly for Windows), we calculate MD5s and patches for authz notifications internally. However unlike the previous default, this requires that the patch be in Unified format (if generated through GNU diff this is the -u option).

Any patch that we cannot parse will result in the notification being rejected and the nodes becoming read-only with the error message "Diff does not appear to be in unified format" in the logs.

If a different format is used, an appropriate patch program and options should be specified via the "-patchtool" argument noted in the key.

Password file

The password file should be in plain text and contain MultiSite's admin authentication details in the following format:

Username
Password

Or, if you use the default 'admin' username, then the password file can just contain the password.

Port

To use a different port, invoke java with -Dauthznotifier.port=xxxx
For example:

java -Dauthznotifier.port=xxxx -jar authznotifier.jar /path/to/password-file
</path/to/diff> <md5-value> [-patchtool </path/to/patch>] [-md5sumtool </path/to/md5sum>]

6.2 Disabling LDAP for Admin Authentication

A stand-alone utility is now provided that can unset the use of LDAP for admin authentication. It can be run from the /svn-replicator/utils directory using the following command:

perl ldapadminreset

This can be changed by creating a file called ldapreset.properties (situated in the "/config" folder), this file can have three values (all optional):

ldapResetKey

Key sent to the shutdown listener - default "LDAP_RESET"

ldapResetUser

Username to create/reset - default "admin"

ldapResetPassword

Password to give the above user - default "password"

6.3 LDAP Admin Authentication URL syntax

The LDAP Admin login functionality lets you specify an LDAP group or subtree that contains users who can login as administrators. The format of the URL is:

ldap(s)://host:port/basedn?attribute?scope?filter

ldap

For regular LDAP, use the string ldap. For secure LDAP, use ldaps instead.

host:port

The name/port of the LDAP server.

basedn

The DN of the branch of the directory where all searches should start from. This would typically specify a group populated with admin users

attribute

The attribute to match user names against. If no attributes are provided, the default is to use uid. It's a good idea to choose an attribute that will be unique across all entries in the subtree you will be using.

scope

The scope of the search. Can be either one, sub or obj. The default is to use a scope of sub.

-one: Entities will be searched below the DN one level only - if an entity with a matching attribute=username is found then the user is considered an admin.
-sub: Entities will be searched below the DN throughout the entire subtree - if an entity with a matching attribute=username is found then the user is considered an admin.
-obj: A specific object is being identified by the DN - if this object is located AND a matching attribute=username is found then the user is considered an admin.

filter

A valid LDAP search filter. If not provided, defaults to (objectClass=*), which will search for all objects in the tree.

Once a user is deemed to be an admin based on the criteria above, authentication is carried out against the LDAP Authority system, so it is important to ensure a relevant LDAP authority has been defined.

6.4 LDAP Timeouts

MultiSite's pre-replication authentication operates with some timeout values which may need to be modified to match your environment. From Build xxxx these timeout values are set from the LDAP Settings screen, under the Security Tab (providing you have the Access Control add-on for MultiSite enabled).

View more about LDAP Timeout settings.

In versions of SVN Multisite 4.2 released before July 2013, LDAP timeouts were set through the JAVA initialization reputils.pm file:

ldapauth.timeout

Use this property to set the timeout for LDAP (in milliseconds), the default value is 15000 (15 seconds).

ldapauth.cachettl

Use this property to set the length of time that credentials are cached if they have been successfully authenticated against the LDAP service. The default is 600,000 (600 seconds).

To override these defaults, ensure that the following statements are apended to SERVER_JVM_ARGS: in the
<WANdisco Installlation>/bin/reputils.pm file:

$SERVER_JVM_ARGS .= " -Dldapauth.timeout=<LDAP timeout> -Dldapauth.cachettl=<Cache expiry time>";

Both timeouts are in milliseconds

Example LDAP Timeout error as it appears in the log:

Mon Feb 28 15:27:24 GMT 2011 1298906844401 org.nirala.util.services.ldap.LDAPAuthenticator getLdapContext
        WARNING: [request-handler-1] LDAP response read timed out, timeout used:1ms.
        Mon Feb 28 15:27:24 GMT 2011 1298906844402 org.nirala.util.services.ldap.LDAPAuthenticator isUserAuthed
        WARNING: [request-handler-1] LDAP Authorization failed: null
        Mon Feb 28 15:27:24 GMT 2011 1298906844402 org.nirala.util.services.ldap.LDAPAuthenticator isUserAuthed
        INFO: [request-handler-1] LDAP Auth - User: test@wandisco.com Authenticated: false

7. Access Control

WANdisco's Access Control replaces Apache Subversion's own access control features to provide full authorization, authentication and audit capabilities that are essential for using Subversion within an enterprise environment. Take control of your organization's Subversion repositories through an easy-to-use web interface. No longer will you need to manage user access through the manual editing of authz files.

WANdisco's Subversion Access Control offers the following capabilities:

• Managing Subversion access across multiple SVNROOTs and modules
• Set repository authorization permissions using hierarchical groups, called Teams
• Bundle specific repository permissions for a subset of team members using subteams
• Delegate control of specific repository resources to Team Leaders using subteams
• Build access rules using Perl-style regular expressions.

Subversion Access Control is part of the SVN MultiSite proxy that sits between Subversion users and the Subversion server. Subversion users connect to Access Control using the standard port 80. Access Control relays user interactions to the Subversion server using port 81. Subversion users never have direct access to Subversion, allowing administrators to maintain exclusive control.

Fig1. How Access Control works with Subversion.

7.1 Access Control Model

Access Control uses Apache's 'mod_authz_svn' module to apply path-based authorization of Subversion repositories through an authz configuration file. Path-based authorization uses the repository's directory structure as the basis for permitting or denying specified users or groups of users read and or write access to repository files and folders. Through sets of user-created rules, precise control can be applied to what a user can access or overwrite. The limitation of path-based authorization is that once a user is granted permission to access and write to a particular path (directory), it's only possible to limit authorization to a specific file using rules based on regular expressions.

To some extent, running with path-based authorization should therefore dictate how repository files are arranged, ensuring that sensitive material is secured within subdirectories.

Specific permissions overrule inherited permissions.
Access Control tries to match the path itself, and then the parent of the path, then the parent of that, and so on. The net effect is that mentioning a specific path in the access file will always override any permissions inherited from parent directories.

7.2 Creating Users

Use this procedure to manually add new Subversion users to Access Control. To add users en bulk, see 7.3 Importing Users.

• 1. Login to the admin console and click create User on the side menu of the Security tab.
  

  



• 2. Enter the new user's details into the form.
  
  If you are logged in as a system administrator you are able to create additional users with system administrator privileges by ticking the "System Administrator" check box. This option is not available to team leaders.
  For more information, see Create Users.

  User Type:
  indicates whether the user created or imported into MultiSite (Local) or imported from an LDAP authority (LDAP).
  

  Once created a user can't change type.
  After a user has imported via LDAP or entered into the Create User screen, it's not possible to the change that user's type. If necessary, delete the user and re-enter using the alternate method.

  Local User

  Users are flagged as local if they are added through the admin console, either through the Create User form or the Import Users tool.

  LDAP

  Users are flagged as LDAP if they were added through an LDAP query.

  

7.3 List Users

The Users List shows all the users that have been entered into SVN MultiSite, including users who have been entered throught he admin console form, through LDAP or through a csv import.

Users are listed with the teams they are currently members of, along with an "Actions" buttom that allows an administrator to Edit, Delete or Disable the applicable user.

User:

list of usernames. Users with Admin permissions are colored orange.

User Type:

Indicates whether a user is imported via LDAP or has been added locally (either through data entry into the admin console or through a csv file import).

Teams:

Shows any teams of which the user is a member.

Last Recorded Access:

This column shows the date that each user last interacted with repository data.

Recording is only done, and this field is only displayed if you enable Enable User Access Tracking.

Recording is done following these rules:

• User reads are only recorded on the local site.
• Access logs are recorded a maximum of once every 24 hours

Actions:

A drop-down list of available actions that can be applied to each user.

View User

7.4 Importing Users

The option is available to bulk import users in a comma delimited file, instead of entering them individually using the Create User screen.

Alert
If "global" case insensitivity is turn on within the System Settings screen then all users imported from a CSV file will be forced into lowercase. If the case insensitivity is disabled in SVN MultiSite then users are imported as they are in the import file.

• 1. Create a comma delimited (CSV) file containing your users, using the following format:

  <username>,<first name>,<last name>,<email address>,<password>,<UserType>[, [Parent/Team/Names/]TeamName]

• 2. Log into the admin console and click on Import users on the side menu of the Security tab.
  

  

• 3. Click on the Browse button and navigate to your import file. When selected click the Import button. A growl message window will appear, that indicates the status of the import.

Local User
Imported users will count as Local Users and will be editable from within SVN MultiSite. It isn't possible to place an imported user under LDAP control, although from build: #3296 it is possible for LDAP users to create local users and teams can support mixed users (LDAP and Local working within the same team) although with this arrangement it is not possible for a local users to be team leaders.

7.5 Creating Teams

Teams are the basic unit of organisation within Access Control. All access policies are applied to teams. To apply permissions to a single user, you will need to add the user to a team, even if the team is used only for that one user.

Teams can contain mixed user types.
It's possible for a team leader who is managed by LDAP to add local user accounts to their team. This can be useful if you need to add build bots using local service accounts to a team that is otherwise managed with an LDAP authority. It's also possible for a local user to be a team leader - prior to build #3296 local users would be removed from the team after a polling of the LDAP service.

• 1. Click on the Security tab and then on the Create Team link.
  

  

• 2. Enter the new team's details, including a name, a description then add users and resources.
  

  Adding Users

  You can build a team from users who have been created locally - enter their names or click Browse to select them.

  

  If you've added LDAP Authorities you can add users through LDAP, either as n entire authority or selectively by applying a filter to the LDAP query.

  

  Click Create Team when you're ready. Read more about the Create Team screen in the Admin Console Guide.

Team Names Must Be Globally Unique
All teams must have a unique name within Access Control. It's currently not possible to have two teams with the same name, even if they are located within different directories.

7.6 Creating Rules

Team rules are the business end of the Access Control system, they define how a user "X", a member of team "Y" can interact with repository resource "Z". A single rule can be useful in itself but the real power of Access Control comes for applying complex policies through a number of rules. The procedure below runs through the creation of the rule, and explains how it's applied.

• 1. Login to the admin console, click on the name of the team for which you wish to add a new access rule. In this example we'll create a rule for senior Red Team developers that ensures that they have read/write access to all the Red Team code. For extra security, application of the rule is restricted to the IP range from which only these developers access Subversion.
  


• 2. On the Team screen, click the Rules tab.
  


• 3.On the Rules tab click on the Create Team Rule button.
  


• 4. On the Rules tab, enter the following details:
  
  

  Rule Name*

  (Required Field) A name for the rule. In this example we'll name the rule ReadWriteAllGreenCode. With you are planning on creating a lot of rules it may also be worth adding some reference to which users for which the rule will applied.

  Users this rule applies to:

  Enter or browse for users that this team rule will be applied to. The available users will be limited to the team's members.

  Define Resources/Paths/Regular Expressions for this rule:

  Enter or browse for repository resources, or type in a regular expression that defines the parameters for resources that should be added. In this example all directories within the zelenyj directory are added, along with Read/Write permissions.
  

  Authz and regular expression based Access Control rules
  If you use Authz to manage user authorization through Apache you need to be aware that Access Control rules that use regular expressions to define resources are not supported by Authz so won't be written to the Authz file.

  File level Access Control
  Although Access Control usually sets permissions on a directory level it is possible to set rules that apply to individual files, or a set of files that match criteria that you build into a regular expression.

  IP Addresses this rule applies to:

  Enter an IP address or an address pattern defined in a regular expression to apply the rule on the basis of the connecting user's IP address. In the example we'll use a regex that limits application of the rule to a small IP range that corresponds with the assigned users. This ensures that even if unauthorized users gain one of the team member's access credentials, the access rule will not be applied unless they access from the authorized user's IP.

  
  

Once all the rule details have been added, click the Create Team Rule button.

• 5. Check that the new rule now appears on the Rules panel on the team Summary tab.
  
• 6. Team rules can conflict causing unexpected results, so Access Control lets you test that a user has the right access and authorization for a given resource. In this case, one of the user's tied to the rule is checked against the "red" directory. The user's Read/Write access as a result of the application of rule RedTeamLeaders-AllReadWrite is confirmed.
  
  

7.7 Creating Subteams

Subteams are teams that have been created under a parent team. The users and repository resources available to a subteam are limited by those that appear in the parent team (which could itself be a subteam). Subteams allow system administrators to delegate responsibility for managing user access for a segment of their repository resources, without having to hand out total control of the repository to subordinates. This is very helpful for organisations working with satellite development sites where managing day-to-day access changes can be made difficult by the distance and time-zone difference between sites.

• 1. Login to the admin console and click on the Actions drop-down button that corresponds with the team for which you want to create a subteam. From the Actions, select Create Subteam. You can also create a subteam from a team team's screen by clicking on the Subteam tab.
  


• 2. On the Team or subteam screen, click the Create Subteam button.
  


• 3. Complete the subteam entry fields, following the same process that was used for setting up parent teams (See Create Team).


When all the users and resources have been added, click Create Subteam.


• 4. The subteam has now been created. The screen will update to show you the summary tab of your new subteam. Remember, unless you create some rules for the new subteam, it doesn't affect how the team members interact with Subversion. So once a subteam has been created, you'll need to add rules. See Creating Team Rules
  
  

7.8 Testing User Access

A Subversion user environment that requires complex access policies across large numbers of repository resources can require the creation of a lot of team rules. When users belong to multiple teams and have their access controlled by many rules it's possible for a user's access to a particular resource be mistakenly blocked. The Lookup/Search tool lets users identify which rules prevail for a given resource.

• 1. Login to the admin console and click the Lookup/Search link.
  


• 2. Enter the username of the user that you want to test, and the repository resource that you want to check against for access rules. Then click Lookup.
  
  


• 3. The Lookup results will display at the bottom of the screen. They will show whether your selected user has either read or write permissions for the specified resource and from which rule and which team the rule is applied.
  
  

7.8 Exporting/Importing Access Control Settings

SVN MultiSite's Export Settings tool, situated under the System tab is used to backup all of SVN MultiSite's settings (although not the Subversion repository data). This includes the users, teams and rules. See Export Settings.

7.9 Common Access Control Scenarios

8. Batch Access Control Updates

8.1 Introduction

Many operations in Subversion Access Control require that the Authz and the Subversion password files be regenerated. In a rapidly changing MultiSite environment, the number and frequency of user, team and rule changes can degrade performance (chiefly from a basic disk I/O perspective).

If you need to ensure that performance is not affected by Access Control traffic, the answer is to use the Batch Access Control Updates feature.

Important
When Access Control Batch Updates is enabled, changes to users, teams and rules are not instant.
In order to see how Access Control is currently behaving (excluding changes that will only take effect after the next scheduled         password/authz rewrite) you need to use the Current View.

Apply changes in batches

When Batch Access Control Updates is enabled, all Access Control changes that would result in a regeneration of the password or Authz files (see the list below) are batched up to be replicated to the other MultiSite nodes at a user-defined frequency (default is 60 minutes).

Access Control Changes that can be batched

• Create/delete users
• Create/modify/delete teams
• Create or modify rules (apply a change to access permissions for either a individual users or teams)

Not affected

• Creation/modification of LDAP Authorities

WARNING: Changes to LDAP Authorities do not trigger password / authz file rewrites, however when an authority is run, any resulting user or team changes will.

8.2 Batch Mode v.s. Enforce Mode

When Batched Updates are enabled the admin console can be viewed in two separate modes:

Pending:

When Batched Updates is enabled, the admin console will present a view of Access Control that includes objects (users, teams and rules) that are not yet enforced, because they're waiting to be applied in the next batch update. This is the mode you use to make any access control changes, which will immediately be added to the current waiting batch.

So the Pending Mode is actually showing you how access control will be applied once the current batch has been updated - which makes it difficult for troubleshooting and testing, where you would only want to see objects as they are actually applied to users here and now.

Current:

The Current view is available for looking at Access Control with all pending changes filtered out, this is specifically for troubleshooting and testing. Most Access Control creation or edit tools are disabled - it wouldn't make sense to create Access Control objects (users,teams or rules) in this mode as they wouldn't appear until the current batch was run.

If you need to make Access Control changes, switch back to the Pending Mode, or disable the Batched Updates feature.

8.3 How to enable Batch Access Control Updates

This procedure shows you how to enable Batch Access Control Updates and then how to manage them:

• 1. From the admin console, click on the System tab.
  
  
  
• 2. On the System screen side menu, click on the System Settings button.
  
  
  
• 3. On the System Settings screen, click on the check box for Run batch Access Control updates. You'll need to select which of your modes will initiate the batch, it's this node that will trigger the passwd and authz regeneration, according to the Access Control update frequency which you can set below.
  
  The default frequency is 60m, so that any Access Control change will take a maximum of one hour to take effect. Set a higher frequency to reduce the maximum delay for Access Control changes, with a corresponding impact on performance. Click Update to save your changes.
  
  

8.4 How to manage Batch Access Control Updates

Once Batch Access Control updates is enabled, all nodes will show the pending changes at the bottom of the side menu on the Security tab.

8.5 View Pending Changes

You can click on View Pending Changes for a list of all Access Control changes that have so far been added to the next batch. Pending User, Team and Rule changes are listed separately.

At any time you can override the countdown to the next run by clicking the Apply Now button. All listed changes will be applied immediately and the countdown to the next batch will start.

8.6 Enable Current View

Enabling "Current view" updates Access Control so that any change still waiting in the current batch (and therefore has not yet taken effect) will not be displayed. This view is intended to make troubleshooting and testing a little easier as you will only see users, teams and rules as they are currently enforced through the Access Control model.

Confirm View Change Moving to the current view will immediately blank any information that has been entered into an open admin console screen, so on clicking Enable Current View you'll get a warning and a prompt to continue. Unless you are currently entering settings on screen you can click Yes.

Viewing: [Current]
A red label appears at the top of the admin console when the "Current View" is enabled. This is to remind you that access control is currently in a viewing only mode.

In the Current View the menu loses all options that result in Access Control changes (creating/importing users, teams etc.) In order to make such changes you will need to change back to the Pending view or normal mode (by disabling the Access Control Batch updates).

Troubleshooting

1. Finding the Last Committed Transaction

Even though committed transactions are always in the same order for each node, the timing of the commits usually varies from node to node. So unless there are no Subversion users logged in, you probably are going to have variations per node for committed transactions.

Go to any node's Dashboard. Type:
http://<IP address>:<WANdisco port number>/dashboard2

You see all the nodes on the Dashboard to compare the listed transactions. Click on a Tx Id (Transaction ID) to view more details about the WebDAV command, along with its transaction number.

2. Repair Repository

Repair Repository is a tool for making a repair of corrupted repositories. Previously, repairing a repository would require that all nodes in the replication group be brought to a stop, making all repositories read-only until the repair was completed.

2.1 Repository Repair Overview

This overview will help you understand the sequence of events that are involved in a typical repository repair, for a step-by-step explanation of how to repair a repository see the Repository Repair Procedure.

• 1. In this overview we have a 3-Node replication group, including NodeSeattle, NodeSeoul and NodeStPetersburg. Each nodes hosts 3 Subversion repositories, svn1, svn2 and svn3.
  
  
• 2. On NodeSeoul, repository svn2 experiences a problem and becomes read-only. Users of the svn2 Repository on other nodes in the replication group can continue to work, but their changes can't be replicated to NodeSeoul.
  


• 3. The administrator works to repair the fallen svn2 repository by using the Repair Repository process. They select NodeStPetersburg as the helper node and begin the repair.
  
  Both NodeSeoul and NodeStPetersburg are suspended from the replication group. The users of these nodes will be unable to write to any of the repositories.
  


• 4. From the helper node, the administrator can make a copy of the repository that needs repairing or use rsync between the two nodes.
  
  In this illustration, a copy of svn2 is made. An alternative to making a copy of svn2 would be to use the rsync program to transfer missing or corrupted revisions from the helper node, NodeStPetersburg to NodeSeoul.
  


• 5. With a copy of the repository made, the administration will be able to reactivate the Helper node, allowing its repositories to catch up with changes made on NodeSeattle. The good copy of svn2 can then be transferred to NodeSeoul where it is used to overwrite the read-only version of svn2.
  


• 6. The Repository Repair procedure is now complete. NodeSeoul now has an up-to-date copy of svn2 in place, and is released from the repair process. All repositories on all nodes are once again replicating.
  

2.2 Repository Repair Procedure

This is a step-by-step guide to doing a repository repair. Before running through this procedure, it may be helpful to read through the overview which explains how the nodes and repositories change their state through the Repository Repair Overview.

WARNING: After starting a repository repair you Must Not navigate away from the repair browser session or open up a new session on the node. To ensure that the repair doesn't fail or stall, don't perform any other action on the node until the repair has been completed.

• 1. With Email settings applied to the replicator, you will receive an alert that a repository has been placed in read-only mode. Click on the dashboard link to open the admin console for the node with the problem.
  



• 2. On the dashboard, click the link "One or more repositories are in read-only mode".
  



• 3. The Repository status screen will list all the repositories on the node. If you are have trouble finding the read-only repository, click the "Show Read-Only" link. Make a note of the name of which repository is read-only. In the example "svn2" is read only after a failed retry.
  
  Click on the Repair link that corresponds with the read-only repository.
  


• 4. The repository repair procedure needs the use of one of your other nodes, which we call the "Helper Node".

  Alert
  The Helper Node must have an up-to-date copy of the repository under repair, and must also not be currently involved in another repair.

  From the drop-down boxes, select the node that you wish to use as the helper, and the node that hosts the repository to be repaired.
  

  Tip
  If you start a repository repair from the Repository Status page, you have already identified which repository is being repaired. If you start the repair from the Repair Repository link on the side menu, you will also need to select the repository to be repaired.

  To move to the next step of the repair procedure, click Continue. Before doing so, make sure that you read through the key information:
  

  Alert
  Things you must know before you continue:
  
  * Inform Subversion users working on either the helper or affected node that they won't be able to commit to any repositories hosted on their node during the repair.
  
  * Repair only one repository at any given time and ensure that it is completed before starting another.
  
  * Whilst using the repair tool, don't navigate away, start another repair or use other admin console features until the repair is complete. We recommend that you arrange exclusive access to the admin console on all nodes during the repair.



• 5. You are presented with a summary of your selection. Check your selection, if the selected node and repository are correct, click Suspend {Helper Node} and {Node to be repaired}.

After clicking the button both the helper node and the node being repaired will go into a suspended mode until the repair is completed.
Nodes put in a suspended mode will:


• still be accessible via the admin console.
• be places in a read-only mode until reactivated.
• not flagged as read-only when viewed from the other nodes.
• not be able to interact with any other nodes in the replication group.

Effect on the remaining (active) nodes in the replication group:

While a repair is underway it will not be possible to perform any global actions on the replication group. That's any actions that depend on interaction with other nodes in the replication group, such as consistency checks, emergency quorum reconfigurations, scheduled distinguished node rotation,etc).


• 6. A progress bar will appear. When it reaches 100%, all in-flight proposals will have completed and both nodes will be suspended. No further traffic will be accepted until the repair procedure is completed or cancelled.
  
  
• 7. Next, you need to choose which method you will use to complete the repair. The two options are:
  
  2.3 Manual overwrite of the corrupted repository
  or
  Manual rsync corrupted repository

2.3 Manual overwrite of the corrupted repository

With this option, you will take an up-to-date copy of the repository from the helper node, once you have the copy, click "I have copied the repository". The helper node will be returned to duty, and will catch up with the remaining active nodes.

We strongly recommend that you organise to have exclusive access to the admin consoles of your nodes during a repair, to ensure that no actions are triggered that might cause the repair to fail.

This section illustrates a possible method for creating a copy of the repository.

• 1. Leave the admin console open, you'll return to it later. Open a terminal window, create a backup of the repository to be repaired.
  
  You might create a backup directory, then create a tar archive:

  root@Fed11-2 Subversion]# ls -l
  total 12
  drwxr-xr-x 7 apache apache 4096 2010-03-24 11:13 Bozo
  drwxr-xr-x 7 apache apache 4096 2010-03-24 11:11 Harpo
  drwxr-xr-x 7 apache apache 4096 2009-11-05 17:26 Repo
  [root@Fed11-2 Subversion]# mkdir backup
  [root@Fed11-2 Subversion]# tar cvf RepoBozoBackup.tar Bozo
  

  Then move the tar file into the backup directory:

  backup  Bozo  Harpo  Repo  RepoBozoBackup.tar
  [root@Fed11-2 Subversion]# mv RepoBozoBackup.tar backup
  

  
  
• 2. Once you have the repository copied you can return to the admin console and click the "I have copied the repository" button.



• 3. The helper node is no longer suspended and will automatically catch up on any repository changes that occurred while it was suspended. You are redirected to the node that is hosting the corrupted/out-of-date repository, which remains suspended.
  While leaving the admin console screen open, transfer the repository copy that you made in step 1 from the helper node to the node currently under repair (which you are currently viewing in the admin console).



• 4. Overwrite the corrupted repository with the copy that you have just transferred onto the node. If you prefer, and have enough disk space, just move the corrupted repository out of the way and copy the replacement in its place.


• 5. Once the repository has been overwritten, click on Repository Repaired.
  


• 6. With the repository repaired, the node is returned to duty and will catch up on any repository changes that have occurred since it was suspended. Click Complete Repair.



• 7. The repair is now complete. We recommend that you do a consistency check of the repaired repository to confirm that it's up-to-date. You can check the status of the repository by clicking on the "View Repository Status" button.
  

2.4 Manual rsync corrupted repository

With this repair method you'll use the rsync command to synchronize the out-of-date repository with the good copy on the helper node.

• 1. Click on the "I will rsync the repository button.
  


• 2. At this point both nodes are suspended, allowing you to complete an rsync of the out-of-date repository.
  There's a guide to synchronizing using the rsync command in the Troubleshooting section 2.7 Using rsync to repair a repository.


• 3. When the rysnc has been completed, it's a good idea to do a verification of the repository, you can use:

  svnadmin verify <REPOSITORY NAME> -r <REVISION NUMBER>

  


• 4. With the rsync completed, the repository is now repaired. Return to the admin console session and click the "Repository Repaired" link.
  


• 5. The node hosting the repository that has been repaired has now been returned to duty. However, the helper node remains suspended. To return the helper node to duty, click "Activate Helper Node".
  


• 6. The repository repair is now complete. We recommend that you do a consistency check of the repaired repository to confirm that it's up-to-date. You can check the status of the repository by clicking on the "View Repository Status" button.
  

2.5 Disabling Access to Subversions

Alert
This makes Subversion read-only. You may need to warn you Subversion users.

Temporarily Disabling Subversion Access At Selected Nodes

You can stop transactions at one or more nodes. For a discussion of stopping one, but not all, nodes, see WANdisco is Listening

Alert
To stop replication you need to run a synchronized stop.

• 1. Open the admin console.
• 2. Select the Proxy tab, click Stop Proxy.
• 3. Select Stop this proxy only.
• 4. Repeat steps 2 and 3 for each node you want to stop.
• 5. To start a node, click Start Proxy. The replication group catches up with the re-started node on any transactions that occurred between the nodes that continued to synchronize.

6. Re-start each node that you stopped.

2.5.1 Disabling Subversion access (all nodes)

To stop all nodes at once, run a synchronized stop.

2.6 Emergency Reconfiguration of Quorum

Use this procedure as a last resort, if the distinguished node (in a singleton quorum) has become permanently unavailable. The procedure basically creates a new replication group that excludes the unreachable node.

If a quorum can't be formed, an alert will appear at the top of the admin console.
Follow the "Click here for more details" link.

Use the emergency reconfiguration procedure only when the recovery of the distinguished node will take an unacceptable amount of time.

Using a Singleton Quorum - If the distinguished node is periodically reachable, instead, try using the Change Distinguished Node while its available.

Using a Majority Quorum - If the quorum is periodically reachable, instead, wait for the quorum to become available and recreate the replication group, using a singleton quorum, instead of using the emergency procedure. You can change back to the majority quorum after stabilising your nodes.

Caution
We don't recommend using this procedure because of the potential for data loss. If the distinguished node is unreachable by other nodes it could still be processing transactions on its own (with singleton quorum, the distinguished node can operate alone). Therefore, if the other nodes form a new quorum that excludes the distinguished node, any transactions made at the distinguished node since it became unreachable to the other nodes would be lost.

• 1. Ensure Subversion users at the nodes which will not be in the new replication group, after the emergency quorum reconfiguration are notified that they won't be able to make changes to any repositories.


• 2. It's essential that the unreachable distinguished node is properly shutdown and has had a reset performed. The emergency quorum reconfiguration can't be completed (specifically a new Replication Group can't be created) if the distinguished node is brought back up during the procedure.


• 3. Be prepared to manually start any Nodes, you may be directed to do so during the procedure.
  


• 4. Read the message. If you decide to continue, click "Emergency Configuration".
  
• 5. Read another warning message about the possible data loss that can occur. It may be better to make another attempt to recover the distinguished node, otherwise click "I Understand, continue".
  
• 6. Tick the checkboxes next to the nodes to be added to the new replication group.
  
• 7. Name the new replication group, and click "Continue to Define Quorum".
  
• 8. When all the nodes are reset, click "Continue".
  
• 9. Verify that the systemdb has been deleted at the other nodes (other than the one you're on now), and click "Restart and Join New Replication Group".
  


• 10. After a few moments, you see that all nodes confirm that they have joined the new replication group. The repositories hosted on these nodes will be in synch. Click "Done! Return to Proxy Status".
  
• 11. The Emergency Reconfiguration is now complete. Nodes that have been removed from the replication group can be restored and reintroduced using the same procedure that is used for adding brand new nodes - 1.1 Adding a Node to the Replication Group You may wish to clean up the retired node and replication group.
  
  There's currently a bug that prevents the deletion of the first (installation) node from the previous replication group. There's a workaround available in our Knowledgebase - Deleting the First Created Node in SVN MultiSite 3.7

2.7 Using rsync to repair a repository using an out-of-date copy

This process can fix a corrupted or out-of-sync repository by restoring it to a replica of the repository that is known to be in a good state.

From the machine with the master repository, type the following commands:

rsync -rvlHtogpc /path/to/local/repo <Repository Name> remoteHost:/path/to/remote/repo

For example:
rsync -rvlHtogpc /Subversion/Repo/db/ root@172.7.2.33:/Subversion

Then follow up with an additional rsync that will ensure that contents of the locks directory are identical (by deleting locks that are not present on the originating server)

rsync -rvlHtogpc --delete /path/to/local/repo/db/locks <Repository Name> remoteHost:/path/to/remote/repo

For example:
rsync -rvlHtogpc --delete /Subversion/Repo/db/locks root@172.7.2.33:/Subversion

Knowledgebase
You can read a more detailed step-by-step guide to using rsync in the Knowledge Base article Reset and rsync Subversion repositories.

2.8 Offline mode

The offline mode makes it possible to stop the java heap from being swamped by marking the failed transaction as eligible for garbage collection. As a result, all nodes can invoke garbage collection and clear the queued transactions. The node with the read-only repository no longer needs to keep the buckets as active, which will free up heap. The trade-off is that it's no longer possible for the repository to automatically recover, instead a repository repair will be required to get it back into sync.

How read-only repositories can lead to performance problems
If a transaction fails on one node, and is not bypassed, the repository is placed in a read-only mode to stop the failure from halting replication between the replicas on the other nodes. The subsequent transactions between the other replicas will stack up until the read-only replica is again writable. Garbage collection is held up and on the node with the read-only repository, all buckets subsequent to the failed transaction remain active and continue to eat into the Java heap. In a heavy traffic environment this can lead to Java heap overflows unless the repository is quickly recovered or taken offline.

Knowledgebase
For a detailed explanation of how SVN MultiSite's Garbage Collection works, see the article Garbage Collection.

Automatic offline mode

A read-only repository will switch offline should the number of transactions exceed a threshold. This threshold is controlled from the Garbage Collection screen, found on the admin console's System tab.

For an explanation of the Garbage Collection page, see the Admin Console Guide - 4.5 Garbage Collection.

The threshold is called the Auto Offline High Water Mark and by default it is set to 100,000 transactions. You can increase the number if there's sufficient Java heap space to handle large amounts of queued transactions. If JAVA resources are tight, you can reduce the threshold so that a read-only repository is taken offline sooner.

Here is an example of the Automatic offline mode in action.

Manual offline mode

A repository that is read-only can be placed offline through the admin console. The option to take a repository offline will coincide with the option to attempt a bypass of a failed transaction.

From the Proxy tab, click on Repository Status which is the second link on the status menu.

3. Talkback script

When you do contact WANdisco with a problem, the first thing WANdisco support asks for is the talkback file. Run the file by typing at svn-replicator/bin
perl talkback

Type in the pathname to SVNROOT when prompted. The output looks like this:

Please open a ticket by visiting http://support.wandisco.com and upload the /talkback-<machine name>.zip, with a description of the issue.

Do not email the talkback files, only attach them via the web case interface.
The zip file is located at the root directory. Do not email the .zip file, just attach them to your case. Read our Knowledgebase article about How to raise a support case.

4. Connection Request Timeout Messages

Sometimes in the WANdisco logs, you see connection request timeout information messages logged. These are informational messages and should be ignored unless it is guaranteed that the connection can be established in xxx milli-seconds and happens often.

In normal operation of MultiSite, two connections are established between each of the replicated machines, MultiSite connection and a DFTP connection. These two connections were established when MultiSite started and are used when required. A keep-alive signal is periodically sent on the WANdisco port. There is no traffic on DFTP until a file transfer begins.

Some lesser routers in the path of the two end points will close an established connection if there is no traffic on the connection without notifying the end points. When end points sent data on this stale connection, they hang forever. To deal with these lesser routers, MultiSite does not keep the DFTP open in its connection pool forever. MultiSite establishes a DFTP connection from receiver to sender when a file transfer was required. This solved the problem dealing with lesser routers.

Some companies have a corporate policy that network connections can only be established in one direction. To deal with this scenario, the replicated machines establish a DFTP connection to other nodes periodically and tear them down if there is no traffic within a known interval. Once a connection is established, any side is free to use the connection regardless of which side initiated the connection. A connection in use is never torn down until it is available as a free connection. This is the current implementation.

It takes between 300 to 400 milli-seconds to establish a network connection even on a slow WAN. By default, MultiSite waits for 500 milli-seconds before giving up that a connection cannot be established to a peer machine and prints this informational message. What if the establishments of connection always take 501 milli-seconds. In this case, a connection is never established. To solve this problem, the timeout value is adjusted in 10% increments of the last timeout, starting at 500 milli-seconds, to a maximum of 10 seconds for each timed out connection.

Upon establishment of a successful connection, this timeout value is used for subsequent connection establishment unless an adjustment is required for failed attempts.

5. VPN, NAT, Firewall Timeouts

This section is useful if you are experiencing issues with slow commits on the non-distinguished node or if you have port-forwarding in your environment.

In a multi-site configuration, most nodes are connected through a WAN. Often, VPN and NAT devices are used to do IP translation and port forwarding. These devices need to maintain state in order to do the port forwarding on-the-fly. This state can grow if not cleaned out. Many devices simply reset the internal state after an inactivity timeout. For example, some Cisco NAT routers reset state after 7200 seconds or 2 hours.

The MultiSite replicator uses persistent TCP connections between the replicators. If these TCP connection are going through a NAT or port forwarding device, it's important to tune the VPN and/or the TCP stack at the replicator host machine. Many NAT devices have buggy implementation that resets the internal state without resetting the TCP connections.

In such a situation, the replicators may see a connection as established but no communication actually happens. The symptoms include a slow commit that is blocking WAN communication. You can run netstat -a | grep <DConEPort> to see if the TCP send queues are backing up. That, in conjunction with slow commits that appear to be hanging, or frozen, typically indicates the NAT is not gracefully resetting TCP connections.
You can further confirm this by using tcpdump or ethereal to check for excessive retransmissions on the DConENet connections. You could also look at your VPN/NAT device log to see if it reset any DConENet connections that appear to be in an ESTABLISHED state via the netstat -a command.

These are a few ways of addressing the issue :

• Specify a connection keepalive timeout in the prefs.xml file as:

  <DConENet>
   <ConnectionKeepAliveTime>1800000</ConnectionKeepAliveTime>
  </DConENet>
  

• This causes inactive connections to be closed and refreshed periodically (after 1800K ms or 30 minutes).
• Increase the keep-alive timeout on the NAT/Port forwarding device. If possible, have the DConENet connections never expire. Some devices let you set port specific QoS.
• On the replicator host, tune the TCP stack to have a smaller fuse on the TCP keep alive timer. For example, on Linux, you can specify a value like 1800 (seconds) in
  /proc/sys/net/ipv4/tcp_keepalive_time to reduce the interval from default 2 hours to 30 minutes.
• You can restart the Subversion replicator to kick out seemingly established connections but broken by NAT internal resets.

6. A node is in read-only mode

If any nodes are in read-only mode,

• 1. Go to the Dashboard. You see the Problem Transactions listed for each node.
• 2. Click on the Problem Transactions link. The System tab displays a message, noting the bypassable transactions and the non-bypassable transactions.
• 3. If there are no bypassable transactions, and you do not know why, see How Do I Get WANdisco Support?
• If there are bypassable transactions, there may be more than one. Click on each transaction.
• 4. Read the returned message that will confirm the bypass status of the transactions.
  
• 5. Verify that the Dashboard does not report any nodes in read-only mode.

7. Updating Apache or Subversion

This procedure requires that your MultiSite deployment, including Subversion servers, are offline during the time it takes to upgrade at each node. Please plan this procedure accordingly.

• 1. Ensure MultiSite supports the version you want to upgrade to. Check on the deployment checklist.
• 2. Test the new version of Apache or Subversion in a test environment.
• 3. Once you are ready to deploy the new version in your production environment, perform a synchronized stop. See Performing a Synchronized Stop.
• 4. Update each of the nodes.
• 5. Resume MultiSite.
• 6. Perform some tests at each node to confirm the update has completed successfully.

8. Missing License Key file

SVN MultiSite depends on a license key file being present in the svn-replicator/config directory for each node. Please get a valid license from WANdisco and copy the file to the config directory. MultiSite won't start without the license file.

8.1 I'm getting a severe exception

I'm getting a SEVERE exception, and replicator is aborting the Subversion transaction and shutting down.

If you get a message in the logs/SVNProxy*.log file similar to
svn: Commit failed (details follow):
svn: File not found: transaction '10-d', path '/development/Hello.txt'

it means the replicator has detected an out of sync condition. Remember, the replicator continuously monitors your repository for any out of sync issues. If it detects this has occurred, it triggers an automatic shutdown to prevent further corruption.

This could happen if someone accidentally committed directly to Subversion, bypassing the replicator, and ramped up the version in one node without giving the replicator any chance of replicating. This can be easily resolved by following the reset procedure outlined in I Directly Committed to Subversion, How Do I Rsync?.

Follow all precaution to avoid bypassing the replicator:

• 1. Ensure only svnreplicator host/IP address is allowed to connect to the Subversion server.
• 2. Protect direct logins in Subversion replicator or Subversion server box from end user.

Admin console guide

SVN MultiSite has a browser-based admin console for the management of replication groups, setting changes or viewing system logs.

Connect to the admin console through a web browser, using your server's hostname or IP address, along with the administration port (defaults to 6444).
e.g.

http://10.2.5.101:6444   (http://(Server IP):(Admin port)

1. Admin console key

The admin console screen consists of the following elements:
Top menu bar: includes the product license and any expiry date, along with links to documentation and support.

Section tabs: The admin consoles different sections are separated on tabs. They consist of the following sections:

Alert
The layout and features available in admin console alter depending on your license agreement. So you may not see all the elements described in this guide.

• Security appears if you have a license for the Access Control feature that may be bundled with SVN MultiSite. If not, the tab is called "Users" and provides functions relating adding and managing Subversion users.


• System: Various housekeeping features are handled within the System Tab. This includes a log viewer and settings exporter.


• Proxy: The proxy tab is the business end of the admin console. It provides access to the Subversion settings and controls to the MultiSite proxy.

Jump to node...: This drop-down link will open the admin console for one of the other nodes. You can identify which node you're viewing by checking the webpage's title.

Section menu: Appears on the left-hand side and contains links to features that are specific to the tab.

Product version line appears at the bottom of the screen and shows the product version and build, the version of Java installed and the operating system.

2. Login

On the authentication screen there are a number of links available, even if you don't login.

Quick Links

Proxy Status - Proxy Status Displays the node's status in the tab's main panel.
Transaction Status - Lets you check on the status of individual transactions.
Dashboard - Lists DAV traffic
Change User Password - Follow this link to change the password of a User, it's not used for changing the Admin password.

3. Security

The security tab handles all user related functionality.

Tip
The Security tab features product capabilities that are available when the Access Control product has been bundled with SVN MultiSite. The documentation for using Access Control can be found in the User Guide Section.

3.1 Create Team

In Access Control a Team is a group of users who are associated with a resource or collection of resources. Teams underpin how user authorization is managed in Access Control.

Name*

A name by which Access Control will refer to the team. A Team name can consist of letters numbers and spaces.

Team Names Must Be Globally Unique
All teams must have a unique name within Access Control. It's currently not possible to have two teams with the same name, even if they are located within different directories.

Description

A description of the team. This can help distinguish the team, possibly referring to the team's purpose or location.

Add Users

Select local users either through Manually Assign Users or click Assign users via LDAP to select an LDAP authority.

Manually Assign Users - Enter usernames into the "Add Users" entry field, will make the user appear. Entering a letter into the field will result in all users whose usernames begin with that character being listed under the entry field.

If you prefer, instead of entering the username of each user you wish to add to the team, you can click the Browse button and select from all available users by ticking check boxes. Tick those users that you want in the team, or if you're adding a lot of users it maybe be faster to click Select All then untick users that you don't want to add. When you've finished selecting users, click the Save button.

Add Resources

Resources are repository folders. Browse the repository and select a folder, or type in the directory path and click the Add button.

Adding a resource to a team doesn't automatically provide the team's members with authorization to access the resource, first Team Rules must be created that will set the specific permissions for selected team members.

3.2 List Team

List Teams is the default screen of the Security tab. All teams and subteams are listed, along with the number of users in the team, the number of subteams and the number of team rules that apply to the team /subteam.

3.2.1 View Team

The details of a team can be viewed by clicking on the team's name on the List Teams screen. This will show the team's summary tab which shows the teams details. The summary screen doesn't allow any changes to the team to be made directly, instead, click on the tab that correspond to the property that you wish to change.

Team Leaders

Lists users who have been assigned as Team Leaders to the team. Team Leaders are able to manage the team's settings, adding or removing users, team leaders or repository resources.

Add Team Leaders -
Open a dialog box for adding additional team leaders.

Default Team Leader Permissions.
When you make a user into a team leader, you enable them to change the team's setup. However, they need to be further assigned specific permissions to Create Users, Create Subteams and Disable users. See the next section for how to enable Team Leader User permissions.

Local Users not eligable for team leadership when the team is managed via LDAP.
It is not possible to promote a local type user to a team leader position if the group contains LDAP users, i.e. is managed by an LDAP authority.

Manage Team Leaders
Open a dialog box for changing team leaders' permissions, or removing team leader status.


From the Manage Team Leaders dialog you can remove a user's team leader permissions or enable/disable the following user permissions:
Create Subteam - Enable the team leader to further delegate a team by creating a subteam, assigning users, team leaders and repository resources from those available in the parent team.

Create Users - Enable the team leader to add new user accounts.

Disable Users - Allow the team leader to disable Users. A disabled user will appear within the team's user list, but will be prefixed with a (D). A disabled user will have all SVN permissions removed. It's not possible to selectively disable a user's access, a disabled users will be disabled in all teams.

Disabled users are not counted towards the user limit of your license.
Close to the user limit of your product license? (User limit is noted at the top of the admin console.) You can make room by disabling inactive users. It's possible to "hot bunk" users in this way, although you'll need to upgrade your license to increase the number of concurrent users.

Team Members

Team Members lists all users who are currently assigned to the team.

Create user - Create a new new users that will automatically be added to the team. See Create User for information.

Add Team Members - Add an available user to the team. Only users who are present in the parent team can be added.

Manage Team Members - The Manage Team Members dialog contains tabs to Enable Users, Disable Users and Remove Users.



Manage LDAP Settings - This option is available if you have at least one LDAP server added.

Resources

The resources tab lists all repository directories that are associated with the team. Team members do not have access to these resources by default; a team leaders or system administrators must first create rules that specify which team members can access the resource, along with a file permission (write access or read-only).

The Resources tab has two buttons:
Add Team Resources -
Opens a browse window to the repository file system from where a directory can be selected.

Delete Team Resources -
Directories can be removed by clicking on the blue crosses, then

Rules

The Rules tab shows all existing rules associated with the team. You can view a rule's settings by clicking on the team's name. You can create a new rule by clicking onCreate Team Rule or delete a rule by clicking on Delete Team Rules, then selecting the rules that you wish to delete.

Subteams

The Subteams tab lists all subteams that have been created under the current team. The list of subteams will work as a 'tree view', allowing you to toggle the visibility of subteams by clicking on the parent team.

It's possible to subdivide teams into smaller 'Subteams', which allows you to delegate areas of team responsibilities to deputies. A subteam can only consist of users from the parent team, plus whatever resources are available in the parent team. Click Create Subteam if you would like to delegate some of your users

Create Team Rule

Rules are used to control how team members (either all of them, or a subset) are able to interact with the team's allocated repository resources.

Denied by default
In the absence of any team rules, all team members are denied access to all repository resources. This might seem like 'bunker mentality' but it's the only sensible strategy for ensuring that a repository's data is secured against unauthorized access.

Team Setting Tabs

The tabbed side panel lists the team's members, the subteams and the associated Team Rules. It's possible to edit these settings by clicking on the Edit button at the bottom of the main panel.

Edit

Click the Edit button to make changes to the Team. Clicking this button has the same effect as clicking on the Edit Action on the Team List screen.

3.3 Create Users

Use the Create User screen to add new Subversion users. Users that you create here can be assigned to teams or subteams, from which they will inherit the teams/subteams repository access permissions.

User Type

(Not editable) Automatically flagged as "Local".

User Type
Two possible values for User Type: Local for a user entered into the system through the admin console or "LDAP" for users entered in through an LDAP Authority. Users entered into the system through a CSV file import can be assigned either value "Local" or "LDAP".

Username

Choose a username for the new user - this be used for their Subversion access.

Password

Choose a password for the new user - they can change this when they first log in.

Password

Enter the password again to confirm that you've typed it correctly.

First Name

Last Name.

Email

The user's email address - this will be used to send any relevant system messages to them.

System Administrator

Tick the System Administrator check box to assign administrator permissions to the new user. As a system administrator the new user will have full access to the product, and will be able to add new users themselves, and also add/modify Subversion repositories.

Admin authority
It's possible that a user who is a member of an admin LDAP group could have his or her admin rights removed from within SVN MultiSite's Access Control tools, these changes would then be overwritten at the next poll of the LDAP service. The solution is to create the LDAP group, then assign System Administrator privilages from the Access Control screens, after the users have been imported. Setting the admin privilages above the LDAP AUTH level ensures there's no conflict of control over user permissions.

Assign User to Teams

Enter a team name, or browse for a team using the Browse button if you would like to immediately add the the new user to an existing team. You can remove the user from any team by clicking on the blue cross that appears after any selected team names.

3.4 Import Users

It's possible to import users instead of entering them manually through the Create User screen. Use a comma delimited (CSV) format for the import, with the following fields:

<username>,<first name>,<last name>,<email address>,<password>,<UserType>, [Parent/Team/Names/]TeamName]

Click on Choose File and navigate to your import file. When selected click the Import button. A message window will appear, indicating whether the import was successful.

Alert
If "global" case insensitivity is turn on within the System Settings screen then all users imported from a CSV file will be forced into lowercase. If the case insensitivity is disabled in SVN MultiSite then users are imported as they are in the import file.

Caution importing users to subteams
You can import users straight into teams by including an optional team name field (see the format, below). This will allow you to import to either top level teams or subteams if the subteam has a unique name.

Unique team names are only enforced at the top level and within each parent team. It's therefore possible to have two teams called "QA", so long as they are subteams for different parent teams. Importing a user to "QA" team when there is more than one "QA" team will result in the user being placed unpredictably into one of the teams.

3.5 Rules Lookup

The Rules Lookup tool lets you test a user's access permissions for a specified repository resource. For users who are members of multiple teams and to whom numerous team rules are applied, the Lookup/Search greatly simplifies the verification of their access permissions.

Username

Enter a username on which to perform a permissions review.

Resource

Enter a repository resource / path for which you want to test the above user's access

Click Lookup to initiate the test. The results are presented as a Read Results and a Write Results. Each result will indicate whether access is granted or denied, and from which rule the permission was granted, and from which team the rule was applied. This is enough information to help you fine-tune permissions without a lot of blind trial-and-error changes.

3.6 Search

The Search tool offers system administrators and team leaders a tool for search through Access Control data for any users, teams, rules and permissions.

Username

Enter a username on which to perform a search.

Team

Enter a Team name or fragment on which to perform a search.

Rule

Enter a rule name or fragment on which to perform a search.

Resource

Enter a resource name on which to search.

Filter checkboxes

Select from the available filters to help limit your search: Search Users, Search Teams, Search Rules and Search permissions

3.7 Create LDAP Authority

It's possible to set up a number of LDAP authorities, if your organisation is runs from more than one LDAP server. Use the Create LDAP Authority screen to define your authorities as well as the order in which they are queried.

Order: The priority given to the LDAP URL. e.g. A Location Order of 1 means that Access Control will look within that defined authority for a user first. If unsuccessful then Access Control will then check the authority with the next order number (e.g. 2).

Regular Expression: Enter a regular expression to set a pattern for matching against available LDAP authorities. The expression, is applied locally to limit unnecessary queries against available LDAP authorities. For instance, setting an expression that specifies a top-level domain can ensure that authentication of a user (whose username is their email address) doesn't need to run through all LDAP authorities, but instead can be checked only against those servers that match the the regular expression - in this case, the same top-level domain.

URL: the URL for the LDAP authority, using the format ldap(s)://hostport/basedn?attribute

Bind User DN: (Optional)

Bind Password: (Optional)

First Name Attribute: (Optional)

Last Name Attribute: (Optional)

Email Address Attribute: (Optional)

Case Insensitive Usernames? Tick this checkbox if you need MultiSite to stop distinguishing between usernames that differ only by their case pattern.

For example, with the "Case Insensitive Usernames?" unticked (it is by default), MultiSite will treat username "MelissaJones" and username "melissajones" as two distinct users. Support for username case insensitivity is a feature that may be required when Active Directory

System Administrator Group?: Tick this checkbox if the authority will return users who are System Administrators (able to access the MultiSite admin console).

Support for Active Directory Nested Groups

Active directory allows the definition of groups that are themselves members of a parent group. This provides the administrator with a useful organizational structure.

We have a query for that

It is possible to handle Active Directory nested groups using the following query (based on the LDAP_MATCHING_RULE_IN_CHAIN OID). it works as follows:

When we run an ldap query to find all the members of a group, we treat all those members as users.
If a group is returned as a member, we run another query to find out the users that are a member of the included group. ascii example:

sanramon (group synchronized to team in access control)
     +- Jon (user)
     +- Kev (user)
     +- support (group)
          +- Gaz (user)

Looking at the above heirarchy which shows the "San Ramon" (sanramon) team, and set a filter to get all users that are members of the 'sheffield' group. Access control will normally create three users in the team: Jon, Kev and support. What this query does is create three users: Jon, Kev and Gaz (through a recursive search of the 'support' group).

Example Query

The query is :

ldap://10.2.2.45:389/CN=Users,DC=sr,DC=wandisco,DC=com?
sAMAccountname?sub?(memberOf:1.2.840.113556.1.4.1941:=CN=shef_devs_grp,OU=Belfast,DC=sr,DC=wandisco,DC=com)

3.8 LDAP Settings

The LDAP settings screen contains all the settings for any LDAP services that you use with SVN MultiSite. By default, LDAP is not activated. Click on Yes to Use LDAP for login authentication.

Toggle changes are immediate
Changes to this toggle setting are immediate, you don't need to click on the Save LDAP Settings for activating or disabling LDAP, only for making changes to the LDAP Settings.

Mixed Mode Login (Supporting local and LDAP users at the same time)
SVN MultiSite supports user login through both internally managed and LDAP managed accounts. To use mixed mode login, after adding an LDAP Authority, click on the option "Allow both LDAP and Local Login".

As the switch to enable LDAP is immediate and occurs before the first LDAP authority is actually setup, it is necessary to enable "Mixed mode" when LDAP is first selected. Otherwise, leaving the admin console before setting up an authority could result in you getting locked out of the admin console.

Running with multiple LDAP Authorities
When you enable multiple LDAP authorities, take care that you properly set how SVN MultiSite behaves when handling user logins.

It is possible to set up LDAP in such a way that you'll effectively block access to the SVN MultiSite admin console. For example, setting up an additional, lower order LDAP authority to handle your admin login, but not enable the features:

Continue onto next LDAP authority on user not found
Continue onto next LDAP authority on user authentication failure

If you get locked out, use the LDAP reset script to Disable LDAP for Admin Authentication.

When enabled, all LDAP authorities will be listed, along with the following options:

Authority Table

Order: LDAP authorities are listed in the order of execution that you set when defining the authorities' properties.
Regular Expression: Lists any regex that are applied to the search.
URL: The URL of the authority.
Admin: Authorities that are set to return Admins will show a tick.

Authority Controls

Delete Authorities: Button to delete one or more selected LDAP authorities from the list.

Removing the last authority will disable LDAP authentication
If you delete the last LDAP Authority SVN MultiSite will automatically switch back to local authentication to ensure that it is still possible for users to authenticate with Subversion or login to the SVN MultiSite admin console.

Test Authorities: To test the connection to one or more authorities, tick the authorities check box and click the Test Authorities button.
Create Authority: To create a new LDAP authority, click the Create Authority button.
Actions: The following, additional actions can be performed on each authority by clicking on the Actions drop-list.

Edit: Edit the authority settings.
Delete: Delete the authority.
Set as Admin Group/Unset Admin Group: Sets or unsets the authority as the admin group.
Change Order: Change the authority's order number to make it execute earlier or later.
Test Connection: Perform a test connection to the authority.
View Members: Connect to the first matching authority and retrieve the selected group members.
Disable Locally: Disable the first matching authority on the current MultiSite node.
Disable Globally: Disable the first matching authority and replicate the action to all other nodes.



Warning:
When you disable an LDAP authority Globally it can take a minute to propagate to other nodes. Until propogation is completed, users will still have access to repositories that correspond with the disabled authority.

LDAP admin/team polling frequency: The frequency (in minutes) that MultiSite performs a check of the status of the admins / team members. (Default is 7 minutes)

LDAP Server Connection Timeout: The timeout value that will apply to SVN MultiSite's attempts to connect to an LDAP server. (Default is 2 minutes)

LDAP Query Execution Timeout: The timeout value applied to any LDAP Queries. (Default is 8 minutes)

LDAP TTL Cache Timeout: The timeout value lets you set the timeout on the LDAP auth cache. (Default is 28 days)


These timeout settings can be set in seconds (s), minutes (m) and hours (h). It's also possible to specify a time in days (d), although this is probably only meaningful for setting the LDAP TTL Cache Timeout.

Continue onto next LDAP authority on user not found: Tick to enable MultiSite to run through all available authorities should a user's credentials are not present on the first authority.

Continue onto next LDAP authority on user authentication: Tick to enable MultiSite to run through all available authorities should a user's authentication fail on the first authority.

Use LDAP authorities to authenticate administrators: Tick to enable MultiSite to authenticate its administrators (users who have access to the MultiSite admin console) via LDAP. This will automatically be ticked and not editable if System Administrator Group? has been ticked in the authority's settings.

Disable non-admin users on removal from last LDAP team: Tick to automatically disable users who no longer belong on any team because LDAP management has removed them.

Allow both LDAP and Local Login Tick to allow users to be managed both local (created and managed via the admin console) and through an enabled LDAP Authority.

Click SAVE LDAP Settings to save any changes you make on the LDAP Settings screen.

After entering and saving LDAP Authority settings it is possible to switch back to not using LDAP while retaining your LDAP settings. Coming back to the LDAP Settings screen and selecting "Yes" will re-enable your earlier settings, you don't need to enter them again.

3.9 Pending Changes

It's now possible to set MultiSite to batch up user, team or rules changes that require modifications to the Authz or htaccess files - modifying these files results in the need for a global transaction which, in large numbers, can result in a drop in performance. This feature is aimed at large scale deployments where large numbers of concurrent file regenerations could result in hardware/disk i/o bottlenecks.

The Pending Changes lists all actions that have been placed in the next batch, along with an indicator of how long until these batch will be replicated to the other nodes.

Click View Pending to see all pending changes (the current number of changes are listed in brackets), the time of the last batch and the time when the current batch will be run (triggering password/authz file rewrites). Any pending changes will be listed in a table that shows the "Action"(type of change) and the corresponding target user or group.

4. System

The system tab provides access to logging and backup features.

4.1 Log Viewer

MultiSite log files can be viewed from here.

console.txt

Logs from the operating system concerning the status of MultiSite's running processes.

reset.log

Logs instances where MultiSite has been reset.

SVNProxyServer-prefs.log.(time-stamp)

The MultiSite log file contains most of the activities of the replicator.

user-import.log

Logs instances where users have been imported.

WebProxy.log.(time-stamp)

Logs errors and changes applicable to the MultiSite replicator.

Knowledgebase
There's a knowledgebase article about WANdisco Log Files.

user-import.txt

Logs each user import that has been completed on the node.

4.1.1 Limiting Log lines

Logs can be generated over multiple lines, which can make reading/performing greps difficult. It's possible to limit each log to a single line by using the following procedure:

• 1.Set the following line in the svn-replicator/config/log.properties file:

  org.nirala.trace.WandiscoLoggingFileHandler.formatter = org.nirala.trace.SingleLineFormatter
  

• 2. Replacing any other formatter for this handler.
• 3. Apply the above change to all nodes.
• 4. Perform a synchronized stop, then restart all nodes.

4.2 Disk Monitor

The disk monitor can be used to lessen the chance of a node running out of disk space, which would result in the loss of replication or data corruption.

Use disk monitor: Tick the checkbox to have the disk monitor running.

Interval between checks (in milliseconds): 900000 (15 minutes) is the default interval time between checks.

Determine disk state by: Choose to measure disk usage either by percentage used or bytes free (in bytes, or append KB, MB or GB).

Warn Level: State the level of usage (either as a percentage or bytes free) at which point an alert email will be sent, and resent each time the disk monitor completes a check. So, if you set the Interval between checks to 900000 milliseconds, a warning email will be triggered every 15 minutes.

Critical Level: State the level of usage (either as a percentage or bytes free) at which the node will send an alert email and then shutdown. This will prevent the data on the node from being corrupted.

Click the Save Settings button to save any changes you make.

Alert
You MUST restart the node for any changes you make to take effect.

4.3 License Info

The License Info page contains the details of your WANdisco product license, useful for confirming the details of a number of license-specific features.
Company: Your company name.

Product: The product type. This is currently either svn (Subversion) or (CVS).

License Type: This indicated whether the license is either for evaluation or production.

Access Control:
Shows if Access Control has been enabled on the product. With Access Control enabled, the User's tab is renamed "Security" and is expanded to include many additional features for Access Control.

Number of Allowed Nodes: Number of replicator instances that have been licensed.

Licensed IP Addresses: The IP addresses of each node.

Number of Allowed Users: Total number of possible users that are covered under the license.

License Expires: The date on which the current license will expire.

Maintenance Start Date: The first date covered by the current maintenance contract.

Maintenance End Date: The final date covered by the current maintenance contract.

4.4 System Settings

The system settings screen provides a number of extra controls that allow you to fine -tune MultiSite for potential performance improvements.

Auto skip bypassable transactions: Tick the box to force the replicator to automatically skip bypassable transactions.

A transaction is bypassable so long as it fails on all nodes within the replication group. In spite of the failure, all repository replicas remain in sync, allowing replication to continue.

Max INFORM Messages: This is a an advanced tunable parameter that can limit network notification traffic in situations where a node is pulling in a large number of changes, such as while recovering from a brief network outage.

Default value: 1000,
For Unlimited INFORM Messages use the setting "-1".

The default value of 1000 is optimal for the majority of network configurations and file sizes, but can be tuned to make best use of higher or lower bandwidth networks handling varying file sizes.

Once the value is altered at one node, it is replicated to other nodes that are part of the replication group. We recommend that you consult with our support team before changing this setting.

HTML format failed pre-replication hook output: The labelling of the output of failed pre-replication hook scripts can cause problems. Untick the checkbox to removed the extra formatting, which will make outputs less human-readable but will support scripts that were written around the previous output.

Use generic file replication: Use this checkbox to activate the RESTful API function that allows for generic file replication.

Alert
Turning on this feature allows you to replicate files within your file system, i.e. files outside of your repositories such as Authz or Apache configuration files. The feature should be activated with caution as it allows the admin console user to overwrite any file.

Automatically send problem repos offline on global transaction:
Tick this option to ensure that any repositories that fail to complete a global transaction are placed in a read-only mode, so that they don't risk halting all of the whole replication group.

Limit commit size:
The commit of a very large file can easily clog DFTP transfer between nodes, which will hold up all other traffic behind a single PUT operation. It's possible to mitigate against attempts to commit giant files using pre-replication hook scripts, however, hooks don't fire commits until the MERGE command is sent, which can be too late to stop the disruption.

Instead, use this check box to set the Maximum commit size. When switched on, the default is 1024 MB (1GB). You can use the units: "KB", "MB" or "GB" to denote your filesize.

Pending transaction warn level: Email notification is now sent out when there are more than 50,000 pending transactions on a node. The number of pending transactions that triggers the notification can be configured here.

Enable Access Control prevayler snapshot: Access Control and MultiSite's internal settings are stored within a prevayler database that uses journaling to store all changes. When used in large organizations this approach can consume a very large amount of diskspace. Enable the prevayler snapshot to take periodic (running at Midnight each day) copies of the database instead of recording all changes. Choose which node you want to perform the snapshot

Run automated backup:
This automates the export setting tool so that MultiSite's backup files are created each day, running at midnight (using the local time of the node that you select). Use the Number of backups to retain field to limit the number of backups that the system keeps before it begins cycling.

Enable Audit Trail logging: Enable this option to record all SCM activity in the AuditTrail logs. This feature is switched off by default as it creates some additional overhead in transaction processing.

Enable User Access Tracking: Enabling this option will save timestamps against each user indicating the last time they interacted with the repository, which can be useful for user management.

User updates are greatly increased
Switching on the "Enable access control prevayler snapshot" setting is highly recommended when you use this feature.

Use Case Insensitivity Usernames: This is a system-wide control for handling usernames in a case insenstive manner. The setting can instead be applied on a per-LDAP authority basis by using the setting in the Create LDAP Authority screen.

Run batch Access Control updates: This option allows you to store up changes that need to be made to the htpasswd and authz files and fire them off in a global transaction at a frequency that you set in the Access Control update frequency field. Use the Batch update proposing node to choose which node in your replication group will control the batch and handle the update. In a large scale implementation with many repositories, users and teams the use of this feature can greatly improve performance.

Don't disable SSL while batched changes are pending
When batching Access Control updates ensure that there are no pending changes before you disable SSL encryption. If SSL is disabled while changes are pending, after restarting, the node will become read-only.
If this occurs you can return the node to operation by performing a reset.

4.5 SSL Settings

The SSL Settings screen controls all settings related to the keystore and truststore management, used when the secure socket layer protocol is used to encrypt traffic between the replicator and clients.

Keystores and Truststores
You might be familiar with the Public-key system that allows two parties to use encryption to keep their communication with each other private (incomprehensible to an intercepting third-party).

The keystore is used to store the public and private keys that are used in this system. However, in isolation, the system remains susceptible to the hijacking of the public key file, where an end user may receive a fake public key and be unaware that it will enable communication with an impostor. Enter Certificate Authorities (CAs). These trusted third parties issue digital certificates that verify that a given public key matches with the expected owner. These digital certificates are kept in the truststore.

An SSL implementation that uses both keystore and truststore files offers a more secure SSL solution.

Use SSL for replication:
Tick the checkbox to enable SSL encryption for your replication traffic.

Don't disable SSL while batched changes are pending
When batching Access Control updates ensure that there are no pending changes before you untick SSL encryption. If SSL is disabled while changes are pending, after restarting, the node will become read-only.
If this occurs you can return the node to operation by performing a reset.

Certificate Alias:
The certificate name of the server.

Key Password:
(Optional) password to control access to keys in the KeyStore. All keys in the keystore must have the same password.

KeyStore Filename:
The file in which private keys are stored. Click on the spyglass icon if you need to browse for the file.

KeyStore Password:
Optional password to control access to the KeyStore. All keys in the keystore must have the same password.

TrustStore Filename:
This is the file in which digital certificates are stored. This can be the same as the KeyStore. Click on the spyglass icon if you need to browse for the file.

TrustStore Password:
Password to control access to the TrustStore.

For more information about running SVN MultiSite with SSL, see 2.7 Using SSL.

4.6 Garbage Collection

The Garbage collection screen lets you tune MultiSite's memory recycling system - the process by which objects that are no longer require get deleted in order to free up resources.

Knowledgebase
For a detailed explanation of how SVN MultiSite's Garbage Collection works, see the article Garbage Collection.

Garbage Collection Interval (ms): The frequency of garbage collection in milliseconds (default 10 minutes or 600,000ms). Changes will take effect after the next scheduled garbage collection completes.

Max Proposal Attempts: The number of times a transaction execution should be attempted before suspending the proposal (default 2).

Max Buckets: The maximum number of buckets to clear in a single garbage collection. (default 1000) This setting can help prevent the system from being overwhelmed if a large number of transactions have built up.

Auto Offline High Watermark: The number of transactions between a problem transaction and the most recent one where the repository should be taken offline and all proposals against it discarded to prevent it occupying excessive heap space (default 100 000).

Auto Offline Low Watermark: After auto-offlining is triggered, the difference between the oldest remaining problem transaction and the most recent transaction when auto-offlining can cease (default 50 000). Troubleshooting section: 2.8 Offline mode.

4.7 Transaction Status

The Transaction Status page lets you query individual transactions to check on their status.
Transaction Number: The transaction's reference number.
Node Submitted From: The originating node of the transaction.

4.8 Log Level

MultiSite uses one log, and the default level is info. The levels vary from severe, where you get only the most severe warnings, to finest, which logs every action.

Logger Name: References the logger object's name.
New Log Level: The following options are available:

Severe

message level indicating a serious failure.

Warning

message level indicating a potential problem.

Info

message level for informational messages.

Config

message level for static configuration messages

Fine

message level providing tracing information.

Finer

message level providing additional tracing information.

Finest

indicates a highly detailed tracing message.

4.9 Free Memory

This command frees the memory (GC stands for garbage collection) for the current node. The command occurs when you click on this menu selection. The display shows information on the command that was just performed.

Max memory used by JVM the maximum memory that JVM can use on the current node.
Free memory before GC the amount of free memory before you ran this command
Free memory after GC the amount of free memory after you ran this command
Memory freed the total amount of memory freed at the command's completion

4.10 Dashboard

Dashboard provides a list of replicated transactions.

By default, the Dashboard doesn't refresh automatically. Click Update to force a refresh, or set an automatic refresh by entering a refresh rate, in seconds in the Auto Refresh Every box.

As soon as MultiSite receives a Subversion transaction request, that transaction joins the replication group's queue. There is one queue for the replication group. Pending transactions are not displayed, but are counted in the Total Transactions Pending box in the upper right.

When MultiSite finishes processing a transaction, its displayed in the TX Id field. The Replicator field shows the node from where the transaction originated.

e.g. 0.0.0.0:6444 means the local node originated the transaction.

4.10 Export Settings

Export Settings Allows you to export settings, including all users, for later importation into a WANdisco product.

The export routine creates a directory containing the files necessary to import user settings back into MultiSite. Backup directories use a name string YYYY-MM-DD.HH-MM-SS::XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX ,
e.g. 2012-01-23.16-21-03::2775e033-45b1-11e1-acb4-21d651998aa9

A typical backup directory will look like this:

File properties                                                                Unique or shared
-rw-r--r-- 1 root root 62251 Jan 23 16:21 access-control.xml                   Common to Replication Group  
-rw-r--r-- 1 root root   101 Jan 23 16:21 ExcludeUsersDB.xml                   Common to Replication Group
-rw-r--r-- 1 root root   579 Jan 23 16:21 mailconfig.properties                Common to Replication Group
-rw-r--r-- 1 root root     6 Jan 23 16:21 MembershipConfigurationRegistry.xml  Node Specific
-rw-r--r-- 1 root root    87 Jan 23 16:21 NewUsersDB.xml                       Common to Replication Group
-rw-r--r-- 1 root root   814 Jan 23 16:21 NodeRegistry.xml                     Node Specific
-rw-r--r-- 1 root root  1238 Jan 23 16:21 NonResettableDynamicPreferences.xml  Node Specific
-rw-r--r-- 1 root root  2789 Jan 23 16:21 prefs.xml                            Node Specific
-rw-r--r-- 1 root root   162 Jan 23 16:21 RepoSettings.xml                     Common to Replication Group
-rw-r--r-- 1 root root  1162 Jan 23 16:21 Repos.xml                            Common to Replication Group
-rw-r--r-- 1 root root     6 Jan 23 16:21 ViewRegistry.xml                     Node Specific

On a successful export you'll see a confirmation message that verifies the location and name of an export directory.

4.11 Import Settings

Important This section applies when upgrading from SVN MultiSite 4.0 to 4.2. When upgrading between builds of 4.2 use the Temporary Import Procedure

The Import Settings screen can be used to import either a backup.xml file (used for MultiSite 4.0 or earlier) or a backup directory (introduced in MultiSite 4.1). These backups let you import previously backed-up user settings.

Important All files for use with WANdisco MultiSite should use UTF-8 encoding (and/or single-byte encoding if there are no none-ASCII characters). Any other encoding e.g. Unicode will not be parsed correctly.

Importing a backup

• 1. From the admin console, click the System tab.
  
  


• 2. Click the Import Settings link.
  
  


• 3. Enter the path to the backup directory (or backup.xml file from MultiSite 4.0 or earlier).
  
  


• 4. If you're unsure of the path, click on the spyglass button and browse to the backup.
  
  


• 5. Click on Import Settings. Backed-up user data will now be imported back into MultiSite.
  
  

4.2 to 4.2 Alternate Import Procedure

4.2 to 4.2 upgrades
This is an alternative to using the upgrade script, although in both cases a full reset and shutdown is required.

4.0 to 4.2 Import Procedure Remember that completing an upgrade between two different product versions requires that you complete an export of your user settings, then apply these to a fresh installation.

• 1. Perform a synchronized stop.
• 2. Reset all nodes: navigate to <path>/svn-replicator/bin: and run:
  

  ./reset

• 3. Confirm the location of your SVN MultiSite backup directory, e.g. <path>/svn-replicator/backups/<chosen backup folder>.
• 4. Create an empty file in <path>/svn-replicator/config/ called "restore.properties".
• 5. Open the new file and add a single line:

  backup.location=<absolute-path-of-the-backup-directory-as-noted-in-step-3>

• 6. Save the file.
• 7. On the node that you are importing the backup, restart the replicator using the command:

  ./svnreplicator

  The replicator restore.properties file will trigger the import of the backed up settings.
• 8.The replicator will start up with all backed-up settings in place.

5. Proxy

The Proxy tab handles proxy settings which alter the way that MultiSite works between Subversion and users.

5.1 Proxy Status

Proxy Status Displays the node's status in the tab's main panel.

5.2 Repository Status

Repository Status gives you the status of all repositories that are hosted on the node.

Show All: Click on Show All to display all hosted repositories.

Show Read-only: Click on Show Read-only to show only those hosted repositories that are in read-only mode.

Auto Refresh: The Repository Status screen will automatically refresh every xxx seconds. Click on Auto Refresh to turn this feature off.

Repository Search: Use this entry box to search for a particular repository.


For each Subversion repository there is a table entry that includes the following information:


Tick the checkbox to select multiple repositories for the Retry Transactions command.

DAV Location:The directory used for access to the repository.

SVN Repository Path: The absolute path to the Subversion repository.

Scheduled TXs: Number of transactions that are scheduled for replication.

Completed TXs: Number of transactions that have been completed.

Bypassable TXs: Number of transactions that can be bypassed. These are transactions that have failed on the node, but can be ignored by the replicator because the transaction has failed on all nodes in the replication group. To force the replicator to bypass the transactions, click on Bypass. You can choose to have any bypassable transactions, bypassed automatically by using the Auto skip bypassable transactions checkbox, situated on the System Settings.

Retry TXs: Number of transactions that can be retried. If write access to the repository was briefly lost, the retry command allows you to replay the transactions and hopefully bring the repository back up-to-date.

Retry Count: Number of attempted retries that have been made.

Details: Click to view the dashboard for the corresponding node.

Bypass: Click to attempt to Bypass the transaction. This command will only be available if the transaction can be bypassed.

Retry: Click to retry any pending transactions.

Take offline: Click to sideline the repository by putting it into the offline mode. This tells the other nodes to press on, and not queue up subsequent proposals. Once a repository has been taken offline, it can never catch up and will require a Repository Repair.

Repair: Click to being a repository repair. There's a full explanation of how to complete a repository repair in Repair Repository.

5.3 Log Viewer

Log Viewer You can view the logs, including the main log SVNProxyServer-prefs.log.0

5.4 SVN Settings

These settings control how Subversion is set up and include host and authorization details.

LDAP Settings
From SVN MultiSite 4.2 the LDAP settings have been moved from the SVN Settings screen to the Security Tab

Use Pre-Replication Hooks

Indicates whether pre-replication hooks are being used. (Default: No). A pre-replication hook is a script that gets triggered by a specified repository event, which runs before the event is replicated to the other nodes. If you are using pre-replication hooks, select Yes.

Knowledgebase
You must select the version of Subversion you are using, otherwise pre-replication hooks will not work. See Setting up hooks.

SVN and WANdisco are on the same server:

There are significant benefits from running both Subversion and MultiSite on the same host, for example, MultiSite can manage the Subversion password.

Subversion Host:

The IP address of the Subversion Server. If Subversion is running locally you can put 'localhost'.

Subversion Server Port:

The port on which Subversion is actually running (8080 by default).

SVN Executable

The fully qualified path to the svn executable. On Linux you can run "which svn" to determine the path to the file.

Replicate the location of the svn executable to all nodes

It's best to have your nodes set up in exactly the same way. However, if you need to run Subversion from different locations on each node, select Only set the location of the svn executable at this node.

Use authz-based access control?

If you intend to use Apache's Authz authorization, tick the check box.

Authorization File:

If you have selected to use Authz, enter the path to the authorization file here.

Restrictive AC Mode

When Access Control uses Authz there are two available modes for applying access control rules, by default Access Control is restrictive and gives more weight to access restrictions. This ensure that a user in a situation with conflicting rules doesn't get the benefit of the doubt as the "deny" rule always takes precedence. I.e.: the order in which rules are applied is DENY, READ-ONLY, READ-WRITE.

If you prefer that Access Control behave like mod_authz_svn then untick Restrictive AC Mode. In which case Access Control applies rules in the opposite way: READ-WRITE, READ-ONLY then DENY. So that in a rules conflict, the most favorable rule is applied.

Save Settings Click to save your settings.

Regenerate Password File(s) Regenerate the SVN password file. If an Authz file is deleted or corrupted, it will also be regenerated.

Repository Directory and DAV Location

You can Edit, Delete or Check the consistency of the Subversion Repository.

Add Location: Click to add an additional location to the directory in which you are managing repositories.

Edit Repository

The edit repository screen lets you modify a number of repository specific settings

Directory on File System: This is the fully qualified path to the Subversion repository. Not the URL used by clients. You can search for the location by clicking on the magnifying glass button.

Manage Password File: Tick the checkbox to have MultiSite control your Subversion password file. You can only use this feature if both MultiSite and Subversion are located on the same server.

Pre-Commit Hook: Equivalent to normal pre-commit hook but it is automatically picked up and invoked in a replication safe manner. If field is left blank and hook exists in repository hooks directory called ms_pre-commit it will be executed.

Pre-lock Hook: Equivalent to normal pre-lock hook (invoked when a user attempts to lock a resource) but it is automatically picked up and invoked in a replication safe manner. If field is left blank and hook exists in repository hooks directory called ms_pre-lock it will be executed.

Pre-Unlock Hook: Equivalent to normal pre-unlock hook (invoked when a user attempts to destroy an exclusive lock) but it is automatically picked up and invoked in a replication safe manner. If field is left blank and hook exists in repository hooks directory called ms_pre-unlock it will be executed.

Authentication Realm: If left blank the realm will default to "wandisco". Enter an alternative realm, subject to your needs. The chief benefit of having the editable realm is that you can have a different authentication realm for each Subversion repository.



DAV Location: the directory in which WebDAV is located.

Multiple SVN Repositories: Choose whether you are using SVNParentPath or SVNPath in the Apache location directive.

If you select Yes, SVNParentPath is used in the apache location directive, you'll be presented with a entry box that allows you to select and edit available Root Names.

Consistency Check

Provides a basic check that replication is working properly by comparing Subversion revision information between nodes. Although this is a fairly superficial test of consistency, it will more often than not pick up on problems.

The following parameters are available, allowing for more selective checks to be performed:

DAV sub path: Select the Repository by its DAV sub path.

From Revision: The most recent revision to take the consistency check from. Can be specified by revision number or "HEAD" to indicate the latest in the repository.

To Revision: The oldest revision to include in the consistency check.

Revision Increment: The increment to display the revisions in e.g. display every 5th, 10th, 100th revision (1 to display all revisions in range, can be used to speed up result retrieval for large revision ranges).

# revisions: The maximum number of revisions to include in the consistency check.

When a consistency check confirms that replicas are consistent the screen will appear as above

When a consistency check finds that there's an inconsistency between replicas the screen will appear as above.

Schedule Consistency Check

Consistency Checks can be scheduled to occur automatically: select a frequency from the dropdown, if applicable, select a sub path (If using SVNParentPath) then click Set Schedule.

Repository DAV location

The DAV location assigned to the repository for which you will perform a consistency check.

Schedule Frequency:

Daily or Hourly are available.

Email consistency results:

Send the results of all checks using the email details entered into SVN MultiSite's Email Settings

Email consistency check results only on failure:

This option lets you limit the amount of email that get sent, in this case an email is only triggered after the consistency check identifies that consistency has been lost.

Set Schedule

Click this button to save the current settings

Send test email now

Click this button to trigger an immediate tests, the results of which will be emailed (subject to which email option you selected).

5.5 Email Settings

Email Settings Email settings that Access Control uses to send status alerts.

5.6 Change Distinguished Node

Allows you to change the node in your replication group that holds distinguished status.

Change the distinguished node immediately. Verify with Proxy Status.

Distinguished Node Rotation Schedule

When running SVN MultiSite with a Singleton Quorum, only one node in the replication group decides on the transaction order, this is called the distinguished node (DN). Subversion users who are local to the DN will experience the best possible performance as the proposals they generate will not be delayed while a far-flung node decides on transaction ordering.

Follow-The-Sun
In a global organization it makes sense to locate the distinguished node to wherever the most Subversion users are currently active, using the workflow that has become popularly known as "following the sun".

While it's possible to use the Change Distinguished Node tool to manually change its location, it's easier to automate the move as part of a scheduled rotation.

To understand how to set up a schedule rotation, read 3.5.3 Scheduled rotation of the distinguished node.

Interference
If you have a rotation schedule in place and had a need to manually change the location of the distinguished node, the next scheduled rotation will continue as normal.

Time
You can set up a rotation schedule from any node, however you need to select a 'Controller Node', this is the node that will create the proposal that sets the schedule. The Schedule times, as displayed on each node will be the 'local time' rather than the time at the controlling node.

Example In a 3-node replication group with the nodes located in U.S.A, Europe and China the following schedule is set from the U.S. node (the U.S. Node being set as the controller):

European Node 02:00
Chinese Node  18:00

The European Node is the distinguished Node from 2am (U.S. Eastern Time) If viewed from the European node, the schedule will display 7.00, which is the equivalent local (Belfast) time. Likewise, the Chinese Node takes over the Distinguished Node at 6pm (U.S. time). Viewing the same schedule from the Chinese Node, the time displayed with be the equivalent local time: 6am.

Setting time
For consistency we recommend that all nodes have their clocks set to use GMT plus or minus a timezone offset. If the clock is set using a city location, then daylight savings may not be accounted for, causing apparent timing inconsistencies in the logs.

5.7 Repair Repository

5.8 Stop Proxy

Stop Proxy Stops MultiSite from and prevents clients from writing to the repository.

5.9 Shut down node

Shut Down Node Shuts down MultiSite completely on the node.

To restart the node:
Go to svn-replicator/bin and type:

perl svnreplicator

Replication

Provides control over node and replication group settings

5.10 Nodes

Lists the available nodes. You can edit or create a new node.

Create / Edit Node

5.11 Replication Group

View or create new replication groups.

5.12 Replication Group History

A table presents the current and any past active replication group activity.

4. Technical Guide

1. Introduction

This technical guide will help you understand the underlying technology and technical concepts that relate to WANdisco's SVN MultiSite software.

We'll be using terms like "node" and "replication group" without explaining what we mean. Check out the Glossary for an explanation of these and other WANdisco terms.

2. MultiSite Overview

Subversion is designed to run as a central server to which multiple Subversion clients connect. WANdisco's replication technology makes it possible to have multiple active replicas of a Subversion repository that are in synch. The Subversion replicas can be anywhere on a WAN - distributed throughout a company's campus or throughout the world. WANdisco users experience the performance of a local Subversion repository, with the semantics of a single shared Subversion repository. We call this "active replication with one-copy-equivalence."

Replication ensures that each replica acts as a hot backup to every other replica. If a local server experiences a problem that takes it offline, only local users are disrupted, the rest of the replication group continues as normal.

Example Replication Group

This illustration shows a replication group with five Subversion severs.

WANdisco offers a High Availability solution based on Zeus Software that ensures no disruption in service. Even if a Subversion server failed, a Zeus cluster ensures that Subversion and repository data continue to be available.

SVN MultiSite acts as a proxy between the Subversion Server and clients. An instance of the proxy runs at each replica. All the communication paths involved in the operation of WANdisco are illustrated in diagram above.

3. WANdisco MultiSite Concepts

All MultiSite nodes are synchronized at all times: each Subversion repository is a functional replica of the others. WANdisco replication technology is the concept of one repository, multiplied. Because there are multiple synchronized repositories, each replicated node is effectively a current hot backup, which makes disaster recovery easy to plan and implement.

The Subversion usernames and passwords on all repository hosts must match. This is required because MultiSite creates a peer-to-peer replication system. Any replica of the Subversion repository is accessible by every valid Subversion user. WANdisco offers the user the option of having MultiSite manage the Subversion password file.

4. How Replication Works

The sites in the replication group are continuously coordinating the Subversion write transactions users are making. The group establishes transaction ordering through the agreement of a quorum of replicas. When you install the first node, that node by default is the distinguished node with Singleton quorum. When you create the replication group that includes other nodes, you select the quorum type best suited to your configuration. For which quorum is best for particular situations, see Quorum Recommendations.

Singleton Quorum Singleton Response quorum - only one of the nodes in the membership decides on the transaction order. With Singleton Response quorum, the node that decides transaction ordering is called the distinguished node. The Singleton quorum offers the fastest response time for those users working at the distinguished node, because as soon as the distinguished node determines that a transaction can be processed in the correct order, the transaction is sent to Subversion. Any replicator except the distinguished node can go down, but the replication group continues. The replication group replays the missing transactions when that node rejoins the group. However, the Singleton quorum also represents a single point of failure, since replication halts if the distinguished node fails.

Majority Quorum Majority Response is another quorum option, whereby you specify that a majority of the sites must agree on transaction order before any transaction is committed. Having a majority quorum ensures that if one site goes down in a replication group, even the distinguished node, the other sites can continue uninterrupted, as long as a majority of the sites remain available. The replication group replays the missing transactions when that site rejoins the group.

In a majority quorum, the distinguished node?s role is that of a tie-breaker. For example, in a four node replication group, three sites make the quorum (three sites must agree about transaction ordering). If two nodes want one transaction first, and the other two want another transaction first, then the distinguished node gets a weighted vote. The group with the distinguished node determines the transaction ordering. With an even number of nodes with majority quorum, you can schedule the distinguished node to rotate to different nodes around the world, to "follow the sun".

Unanimous Quorum The last quorum option is unanimous response, which requires that all replicators must be reachable to accomplish transaction ordering.

5. Replication Example

Here is an overview of what occurs when a write transaction is received by any replicator in the replication group.

1. The originating client sends the transaction to SVN MultiSite, which passes it along through the replication group.
2. Transaction data is successfully received by the quorum (the distinguished node for Singleton quorum, or a majority of sites for Majority quorum). The quorum assigns the transaction a Global Sequence Number (GSN).
3. After receiving the transaction, each SVN MultiSite passes the transaction data to its local Subversion server.
4. Each local Subversion server processes the transaction.
5. SVN MultiSite waits for Subversion to complete the transaction. SVN MultiSite only marks the transaction complete when Subversion returns a completion status. If for some reason replication goes down during this process (the replicator crashes, is stopped by an admin or the server it's running on shuts down), SVN MultiSite does not mark the transaction as complete, and it gets reprocessed upon replication restart.

6. WANdisco is Listening

There is a field in the Admin Console that tells you whether WANdisco is accepting any incoming Subversion client requests. Replication still continues among the WANdisco sites, whether WANdisco is listening or not at one or more sites.

You can turn the listening on and off through the Admin Console (through the Start Proxy and Stop Proxy commands). Issuing the Stop Proxy command on a site puts that Subversion server in read-only mode.

The following illustration shows Sites 2 and 5 are not listening. (An administrator executed the Stop Proxy command for those sites.) Replication continues, and Sites 2 and 5 are still receiving and processing replicated transactions originating from the other sites. However, Subversion users at Sites 2 and 5 cannot make any write transactions. Once an administrator issues the Start Proxy command for Sites 2 and 5, the local Subversion users can again issue Subversion commands.

Please follow your company guidelines in regards to notifying Subversion users of maintenance.

7. Synchronized Stop of All Sites

When an administrator issues a synchronized stop command, the Subversion servers stop accepting write commands from clients. Pending transactions are processed, but no new write transactions are accepted. Subversion users continue to have read access to the repository, but cannot perform write operations, such as commit or lock.

When an administrator issues a resume command, the WANdisco proxies restart and begin accepting write transactions.

8. Handling Node and Network Failure

8.1 Node Failure

For Singleton Response quorum: If a nodes goes down, replication continues at the remaining nodes, as long as the quorum* is reached, although users connecting to the downed node will be unable to write to repositories until the node is repaired (and the quorum reached).

In a Singleton Response quorum, the distinguished node must be available.

As soon as a node comes back up, the replication group catches up the node on its missing transactions, so that all nodes are again synchronized.

For Majority Response quorum: say you have a five node replication group. One or two nodes could go down, and replication would continue at the other nodes, as long as a majority of nodes remain up. The one or two downed nodes go into read-only mode. As soon as a node comes back up, the replication group catches up the node on its missing transactions, so that all nodes are again synchronized.

8.2 Per-repository scheduling

SVN MultiSite 4.0/GA includes some changes to transaction scheduling and failure handling. In previous versions, a single scheduler handled all transactions on a node, so that in the event of a single repository becoming corrupted, and entering read-only mode, all the repositories hosted on the node would also become read-only.

Previous model
Transaction failures would result in the replicator restarting and retrying an arbitrary number of times. If the transaction still failed, a check would be made to see if the transaction was bypassable (this is so if it failed on all nodes). If the transaction is bypassable, replication would continue after a restart, if the transaction was not bypassable, the node would restart in read-only mode.

New model
Instead of a single scheduler per node, Subversion transactions now use a scheduler per svnroot, with AuthzProposal and UserPasswordProposal shared between all schedulers.

There are two key changes in behavior, as a result of this change:

1. 1. Failed transactions, or moving in or out of read-only mode no longer triggers the node to restart.
2. 2. Failed transactions will now only interfere other transactions going through the same scheduler, and with global transactions. This allows the other repositories to continue to replicate, greatly increasing resilience in environments where large numbers of repositories are hosted on a single node.

8.3 Network Failure

If a network link goes down for one node, and outside connectivity is completely lost, there are two possible scenarios, depending on your quorum:

• If you have Singleton quorum, and the distinguished node's network link goes down, the distinguished node alone can make progress. The Subversion users local to the distinguished node continue working uninterrupted, while users at other sites in the replication group can make only read operations (like up, co, log, etc.) working with stale data.
• If you have Majority quorum, and one site's network link is lost, then users at that site can execute only read operations (like up, co, log, etc.) working with stale data. Providing that the remaining sites can still meet quorum (having a majority of sites responding), the other sites continue working uninterrupted

When connectivity is restored or the errored node is back online, the local node syncs up with the replication group automatically. First, the local node consults its local recovery journal (similar to a database redo log), and then, if necessary, attempts recovery from any of the quorum sites.

The recovery infrastructure and details of WANdisco fault-tolerance can be found at http://www.wandisco.com/get/?f=documentation/whitepapers/WANdisco_DConE_White_Paper.pdf.

9. LDAP Authentication

Authentication is handled by the MultiSite replicator performing an LDAP login attempt from the originating node, using the credentials supplied by the user. As users successfully authenticate, they are added to the replicator's user database and added to the (replicated) Apache password file with a known common password.

Apache is configured to use the password file containing common passwords and the WebDAV requests are populated with an authentication string from that password file. The remainder of the replication work flow remains unchanged. The key difference is that the authentication credentials used by Apache will not change over time and thus authenticated transactions can be executed on each replicated node.

Caution
If you are using LDAP based authentication, do not make changes to users using the admin console User Administration screen.

LDAP passwords
Use only alphanumeric characters in LDAP passwords. Due to a bug in Apache/mod_svn_dav, the use of non-alphanumeric characters in passwords could result in replication failure.

9.1 Requirements

• Incoming requests must contain accessible username and password credentials.
• The LDAP server must be available for authentication to succeed.
• Only WANdisco is modifying the htpasswd file

9.2 LDAP data flow

Copyright © 2010-2013 WANdisco
All Rights Reserved

This product is protected by copyright and distributed under licenses restricting copying, distribution and decompilation.