Reference Guide

1. UI tabs

1.1 DASHBOARD

The dashboard enables an administrator to track changes on selected repositories. You can see detailed log messages as SVN MultiSite Plus is working.

Example Dashboard - situation normal

Dashboard

Example Dashboard - indicating errors

Something's wrong! Dashboard may tell you more

System Status

A single line status message that indicates whether replication is running successfully.

Replication Groups

The status of each running replication group is listed. Click on the dropdown button to indicate which nodes are at fault.

Pending Tasks

List all tasks that are currently pending. It is possible to cancel tasks by clicking on the corresponding button.

Important: Don't cancel replication group creation tasks
If you create a new replication group, then find that the task is stuck in pending because one of your nodes is down, do not use the Cancel Tasks option on the Dashboard's Pending Tasks table.
If, when all nodes are up and running, the replication group creation tasks are still not progressing, please contact the WANdisco support team for assistance.

Failed Tasks

Lists the replicator tasks that have failed, along with the task's unique Id, which can be used to search the logs for more details.

Disconnected Nodes

Logs all nodes that are been disconnected, along with a duration.

1.2 REPOSITORIES

Click this tab to manage your replicated SVN Repositories: Add, Edit, Consistency Check, Repair, Sync Stop, Reset all Stats, Remove

REPOSITORIES tab

1.2.1 Repository table

The repository table lists all the repositories that you add to SVN MultiSite Plus.

Name

The Repository Name

Known issue: duplicate repository names allowed
It's currently possible to add multiple repositories with the same name, although they need different paths though. Ensure that you do not use the same name for multiple repositories. This is bad practice and will be prevented in future releases.

Path

The local path to the repository. This needs to be the same across all sites

Replicator Group

The replication group in which the repository is replicated

Size

The file size of the repository

Size field may show "-1"
The Size field may show "-1". To prevent slowdown or CPU spikes, when a repository is freshly added or its stats are reset, there is a limit of 10,000 on the number of revisions that MSP gathers. For a repository with 10,000+ revisions, the size is not calculated and "-1" is displayed.
If you want an accurate value, add the following line to application.properties:

stats.collection.revisions=<value>

Restart the node. You can repopulate existing repositories after increasing this limit by resetting the stats.

Youngest Rev

This is the latest revision number for the repository

Transactions

Lists any pending transactions. The transactions link to the last transactions played out for the repository:

Click on a transaction box.

Transactions list revealed!

Last Modified

The date and time of the last modification to the repository

Global RO

Indicates if the repository is globally read-only

Stops any further commits from SVN users
The term Global Read-only doesn't accurately reflect what happens at the repository-level. When a repository enters a Global Read-only state it will no longer accept any commits from SVN clients. However, proposals that are flying around within the state machine can still be written. It is this state that allows nodes to reach a synchronized stop.

Local RO

Indicates if the repository is locally read-only. A locally read-only repository is completely locked down, it does not accept new commits from SVN clients but does accept futher changes from within the replication system.

Status

Indicates whether the repository is replicating or has stopped. A stopped repository will be in a read-only state, either globally or locally

Consistency Check

The Consistency Check tool provides a quick method for confirming that the distributed copies (replicas) of a repository are all in an identical state - a requirement for replication. Clicking the button will trigger series of checks, the results of which will appear on the Replicator Tasks widget, available on the dashboard. For more information on its use, see Consistency check.

Remove

Use the remove button after selecting a repository to remove the selected repository from MultiSite's control. The repository data will not be deleted but once deleted, changes made to the repository locally will no longer be replicated to other nodes. See 8. Removing Repositories

Edit Repositories

It's possible to make changes to a repositories settings after it has been added. This is how:

• Click on the REPOSITORIES tab and then click on a Repository Name. Alternatively you can click anywhere on the Repository's bar, causing it to highlight in yellow. Click the EDIT button on the repositories menu bar, which has now turned blue.
  


• The repository's screen will open. You can change the any of the settings that were applied when the repository was setup.
  

  



Consistency Check
This button will open the consistency checker tool that you can use to verify that the repository matches its replicas on the other nodes.
Sync Stop
Bring replication of the repository to a stop accross all nodes.
Repair
Use the repair button to initiate a repository repair procedure.
Reload
Refresh the screen to pick up any changes.
Repository Information
Available information about the repository.
Remove Repository
Use this tool to remove a repository from SVN MultiSite Plus's control, note that the repository data will not be moved or deleted, it will just not be tracked by MultiSite.

Add Repository

Click ADD in order to add a new repository to SVN MulitiSite Plus - the repository must already exist and be tested before you place it under the control of MultiSite.

Repo name

Choose a descriptive name. This doesn't need to be the folder name, it can be anything you like.

FS Path

The local file system path to the repository. This needs to be the same across all sites.

Replicator Group

The replication group in which the repository is replicated. It is the replication group that determines which sites hold repository replica, and what role each replica plays.

Global Read-only

Check box that lets you add a repository that will be globally read-only. In this state SVN MultiSite Pluscontinues to communicate system changes, such as repository roles and scheduling, however, no repository changes will be accepted, either locally or through proposals that might come in from other sites - which in most cases shouldn't happen as by definition the repository should also be read-only at all other sites.

Global Read-only
You can think of the Global Read-only flag as quick means of locking down a repository, so that no commits will be accepted at any site.

Add Repo

Click the Add Repo button when you have entered all the required fields for the repository that you are adding. You can cancel the addition of the repository by clicking on the circular cross icon that appears on the left-hand side of the entry fields.

Filter Repositories

You can use this search box to filter the list of available repositories, useful if you're running with a large number of repositories.

Reset all stats

SVN MuliSite Plus captures basic repository stats for all the repositories placed under it's control. The stats for selected repositories are displayed on the dashboard.

Click the RESET ALL STATS button to blank all the repository statistics on a node. The action is not replicated and the stats that are stored on the other nodes will not be affacted.

Repair

The Repository Repair tool is used when a repository on one of your nodes has been corrupted or similarly requires repair or replacement. Selecting a repository to repair, the tool asks to you select a 'helper' node. This node will briefly stop replicating as it will be used to copy or rsync an up-to-date replica of the broken repository onto the current node.

Sync Stop

The Sync Stop tool lets you bring replication to a stop for a selected repository. The tool is required to ensure that when replication has stopped all repository replica remain in exactly the same state. This requirement is complicated within distributed systems where proposals may be accepted on some nodes while still in-flight to other nodes. See Performaing a synchronized stop

1.3 REPLICATION GROUPS

Replication Groups are units of organization that we use to manage replication of certain repositories between certain sites. In order to replicate a SVN repository between two or more nodes, you would need to associate by adding them to a replication group:

1.3.1 Example replication groups

An organization with developers working in Chengdu and San Fransisco need to collaborate on projects stored in two SVN repositories, Repo1 and Repo3. An administrator in the Chengdu office creates a replication group called "DragonGroup". The SVN MultiSite Plus nodes corresponding with each of the two sites are added to the group.

Replication Group Example 1

The Chengdu office is the location of largest development team, where most repository changes occur. For this reason the node is assigned the role of Tie-breaker. In the event that there is disagreement between the nodes in the group over transaction ordering, NodeChengdu will carry the deciding vote.

The node in San Fransisco hosts a standard active node. Changes to the local repository are replicated to NodeChengdu, changes made on the Chengdu node are replicated back to San Fransisco.

In addition to the two active nodes, a third node, NodeParis is added, located at a management site that plays no active part in development. However, the node has been added to the group as a "pure voter". This means that NodeParis takes part in the vote for transaction ordering, even though the payload of those transactions are not written to repository replicas stored at the Paris office. The purpose of NodeParis is simply to add resilience to the replication system, in the event of a short-term disruption to traffic from one of the other two nodes, agreement can still be reached and replication could continue.

The organization might choose to make the Paris node "Passive" instead. With ParisNode running a passive node, replicas of Repo1 and Repo3 would also be stored in Paris, although these copies could only be changed by updates from the other (active) repository copies, they wouldn't be accessible to SVN clients.

Types of node

There's another element controlled by replication groups, the role that each repository replica plays the replication system, where each replica may be casting a vote in order to determine the correct order in which proposals are played out. See more about Types of Sites.

1.3.2 Create replication group

You can create a replication group providing that you have at least two sites connected.

1. Click Create Replication Group.
   

Replication Groups



2. Enter a Group Name, it can be anything but it's a good idea to keep it simple.
   

Group names



3. Select from the available sites. As you select sites they'll be listed in the add node's field.

   Add some sites - you need a minimum of two

The local node is automatically made the first member
It's not possible to create a replication group remotely, the initiating node is alsways added automatically.
4. Added sites are by default given the role Active Voter. Click on a node's label to bring up a list of available roles. See Types of Sites.
   

Replication groups

5. When you have added all the nodes that you need, click on the CREATE REPLICATION GROUP. Replication groups are listed in boxes on the Replication Group tab.

View

You can view and partly edit a replication group by clicking on the button.

1. Click on the Quick View button.
   
2. The replication group's screen will open showing the member nodes of the group.
   
3. Each node is desplayed as a color-coded circle. Click on the circle to see what other node types are available. Read more about node types.
   
4. The Configuration screen provides access to the each node's type, along with a list of repositories and a link to the Configure Schedule screen.
   

Add Nodes

It's possible to add additonal nodes to a replication group. Click on the Add Nodes button to start the procedure, you can read about it in the Admin Guide - 13. Adding a node to a replication group

Save Node Roles

Use this button to save any changes that you make to the member nodes.

Configure Schedule

The Schedule screen lets you set the roles of sites to change over time, specifically changing according to a schedule.

Why change a node's role?

At the heart of WANdisco's DConE2 replication technology is an agreement engine that ensures that SVN operations are performed in exactly the same order on each replica, on each node. Any node that has the role of voter becomes part of the agreement engine and together with other voters determine the correct ordering. If there's high latency between any voters this may adversely affect replication performance. Fortunately it isn't a requirement that every site takes part in forming agreement. An Active site can still create proposals (i.e. initiate repository changes) but the agreement engine doesn't need to wait for its vote. Read more about how replication works in the Reference Section.

Follow the Sun

So in order to optimize replication performance it's common for administrators to remove voter status from node after their staff leave for the day - a practice commonly known as "Follow The Sun" where far-flung organisations transfer roles and privileges between locations so that they are always held by an active, manned site.

Role Schedule

The Role Schedule window shows all the sites in the replication group, along with each node's current roll (denoted by a circular buttons). It's possible to click on a node and change its role as a Schedule Manager.

If you change any sites you need to click the SAVE SITE ROLES button. to ensure that your changes are saved.
There's an indicator as to whether a Role Schedule has been enabled on the group. If there's a schedule in place the indicator will show ENABLED, otherwise it will show DISABLED.

Clicking on the indicator will open up the Schedule screen, the same as if you clicked on the CONFIGURE SCHEDULE button.

Schedule

A drop-down selector for viewing a particular day of the week. This filter is only for viewing an established schedule, it isn't used to select the day that you intend to schedule.

Clear Schedule

Use this button to blank out settings you have changed, returning to the default schedule.

Save Schedule

After making any changes to the schedule, you need to click the SAVE SCHEDULE to apply them. Next to the button is a reminder of the server's timezone, this is derived from the server's system clock, not the location information provided during installation.

To see how to setup a schedule, see the SVN MultiSite Plus Admin Guide - How to configure a schedule

1.4 SETTINGS

The server's internal settings are reported on the Settings tab, along with a number of important editable settings.

Administrator Settings

User Interface Ports

Change the ports that you want to use to access the User Interface. Enter valid port numbers and click SAVE.

Shutdown Replicator / Restart Replicator

If you want to shut down the replicator, click Shutdown Replicator. Note the message you receive:

This action will shutdown the Replicator process.
 The Replicator can only be restarted from the command line and cannot be restarted through the User Interface.

If you shut down from this button, you cannot restart using the UI as the restart command only works if the replicator is currently running. However, you can stop and restart the replicator process by clicking Restart Replicator. You receive a message saying this takes about 15 seconds. Click Restart to confirm.

Resource Monitoring Data

The Resource Monitoring Data settings provide a basic tool for monitoring available disk storage for MultiSite's resources.

Monitor Interval (mins)

If the disk space available to a monitored resource is less than the value you have for a "Severe" event then the event is logged and MultiSite's replicator will shut down after this interval, currently set at 10 minutes by default. You can configure the interval in application.properties file:

/opt/wandisco/svn-multisite-plus/replicator/properties/application.properties

monitor.period.min=10L

Value is in minutes, and only run through the UI, it is not handled directly by the replicator.

Add New Resource Monitor

Enter the path to a resource that you wish to monitor, then click "Add".

Resource Monitors

This section lists all resources currently being monitored. Click on "Configure" to change monitor settings, "Delete" to remove a monitor.

For more information about setting up monitors, read 22. Setting up resource monitoring.

Notifications

The notifications system provides SVN administrators with the ability to create event-driven alert emails. Set up one more more gateway (mail servers), add destination emails to specify recipients, create email templates for the content of the alert emails, then set the rules for which event should trigger a specific email.

Gateways

The Gateways section stores the details of those email relay servers that your organization uses for internal mail delivery. You can add any number of gateways, SVN MultiSite Plus will attempt to delivery notification emails using each gateway in their order on the list, #0, #1, #2 etc.

SVN MultiSite Plus will attempt delivery via the next gateway server once it has attempted delivery a number of times equal to the Tries number. It will repeat a try after waiting a number of settings equal to the Internval setting.

IP/Hostname of SMTP Server:

your email server's address.

SMTP Server Port

The port assigned for SMTP traffic (Port 25 etc).

Encryption Type

Indicate your server's encryption type - None, SSL (Secure Socket Layer) or TLS (Transport Layer Security). SSL is a commonly used. For tips on setting up suitable keystore and truststore files see Setting up SSL Key pair.

If you're not familiar with the finer points of setting up SSL keystores and truststores it is recommended that you read the following article: Using Java Keytool to manage keystores.

Authentication Required

Indicate whether you need a username and password to connect to the server - requires either "true" or "false".

User Name

If authentication is required, enter the authentication username here.

Password

If authentication is required, enter the authentication password here.

Sender Address

Provide an email address that your notifications will appear to come from. If you want to be able to receive replies from notifications you'll need to make sure this is a valid and monitored address.

Number of Tries Before Failing

Set the number of attempts SVN MultiSite Plus makes in order to send out notifications.

Interval Between Tries (Seconds)

Set the time (in seconds) between your server's attempts to send notifications.

Destinations

In the Destinations panel is used to store email address for notification recipients. Add, Edit or remove email addresses.

Templates

The templates panel is used to store email content. You create messaging to match those events for which you want to send user notifications.

Template Subject

Use this entry field to set the subject of the notification email. You'll want this subject to be descriptive of the event for which the email will be triggered. .

Body Text

Enter the actual message that you want to send for a particular situation/event.

Rules

Use the Rules panel to actually setup up your notification emails. Here you'll associate email templates and destination emails with a particular system event. For example, you may create an email message to send to a particular group mailing list in the event that a repository goes into Read-only mode. Selecting descriptive subjects for your templates will help you to select the right templates here.

Event

Choose from the available list of trigger events.

Template

Destination

Logging Setting

The Logging Setting lets you quickly add or modify Java loggers via the admin console, rather than making manual edits to the logger file:

<install-dir>/replicator/properties/logger.properties.

Loggers are usually attached to packages, here, the level for each package is available to modify or delete. The global level is used by default, so changes made here are used to override the default values. Changes are applied instantly but in-memory only and are forgotten after a restart of the replicator (unless they are saved). For information about adding or changing loggers, see Logging Setting Tool.

com.wandisco.fsfs.logger.FSFSFileHandler

The default logging package. This cannot be deleted.

All

All levels including custom levels.

Debug

Detailed information on events that are most useful to debug the application.

Info

Informational messages that highlight progress.

Warning

Potentially harmful situation.

Severe

Very severe error events that will probably lead the application to abort.

Reload All Settings From File

Ditches all changes by reloading the logger settings from the <install-dir>/replicator/properties/logger.properties file.

Save All Settings To File

Applies your changes to the above logger.properties files.

System Data

The System Data table provides a list of the editable and read-only settings.
Changing the editable settings causes a replicator restart:

Node Name

This is the human-readable form of the node’s ID. You can change the Node Name and reuse it after it has been removed from the replication network. You cannot have two nodes with the same name, but you can reuse a previously removed node name.

Location Latitude

The Node’s geographical location is no longer recorded during installation. Instead you enter the details here.

Location Longitude

Along with Longitude, this value places the node on the internal map and helps the application determined the local time for the node based on the timezone in which it falls.

Host Name/IP Address

The Hostname / IP address of the server hosting the node.

DConE port

DConE port (actually it's DConE 2) handles agreement traffic between sites. The Default is 6444.

Dashboard Polling Interval (Minutes)

Sets how often the dashboard messaging is updated. The messaging is populated by Warnings and Errors that appear in the replicator logs file. The default frequency is every 10 minutes.

Dashboard Item Age Threshold (Hours)

The number of hours a logged event is displayed on the Dashboard for. We recommend not setting this value lower than 96 hours so you don’t miss an important issue over a 3-day weekend.

Number of Revisions in Default Consistency Check

The number of revisions to compare between nodes. Ensure this number is the same between nodes before starting a check.

Scheduled Consistency Check Enabled?

Tick this box to schedule a consistency check.

Scheduled Consistency Check Frequency (Hours)

The default is 24 hours so repositories are checked for consistency once a day. The allowed value however is 1-999.

The read-only settings were either provided during setup or have since been applied:

Node ID

A unique string that is used to identify the node during an induction.

Location ID

A unique string that is used to identify the server during an induction.

Database Location

The full path to MultiSite's database. By default this will be <install-dir>/replicator/database.

Delegate Port

The delegate port is used by SVN to delegate write operations to the WANdisco Replicator (via the contact.server.port (described below).

Jetty HTTP Port

The HTTP port is used for browser access to the User Interface.

Content Server Port

The port that will be used to transfer replicated content (repository changes). This is different from the port used by WANdisco's DConE agreement engine.

Content Location

The directory in which replication data is stored (prior to it having been confirmed as replicated).

License information

Details concerning SVN MultiSite Plus product license - such as the date of expiry.

View REST API Documentation - This link takes you to your node's local copy of the API documentation. The link goes to the following location: http://<Node IP>:8080/apidocs/. This documentation is generated automatically and ties directly into your server's local resources. There is an only copy of the latest API documenation available in this admin guide, note though that it has been lifted from an installation and will link to resources that will not be available on the website (resulting in dead links).

Module Versions

The module versions provides a list of the component parts of the SVN MultiSite application. This is useful if you need to verify what version of a component you are using - such as if you need to contact WANdisco for support.

1.5 SECURITY

The security tab is used to manage admin accounts, either entered manually into SVN MultiSite Plus or managed through an LDAP authority. On the tab is an entry form for multiple admin accounts, along with LDAP Settings for binding MultiSite to one or more LDAP services.

SVN MultiSite Plus - Security

Add User

Enter the details of an additional administrator who will be able to login to the SVN MultiSite Admin UI. See Adding additional users for more information.

Add Authority

Enter the details of one or more LDAP authorities for managing administrator access. See Adding LDAP authorities for more information.

Disable Managed Users

This feature lets you block access to the SVN MultiSite Admin UI by non-LDAP users. See Disabling (Internally) Managed Users below.

Enable SSO

This button will only be available to click if you have entered valid Kerberos settings. When enabled it places SVN MultiSite Plus's admin console into Single Sign-on mode. When enabled accessing the admin UI will use Kerberos instead of the username and password login form. In the enabled state the button will change to say Disable SSO.

Export Security Settings

The data entered into the Securities tab can be backed up for later re-importing by clicking the Export Security Settings button. The data is stored in /opt/wandisco/svn-multisite-plus/replicator/export/security-export.xml which should be included in any backup procedures you are running. You will need access to the file from your desktop during a re-import.

Import Security Settings

Click the Import Security Settings button if you need to restore your Security settings, such as after a re-installation of SVN MultiSite Plus. The import will proceed providing that you can enter a file path to the security-export.xml file.

Reload

Click on the reload button to refresh the Admin UI screen, you will need to do this in order to view any changes that you make.

Admin Account Precedence

SVN MultiSite Plus uses the following order of precedence when checking for authentication of users:

• First: Internally managed users (if they are enabled - see Disable Managed Users)
• Second: Local LDAP authorities by order
• Third: Global LDAP authorities by order

Explanation

This provider implementation tries to authenticate user credentials against either the list of internally managed users, or against any number of LDAP authorities, or both -- depending on how the administrator has configured the application.

When authenticating against LDAP authorities, each one is tried in sequence until one either grants access or they all deny access. In the event that they all deny access, only the error from the last authority tried will be returned.

Disable (Internally) Managed Users

Click the Disabled Managed Users button if you want to control access to SVN MultiSite Plus exclusively through LDAP. Once clicked, any Internally managed users will no longer be able to log into the Admin UI after they next log out. From that point only LDAP managed users will have access to the SVN MultiSite Plus Admin UI.

/opt/wandisco/svn-multisite-plus/replicator

java -jar resetsecurity.jar

Internally Managed Users

SVN MultiSite Plus - Internally Managed Users

This table lists those admin users who have been entered through the Admin UI or imported using the Import Security Settings, along with the first admin account.

Admin Account #1

Note that the first admin account is the one set up during the installation of your first node. The credentials specified during this installation are stored to the users.properties file which is then used during the installation of all subsequent nodes.

Admin Account #1 can be removed but the last admin account remaining on the system will not be deletable to ensure that it isn't possible for an administrator to be completely locked out of the admin UI.

Kerberos

Support for the Kerberos protocol is now included. When enabled, Kerberos handles authentication for access to the admin UI, where the administrator is automatically logged in if their browser can retrieve a valid Kerberos ticket from the operating system.

kerberos settings entry form

Service Principal

A service principal name (SPN) is the name that a client uses to identify a specific instance of a service.

Example:

HTTP/host.example.com

Keytab File

The keytab is the encrypted file on disk where pairs of Kerberos principals and their keys are stored.

Example:

/tmp/krb5.keytab

Kerberos 5 Realm Configuration File

The krb5 configuration file location of the replicator host's Kerberos 5 realm configuration.

Example:

/etc/krb5/krb5.conf

/etc/krb5.keytab

LDAP Authorities

LDAP Authority entry forms

Node-Local LDAP Authorities

If chosen, then only the local node will use the LDAP authority for authentication.

Replicated LDAP Authorities

If replicated is chosen, all nodes in the replication network can use the LDAP authority for authentication.

Mixing Local and WAN-based authorities

Both kinds of authority are supported simultaneously, with the node-specific LDAP authorities taking precedence over WAN-based authorities in order to support the use-case where, for example, a particular node may prefer to use a geographically closer LDAP directory. Also, if multiple LDAP authorities of either type are configured then the order in which they are consulted is also configurable, using the +/- buttons at the end of each entry.

Order

LDAP authorities are listed in the order of execution that you set when defining each authority's properties.

Url

The URL of the authority. The protocol "ldap://" or "ldaps://" are required.

Bind User DN

Identify the LDAP admin user account that SVN MultiSite will use to query the authority.

Search Base

This is the Base DN, that is the location of users that you wish to retrieve.

Search Filter

A query filter that will select users based on relevant LDAP attributes. For more information about query filter syntax, consult the documentation for your LDAP server.

Remove

Click to remove the authority from SVN MultiSite Plus.

Edit

Click to make changes to the authority's settings.

The usual configuration options are supported for each configured LDAP authority: URL, search base and filter and bind user credentials. Note that the bind user's password cannot be one-way encrypted using a hash function because it must be sent to the LDAP server in plain text, so for this reason the bind user should be a low privilege user with just enough permissions to search the directory for the user being authenticated. Anonymous binding is permitted for those LDAP servers that support anonymous binding.

LDAP Home or away

When adding an LDAP authority, the configuration can be selected to be either replicated or node-specific.

Replicated LDAP Authorities

If node-specific is chosen, then only the local node will use the LDAP authority for authentication. Both kinds of authority are supported simultaneously, with the node-specific LDAP authorities taking precedence over WAN-based authorities in order to support the use-case where a particular node may prefer to use a geographically closer LDAP directory, for example. Also, if multiple LDAP authorities of either type are configured then the order in which they are consulted is also configurable.

The usual configuration options are supported for each configured LDAP authority: URL, search base and filter and bind user credentials.

2. Architecture overview

The diagram below outlines the SVN MultiSite Plus architecture, in term of how the application is split up and how those component parts communicate with each other and the outside world.

Key points

3. Install directory structure

SVN MultiSite Plus is installed to the following path by default:

/opt/wandisco/svn-multisite-plus/

Inside the installation directory you'll find the following files and directories:

[DIRECTORY] drwxr-xr-x 2 wandisco wandisco 4096 Dec  5 01:16 bin
        -r-xr-xr-x 1 root root 9130 Feb 17 17:35 backup
        -r-xr-xr-x 1 root root 1630 Feb 17 17:35 rollback
        -r-xr-xr-x 1 root root 1569 Feb 17 17:35 svn-multisite-plus
        -r-xr-xr-x 1 root root 12571 Feb 17 17:35 talkback
        -rwxr-xr-x 1 root root 3764 Feb 17 17:35 watchdog

[DIRECTORY]drwxrwxr-x 2 wandisco wandisco 4096 Dec  5 13:21 config
        -rw-r--r-- 1 wandisco wandisco 240 Feb 18 17:33 main.conf

[DIRECTORY] -rw-rw-r-- 1 wandisco wandisco  256 Dec  5 13:56 lib
        -rw-r--r-- 1 root root 7142 Feb 17 17:35 init-functions.sh

[DIRECTORY] -rw-rw-r-- 1 wandisco wandisco  256 Dec  5 13:56 local-ui
        -rw-r--r-- 1 wandisco wandisco 1049 Feb 18 17:36 ui.properties

[DIRECTORY] drwxrwxr-x 2 wandisco wandisco 4096 Dec  5 14:20 logs
   
        -rw-r--r-- 1 wandisco wandisco    88 Feb 18 17:35 multisite.log
        -rw-r--r-- 1 wandisco wandisco   220 Feb 18 17:35 replicator.20140218-173532.log
        -rw-r--r-- 1 wandisco wandisco 15822 Feb 18 17:35 ui.20140218-173305.log
        -rw-r--r-- 1 wandisco wandisco  1902 Feb 18 17:35 watchdog.log
   

[DIRECTORY] drwxr-xr-x 8 wandisco wandisco 4096 Dec  5 14:20 replicator
[DIRECTORY]drwxr-xr-x 2 wandisco wandisco   4096 Feb 18 17:33 content
[DIRECTORY]drwxr-xr-x 5 wandisco wandisco   4096 Feb 18 17:35 database
[DIRECTORY]drwxr-xr-x 4 root     root       4096 Feb 18 17:33 docs
[DIRECTORY]drwxr-xr-x 2 wandisco wandisco   4096 Feb 18 17:33 export
[DIRECTORY]drwxr-xr-x 6 root     root       4096 Feb 18 17:33 gfr
[DIRECTORY]drwxr-xr-x 2 root     root       4096 Feb 18 17:33 lib
[DIRECTORY]drwxr-xr-x 5 wandisco wandisco   4096 Feb 21 10:12 logs
[DIRECTORY]drwxr-xr-x 2 wandisco wandisco   4096 Feb 18 17:35 properties
[DIRECTORY]drwxr-xr-x 2 root     root       4096 Feb 18 17:33 properties.dist

        -rwxr-xr-x 1 root     root       7679 Feb 17 17:35 resetSecurity.jar
        -rw-r--r-- 1 root     root     315294 Feb 17 15:49 svn-ms-replicator-fsfsrestore.jar
        -rw-r--r-- 1 root     root     315280 Feb 17 15:49 svn-ms-replicator.jar
        -rw-r--r-- 1 root     root     315292 Feb 17 15:49 svn-ms-replicator-updateinetaddress.jar
        -rwxr-xr-x 1 root     root      23731 Feb 17 17:35 transformer-tool.jar
        -rw-r--r-- 1 root     root        605 Feb 17 15:49 VERSION-TREE

[DIRECTORY] drwxr-xr-x 3 root   root   4096 Oct 17 15:29 resources

 
[DIRECTORY]drwxr-xr-x 2 root root     4096 Oct 17 15:28 svn

4. Properties files

The following files store application settings and constants that may need to be referenced during troubleshooting. However, you shouldn't make any changes to these files without consulting WANdisco's support team.

/opt/wandisco/svn-multisite-plus/replicator/properties/application.properties

file contains settings for the replicator and affects how MultiSite performs. [view sample]

Temporary requirement:
If you (probably under instruction from WANdisco's support team) manually add either connectivity.check.interval or sideline.wait to the applications property file then you must add an "L" (Long value) to the end of their values so they are converted correctly. View our sample application.properties file to view all the properties that are suffixed as "Long".

/opt/wandisco/svn-multisite-plus/replicator/properties/logger.properties

handles properties that apply to how logging is handled. [view sample]

/opt/wandisco/svn-multisite-plus/replicator/properties/users.properties

contains the admin account details which will be required when installing second and subsequent nodes. [view sample]

/opt/wandisco/svn-multisite-plus/local-ui/ui.properties

contains settings concerning the graphical user interface such as widget settings and timeout values. Stored in this file is the UI Port number and is considered the defacto recording of this value, superceding the version stored in the main config file /opt/wandisco/svn-multisite-plus/config/main.conf. You can [view a sample]

5. Replication strategy

SVN MultiSite provides a toolset for replicating SVN repository data in a manner that can maximise performance and efficiency whilst minimising network and hardware resources requirements. The following examples provide you with a starting point for deciding on the best means to enable replication across your development sites.

5.1 Replication Model

In contrast with earlier replication products, SVN MultiSite Plus is no longer based upon a network proxy that handles file replication between replica. Now, replication is handled at the filesystem level, via FSFS.

SVN MultiSite Plus differs from earlier WANdisco replication products on a number of levels.

Limitations of the old model

• The replicator sits in front of Apache
• All reads have to pass through the replicator
• Requires some functionality to be added so that certain operations can be performed prior to scheduling a transaction for replication
• Authentication against external data stores
• Hooks firing

SVN MultiSite Plus differs from earlier versions in that it replicates at the file system level.

5.1.1 Per-Repository Replication

SVN MultiSite is able to replicate on a per-repository basis. This way each site can host different sets of repositories and replicate repositories this means that you can have different repositories replicate as part of different replication groups.

5.1.2 Dynamic membership evolution

No need for a synchronized stop - SVN MultiSite Plus allows replication groups to change their membership on-the-fly.

A repository can only replicate to a single replication group at any one time, although it is possible to move between replication groups as required - this can now be done on-the-fly, nodes can be added or deleted without the need to pause all replication (with a synchronized stop)

SVN MultiSite Plus offers a great deal of flexibility in how repository data is replicated. Before you get started it's a good idea to map out which repositories at which locations you want to replicate.

5.1.3 WANdisco replication and compression

There are a number of WAN network management tools that offer performance benefits by using data compression. The following guide explains how data compression is already incorporated into WANdisco's replication system, and what effect this built-in compression may have on various forms of secondary compression.

Network management tools may offer performance benefits by on-the-fly compression of network traffic, however it's worth nothing that WANdisco's DConE replication protocol is already using compression for replicated data. Currently Zip compression is used before content is distributed using the Content Distribution component of DConE.

Traffic Management systems that provide WAN optimization or "WAN Acceleration" may not provide expected benefits as a result of WANdisco's compression. The following list highlights where duplication or redundancy occurs.

Compression

Encoding data using more efficient storage techniques so that a given amount of data can be stored in a smaller file size.
WANdisco effect: As replicated data is already compressed, having a WAN accelerator appliance compress the data again is a waste of time - however, as long as it can "fill the pipe", i.e. keep the throughput of traffic faster rate than the network can consume it then its not going to negatively impact data transfer.

Deduplication

Eliminating the transfer of redundant data by sending references instead of the actual data. By working at the byte level, benefits are achieved across IP applications. In truth, Data-deduplication offers the most benefit when there's a lot of repetition in the data traffic.
WANdisco effect: Because the data is already compressed then data-deduplication (by whatever WAN optimization solution) will not be effective, when data is compressed any small change at the start of the data stream propagates through the data stream and defeats data-deduplication. You can read more about this effect in this external article - rsyncable-gzip

Latency optimization

Various refinements to the TCP implementation (such as window-size scaling, selective Acknowledgement etc.
WANdisco effect: DConE does not use TCP / network layer techniques. This form of optimization won't have any impact on WANdisco Replication.

5.2 Creating resilient replication groups

SVN MultiSite Plus can maintain SVN repository replication (and availability) even after the loss of nodes from a replication group. However, note these configuration rules:

Rule 1: Understand Node roles

The unique Active-Active replication technology used by SVN MultiSite Plus is an evolution of the Paxos algorithm. Different node roles are set by particular mechanics at play in the replication system:

• Active/Pass Nodes:
  Active/Passive nodes are required for the storage and replication of repository data. You need these at any location where SVN users are working or where you wish to store hot backups of repositories. Active/Passive nodes learn proposals from other nodes and take action accordingly, for example, updating repositories based on proposals (i.e. replication).
  




• Active Voter/Voter-only Nodes:
  All changes being made on each repository in exactly the same order is a crucial requirement for maintaining synchronization. All voter nodes that take part in the vote for the order in which proposals are played out.
  Voters are required for keeping replication going. You need enough of them to ensure that agreement over proposal ordering can always be met, even accounting for possible node loss. For configurations where there are an even number of voters, voting could be tied. Therefore, you can make a voter node into a tie-breaker which has slightly more voting power so that it can outvote any other single voter node.

Rule 2: Replication groups should have a minimum membership of three Active/Passive nodes

Two-node replication groups are not fault tolerant, you should strive to replicate according to the following guideline:

• The number of active/passive nodes required to survive population loss of N nodes = 2N+1
  where N is your number of nodes.

  So in order to survive the loss of a single node you need to have a minimum of 2x1+1= 3 nodes
  In order to keep on replicating after losing a second node you need 5 nodes.

Rule 3: Learner Population - resilience vs rightness

• During the installation of each of your nodes you are asked to provide a Content Node Count number, this is the number of other active/passive nodes in the replication group that need to receive the content for a proposal before the proposal can be submitted for agreement.

  Setting this number to 1 ensures that replication won't halt if some nodes are behind and have not received replicated content yet. This strategy reduces the chance that a temporary outage or heavily loaded node will stop replication, however, it also increases the risk that repositories will go out of sync (requiring admin-intervention) in the event of an outage.

Rule 4: 2 nodes per site provides resilience and performance benefits

Running with two nodes per site provides two important advantages.

• Firstly it provides every site with a local hot-backup of the repository data.
• Enables a site to load-balance repository access between the nodes which can improve performance during times of heavy usage.
• Providing the nodes are Voters, it increases the voter population and improves resilience for replication.

5.3 Content distribution policy

WANdisco's replication protocol separates replication traffic into two streams, the coordination stream which handles agreement between voter nodes, and the content distribution stream through which SVN repository changes are passed to all other nodes (that is active/passive nodes that store repository replicas).

Content in this setting is the the data required for an agreement to be delivered. The content distribution policy determines how many other nodes must have this content before the voting process is initiated. Without this, if an agreement is scheduled and the node(s) that have the content are lost before the other nodes can obtain that content then a disaster recovery will be required as the content is no longer available.

3 Paxos roles are used in content distribution:

• Acceptor - votes on proposals
• Proposer - creates proposals, resolves proposal conflicts (e.g. propose to make a change to a repository)
• Learner - delivers/executes proposals (e.g. updates a repository) - node with a repository replica

Proposals turn into agreements when they have sufficient votes from acceptors. The agreements are in a fixed order relative to each other and are delivered at each node in that fixed order.

SVN MultiSite Plus lets you apply different policies to content distribution on a per-node basis.

Contact WANdisco support if you have any questions about the Content Distribution policy.

5.3.1 Changing content distribution policy

In SVN MultiSite Plus there are 2 settings that govern the behavior of the Content Distribution Policy. Their names and defaults are:

content.min.learners.required=true
content.learners.count=1

These settings are modified on a per-node basis via the application.properties file.

/opt/wandisco/svn-multisite-plus/replicator/properties/application.properties

A restart of the application is required after any change is made to the application.properties file.

Reliable Policy

The "Reliable Policy" is the default setting.

content.push.policy=reliable

The content.learner.count represents the number of learner nodes excluding the originating node that must have the data before any repository change will be put to the vote.

• If content.learner.count is larger than the number of non-originating replicas then it will automatically be reduced to the number of non-originating replicas.

If content.min.learners.required is true and there are an insufficient number of available replicas (based on the content.learner.count value) then the repository modification will fail without being put to a vote.

If content.min.learners.required is false then the value of content.learner.count will be adjusted to the number of non-originating available replicas. However, if the number of non-originating available replicas is zero and content.learner.count is non-zero then the repository modification will fail.

The number of simultaneous failures that can occur without requiring disaster recovery is strictly governed by the content.learner.count value. The default number is 1 so either the originating node OR the non-originating node that had the content delivered could be lost, but not both, before disaster recovery would be necessary. If both nodes in this case were down for maintenance then the other nodes would be stalled until one of the 2 nodes that have the content are once again available - at least for that repository family.

• During an outage there are only 4 learner nodes available in the replication group - requests to modify the repository will fail because there aren't enough available learner nodes to validate a content distribution policy check.

• During an outage there are now only 4 learner nodes in the replication group - requests to modify the repository will be successful because SVN MultiSite Plus will automatically drop the required learner count to ensure that the required learner count doesn't exceed the total number of learner nodes in the group.

5.3.2 Steps for a policy change

Use this procedure to change between the above Content Distribution policies.

1. Make a back up and then edit the /opt/wandisco/svn-multisite-plus/replicator/properties/application.properties file (Read more about the properties files).


2. Change the value of content.min.learners.required, make it "true" for reliability, "false" for speed (default is true).


3. Save the file and perform a restart of the node.

5.3.3 Set policy on a per-state machine basis

When editing the property, add the state machine identity followed by .content.push.policy. e.g.

<machine_identity_name>.content.push.policy=reliable

The system assigns policy by looking up the state machine policy followed by content.push.policy. If none are available, "reliable" is chosen.

content.thread.count

Content Distribution will attempt parallel file transfer if there are enough threads available. The number of threads is controlled by a configuration property content.thread.count which is written to the application.properties file.

content.thread.count=10

The default value is 10. This provides plenty of scope for parallel file transfer. However, as each thread consumes system overhead in the form of a file descriptor and some memory space, servers that are under regular heavy load should lower the count to 2.

5.3.4 Change the content maximum idle time

content.max.idle.time=2147483647

Set this in milliseconds. If content connection, either push or pull, is idle for this time, it is considered unreliable and closed. A new connection is then opened when needed. TCP/IP itself does not time-out the connections, however many network components (routers and firewalls) do. This timed-out connection then can behave as dead-hole, which blocks writes for tens of seconds timeouts. This can lead to spikes in transmission (push or pull) times after a period of inactivity (or even during activity if the number of connections is large and under-utilised).

Set the content.max.idle.time to, for example, 10 minutes, or whatever expiration the network infrastructure uses. We recommend that you set this to whatever your routers/firewalls are set to. This can avoid delays. If you set the value too low, connections may be closed unnecessarily and cause delays on new connection creation (roughly 1 RTT, but more for ssh connections).

You should set, or lower, this value if you get a large number of "Failed to send" info logs from PrioritizingSender, occurring especially after some time of commit inactivity.

5.3.5 Set the memory chunk size

content.in.memory.chunk.size=16K

For file transfer, compression or zip, a chunk is read from the physical disk. This property defines the chunk size in bytes. The default is 16K. Note: If you make the size bigger, then you need a bigger heap space.

5.4 Replication lag

There are some time-sensitive activities where you need to work around replication lag. For example:

1. You have a 2-node replication group, NodeA and NodeB, and Repository Repo01 is replicated between them.
2. A commit to NodeA puts Repo01 at Revision N. The proposal for this commit is agreed but NodeB is still waiting for the changes to arrive so lags slightly behind NodeA at revision N-1.
3. A user on NodeB does an svn cp http://nodeB/repo01/trunk http://nodeB/repo01/tags/TAG_X.
4. This tag does not include changes that occured in the latest revision. WANdisco's replication technology ensures that all nodes are in the same state in the short to medium term. However, at any moment changes may be in transit. A larger volume of traffic and less available network capacity increases this still in transit state.

This lag is unavoidable in a real-world application and all replica should soon be back in sync.

6. Guide to node types