WANdisco
 Navigation:  v1.6.1.2 Build eba41260 | Release Notes | Install | Upgrade | Administration | Reference | Gerrit | API | Glossary | Archive

Administrator Guide

1. Running Git MultiSite

This guide describes how to use Git MultiSite (GitMS).

1.1 Start up

To start the GitMS replicator:

  1. Open a terminal window on the server and log in with suitable file permissions.
  2. Run the git-multisite service, located in the init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 git-multisite -> /opt/git-multisite/bin/git-multisite
  3. Run the start script:
    [root@localhost init.d]#  ./git-multisite start
    20130520-164811 (24088) [INFO]: Starting WANdisco MultiSite 20130520-164811 (24088) [INFO]: Started replicator (24100) 20130520-164811 (24088) [INFO]: Started ui (24110) 20130520-164811 (24088) [INFO]: Number of errors: 0 20130520-164811 (24088) [INFO]: Number of warnings: 0
  4. The two components of GitMS, the replicator and the UI, start up. Read more about the git-multisite init.d script.

1.2 Shut down

To shutdown:

  1. Open a terminal window on the server and log in with suitable file permissions.
  2. Run the git-multisite service, located in the init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 git-multisite -> /opt/multisite-plus/bin/git-multisite
  3. Run the stop script, i.e.:
    
    [wandisco@ip-10-0-100-7 bin]$  ./git-multisite stop
    
    
    20130520-165704 (24767) [INFO]: Stopping WANdisco MultiSite
    20130520-165704 (24767) [INFO]: Request received to shut down replicator
    20130520-165704 (24767) [INFO]: replicator processes ended
    20130520-165704 (24767) [INFO]: Request received to shut down ui
    20130520-165704 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 1)...
    20130520-165707 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 2)...
    20130520-165710 (24767) [INFO]: ui processes ended
    20130520-165710 (24767) [INFO]: Number of errors: 0
    20130520-165710 (24767) [INFO]: Number of warnings: 0
    
          
  4. Both the replicator and the UI processes shut down. Read more about the git-multisite init.d script

1.3 Startup script

The startup script for persistent running of GitMS is in the /etc/init.d folder. Run the script with the help command to list the available commands:


[root@localhost init.d]# ./git-multisite help
    
  Usage: git-multisite {start|stop|status|uistart|uistop|repstart|repstart}

  start      Start WANdisco Multisite service
  stop       Stop WANdisco Multisite service
  status     Display running WANdisco Multisite services
  uistart    Start WANdisco Multisite UI service
  uistop     Stop WANdisco Multisite UI service
  repstart   Start WANdisco Multisite Replicator service
  repstop    Stop WANdisco Multisite Replicator service
    
  

1.4 Change the admin console password

You can change GitMS's login password at any time:

  1. Log in to the MultiSite admin console.
    Login

    Login

  2. Click the Security tab.
    Settings Screen

    Security

  3. On the security tab screen you see the Internally Managed Users table. Click the Edit link that corresponds with the Admin account. In the Edit User window that opens, enter a new password. Repeat the entry in the box immediately below.
    Password Change Form

    Changed password

  4. Click the SAVE button to store the new password. The new password has been accepted if you see a growl message on screen.
    Save button to store

    Growl

Changing Username
You cannot currently change the Administration username. To change the username you would need to re-install GitMS.

1.5 Update your license.key file

Follow this procedure if you need to change your product license, e.g. if you need to increase the number of Git users or the number of replication nodes.

  1. Log in to your server's command line, navigate to the properties directory: /opt/wandisco/git-multisite/replicator/properties and rename the license.key to license.20130625:
    
    total 16
    -rw-r--r-- 1 wandisco wandisco 1183 Dec  5 15:58 application.properties
    -rw-r--r-- 1 wandisco wandisco  512 Dec  5 15:05 license.key
    -rw-r--r-- 1 wandisco wandisco  630 Dec 17 15:43 logger.properties
    -rw-r--r-- 1 wandisco wandisco  630 Dec 17 15:45 log4j.properties
    	    
  2. Get your new license.key and drop it into the /opt/git-multisite/replicator/properties directory.
  3. Restart the replicator by running the GitMS script with the following argument:
    /etc/init.d/git-multisite restart
    This triggers a GitMS replicator restart, which forces GitMS to pick up the new license file and apply any changes to permitted usage.
If you don't restart
If you follow the above instructions but don't do the restart, GitMS continues to run with the old license until it performs a daily license validation (which runs at midnight). If your new license key file is valid and is in the right place, then GitMS updates its license properties without the need to restart.

If you have problems, check the replicator logs (/opt/git-multisite/replicator/logs) for more information.

PANIC: License is invalid com.wandisco.fsfs.licensing.LicenseException: Failed to load filepath. 

1.6 Update a node's properties

The System Data section of the Settings tab lists editable properties that you can quickly update by re-entering, saving, and allowing the GitMS replicator to restart. Note: This may cause a brief disruption for users because in-flight commits will fail.

Cha cha changes...

Editable node properties, involving replicator restart

Node Name
This is the human-readable form of the node's ID. You can change the value of Node Name and reuse it after it has been removed from the replication network. You cannot have two nodes with the same name, but you can reuse a previously removed node name.
Location Latitude
Enter the node's latitude here.
Location Longitude
Enter the node's longitude here.
Hostname / IP Address
Enter/update the hostname or underlying IP address.
DConE Port
The TCP port used for DConE agreement traffic. Do not confuse this with the Content Distribution port which carries the payload repository data.
Dashboard Polling Interval (Minutes)
Sets how often the dashboard messaging is updated. The messaging is populated by Warnings and Errors that appear in the replicator logs file. The default frequency is every 10 minutes.
Dashboard Item Age Threshold (Hours)
Sets how long dashboard messages are maintained. After this amount of time messages are flushed from the dashboard. The default is 96 hours (4 days).

After entering a new value, click Save. A growl message confirms that the change is being replicated. This results in a restart of the replicator which may cause brief disruption to SVN users.

1.6.1 Other property changes

You can also modify other properties in the configuration files.

Take care when making changes to "hidden" properties
An error can affect product behavior and be difficult to trace. In most situations, you should only make changes with the assistance of WANdisco's support team.

Content delivery port

To change the Content Delivery port:

  1. Use the command:
    content.port.<Node id>=<new port>
  2. When the file is in place, run the following command (on all the nodes except the one you have changed):
    java -jar git-ms-replicator-updateinetaddress.jar -c <path to application.properties>
  3. Go back to the node with the updated properties and Restart MultiSite.
  4. Log in to the updated node and check its System Data at the bottom of the Settings tab. Do some test commits to ensure that replication continues successfully.

Task garbage collection

Two configurable properties control how often the task garbage collection process runs. These properties are set during installation. To modify their values, add them to the application.properties file:

task.removal.interval
This controls how often the task garbage collection process should run. The default is 96 hours, noted in milliseconds (i.e. 345600000 milliseconds for 96 hours).
task.expired.interval
This controls how old a successfully run task must be before it is made available for garbage collection. The default is 96 hours, noted in milliseconds (i.e. 345600000 milliseconds for 96 hours).

Recommended Settings

Summary: For large deployments reduce the time from 96 to 24 hours
The recommended settings are suitable for most deployments. However, for deployments with very large numbers (thousands) of repositories and where repository consistency checks are automated, then we recommend that you reduce the setting times, initially to 24 hours (86400000 ms).

Shorter periods result in a corresponding reduction in your ability to troubleshoot problems that involve replicator task history. If you notice large numbers of failed tasks accumulating over time or have any concerns about what settings are right for your specific deployment, contact WANdisco's support team.

Example

For a deployment that replicates several thousand repositories and schedules daily consistency checks it's decided to reduce the task expiry to 48 hours and the garbage collection frequency to 24 hours. The settings would therefore be:


task.removal.interval   86400000L
task.expired.interval   172800000L
  
IMPORTANT!
Make sure that you add an "L" to the end of your value.

Node content distribution timeouts

Two configurable properties enable you to balance best possible performance against the tolerance of a poor WAN connectivity. The properties are in the application properties file, by default located in /opt/wandisco/git-multisite/replicator/properties/application.properties.

socket.timeout
socket.timeout=90000
This is the amount of time in milliseconds that the local node waits for the connection to be established before throwing an exception. The exception shows that it failed to connect within that timeout. Default value is 15 minutes (90,000 milliseconds).
Not less than 10 minutes!
DO NOT set socket.timeout to less than 10 minutes (60,000 milliseconds) or you may encounter problems.
content.pull.timeout
content.pull.timeout=300000
This sets how long the Content Distribution system waits for new content to be pulled fully over from a remote node. The default value is 5 minutes (300,000 milliseconds). This default is set with the assumption that there are no problems with the deployment's WAN connectivity.

1.7 Set up data monitoring

The Monitoring Data tool monitors the disk usage of GitMS's database directory, providing a basic level of protection against GitMS consuming all disk space. The tool also lets you set up your own monitors for user-selected resources.

Monitoring Data - not intended as a final word in system protection
Monitoring Data - not intended as a final word in system protection
Monitoring Data is no substitute for dedicated, system-wide monitoring tools. Instead, it is intended to be a 'last stand' against possible disk space exhaustion that could lead to data loss or corruption.

Read our Recommendations for system-wide monitoring tools.

1.7.1 Default settings

Resource Monitor Tool

Click the "View" link to go to a monitor's settings.


By default MultiSite's database directory (/opt/wandisco/git-multisite/replicator/database) is monitored - this is the location of MultiSite's prevayler database where all data and transactions files for replication are stored.

This built-in monitor runs on all nodes. Any additional monitors that you set up will monitor on a per-node basis. Monitors are not replicated so a monitor set up on one node is not applied to any other node.

1.7.2 Additional monitors

As well as GitMS's own database folder, there are several directories that might grow very large and potentially consume all available file space.

Consider monitoring the following MultiSite directories:

Also monitor /path/to/authz. If you are using Authz to manage authorization and your Authz file is situated on different file system from GitMS, then we recommend that you set up monitoring of the authz file.

For most deployments all these directories will reside on the same file system, so that our default monitor would catch if any of them were consuming the available space. However, there are two scenarios where we'd recommend that you set up your own monitor for the content directory:

1) You wish to set a higher trigger amount than the default monitor (1GiB for warning, 0.09GiB for emergency shutdown).
2) You have placed the content directory on a different filesystem with its own capacity that wouldn't be tracked by the default monitor.

In either case you should follow up the setting up of a monitor with a corresponding email notification that will be sent if some or all of your monitor's trigger conditions are met.

Create additional resource monitors using the following procedure:

  1. Log in to the Administrator user interface.
  2. Click the "SETTINGS" link on the top menu bar.
  3. Monitoring Data is situated below the Administrator Settings. Enter the full path to the resource that you wish to monitor. For example, you might wish to monitor the replicator logs: /opt/wandisco/git-multisite/replicator/logs. Enter the path and click "Add".
    Resource Monitor Tool

    Add resource path

  4. The new resource monitor appears as a new box - it will display "No records found", indicating that it doesn't yet have any monitoring rules set. Click its corresponding "Configure" link. Resource Monitor Tool

    Configure

  5. The screen will update to show the Resource Monitoring screen for your selected resource.
    Resource Monitor Tool

    Settings

    File Path:
    The full path for your selected resource
    Monitor Identity:
    The unique string that will identify the monitor
    Edit Condition and Event List
    Lists current resource monitors, initially this will state "No records found"
  6. Add Conditional and Event to list.
    Storage amount entry field
    Enter an amount of disk space in Gigabytes. e.g. 0.2 would be equal to 200 Megabytes of storage.
    Select an Event from the dropdown:
    SEVERE
    Initiates a shutdown of GitMS and will also write a message to the log and the "SEVERE" logging level. See "When a Shut down is triggered" for more information.
    WARNING
    Writes a message to the log and the "WARNING" level of severity.
    DEBUG
    Writes a message to the log and the "DEBUG" level of severity.
    INFO
    Writes a message to the log and the "INFO" level of severity.
  7. When you have added all the trigger points and events that you require for the resource, click "Update". You can then navigate away: Click "Resource Monitoring" on the breadcrumb trail to return to the settings screen.

1.7.3 When a shutdown is triggered

If the disk space available to a monitored resource is less than the value you have for a "Severe" event then the event is logged and MultiSite's replicator will shut down after a set interval of 10 minutes. You can configure the interval in the application.properties file in /opt/wandisco/git-multisite/replicator/properties/application.properties.

resourcemonitor.period.min=10L
Value is minutes.
Edits to property files require a replicator restart
Any change that you make to the application.properties file will require that you restart GitMS's replicator.

When shut down, all Git repositories become unavailable to users. You should immediately make more disk space available. The replicator can be restarted using GitMS's service as soon as the resource that triggered the shutdown has enough available disk space not to shut down again.

Overriding the forced shutdown
You may need to override the forced shutdown if you can't start a node to resolve the cause of the forced shutdown. For example, you might have created a data monitor that triggers a severe log message if there's less disk space than the disk's actual capacity. You then cannot free up space, apart from swapping for a bigger disk.

To unlock the forced shutdown.

  1. Log in to the locked node using a terminal.
  2. Navigate to the properties folder. By default this is here:
    /opt/wandisco/git-multisite/replicator/properties/application.properties
  3. Create a backup, then edit the file, changing the line:
    monitor.ignore.severe=false
    to say
    monitor.ignore.severe=true
    Save the change to the file.
  4. Restart the replicator (see Starting up). During the restart the replicator will now ignore the severe warning (which are still written to the log file) allowing you to delete the offending monitor.
    You cannot use this procedure to override the default monitor. Its emergency shutdown limit of <100MiB always shuts down the replicator.

1.8 Set up email notifications

Email notification is a rules-based system for delivering alerts (based on user-defined templates) over one or more channels to destinations that are based on triggers that are activated by arbitrary system events. Put simply, email notification sends out emails when something happens within the GitMS environment. The message content, trigger rules and destinations are all user-definable.

Email Notifications

Automated alert emails

Read about

1.8.1 Set up a gateway

The Gateway settings panel stores your email (SMTP) server details. You can set up multiple gateways to ensure that the loss of the server doesn't prevent alert notifications from being delivered.

  1. Log in to the admin UI, then click the Settings tab.
  2. Click the Gateway section of the Notifications area.
    Email Notifications

    Add Gateway

  3. Enter your email gateway's settings:
    Email Notifications

    Enter settings

    IP/Hostname of SMTP Server:
    your email server's address.
    SMTP Server Port:
    The port assigned for SMTP traffic (Port 25 etc).
    Encryption Type:
    Indicate your server's encryption type - None, SSL (Secure Socket Layer) or TLS (Transport Layer Security). SSL is a commonly used. For tips on setting up suitable keystore and truststore files see Setting up SSL Key pair.
    Keystores?

    If you're not familiar with the finer points of setting up SSL keystores and truststores it is recommended that you read the following article: Using Java Keytool to manage keystores.

    Authentication Required:
    Indicate whether you need a username and password to connect to the server - requires either "true" or "false".
    User Name:
    dd>If authentication is required, enter the authentication username here.
    Password:
    dd>If authentication is required, enter the authentication password here.
    Sender Address:
    dd>Provide an email address that your notifications will appear to come from. If you want to be able to receive replies from notifications you need to make sure that this is a valid and monitored address.
    Number of Tries Before Failing:
    Set the number of attempts GitMS makes to send out notifications.
    Interval Between Tries (Seconds):
    Set the time (in seconds) between your server's attempts to send notifications.
  4. Click the +Add button. Your gateway appears in the table.
    You can add any number of gateways. GitMS exhausts the "Number of Tries Before Failing" for each registered gateway before moving on down the list to the next. You can use the Test button to verify that your entered details will connect to a mail gateway server.

1.8.2 Set up a destination

The destinations panel stores the email addresses for your notification recipients.

  1. Click the Destinations line.
    Email Notifications

    Click Destinations

  2. Enter an email address for a notification recipient. Click the + Add link.
    Email Notifications

    Email addresses you target for alerting.

  3. The destination will appear in a table. Click the Edit or Remove links to change the address or remove it from the system.

1.8.3 Set up a template

The template section is used to store email messages. You can create any number of templates, each with its own notification message, triggered by one of a number of trigger scenarios that are set up in the Rule section.

  1. Click the Templates line.
    Email Notifications

    Click Templates

  2. Enter a Template Subject line, this will be the subject of the notification email.

  3. Enter some Body Text, this will be the message that is sent out when the notification is triggered. The message has a 1024 character limit, you can track the available number of characters at the bottom of the text box.
    Email Notifications

    The message you want to send out

  4. When the message has been entered, click the + Add link to save the message template.

Available variables

When writing email notification templates, you can insert variables into the template that will be interpolated when the notification is delivered. The following variables are available for ALL event types:

{node}
This returns the node name.
{timestamp}
This returns the time at which the event is received (not the time at which the notification is delivered).
{event}
This returns the raw dump of the event.

For the event types Disk Monitor Info, Disk Monitor Severe and Disk Monitor Warning:

{event.message}
This returns information about the disk monitoring threshold that was exceeded.
{event.resource.absolutePath}
This returns the monitored file path..

For the event types Deploy Repository Succeeded and Deploy Repository Failed, the following additional variables are available:

{event.proposerNodeId}
This returns the ID of the node that sent the event.
{event.repository.name}
This returns the repository name.
{event.originalProposal.repository.name}
This returns the user-specified name of the repository to which the event pertains.
{event.originalProposal.repository.fSPath}
This returns the location on-disk of the repository that the event pertains to. More events types and event variables will be added in the future.

1.8.4 Set up a rule

Use the Rules section to define the system event that should trigger a notification, the message template that should be used, and the recipients that should receive the notification.

  1. Click the + on the Rule line.
  2. Choose an Event from the Event drop-down list:
    Email Notifications

    Rules options for email notification

  3. Any Repository Global Read-Only Event
    In case of any repository entering a global read-only mode.
    Global Read-Only Due to Admin Action
    In case of any repository entering a global read-only mode as a result of administrator interaction through the admin UI.
    Disk Monitor Info
    Disk Storage has dropped below the Info level. This will trigger if any data monitor message is written to the logs at the "INFO" level.
    Disk Monitor Warning
    Disk Storage has dropped below the Warning level.This will trigger if any data monitor message is written to the logs.
    Disk Monitor Severe
    Disk Storage has hit the Severe level. This will trigger if any "Severe" level data monitor message is written to the logs. At this level, GitMS will have shutdown to ensure that disk space exhaustion doesn't corrupt your system and potentially your Git repositories.
    Deploy Repository Failed
    A repository added to GitMS has failed to deploy, in which case the repository will not be replicated.
    Deploy Repository Succeeded
    A repository added to GitMS has successfully deployed. Such an event might be sent to a mail group received by Git users, telling them that their repository is now accessible.
    Global Read-Only Due to Consistency Check Failure
    In case of any repository entering a global read-only mode as a result of failing a consistency check with its replicas.
    Generic file replication error occurred
    An error occured with the Generic Replication script (When running GitMS in conjunction with Access Control Plus).
  4. Choose a Template from the drop-down list. These are the templates that you have already set up under the Templates section.
  5. Choose destinations for your notification from the available destination email addresses. You can make multiple selections so that a message is sent to more than one recipient address.
  6. Click on the + Add link to save your rule.
    Email Notifications

    Rules you can edit or remove

2. Manage access to GitMS

GitMS supports the following mechanisms to manage access to its admin UI:

You can set up multiple administrator accounts to access the GitMS admin console. You can set up the accounts from the admin UI (via the Security tab). These users can then log in to any node's admin UI by providing their username and password.

This section describes how to set up multiple accounts, set up managing LDAP authorities, and export/import the resulting data.

2.1 Add additional users

  1. Log in to the Admin UI using an existing admin account.
    security tab

    Log in

  2. Click the SECURITY tab, then click Add User.
    Security tab

    Add User

  3. Enter details for the new administrator, then click the Add User button at the end of the entry bar.
    Security tab

    Click Add User to save their details

  4. You see a growl message confirming that the user has been added. They are listed on the Internally Managed Users after clicking the Reload button (or refershing your browser session).
    Security tab

    New user appears

2.2 Remove or edit user details

You can modify any user details by clicking their corresponding Edit button on the Internally Managed Users table

    Acc

    Remove or Edit users

2.3 LDAP authorities

GitMS supports the use of LDAP authorities for managing admin logging accounts. See our Guide to LDAP.

When connecting GitMS to available LDAP authorities it is possible to classify the authority as "Local", i.e. specific to the node in question or not. In this case, the authority details are replicated to the other nodes within the replication network.

You can run multiple LDAP authorities that are of mixed type, i.e. using some local authorities along with other authorities that are shared by all nodes. When multiple authorities are used, you can set the order that they are checked for users.

The standard settings are supported for each configured LDAP authority: URL, search base and filter and bind user credentials. Note that the bind user's password cannot be one-way encrypted using a hash function because it must be sent to the LDAP server in plain text, so for this reason the bind user should be a low privilege user with just enough permissions to search the directory for the user being authenticated. Anonymous binding is permitted for those LDAP servers that support anonymous binding.

2.3.1 Add authority

Use the Add Authority feature to add one or more LDAP authorities, either local to the node or connected via WAN. Locally LDAP services are treated as having presedence. When Internally managed users are enabled they are first checked when authenticating users - see Admin Account Precedence

To add an authority:

  1. Log in to the admin UI, click the Security tab.
  2. Click Add Authority.
    Add auth

    Add Authority

  3. The Authority entry form appears. Enter the following details:
    Add auth

    Add Authority

    URL
    Enter your authorities URL. You need to include the protocol ldap:// or ldaps://
    Example (Active Directory:)
    ldap://<server IP>:389
    Bind User DN
    Enter a LDAP admin user account that will be used to query the authority
    Example (Active Directory:)
    cn=Administrator,cn=Users,dc=windows,dc=AD
    Search Base
    Enter the Base DN, that is the location of users that you wish to retrieve.
    Example (Active Directory:)
    CN=Users,DC=sr,DC=wandisco,DC=com?sAMAccountName
    Search Filter
    Optionally add A query filter that will a select user based on relevant LDAP attributes. Currently the search filter must be created so that it filters LDAP query result into unique result. A work around might look something like:
    Example: (Active Directory)
    (&(memberOf=[DN of user group])(sAMAccountName={0}))
    This dynamically swaps the {0} for the ID of the active user. For more information about query filter syntax, consult the documentation for your LDAP server.
    Is Local?
    Tick this checkbox if you want the authority to only apply to the current node and not be replicated to other nodes (which is otherwise done by default).
  4. Click the Add Authority. This will save the authority settings that you have just entered. You can click the Test button to verify that the details will successfully connect to the authority without yet adding the authority.
  5. When running with multiple authorities, you should determine the order by which MultiSite polls the authorities. Use the +- symbols at the end of each authority entry to push it up (+) or down (-) the list.
    order auth

    Order authorities

2.3.2 Edit authority

Modify an existing authorities settings:

  1. Log in to the admin UI, click the Security tab.
  2. Click the edit link on the line that corresponds with the authority that you wish to edit.
    order auth

    Edit authorities link

  3. Update the settings in the popup box, then click Save.
    order auth

    Edit authorities box

Kerberos security

This section describes the basic requirements for integrating GitMS with your existing Kerberos systems. The procedure requires the following:

Ensure that time synchronization and DNS are functioning correctly on all nodes before configuring Kerberos.
A time difference between a client and the master Kerberos server that exceeds the Kerberos setting (5 mins default) automatically causes auth failure.

2.3.3 Configuration

This procedure assumes that you have already set up your DNS service and master Key Distribution Center.

  1. On each node, add the service principal:
    # kadmin -p root/admin -q "addprinc -randkey HTTP/node1.example.com"
    # kadmin -p root/admin -q "ktadd -k /opt/krb5.keytab HTTP/node1.example.com"
    # chmod 777 /opt/krb5.keytab
    	    
  2. Each node should have installed the add-on JCE Java 6 or Java 7 Unlimited Strength Jurisdiction Policy Files". These can be downloaded from Oracle, subject to your local import rules concerning encryption technology. Once downloaded, extract to the the Java security library, i.e.
    $JAVA_HOME/lib/security/
  3. At this point you can install GitMS on each node. If that's already done, then configure the Kerberos settings under the Security tab.
    kerb

    Edit Kerberos box

    Serivce Principal:
    This unique name for an instance of a service, such as HTTP/node1.example.com
    Keytab Location:
    This is the location of the keytab, a file containing pairs of Kerberos principals and encrypted keys (often derived from the Kerberos password). It's used for logging into Kerberos without being prompted for a password.
    Kerberos Config Location:
    The krb5.conf file contains Kerberos configuration information, including the locations of KDCs and admin servers for the Kerberos realms of interest, defaults for the current realm and for Kerberos applications, and mappings of hostnames onto Kerberos realms. Normally, you should install your krb5.conf file in the directory /etc. i.e. /etc/krb5.conf
    Save the settings. Log out. Return to the node in your browser, this time you should log in automatically (in this as user sally@EXAMPLE.COM.)

See Security Reference: Kerberos settings.
See Configure browsers for Kerberos authentication.

3. Nodes

This section describes the functions that manage repository data replication.

3.1 Add a node

To replicate Git repository data between sites, you first first tie the sites together. Start by adding (connecting) sites in an induction process:

  1. Log in to the GitMS admin console of the new node that you are connecting to your existing servers.
  2. Click the Sites tab.
  3. Click the Connect to node button.
    Acc

    Connect to node

  4. Enter the details of an existing node. You can get these details from the Settings tab of the existing node.
    Acc

    Enter details from an existing, connected node

    Node ID
    This is the name that you gave the Node during installation. If you log into the node you can see the Node ID on the title of any screen that you view, it also appears in the logged in message: "Welcome to MultiSite, admin. You are connected to <NODE ID>"
    Node Location ID
    A unique string that that a node creates for use as in identifier. You can get this from the node's Settings tab:
    Indentifer string for a node

    System Data table, found on the Settings tab.

    Node IP Address
    The IP Address of the node's server.
    Node Port No
    The TPC port that the node uses for DConE, which handles agreement traffic. The default is 6444. See Reserved Ports.
  5. Click the SEND CONNECTION REQUEST button. The new node appears on the active list of Sites. The new node may get stuck in a pending state. If this happens click the Cancel button.

3.2 Remove a node

It's useful to remove a node from a GitMS replication group if you are no longer replicating repository data to its location and want to tidy up your replication group settings.

No ties allowed
You only have the option to remove a node if it is not a member of a replication group. Therefore, you may need to remove and recreate replication groups to make it eligible for removal.

Known issue:
NOTE: If a node is inducted but not in a replication group then you can, from that node, remove other inducted nodes that are in a replication group. Currently, a node is not aware of the membership of any replication group of which it is not a member. This means that it is possible to remove a node that is a member of a replication group, if done from another node that does not have knowledge of the replication group.

Until we fix this issue, do a manual check of any nodes that you want to remove to make sure that it is not a member of a replication group.
A removed node cannot return
Take care when removing nodes! To ensure that your replication network is kept in sync, removed nodes are barred from being re-inducted. The only way that you can bring back a node is to reinstall GitMS using a new Node ID.
  1. Log in to the GitMS admin console of any connected node.
  2. Click the Nodes tab.
  3. Nodes that are eligible for removal have the Remove Node option available in the Action column. In this example, NodeSanFrancisco is eligible for removal because we have removed it from replication groups.
    Acc

    Nodes table under the Nodes tab

  4. Click the Remove Node button. Don't forget that this action is irreversible. You must be sure that you want to permanently remove the node. Acc

    Ready to remove NodeSanFrancisco

  5. After a refresh of the admin user interface you can still see the removed node if you click the Display Removed Nodes button. Removed nodes are identified by their Removed status.
    Acc

    Node removed

3.3 Stop all nodes

You can bring all nodes to a stop with a single button click (if all associated repositories are replicating/writable).

A stop can't be synchronized if associated repositories are Local Read-only
Before starting a Sync Stop All, make sure that none of your nodes have repositories in a local read-only state.
  1. Log into the admin UI and click the Nodes tab.
  2. Click the Sync Stop All button.
    Acc

    Stop all nodes

    You get a growl message confirming the stop has been triggered. You see the results when you refresh your browser session.
    Acc

    Stopped

  3. On the Node table all nodes show as Stopped. In this state you can do maintenance or repairs without risking your replication getting out-of-sync.
    Acc

    Node removed.

  4. The Sync Stop All button has changed to Start all. However, you can start selected nodes by logging in to the admin console of each node that you want to start. Use the Start Node link that appears in the Action column of the nodes table.
Important!
We strongly recommend that you watch the log messages and confirm that all nodes report as stopped. If you suspect that one or more nodes are not going to stop you should investigate immediately. The DASHBOARD messages should report the stop, for example:
Aborted tasksType PREPARE_COORDINATE_STOP_TASK_TYPE
Delete Task
Originating Node: Ld5UYU
tasksPropertyTASK_ABORTING_NODE: Ld5UYU
tasksPropertyTASK_ABORT_REASON: One or more replicas is already stopped.
The replica was: [[[Ld5UYU][bf0c6395-77b6-11e3-9990-0a1eeced110e]]]
    

Look for the message:

Aborted tasksType PREPARE_COORDINATE_STOP_TASK_TYPE

In the replicator.log file you might see the error type:

DiscardTaskProposal <task id etc> message: One or more replicas is already stopped.

3.4 Start all nodes

  1. If all nodes have been brought to a stop, click the Start All button to start them replicating again.
    Acc

    Stopped!


  2. After a browser refresh, all nodes will now show as running.

3.5 Disconnected/offline nodes

If a node is disconnected you can see this from the UI:

4. Replication groups

When you click the REPLICATON GROUPS tab you can see if there are disconnected nodes:
See Disconnected/offline nodes.

4.1 Add a replication group

Follow this procedure to add a new Replication Group. You need to add a new replication group when you want to replicate between a new combination of sites, i.e. sites that are not currently replicating in an existing group. If, instead, you want to replicate a new repository between existing sites, you can add a new repository to those sites. In this case, see Add a new repository.

  1. Log in to the GitMS browser-based user interface. Click the REPLICATION GROUPS tab, then click the CREATE REPLICATION GROUP button.
    Create the repo group

    Create a replication group

  2. Enter a name for the group in the Replication Group Name field, then click the drop-down selector on the Add Sites field. Select the sites that you want to replicate between.
    Identifier string for a node

    Replication group details

    Replication Ground Rules
    • A node can belong to any number of replication groups.
    • A repository can only be part of a single active replication group at any particular time.
    • You can change membership on the fly, moving a repository between replication groups with minimal fuss.
  3. Click each node label to set its node type.
    Identifier string for a node

    Click node labels to change their type

    Advice on creating effective replication groups
    For an understanding of some of the ground rules for replication you should read the section Creating resilient Replication Groups.
    Nodes are automatically added to a group as Active Voters. To understand the differences between the different types of nodes, read Guide to node types.
  4. When all sites are in place with your required settings, click CREATE REPLICATION GROUP.
    CREATE REPLICATION GROUP

    Create Replication Group

  5. Newly created replication groups appear on the Replication Group tab, but only on the admin UI of nodes that are themselves members of the new group.
    GROUPGLOBAL

    The new replication group appears if you are logged into one of its constituent nodes

Important: Don't cancel replication group creation tasks
If you create a new replication group, then find that the task is stuck in pending because one of your nodes is down, do not use the Cancel Tasks option on the Dashboard's Pending Tasks table. not with a missing node
If, when all nodes are up and running, the replication group creation tasks are still not progressing, please contact the WANdisco support team for assistance.

4.2 Delete a replication group

You can remove replication groups from GitMS, as long as they they have no repositories. For example:

  1. We have identified that replication group VineyanRepos is to be removed from GitMS. We can see that it has a single repository associated with it. Click the View to see which one.
    Deleting Replication Group

    View

  2. On the Replication Group configuration screen we can see that Repo5 is associated with the group. We can see that currently the Delete Replication Group (VinyardRepos) is disabled. You can follow the link to the repositories page to remove the association.
    Deleting Replication Group

    Repositories

  3. On the Repositories screen, click the associated repository, in this example it's Repo5, then click the EDIT button.
    Deleting Replication Group

    Select and Edit

  4. On the Edit Repository box, use the Replication Group drop-down to move the repository to a different Replication Group. Then click SAVE.
    Deleting Replication Group

    Edit

  5. Repeat this process until there are no more repositories assoicated with the Replication Group that you wish to delete. In this example VinyardRepos only had a single repository, so it is now empty, and can be deleted. Click View, then on Configure.
    Deleting Replication Group

    Move it

  6. Now that Replication Group VinardRepos is effectively empty of replication payload the Delete link is enabled. Click the link Delete Replication Group (VinyardRepos) to remove the replication group, taking note that there's no undo - although no data is removed when a replication group is deleted, it should be easy enough to recreate a group if necessary.
    Deleting Replication Group

    Click the Delete link button

  7. A growl will appear confirming that the replication group has been deleted.
    Deleting Replication Group

    Deleting the replication group

4.3 Add node to replication group

Don't add a node during a period of high replication load
When adding nodes to a replication group that already contains three or more nodes, ensure that there isn't currently a large number of commits being replicated.

Adding a node during a period of high traffic (heavy level of commits) going to the repositories may cause the process to stall.

To add additional nodes to an existing replication group, so that there's minimal disruption to users:

  1. Log in to a node, click the REPLICATION GROUPS tab. Go to the replication group to which you will add a new node, click its VIEW.
    Add node

    Replication Groups

  2. The replication group screen will appear. Click Add Nodes.
    Add Site

    View the group settings

    Why is the Add Nodes button is disabled?
    The Add Nodes button may be grayed out if the current replication group configuration won't support the addition of a new voter node.
    Add node
    It is also possible that a configuration that is scheduled in the future may block the addition of a new node. Check the schedule if you think that you should otherwise be able to add a new node to the replication group.
  3. Select the node that you wish to add to the replication group.
    Add node

    Select

  4. When there are no further nodes to add to the group, click the Add Nodes button.
    Add node
  5. At this stage in the process we're ready to select a Helper node from which we'll synch repository data to the new node - select a Helper Node.
    Add node

    Helper node

  6. Note the warning about not closing the browser or logging out during this process. Otherwise you will need to perform a more lengthy repair procedure. Click the Start Sync button.
    Add node

    Start sync

  7. You now need to manually synchronize the repositories from the helper node, which is temporarily offline until this process is finished. See Synchronizing repositories using rsync.

    The process lets you either do a complete sync or select specific repositories that you wish to sync. Assuming that you have synced all repositories you click Complete All. The helper node is then released from the process, allowing it to catch up with any transactions it missed while taking part in the procedure.
    Add node

    Complete all

  8. A growl message confirms that the new node is added to the replication group.
    Add node

    new node!

  9. Return to the Replication Group screen and you can see the new node count.
    Add node

    Adding new node complete

4.4 Remove node from replication group

You can remove a node from a replication group, for example, if the developers at one of your nodes are no longer going to contribute to the repositories handled by a replication group. Removing a node from a replication group stops more updates to its repository replicas.

Remove stray repositories
If you remove a node from a replication group, you must delete its copy of the repositories managed by the replication group. An out-of-date stray copy can result in confusion or users working from old data.

You cannot remove a node that is currently assigned as the Managing Node. To remove the managing node, go to the Configure Schedule page and assign a different node as managing node.

  1. Log in to the admin console of one of your nodes. The node needs to be a member of the relevant Replication Group, otherwise it does not appear on the tab. Click the Replication Groups tab.
    , then click the View button for the Replication Group from which you plan to remove a node.
    Rmove Node from group 01

    Log in and go to REPLICATION GROUPS

  2. Click the node that you want to remove from the group. Providing that the removal of the node doesn't invalidate the remaining configuration you see a Remove node from replication group link. Click the link.
    Remove Node from group 03

    Remove

  3. A dialog opens asking you to confirm the removal of the selected node from the Replication Group. Click Remove. Rmove Node from group 04

    Confirm remove

  4. A growl message confirms that the removal is in progress. You many need to click the Reload button to ensure that the action has been completed on all nodes. Rmove Node from group 04

    Reload to confirm the updated state

  5. The node is removed from the Replication Group. On the Replication Groups panel you now see that the number of nodes has reduced by one. Rmove Node from group 06

    Less one member node

4.5 Schedule node changes: follow the sun

You can schedule the member nodes of a replication group to change type according to when and where it is most beneficial to have active voters. To understand why you may want to change your nodes read about Node Types.

4.5.1 Schedule node type changes via the public API

Instead of manually setting up schedules through a node's UI you can do it programmatically through calls to the public API.
See Public API ScheduledNodeAPIDTOList element and scheduledNodeAPIDTOList Datatype.

Use the following API call:

http://<ip>:8082/public-api/replicationgroup/{repgroupID}/schedule
e.g.
http://10.0.100.135:8082/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule
PUT with ReplicationGroupAPIDTO XML as body:

To make Node N3 a tie-breaker 'T' FROM 10:00 - 16:00 (GMT) every day of the week with Node N1 as tie-breaker 'T' afterwards.

Times are always in UTC (GMT)
When viewed on a node, times are shifted to the local timezone although internally they are always recorded in UTC.

Example curl command:

Make a text file containing ReplicationgroupAPIDTO XML (as above) called schedule.xml

curl -u username:password -X PUT -d @schedule.xml http://[IP]:[PORT]/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule

Sample schedule.xml file

<ReplicationGroupAPIDTO>
       <replicationGroupName>global</replicationGroupName>
     <replicationGroupIdentity>97913c04-bbad-11e2-877a-028e03094f8d</replicationGroupIdentity>
       <scheduledNodes>
           <dayOfWeek>1</dayOfWeek>
           <hourOfDay>14</hourOfDay>
           <schedulednode>
               <nodeIdentity>N1</nodeIdentity>
               <locationIdentity>c0e486a0-bbab-11e2-863b-028e03094f8e</locationIdentity>
               <isLocal>true</isLocal>
               <isUp>true</isUp>
               <lastStatusChange>0</lastStatusChange>
               <role>AV</role>
           </schedulednode>
           <schedulednode>
               <nodeIdentity>N3</nodeIdentity>
               <locationIdentity>5480f515-bbad-11e2-8301-028e03094f8c</locationIdentity>
               <isLocal>false</isLocal>
               <isUp>true</isUp>
               <lastStatusChange>0</lastStatusChange>
               <role>T</role>
           </schedulednode>
           <schedulednode>
               <nodeIdentity>N2</nodeIdentity>
               <locationIdentity>478c766f-bbad-11e2-877a-028e03094f8d</locationIdentity>
               <isLocal>false</isLocal>
               <isUp>true</isUp>
               <lastStatusChange>0</lastStatusChange>
               <role>AV</role>
           </schedulednode>
  

Download the full sample schedule.xml file.

  1. Log in to a node, then click the REPLICATION GROUPS tab. Click the VIEW link for the replication group that you wish to make a schedule.
    Schedule for you

    Scheduling is done through replication group settings

  2. The replication group's pop-up window opens, showing the member nodes together, along with their current (scheduled) roles. Click the CONFIGURE button.
    Schedule for you

    Configure

    Membership views show what is scheduled, not necessarily what is currently active
    The roles and membership displayed in the popup is based upon the agreed schedule. It is the setup that should be in place if everything is running smoothly. It may not accurately represent the state of the replication group, due to a delay in processing on a node or if a process has hung. This is not a cause for concern but you must be aware that the displayed membership is an approximation based on the information currently available to the local node.
  3. The replication groups configuration screen will appear. You may notice that to the left a Role Schedule is noted. By default this will show as DISABLED. Click the Configure Schedule button, in the right-hand corner.
    Schedule for you

    Role Schedule: Disabled (for now)

  4. The Schedule screen appears. The main feature of the screen is a table that lists all the nodes in the replication group, set against a generic day (midnight to midnight) that is divided into hourly blocks. Each hourly block is color-coded to indicate the specific node's type. Schedule for you

    Vanilla Scheduling - no changes to type over time

    Note: In the image below, NodeSanFrancisco is blue, indicating that it is a Passive Voter. The hourly blocks associated with NodeChengdu are magenta, indicating that it is a pure voter. The blocks for NodeParis are yellow, indicating that it is an Active Voter.

  5. To change the schedule, click a block. You can click any block because the New Scheduled Configuration form lets you modify any hours for any available node.
    Schedule for you

    New Schedule form

      Frequency
      Select from the available frequency patterns: Daily, Weekly, Monday-Friday or Saturday to Sunday.
      From
      The starting hour for the new schedule, e.g. 00 for the start of the day.
      To
      The hour at which the scheduled changes end, e.g. 24 would effectively end the scheduled change at midnight.
      Node list
      The member nodes are listed, in graphical form, colour coded to their type.
  6. Click the node icon to change its type.

    Note: In this example NodeSanFransisco is changed to a Tie-breaking Passive Voter, then NodeAuckland is changed into a Tie-breaker.

    Schedule for you

    Swapping roles

    When all node changes have been made, click the SAVE button to continue, or the CANCEL button if you change your mind.
  7. The schedule view will now change to show the changes that you make. You must click the Save Schedule button for the changes to be applied.
    With all necessary changes made, you need to review the change to the schedule table and then click SAVE SCHEDULE button.
    Changing role of the managing node
    You can change the managaing node to Active, Active Voter and Active Voter Tiebreaker, though not to any passive roles. If you want to make it a passive node you must switch the manager to an active node (A, AV, AVT) because the manager needs to be able to propose schedule changes and therefore be active.

4.5.2 Disable the schedule

If you need to stop any and all scheduled rotations, e.g. in an emergency to prevent losing quorum:

  1. Click the REPLICATION GROUPS tab, then select your group:
    Disable schedule
  2. Click Disable Schedule:
    Schedule warning message
    Note the warning message.

4.6 Single-node configurations

No replication takes place when you have a single node. Other functionality works, but you may have issues relating to quorum.

You may have a single-node configuration in the following situations:

5. Repositories

5.1 Add a repository

When you have added at least one Replication Group you can add repositories to your node:

  1. Click the REPOSITORIES tab, then the ADD button.
    ** Add repository 1 **

    Repositories > ADD

  2. Enter the Repository's name, the file system path (full path to the repository), and use the drop-down to select the replication group.
  3. Tick the Global Ready-only checkbox to set the repository to be read-only. You can deselect this later. Click ADD REPO.
    ** Add repository 1 **

    Repositories > Enter details then click ADD REPO

    Repository stuck in Pending state

    If a repository that you added gets stuck in the deploying state, you see this on the Dashboard, in the Replicator Tasks window. You can cancel the deployment and try adding the repository again. To cancel a deployment, go to the Replicator Tasks window and click the Cancel Task link.

  4. When added, a repository appears in a list on the REPOSITORIES tab. The list provides the following details.
    ** Add repository 1 **

    Repositories listed

    Name
    The name of the repository - this will be the same as the folder name in the Git directory.
    Path
    The full path to the Repository.
    Replication Group
    The Replication Group in which the repository may be replicated.
    Size
    The file size of the repository.
    Youngest Rev
    The youngest (latest) revision in the repository. Comparing the youngest revisions between replicas is a quick test that a repository is in the same state on all sites.
    Last Modified
    The timestamp for the last revision, which provides a quick indicator for the last time a Git user made a change.
    Global RO
    Checkbox that indicates whether the repository is globally Read-only, that is Read-only at all sites.
    Local RO
    Checkbox that indicates whether the repository is locally Read-only, that is Read-only to users at this node. The repository receives updates from the replicas on other sites, but never instigates changes itself.

5.1.1 Automated repository deployment

Although you can deploy new repositories automatically using the REST API, the changing state of repositories is not chained to the deployment task. It's therefore possible that a repository that is successfully created may not be fully deployed, and this could result in errors if your automation moves ahead too soon.

Temporary solution for automated repository management
Ensure that a short delay is added to the deployment of new repositories. You can query the newly added repository through the REST API (using the repository endpoint) in order to verify that it is in fact deployed.

5.2 Remove a repository

You can remove repositories from GitMS. Follow this quick procedure.

Important requirement when performing a repository dump / load
Commits may fail if a repository is removed and, in some way, historically edited before re-introducing the repository to GitMS. This may happen, for example, if you use a Git dumpfilter to remove content/revisions due to sensitive content, while maintaining the same UUID. This is caused by existing full-text cache references no longer being applicable.
Workaround: Restart Apache and GitMS before you use the repository.
  1. Log in to the admin console of one of your nodes. The node must be a member of a replication group in which the repository is replicated, otherwise it is not listed. Click the Repositories tab to see it.
    Rmove Repository 01

    Login

  2. On the Repositories tab, click the line that corresponds with the repository that you want to remove.
    Schedule for you

    Repositories.

  3. When a repository is highlighted (in yellow), the REMOVE button becomes available. Click it. Schedule for you

    Remove

    A dialog box appears called Remove repository from replication group. This confirms that removing a repository from a replication group stops any changes that are made to it from being replicated. However, no repository data is removed.

5.3 Edit a repository

To edit a repository's properties after they have been set up in GitMS:

  1. Log in to the admin console of one of your nodes. The node must be a member of a replication group in which the repository is replicated, otherwise it is not listed. Click the Repositories tab to see it.
    Rmove Repository 01

    Log in

  2. On the Repositories tab, click the line that corresponds with the repository that you want to edit. Then click the Edit button. You can also click the repository's name to go directly to the Edit screen.
    Schedule for you

    Repositories

  3. You now see the Repository screen appear. For here you can trigger a consistency check, bring the repository to a coordinated stop or start a repair if a problem has been detected.
    Schedule for you

    Edit Repository.

    Name
    The name of the repository assigned when it was placed under MultiSite control.
    Path
    The absolute path to the repository on each node.
    Replication Group
    I
    Last Modified
    The filesize of the repository, noted here in bytes.
    Size
    The timestamp of the last modification to the repository.
    Sized added today
    Indicates the change in repository size over the last 24 hours, which provides a quick cue for recent traffic levels.
    Youngest Revision
    This entry is not used by GitMS. It is used when MultiSite is configured to run with Subversion.
    Commits made today
    Provides a count of the repository changes that have been made over the last 24 hours. This is used to measure recent activity levels on the repository.
    Global Read-only
    Changes the Read-only setting, enable or disable the repository Global Read-only setting. When enabled, the repository will not be writable either locally or globally. This is used to lock a repository from any changes.
    Local Read-only
    Changes the Read-only setting, enable or disable the repository Local Read-only setting. When enabled, the repository will not be writable, either for local users or for the replication system (that would push changes made to the repository on other nodes). However, changes that come from the other nodes are stored away to be played out as soon as the read-only state is removed.
    Reload
    For a refresh of the repository information to pick up any changes that may have occured since loading the screen.

5.4 Repository synchronized stop

Use the repository Sync Stop function to stop replication between repository replicas. You can do this either:

To stop all nodes, use the Sync Stop All command via the Nodes tab.

Repository stops are synchronized between nodes using a stop proposal to which all nodes need to agree. Although not all nodes stop at the same time they do all stop at the same point.

  1. Log in to a node's browser-based UI and click the Repositories tab. Click the repository that you want to stop replicating.
    tip
  2. With the repository selected, click the Sync Stop button. A growl message confirms that a synchronized stop has been requested. Note that the process may not be completed immediately, especially if there are large proposals transferring over a WAN link.
    tip
  3. Refresh the screen to see that a successfully sync stopped repository has a status of Stopped and is Local RO (locally read-only) at all nodes.

5.5 Repository synchronized start

When you restart replication after a Synchronized Stop, you must start the stopped replication in a synchronized way.

  1. Click a stopped repository and click the Sync Start button.
    tip
  2. The repository will stop being Local Read-only on all nodes and will restart replicating again.

5.6 Stop repositories on a node

In some situations, such as performing a repository backup, you may need to stop writes to the local repository replica. You can do this in the following ways:

5.6.1 Stop node via API call

With this method, proposals are still delivered to the node and the node can still participate in voting, but the proposals are not executed until the output is restarted.

This is supported in the API with the RepositoryResource methods:

PUT <server>:<port>/api/repository/{repositoryId}/stopoutput

Read more about stopoutput.

This command takes one argument, NodeListDTO nodeListDTO, which is the list of nodes where the repo output will be stopped. In this case the list only includes NodeX.

Note that while the output is stopped it shows in the UI as LocalReadOnly.

PUT <server>:<port>/api/repository/{repositoryId}
/startoutput

Read more about startoutput.

This command takes one argument, NodeListDTO nodeListDTO, which is the list of nodes where the repo output will be started. In this case the list only includes NodeX.

PUT: <server>:<port>/api/replicator/stopall

If you don't need the stop to be coordinated then the following method is simpler and immediately stops the output of proposals on all repositories on the node it is executed against:

PUT: <server>:<port>/api/replicator/stopall

This stops all repositories on the node on which it is invoked (with no coordination).

To start them all again call:

PUT: <server>:<port>/api/replicator/startall

5.7 Reuse a repository

Take care when reusing a Git repository.
You need to edit the Git configuration file.

You might reuse a repository after improperly removing it from replication, by copying it from backup/restore for example. If so, you need to hand edit the Git configuration file (in repo/config) to change replicated = true to replicated = false or remove the line. Do this before putting it in place on any replicated server as a local, i.e. non-replicated, repository.

6. Back up GitMS data

You can back up GitMS's own database if you want to quickly restore a node.

Only MultiSite Settings are backed-up
This procedure backs up GitMS's internal Prevayler database. It does not back up your Git repository data or any other system files that you should also be backing up.

Create a backup of the current installation by invoking the following API call:

curl --user <username>:<password> -X POST http://[node_ip_address]:8082/api/backup
This creates a backup folder in the [install dir]/git-multisite/replicator/database/backup directory.

6.1 Restore GitMS data

Use the following procedure to restore GitMS settings from your backup after reinstalling and starting a node:

  1. Shut down the node.
  2. Run the following jar file. This needs to be run from the replicator directory.
    java -jar git-ms-replicator-gitmsrestore.jar -c path/to/application.properties -r path/to/back-up-folder

    path/to/application.properties is properties/application.properties

    path/to/backup by default is /opt/wandisco/git-multisite/replicator/database/backup/

    For example

    java -jar git-ms-replicator-gitmsrestore.jar -c properties/application.properties -r /opt/wandisco/git-multisite/replicator/database/backup/2017-03-15T14:00:07+0000_DConE_Backup

  3. Restart the node.

7. Running GitMS with Apache

This section describes how to set up Apache with GitMS.

Using mod_dav or WebDav to administer GitMS is not supported.

7.1 Before you start

Note the following requirements:

Notes

If you are using ONLY Apache to access both GitMS and SVN MSP, then you can use just one account.
If you are ONLY accessing either GitMS or SVN MSP via SSH then you can use just one account.
However, if you are using Apache and SSH to access both Git and SVN repositories you need to use two accounts (because each account has only a single authorized_keys file).
When using SSH for both SVN and Git access, the account used to run Apache must be the same account that is used to install SVN MSP.
Running GitMS with the same system account as Apache (for example, via a dedicated apache user account) can cause problems. For production we recommend that you set up a dedicated user account: in Red Hat that's an account with a UID of 500 or higher.

In some cases you may need to run GitMS using the apache user, for example if you are deploying with both Git and SVN repository replication. To make the apache user account suitable for running GitMS you need to ensure that:

  • user has a valid shell.
  • Apache doesn't invoke the SuexecUserGroup directive, which ensures that apache user is set to run CGI programs.
  • Repositories should be set to be owned by apache.

7.2 Install Apache

yum install httpd mod_ssl

Next, ensure that Apache starts on system restart:

chkconfig --add httpd
service httpd start

7.3 Configure SeLinux

If SeLinux is running:

  1. Enable access to home directories, i.e. /usr/home/gitms:
    setsebool -P httpd_enable_homedirs on 
  2. Install semanage:
    usermod -a -G apache gitms
    yum -y install policycoreutils-python 
  3. Allow httpd read/write access to /home/gitms:
    chcon -R -t httpd_sys_rw_content_t /home/gitms
    chcon -R -t httpd_sys_rw_content_t /opt/wandisco/git-multisite/replicator/content_delivery
  4. Allow the update script to make a network connection to the Java service:
    setsebool -P httpd_can_network_connect on
    setsebool -P git_system_enable_homedirs on

7.4 Configure IPTables

/etc/sysconfig/iptables should look like:


*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [11:12222]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 6789 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8082 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8085 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9001 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Jul 29 13:52:30 2013
Increased the OUTPUT ACCEPT from 2222 to 12222 (allows outgoing connections up to port 12222)
Allow incoming connections on 6789, 8082, 8085, 9001 and 443.
  

7.5 Create your HTTP password file

htpasswd -c /var/www/passwd username

Change the ownership of the passwd file:

chown apache:apache /var/www/passwd

7.6 Create git-http-backend wrapper script

The script executed by suexec must be under /var/www:

mkdir -p /var/www/bin

Add the following to /var/www/bin/git-http-backend:


#!/bin/bash
GIT_PROJECT_ROOT=/home/gitms/repos
# This value should be configured to match the base location of repos on disk
export GIT_BASEDIR=$GIT_PROJECT_ROOT
# This should be set to the gitms users home directory - reset here because SUexec strips it out
export HOME=/home/gitms
export GIT_HTTP_EXPORT_ALL=true
# Execute gitms_shell script
exec /opt/wandisco/git-multisite/bin/gitms_shell $REMOTE_USER
  

The script and the directory it is in must be owned by the user who will be executing the script:

chown -R gitms:gitms /var/www/bin

Make the wrapper script executable:

chmod +x /var/www/bin/git-http-backend

Tell selinux that /var/www/bin has httpd exectuable scripts:

semanage fcontext -a -t httpd_sys_script_exec_t /var/www/bin  
restorecon /var/www/bin

7.7 Add Apache config

Copy the following into /etc/httpd/conf.d/repo.conf:


<VirtualHost *:80>
DocumentRoot /home/gitms/repos
ServerName git.example.com
  

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>
  

<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>
  

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

7.8 Allow Git pushes

Change to the directory your repo is located in:


su - gitms -s /bin/bash
cd /home/gitms/repos/bar.git
git config http.receivepack true
git config core.sharedRepository group
  

Do this in each repo and on all replicas.

7.9 Restart Apache and test


service httpd restart
git clone http://192.168.122.1/repos/bar.git
  

7.10 HTTPS support: Generate certificates

You can use tools such as easy-rsa to generate certificates.

You need to have the Epel rep installed to use this:


wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
yum install easy-rsa
cp -r /usr/share/easy-rsa/2.0 .
cd 2.0/
source vars
./clean-all
./build-ca
./build-key-server git-http1     (when prompted for the CommonName use the system IP address if the system does not have a register DNS name)
./build-key-server git-http2
./build-key-server git-http3
  

This generates three server certs/keys as well as the ca cert/key in the ./keys directory.

Copy a cert and key into /etc/apache on each node.

cp ./keys/git-http1.crt /etc/httpd
cp ./keys/git-http1.key /etc/httpd
chown apache:apache /etc/httpd/git-http1.crt /etc/httpd/git-http1.key

7.11 Modify Apache config to support SSL

Edit /etc/httpd/conf.d/repo.conf:


<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

Restart service:

service httpd restart

7.12 Add the CA certificate to client system

The server certificate generated will not be recognized by your client. You can either:

7.13 Test

$ git push

WARNING: gnome-keyring:: couldn't connect to: /tmp/keyring-P25n4M/pkcs11: No such file or directory
Password for 'https://test@192.168.122.207':
remote: GitMS - update replicated.
To https://test@192.168.122.207/repos/bar.git
 d434f14..8b7bab0 master -> master$
  

7.14 Manual setup for audit logging

Use this procedure to account for some configuration relating to the audit feature that is currently missing from the installer.

7.14.1 Sender configuration

Setting sources
This value sets the sources that flume will monitor: acpSender.sources =
  • Example: To monitor all three set: acpSender.sources = svnServeSource svnWebdavSource gitmsSource
  • Example: To monitor just Webdav: acpSender.sources = svnWebdavSource
Setting Log locations
Settings that apply to Webdav and GitMS:
acpSender.sources.svnWebdavSource.type = exec
acpSender.sources.svnWebdavSource.command = tail -F /var/log/httpd/access_log
acpSender.sources.svnWebdavSource.restart = true
acpSender.sources.svnWebdavSource.channels = memChannel

acpSender.sources.gitmsSource.type = exec
acpSender.sources.gitmsSource.command = tail -F /opt/wandisco/git-multisite/replicator/log/gitms.log
acpSender.sources.gitmsSource.restart = true
acpSender.sources.gitmsSource.channels = memChannel
      

The system user that runs GitMS MUST have permissions to read all the locations that you configure.

7.14.2 Avro settings

If you're running with Apache Avro, apply these settings:

Receiver: /opt/wandisco/flume-scm-access-control-plus/conf/flume_acp_receiver.properties
acp_agent.sinks.acpSink.acp_receiver_host = <Access Control Plus IP>
acp_agent.sinks.acpSink.acp_receiver_port = <Access Control Plus PORT>
acp_agent.sources.avroSrc.bind = <Git MultiSite IP>
acp_agent.sources.avroSrc.port = <FLUME PORT>

Sender: /opt/wandisco/flume-git-multisite/conf/acp_sender.conf
acpSender.sinks.acpSink.hostname = <Git MultiSite IP>
acpSender.sinks.acpSink.port = <FLUME PORT>
  

8. GitMS authentication and authorization

GitMS authentication is performed by a third party service. When the authentication is complete, details of the requested git operation and a username are passed back to GitMS.

The operation and username are then checked internally by GitMS to ensure sufficient permissions are available to perform the operation.

Authentication can use either:

8.1 Authentication

Authentication can be by either SSH directly or via an HTTP authentication mechanism. It is normal to only use one or the other, but it is possible to use both in parallel. This section describes examples of using either SSH or Apache for authentication.

8.1.1 Authentication through Apache

Apache authentication allows users to communicate via the HTTP(S) protocol. This is beneficial in environments with heavily restricted firewalls, as the usual ports 80 and 443 are used for communication.

Git has two HTTP-based protocols, Git over HTTP and Smart HTTP. GitMS only supports the Smart HTTP protocol, because of its better functionality, especially in speed of operation.

The information given here assumes that:

You can configure Apache to authenticate against either an internal file (htpasswd) or an LDAP directory service.

Authentication by htpasswd

Authentication by LDAP

Apache can use an LDAP directory to authenticate against. Unlike htpasswd, you do not have to maintain a separate passwd file.

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:


<VirtualHost *:80>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
RequestHeader set X-Remote-User %{R_U}e

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Git Repos"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL "ldap://LDAP-IP:389/CN=CN-details,DC=DC-details,DC=DC-details?uid"

#If the LDAP directory requires a bind user and password:
AuthLDAPBindDN "CN=Administrator,CN=Users.DC=sr,DC=wandisco,DC=com"
AuthLDAPBindPassword password

Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  

Configuration using HTTPS

You can set up Apache to use HTTPS rather than HTTP. This is preferred in Enterprise settings due to the security benefits.

Using HTTPS with htpasswd

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:


<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
# The following two lines will redirect port 80 (HTTP) to 443 (HTTPS)
# if SSL/TLS is always required:
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
RequestHeader set X-Remote-User %{R_U}e
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  
Using HTTPS with LDAP

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
#  The following two lines will redirect port 80 (HTTP) to 443 (HTTPS)
# if SSL/TLS is always required:
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
RequestHeader set X-Remote-User %{R_U}e
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Git Repos"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL "ldap://LDAP-IP:389/CN=CN-details,DC=DC-details,DC=DC-details?uid"

#If the LDAP directory requires a bind user and password:
AuthLDAPBindDN "CN=Administrator,CN=Users.DC=sr,DC=wandisco,DC=com"
AuthLDAPBindPassword password

Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
  
Create git-http-backend script

Create a script called git-http-backend as follows:

#!/bin/bash
GIT_PROJECT_ROOT=/home/gitms/repos
# This value should be configured to match the base location of repos on disk
export GIT_BASEDIR=$GIT_PROJECT_ROOT
export GIT_HTTP_EXPORT_ALL=true
# Execute gitms_shell script
exec /opt/wandisco/git-multisite/bin/gitms_shell $REMOTE_USER
  

This script location should match where you set it to be called from in the repo.conf file on the ScriptAlias line - we have used /var/www/bin/git-http-backend. It is important to ensure that this script is executable, and the script and directory it is in are both owned by the "gitms" user - not "apache".

This script is run by suexec, and will therefore run as the user that owns it, which we require to be "gitms":


chmod +x /var/www/bin/git-http-backend
chown -R gitms:gitms /var/www/bin
  

Configuration of SELinux

If you are running SELinux, additional configuration is required, as follows:

#enable access to home directories, i.e. /home/gitms
setsebool -P httpd_enable_homedirs on

usermod -a -G apache gitms

#install semanage
yum -y install policycoreutils-python

#allow httpd read/write access to /home/gitms
chcon -R -t httpd_sys_rw_content_t /home/gitms

#allow httpd read/write access to /home/gitms
chcon -R -t httpd_sys_rw_content_t /opt/wandisco/git-multisite/replicator/content_delivery

#allow the update script make network connection to the Java service
setsebool -P httpd_can_network_connect on

setsebool -P git_system_enable_homedirs on

#configure selinux for suexec on git-http-backend
semanage fcontext -a -t httpd_sys_script_exec_t /var/www/bin
restorecon /var/www/bin
  

9. Install GitWeb

GitWeb is an optional open source component that provides read-only access to your Git repositories via a Web interface. It allows you to review recent commits, logs, history, and other metadata about the git repositories.

  1. Install gitweb from yum:
    yum install gitweb 
  2. Locate gitweb installation files, e.g. in their own folder:
    mkdir /var/www/gitweb
  3. Copy gitweb.cgi, gitweb.perl and the static directory into here.
  4. Set permissions (because you are using suexec, this must be wandisco):
    chown -R wandisco:wandisco /var/www/gitweb
  5. Modify /etc/gitweb.conf to reflect site changes:
    our $projectroot = "/opt/repos";
    our @git_base_url_list = qw(http://<server-ip-or-name>/gitweb)
          
    Save changes.
  6. Configure Apache to use gitweb (see Appendix A example) and restart Apache after making any changes.
  7. Browse to http://<server-ip-or-name>/gitweb to verify installation and configuration.

Example GitWeb Configuration

#GitWeb Configuration
Alias /gitweb /var/www/gitweb

<Directory /var/www/gitweb>
  Options +ExecCGI
  AddHandler cgi-script.cgi
  DirectoryIndex gitweb.cgi
</Directory>
  

Final step

After you've finished updating your configuration, the last step is to restart apache:

service httpd restart

10. SSH authentication

SSH authentication is available for Git and supported by GitMS. It is simple to set up and is attached to a service which is often already enabled. Occasionally, firewall rules may block clients and and so this may not be popular with Windows users.

10.1 Requirements

SSH authentication is done using the SSH daemon and public/private keypairs. Requirements are as follows:

10.2 Authorized keys

GitMS requires use of the 'command' keyword to be attached to each key in the authorized_keys file. This associates a username with the key used to log in by using it as an argument to the gitms_shell script. By default, the script is found in /opt/wandisco/git-multisite/bin/gitms_shell, but this can vary between installations.

For example:

command="/opt/wandisco/git-multisite/bin/gitms_shell user1" ssh-rsa <SSH_KEY>

If your script is in a different location, update this line accordingly.

SSH and Gitolite
If you're running both Gitolite and GitMS over SSH, both applications may attempt to use the same system account for SSH, this would introduce the risk of conflicts. We therefore recommend that you set up separate system accounts for GitMS and Gitolite.

11. Authorization

11.1 GitMS configuration

Authorization is not enabled by default. To change this, and any other authorization settings, edit the application.properties file. The location is /opt/wandisco/git-multisite/replicator/properties/ for default installations.


# enable/disable authz module
# Default:
#gitms.authz.enabled=false
gitms.authz.enabled=true

# Set the file location of the authz file
# Default: /opt/wandisco/git-multisite/replicator/properties/auth.authz
#gitms.authz.file=/opt/wandisco/git-multisite/replicator/properties/auth.authz
gitms.authz.file=<filepath>

# Set the default permissions policy to DENY or ACCEPT
# for requests without a specific rule
# Default:
#gitms.authz.policy=DENY;
gitms.authz.policy=<policy>

# Set the polling period to detect changes in the AuthZ file
# NOTE: The trailing L is important, as it indicates this
# number is a Long value
# Default:
#gitms.authz.poll.timer=50L
gitms.authz.poll.timer=<numberInMilliseconds>L
  

11.2 AuthZ File format

Begin with the following header:

# Git AuthZ Version:1.0

To define teams, add entries such as:


[groups]
team1 = user1, user2
team2 = user3, user4
  

To define rules against individual repositories, the repository path is used as a unique identifier, which is added in square brackets. For example:


[/home/gitms/repos/Repo1.git]
user1 = R+W+
@team1 = R+W+C+D+
  

AuthZ rules are applied to users and teams using the following tokens:

Note: To create a branch, a user must have both Create and Write access, i.e. C+W+ .

Rules can also be applied at branch level, as follows:

For a branch-level rule to be effective, at least read access is required at the repository level.

[/opt/gitRepos/Repo1.git:BRANCH/secure]
  rick = W-

A complete example:


# Git AuthZ Version:1.0
[groups]
team1 = wayne, rick
team2 = wayne, allan
[/home/gitms/repos/Repo1.git]
rick = R+W+
@team2 = R+W+C+D+
[/home/gitms/repos/Repo1.git:BRANCH/secure]
rick = W-
  

11.2.1 AuthZ rule application

The AuthZ rules have 2 hierarchies which determine whether a user has a requested level of access:

Rule conflicts within these hierarchies are resolved by picking the rule which is most granular. In our above example, "rick" would have Read and Write access to the whole repo, except for the "secure" branch, on which he wouldn't have write access. Apply these rules as follows:

For a branch-level rule to be effective, at least read access is required at the repository level.

Apply these rules as follows. If a user called Tom is requesting Write access to branch master on repo Repo1.git, the AuthZ rule resolution is:

  1. Determine that the repo Repo1.git exists on the local node. If not, error out.
  2. Lookup Tom's rules for branch master on Repo1.git
  3. If rules exist for Tom which grant/deny the access he needs, apply them
  4. If rules do not exist for Tom, check each of the teams Tom is a part of.
  5. Tom may be a part of multiple teams which have conflicting rule permissions. One could grant, and one could deny access. In the case where the permissions conflict at the same point on the hierarchy, we always pick the most permissive rule.
  6. If Tom's teams have no rules for the master branch either, we move up the Resource hierarchy and check the permissions assigned against Repo1.git.
  7. If Tom has permissions assigned against Repo1.git, apply them.
  8. If no relevant rules exist for Tom, check again for each of the teams he is a part of.
  9. If no permissions exist at this point, apply the default policy permissions. (gitms.authz.policy)

If read access is provided for a repository, all branches are readable, even if a rule is added at branch level to deny access.

Logging

The GitMS replicator makes several log entries relating to AuthZ acitivities.

Activity Log entry
Detected an AuthZ file change
When an AuthZ file change is detected
INFO: AuthZ: File change detected for <filename>
When there has been an error parsing the new AuthZ file
WARNING: AuthZ: Auth file invalid <errorMessage>
When the new AuthZ file has been successfully parsed
INFO: AuthZ: Time taken for parsing: <timeTaken>
Authorization request received
Request received - authorization disabled
DEBUG: AuthZ: Authorization disabled, accepting request
Request received - authorization enabled
INFO: "AuthZ: Request [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>] received"
Request received - authorization configuration error
WARNING: AuthZ: There was an error with the Authorization setup, request declined
Authorization response
Permissions Specified in AuthZ and applied
INFO: AuthZ: <ACCEPT/DENY> Permissions applied for [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>]
No specific permissions found - using default policy
INFO: AuthZ: No permissions specified for [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>] using default policy: <ACCEPT/DENY>
No matching user found - using default policy
AuthZ: Request received for non-existent user: <username> applying default policy: <ACCEPT/DENY>

Note: Logging descriptions are subject to change between versions and patches.