WANdisco Git MultiSite logo

5. Administrator's Guide

5.1 Running Git MultiSite

This guide runs through everything you need to know to get Git MultiSite deployed. First we'll cover all the things that you need to have in place before you install. We'll then cover a standard installation and setup. Finally we'll look at some possible problems you might experience with some troubleshooting tips.

Starting up

To start the Git MultiSite replicator, follow these steps:

  1. Open a terminal window on the server and login with suitable file permissions.
  2. Run the git-multisite service, located in the init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 git-multisite -> /opt/git-multisite/bin/git-multisite
    
  3. Run the start script:
    [root@localhost init.d]#  ./git-multisite start
    20130520-164811 (24088) [INFO]: Starting WANdisco MultiSite 20130520-164811 (24088) [INFO]: Started replicator (24100) 20130520-164811 (24088) [INFO]: Started ui (24110) 20130520-164811 (24088) [INFO]: Number of errors: 0 20130520-164811 (24088) [INFO]: Number of warnings: 0
  4. The two components of Git MultiSite; the replicator and the UI will start up. Read more about the git-multisite init.d script

The Dashboard

The dashboard is the first page you'll see when Git MultiSite starts. You can add widgets to your dashboard to display information about your repositories, such as size or latest activity, or about replicator tasks. We'll be adding more widgets in future releases to make your dashboard even more configurable.

Security

The security tab is used to manage admin accounts, either entered manually into Git MultiSite or managed through an LDAP authority. On the tab is an entry form for multiple admin accounts, along with LDAP Settings for binding MultiSite to one or more LDAP services.

Add User

Enter the details of an additional administrator who will be able to login to the Git MultiSite Admin UI. See Adding additional users for more information.

Add Authority

Enter the details of one or more LDAP authorities for managing administrator access. See Adding LDAP authorities for more information.

Disable Managed Users

This feature lets you block access to the Git MultiSite Admin UI by non-LDAP users. See Disabling (Internally) Managed Users below.

Export Security Settings

The data entered into the Securities tab can be backed up for later re-importing by clicking the Export Security Settings button. The data is stored in /opt/wandisco/git-multisite/replicator/export/security-export.xml which should be included in any backup procedures you are running. You will need access to the file from your desktop during a re-import.

Import Security Settings

Click the Import Security Settings button if you need to restore your Security settings, such as after a re-installation of Git MultiSite. The import will proceed providing that you can enter a file path to the security-export.xml file.

Reload

Click on the reload button to refresh the Admin UI screen, you will need to do this in order to view any changes that you make.

Admin Account Precedence

Git MultiSite uses the following order of precedence when checking for authentication of users:

This provider implementation tries to authenticate user credentials against either the list of internally managed users, or against any number of LDAP authorities, or both -- depending on how the administrator has configured the application.

When authenticating against LDAP authorities, each one is tried in sequence until one either grants access or they all deny access. In the event that they all deny access, only the error from the last authority tried will be returned.

Admin account changes are replicated to all nodes.

Changes to admin accounts are handled as proposals that require agreement from a majority of every node in the replication network. Admin account changes are reported into the audit log.

Internally Managed Users

This table lists those admin users who have been entered through the Admin UI or imported using the Import Security Settings, along with the first admin account.

Admin Account #1

Note that the first admin account is the one set up during the installation of your first node. The credentials specified during this installation are stored to the users.properties file which is then used during the installation of all subsequent nodes.

Admin Account Mismatch

The users.properties file is used to ensure that exactly the same username/password is used on all nodes during installation. In the event that there's a mismatch then you wouldn't be able to connect the nodes together (through the Induction process). Rather than clean-up and reinstall you can fix this by manually syncing the password files.

Disable (Internally) Managed Users

Click the Disabled Managed Users button if you want to control access to Git MultiSite exclusively through LDAP. Once clicked, any Internally managed users will no longer be able to log into the Admin UI after they next log out. From that point only LDAP managed users will have access to the Git MultiSite Admin UI.

Admin Account #1 can be removed but the last admin account remaining on the system will not be deletable to ensure that it isn't possible for an administrator to be completely locked out of the admin UI.

Re-enable Internally Managed users

If, after disabling Internally Managed Users you need to enable them again -- should there be a problem with your LDAP authorities -- then it is possible to enable access again by logging into the node via a terminal window (with suitable permissions), navigate to the following directory:

/opt/wandisco/git-multisite/replicator

and run the reset script:

java -cp lib/*:./resetSecurity.jar com.wandisco.multisite.ui.local.util.security.ResetSecurity

Any internally managed users who remain in Git MultiSite 's database will have their access restored.

Shutting down

To shutdown:

  1. Open a terminal window on the server and login with suitable file permissions.
  2. Run the git-multisite service, located in the init.d folder:
    lrwxrwxrwx  1 root root    37 May  9 10:37 git-multisite -> /opt/multisite-plus/bin/git-multisite
  3. Run the stop script, i.e.:
    [wandisco@ip-10-0-100-7 bin]$  ./git-multisite stop
    
    20130520-165704 (24767) [INFO]: Stopping WANdisco MultiSite
    20130520-165704 (24767) [INFO]: Request received to shut down replicator
    20130520-165704 (24767) [INFO]: replicator processes ended
    20130520-165704 (24767) [INFO]: Request received to shut down ui
    20130520-165704 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 1)...
    20130520-165707 (24767) [INFO]: Sending signal 15 to watched ui process (attempt 2)...
    20130520-165710 (24767) [INFO]: ui processes ended
    20130520-165710 (24767) [INFO]: Number of errors: 0
    20130520-165710 (24767) [INFO]: Number of warnings: 0
    
  4. Both the replicator and the UI processes will shut down. Read more about the git-multisite init.d script

init.d script

The 'start-up' script for persistent running of Git MultiSite can be found in the /etc/init.d folder. Run the script with the help command to list the available commands:

[root@localhost init.d]# ./git-multisite help
Usage: git-multisite {start|stop|status|uistart|uistop|repstart|repstart}

  start      Start WANdisco Multisite service
  stop       Stop WANdisco Multisite service
  status     Display running WANdisco Multisite services
  uistart    Start WANdisco Multisite UI service
  uistop     Stop WANdisco Multisite UI service
  repstart   Start WANdisco Multisite Replicator service
  repstop    Stop WANdisco Multisite Replicator service    

Changing the admin console password

You can change Git MultiSite's login password at any time by following this procedure:

  1. Login to the MultiSite admin console.
    Login

    Login.

  2. Click on the Settings tab.
    Settings Screen

    Settings.

  3. At the top of the settings screen is the password change form. Enter the current password, along with a new password.
    Password Change Form

    Changed password

  4. Click the SAVE button store the new password. You can be sure that the new password has been accepted if you see a growl message appear on screen.
    Save button to store

    Growl!

Changing Username
It's not currently possible to change the Administration username. In order to change the username you would need to re-install Git MultiSite

Updating your license.key file

Follow this procedure if you ever need to change your product license. You would need to do this if, for example, you needed to increase the number of Git users or the number of replication nodes.

  1. Login to your server's command line, navigate to the replicator directory (by default: /opt/git-multisite/replicator ) and rename the license.key to license.20130625.
    i.e.
    total 1120
    drwxr-xr-x 2 root root   4096 Jun 19 17:21 content
    drwxr-xr-x 5 root root   4096 Jun 19 17:21 database
    drwxrwxr-x 4 root root   4096 Jun 19 17:13 docs
    drwxrwxr-x 2 root root   4096 Jun 19 17:13 lib
    -rw-r--r-- 1 root root    256 Jun 19 17:18 license.key <-
    drwxrwxr-x 3 root root   4096 Jun 25 14:06 logs
    drwxrwxr-x 2 root root   4096 Jun 19 17:13 properties
    
  2. Get your new license.key and drop it into the /opt/git-multisite/replicator directory.
  3. Restart the replicator by running the SVN MultiSite script with the following argument:
    perl /etc/init.d/git-multisite repstart
    This will trigger a replicator restart, which will force Git MultiSite to pickup the new license file and apply any changes to permitted usage.
If you run into problems, check the replicator logs (/opt/git-multisite/replicator/logs ) for more information.
PANIC: License is invalid com.wandisco.fsfs.licensing.LicenseException: Failed to load filepath>

Application Properties file

The application.properties file contains configurable options for your Git MultiSite installation.

You should only make changes to this file if you are confident of the outcome, or have been instructed to do so by WANdisco's Support Team.

If you manually add monitor.period.min, connectivity.check.interval or sideline.wait to the applications property file then you must add an "L" (Long value) to the end of their values so they are converted correctly.

An example application.properties file is below:

#Mon Feb 03 16:55:07 GMT 2014
content.location=/opt/wandisco/git-multisite/replicator/content_delivery
application.hostname=10.4.4.64
beacon.period=1000L
jetty.https.port=8445
database.location=/opt/wandisco/git-multisite/replicator/database
content.min.learners.required=true
content.push.policy=Faster
deadlock.detect.interval=300000L
stats.collection.period=300
content.learners.count=1
content.transfer.md5sum=true
ssl.enabled=false
database.isPersistent=true
resourcemonitor.location=/opt/wandisco/git-multisite/replicator/properties
location.property.pending=true
content.server.port=4321
application.location=/opt/wandisco/git-multisite/replicator
ssl.debug=false
application.port=6444
prevayler.snapshot.interval=900000L
location.property.latitude=53.3833
jetty.http.port=8082
delegate.port=7777
membership.rotation.period=3600000
communication.hostName=10.4.4.64
node.id=Node1
location.property.longitude=-1.4667
gitms.local.jetty.port=9999

5.2 Managing Replication

This section covers the various functions that handle how repository data is replicated.

Adding a Node

To replicate Git repository data between sites, you first first tie the sites together, this process starts with adding (connecting) sites in a process we call induction.

Removing a Node

The method of removing a node depends on the current state of the node you want to remove:

If you remove a node it is not possible to reinduct it with the same node ID (name). You will either need to reinstall Git MultiSite on the removed node (causing it to be given a new location ID), or you will need to give the node a new, previously unused node ID.

Adding a Replication Group

Use the procedure to add a new Replication Group. You need to add a new replication group when you need to replicate between a new combination of sites - i.e. sites that are not currently replicating in an existing group. If you are, instead, looking to replicate a new repository between existing sites, it's possible to add new a new reposity to those sites. In this case see Add a new repository.


  1. Log in to the Git MultiSite browser-based user interface. Click on the REPLICATION GROUPS tab, then click on the CREATE REPLICATION GROUP button.
    Create the repo group

    Creating a replication group.


  2. Enter a name for the group in the Group Name field, then click on the drop-down selector on the Add Sites field. Select the sites that you want to replicate between.
    Indentifer string for a node

    replication group details.


    Replication Ground Rules
    - A node can belong to any number of replication groups
    - A repository can only be part of a single Active replication group at any particular time.
    - It's possible to change membership on the fly, moving a repository between replication groups with minimal fuss.
  3. Ensure that member sites are each set as an appropriate type.
    Indentifer string for a node

    Click on node labels to change their type.


    Quick guide to setting node types
    All sites are automatically added as the Active Voter type. To understand the differences between the different types of sites, read Guide to Node Types
  4. Once all sites are in place and their settings adjusted to your needs, click CREATE REPLICATION GROUP.
    CREATE REPLICATION GROUP

    Create Replication Group.


  5. Newly created replication groups will appear on the Replication Group tab, but only at the node that are themselves members of the new group.
    GROUPHANZO

    The new replication group now appears - if you are logged into one of its constituent sites.


Scheduling node changes - follow the sun

You can schedule the member sites of a replication group to change type according to when and where it is most beneficial to have active voters. To understand why you may want to change your sites, read about Node Types


Schedule node type changes via the public API
Instead of manually setting up schedules through a node's UI you can do it programmatically through calls to the public API.
See Public API ScheduledNodeAPIDTOList element and scheduledNodeAPIDTOList Datatype


Use the following API call

http://<ip>:8082/public-api/replicationgroup/{repgroupID}/schedule
e.g.
http://10.0.100.135:8082/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule
PUT with ReplicationGroupAPIDTO XML as body:

To make Node N3 a tie-breaker 'T' FROM 10:00 - 16:00 (GMT) every day of the week with Node N1 as tie-breaker 'T' afterwards: (N.B Times are in GMT -4 hours so 14 = 10:00 GMT)

Example curl command:

Make a text file containing ReplicationgroupAPIDTO XML (as above) called schedule.xml

curl -u username:password -X PUT -d @schedule.xml http://[IP]:[PORT]/public-api/replicationgroup/97913c04-bbad-11e2-877a-028e03094f8d/schedule

sample 'schedule.xml' file

  1. Login to a node, click on the REPLICATION GROUPS tab. Click on the QUICK VIEW link for the replication group that you wish to make a schedule.

  2. The replication group's pop-up window will open, showing the member groups together, along with their current roles. Click the CONFIGURE button.
    Shedule for you

    Configure.


  3. Membership views are what is scheduled not necessarily what is currently active The roles and membership displayed in the popup is based upon the agreed schedule, it's the setup that should be in place if everything is running smoothly. It is always possible that it doesn't accurately represent the state of the replication group, due to a delay in processing on a node, or if something has caused a process to hang. This should not be a cause for concern but it's important to be aware that the displayed membership is an approximation based on the information currently available to the local node.
  4. The replication groups configuration screen will appear. You may notice that to the left a Role Schedule is noted. By default this will show as DISABLED. Click on the CONFIGURE SCHEDULE button, in the right-hand corner.

  5. The Schedule screen will appear. The main feature of the screen is a table that lists all the sites in the replication group, set against a generic day (midnight to midnight) that is divided into hourly blocks. Hourly block is color-coded to indicate each site's type.
    In the image below NodeSanFrancisco is coded as turquoise (blue-green) which indicates that it is set as a tie-breaker. The hourly blocks associated with NodeChengdu and NodeAuckland are colored yellow, indicating that these sites are Active Voters. Finally, NodeParis is blank, indicating that it is Active.

    Shedule for you

    Vanilla Scheduling - no changes to type over time.



  6. To make a change to the shedule, click on a block. It doesn't matter which block you select as the New Scheduled Configuration form will let you modify any hours for any available node.
      Frequency
      Select from the available frequency patterns: Daily, Weekly, Monday-Friday or Saturday to Sunday.
      From
      The starting hour for the new schedule, e.g. 00 for the start of the day.
      To
      The hour at which the scheduled changes end, e.g. 24 would effectively end the scheduled change at midnight.
      Sites list
      The member sites are listed, in graphical form, colour coded to their type.
  7. Click on the node icon to change its type.

    In this example NodeSanFransisco is changed to an Active Voter, then NodeAuckland is changed into a Tie-breaker.

    Shedule for you

    Swapping roles.

    When all node changes have been made, click on the SAVE button to continue, or the CANCEL button if you change your mind.

  8. The schedule view will now change to show the changes that you make. You must click the SAVE SCHEDULE button for the changes to be applied.
    Shedule for you

    Swapping roles.


    With all necessary changes made, you need to review the change to the schedule table and then click SAVE SCHEDULE button.

Hooks

There are two types of hooks which need to be taken into account for replication, pre- hooks and post- hooks.

When used in a Git MultiSite setup these hooks will only fire at the node which received the push request. A new hook specific to Git MultiSite, rp-post-receive will fire at all the other nodes in the replication group.

The rp-post-receive hook will receive the same stdin as the hook on the originating node, and can be found in the hooks directory.

5.3 Managing Repositories

Add Repositories

Once you have added at least one Replication Group you will be able to add repositories to your node. Here's how:

Automated Deployment of Repositories

While it is possible to deploy new repositories through automation, using the REST API, currently the changing state of repositories is not chained to the deployment task. It's therefore possible that a repository that is successfully created will not actually be fully deployed and this could result in errors if your automation moves ahead too soon.

Temporary solution for automated repository management
Ensure that a short delay is added to the deployment of new repositories. You can query the newly added repository through the REST API (using the repository endpoint) in order to verify that it is in fact deployed.

Edit a repository

It's possible to Edit a repository properites after they have been set up in Git MultiSite. Follow this quick procedure.

  1. Login to the admin console of one of your sites. The node will need to be the member of a replication group in which the repository is replicated, otherwise it won't appear on the tab. Click on the Repositories tab to see it.
    Rmove Repository 01

    Login.

  2. On the Repositories tab, click on the line that corresponds with the repository that you want to remove.
    Shedule for you

    Repositories.

  3. Once a repository has been highlighted (Shown as a blue line), the EDIT button will become available. Click it. Shedule for you

    Edit.


    The repository edit window will open up.
    Local Read-only
    Change the Read-only setting, enable or disable the repository Local Read-only setting. When enabled, the repository will not be writable to local users. However, changes that are made on replica at other sites will be applied through inter-site replication.
    Global Read-only
    Change the Read-only setting, enable or disable the repository Global Read-only setting. When enabled, the repository will not be writable either locally or globally. This is used to lock a repository from any changes.
    Replication Group
    Use the drop-down selector to change the replication group to which the repository belongs.

Remove a repository

It's possible to remove repositories from Git MultiSite. Follow this quick procedure.

  1. Login to the admin console of one of your sites. The site will need to be the member of a replication group in which the repository is replicated, otherwise it won't appear on the tab. Click on the Repositories tab to see it.
    Rmove Repository 01

    Login.

  2. On the Repositories tab, click on the line that corresponds with the repository that you want to remove.
    Shedule for you

    Repositories.

  3. Once a repository has been highlighted (Shown as a blue line), the REMOVE button will become available. Click it. Shedule for you

    Remove.


    A dialog box will appear entitled "Remove repository from replication group". It will confirm that removing a repository from a replication group will stop any changes that are made to it from being replicated. However, no repository data is removed.

5.4 Back up Git MultiSite Data

It's possible to back up Git MultiSite's own database in case you need to quickly restore a node.

Only MultiSite Settings are backed-up
This procedure backs up Git MultiSite's internal Prevayler database, it doesn't touch your Git repository data or any other system files that you should also be backing up.
    Create a backup of the current installation by invoking the following API call:
    curl --user <username>:<password> -X POST http://[node_ip_address]:8082/dcone/backup
    This will create a backup folder in [INSTALL-DIR]/git-multisite/replicator/db/backup/X.X.X_DConE_Backup directory.

Back up while shut down

(run from within /replicator):
java -cp ./fsfsrestore.jar com.wandisco.fsfs.backup.FsfsBackup -c ./properties/application.properties

Use this to back up the current state of all pervailers when Git MultiSite is shut down - you don't therefore need to start the replicator in order to create a backup of the database.

Back up selected prevailers

(run from within /replicator):
java -cp ./fsfsrestore.jar com.wandisco.fsfs.backup.FsfsClear -c ./properties/application.properties

This class clears selected prevaylers only, when the replicator is shut down. It does not clear all database instances, only those that are being restored during restore process.

5.5 Restore Git MultiSite Data

Use the following procedure to restore Git MultiSite settings after reinstalling and starting a node:

  1. Shutdown the node
  2. Run the following jar file:
    java -jar fsfsrestore.jar path/to/application.properties path/to/back-up-folder

    the first argument, path/to/application.properties is
    <INSTALL-DIR>/multisite-plus/replicator/properties/application.properties

    path/to/backup by default is
    <INSTALL-DIR>/multisite-plus/replicator/db/backup/

    FsfsBackup.class path/to/application.properties
  3. Restart the node

5.6 Running Git MultiSite with Apache

This section will guide you through setting up Apache with Git MultiSite.

Using mod_dav or WebDav to administer Git MultiSite is not supported.

Before you start:

Running Git MultiSite with the same system account as Apache (often via a dedicated "apache" user account) can result in various problems. For production we recommend that you set up a dedicated "user" account - in Red Hat that's an account with a UID of 500 or higher.

In some cases you may need to run Git MultiSite using the apache user -- maybe you are deploying with both Git and SVN repository replication -- to make the apache user account suitable for running Git MultiSite you need to ensure that:

  • User has a valid shell.
  • Apache doesn't invoke the SuexecUserGroup directive, which ensures that 'apache' user is set to run CGI programs.
  • Repositories should be set to be owned by 'apache'.

Install Apache

 yum install httpd mod_ssl

Next, ensure Apache will start on system restart

 chkconfig --add httpd
 service httpd start

Configure SeLinux

If SeLinux is running:

Enable access to home directories - ie /usr/home/gitms

setsebool -P httpd_enable_homedirs on 

Install semanage

usermod -a -G apache gitms
yum -y install policycoreutils-python 

Next, allow httpd read/write access to /home/gitms

chcon -R -t httpd_sys_rw_content_t /home/gitms
chcon -R -t httpd_sys_rw_content_t /opt/wandisco/git-multisite/replicator/content_delivery

Then we need to allow the update script to make a network connection to the Java service

setsebool -P httpd_can_network_connect on
setsebool -P git_system_enable_homedirs on

Configure IPTables

/etc/sysconfig/iptables should look like:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [11:12222]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 6789 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8082 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8085 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9001 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Mon Jul 29 13:52:30 2013
Increased the OUTPUT ACCEPT from 2222 to 12222 (allows outgoing connections up to port 12222)
Allow incoming connections on 6789, 8082, 8085, 9001 and 443.

Create your HTTP password file

htpasswd -c /var/www/passwd username

Change the ownership of the passwd file:

chown apache:apache /var/www/passwd

Create git-http-backend Wrapper Script

The script executed by suexec must be under /var/www

mkdir -p /var/www/bin

Add the following to /var/www/bin/git-http-backend

#!/bin/bash
GIT_PROJECT_ROOT=/home/gitms/repos
# This value should be configured to match the base location of repos on disk
export GIT_BASEDIR=$GIT_PROJECT_ROOT
# This should be set to the gitms users home directory - reset here because SUexec strips it out
export HOME=/home/gitms
export GIT_HTTP_EXPORT_ALL=true
# Execute gitms_shell script
exec /opt/wandisco/git-multisite/bin/gitms_shell $REMOTE_USER

The script and the directory it is in must be owned by the user who will be executing the script:

chown -R gitms:gitms /var/www/bin

You'll need to tell selinux that /var/www/bin has httpd exectuable scripts)

semanage fcontext -a -t httpd_sys_script_exec_t /var/www/bin  
restorecon /var/www/bin

Add Apache Config

Copy the following into /etc/httpd/conf.d/repo.conf

<VirtualHost *:80>
DocumentRoot /home/gitms/repos
ServerName git.example.com
<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>
<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>
SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend

Allow git pushes

Change to the directory your repo is located in

su - gitms -s /bin/bash
cd /home/gitms/repos/bar.git
git config http.receivepack true
git config core.sharedRepository group

This needs to be done in each repo and on all replicas.

Restart Apache and Test

service httpd restart
git clone http://192.168.122.1/repos/bar.git

HTTPS Support

Generate Certificates

You can use tools such as easy-rsa to generate certificates.

You need to have the Epel rep installed to use this:

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -Uvh epel-release-6-8.noarch.rpm
yum install easy-rsa
cp -r /usr/share/easy-rsa/2.0 .
cd 2.0/
source vars
./clean-all
./build-ca
./build-key-server git-http1     (when prompted for the CommonName use the system IP address if the system does not have a register DNS name)
./build-key-server git-http2
./build-key-server git-http3

This will generate three server certs/keys as well as the ca cert/key in the ./keys directory.

Copy a cert and key into /etc/apache on each node.

cp ./keys/git-http1.crt /etc/httpd
cp ./keys/git-http1.key /etc/httpd
chown apache:apache /etc/httpd/git-http1.crt /etc/httpd/git-http1.key

Modify Apache Config to support SSL

Edit /etc/httpd/conf.d/repo.conf

<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>
 
SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>

Restart service:

service httpd restart

Add the CA certificate to client system:

The server certificate generated will not be recognized by your client. You can either turn of strict SSL checking:

git config --global http.sslVerify false

Or the CA certificate can be added to the client machine, this means adding the ./keys/ca.crt file to the set of CA certificates the client system accepts. Instructions for doing this can be found here: http://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate

Test:

$ git push

WARNING: gnome-keyring:: couldn't connect to: /tmp/keyring-P25n4M/pkcs11: No such file or directory
Password for 'https://test@192.168.122.207': 
remote: GitMS - update replicated.
To https://test@192.168.122.207/repos/bar.git
 d434f14..8b7bab0 master -> master
$

5.7 Git MultiSite Authentication and Authorization

Overview

Git MultiSite authentication is performed by a third party service. Once the authentication is complete, details of the requested git operation and a username are passed back to Git MultiSite.

The operation and username are then checked internally by Git MultiSite to ensure sufficient permissions are available to perform the operation.

Authentication can use either:

5.7.1 Authentication

As mentioned already, authentication can be by either SSH directly or via an HTTP authentication mechanism. It is normal to only use one or the other, but it is possible to use both in parallel.

Here are examples of using either SSH or Apache for authentication:

Authentication through Apache

Apache authentication allows users to communicate via the HTTP(S) protocol. This is beneficial in environments with heavily restricted firewalls, as the usual ports 80 and 443 are used for communication.

Git has two HTTP-based protocols, namely 'git over HTTP' and 'Smart HTTP'. Git MultiSite only supports the 'Smart HTTP' protocol, due to improved functionality compared with the older 'git over HTTP' protocol, especially with regards to speed of operation.

Assumptions

The following assumptions are made:

Authentication providers

We'll show you how Apache can be configured to authenticate against either an internal file (htpasswd) or an LDAP directory service.

Authentication by htpasswd

This requires an htpasswd file to be created to store usernames.

Creating the htpasswd file

htpasswd -c /var/www/passwd <username>

Adding users to an existing file:

htpasswd /var/www/passwd <username>

The -c option should only be used when first creating an htpasswd file. If you use this to reference a pre-existing file, any details in the file will be overwritten with the username you specify.

Configuration for htpasswd

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:80>
# 80 is the port the webserver will bind to
DocumentRoot /home/gitms/repos
# The base directory for repositories managed by Git MultiSite
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
RequestHeader set X-Remote-User %{R_U}e
 
<Directory "/home/gitms/repos">                        
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>
 
<Location /repos>
# This matches the location in the requesting url,
# for example, matches against request http://<ip>/repos/
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>
 
SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
# This script alias redirects matches made
# earlier to a script we will create later
</VirtualHost>

Authentication by LDAP

Apache can use an LDAP directory to authenticate against. Unlike htpasswd, this does not require the maintenance of a separate passwd file.

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:80>                                           
DocumentRoot /home/gitms/repos                             
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
RequestHeader set X-Remote-User %{R_U}e
 
<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>
 
 
<Location /repos>
AuthType Basic
AuthName "Git Repos"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL "ldap://LDAP-IP:389/CN=CN-details,DC=DC-details,DC=DC-details?uid"
 
#If the LDAP directory requires a bind user and password:
AuthLDAPBindDN "CN=Administrator,CN=Users.DC=sr,DC=wandisco,DC=com"
AuthLDAPBindPassword password
 
Require valid-user
</Location>
 
SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend   
</VirtualHost>

Configuration using HTTPS

You can set up Apache to use HTTPS rather than HTTP. This is preferred in Enterprise settings due to the security benefits.

Using HTTPS with htpasswd:

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
# The following two lines will redirect port 80 (HTTP) to 443 (HTTPS)
# if SSL/TLS is always required:
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
RequestHeader set X-Remote-User %{R_U}e
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key
 
<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>
 
<Location /repos>
AuthType Basic
AuthName "Private Git Access"
AuthUserFile "/var/www/passwd"
Require valid-user
</Location>
 
SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>

Using HTTPS with LDAP

Add a file called repo.conf in the /etc/httpd/conf.d/ directory with the following contents:

<VirtualHost *:443>
DocumentRoot /home/gitms/repos
ServerName git.example.com
RewriteEngine On
RewriteCond %{REMOTE_USER} ^(.*)$
RewriteRule ^(.*)$ - [E=R_U:%1]
#  The following two lines will redirect port 80 (HTTP) to 443 (HTTPS)
# if SSL/TLS is always required:
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}%{REQUEST_URI}
RequestHeader set X-Remote-User %{R_U}e
SSLEngine on
SSLCertificateFile /etc/httpd/git-http1.crt
SSLCertificateKeyFile /etc/httpd/git-http1.key

<Directory "/home/gitms/repos">
Allow from All
Options +ExecCGI
AllowOverride All
</Directory>

<Location /repos>
AuthType Basic
AuthName "Git Repos"
AuthBasicProvider ldap
AuthzLDAPAuthoritative off
AuthLDAPURL "ldap://LDAP-IP:389/CN=CN-details,DC=DC-details,DC=DC-details?uid"

#If the LDAP directory requires a bind user and password:
AuthLDAPBindDN "CN=Administrator,CN=Users.DC=sr,DC=wandisco,DC=com"
AuthLDAPBindPassword password

Require valid-user
</Location>

SuexecUserGroup gitms gitms
ScriptAlias /repos /var/www/bin/git-http-backend
</VirtualHost>
Create git-http-backend script

Create a script called git-http-backend as follows:

#!/bin/bash
GIT_PROJECT_ROOT=/home/gitms/repos
# This value should be configured to match the base location of repos on disk
export GIT_BASEDIR=$GIT_PROJECT_ROOT
export GIT_HTTP_EXPORT_ALL=true
# Execute gitms_shell script
exec /opt/wandisco/git-multisite/bin/gitms_shell $REMOTE_USER

This script location should match where you set it to be called from in the repo.conf file on the ScriptAlias line - we have used /var/www/bin/git-http-backend. It is important to ensure that this script is executable, and the script and directory it is in are both owned by the "gitms" user - not "apache".

This script is run by suexec, and will therefore run as the user that owns it, which we require to be "gitms":

chmod +x /var/www/bin/git-http-backend
chown -R gitms:gitms /var/www/bin
Configuration of SELinux

If SELinux is being run, additional configuration is required, as follows:

#enable access to home directories - ie /home/gitms
setsebool -P httpd_enable_homedirs on
 
usermod -a -G apache gitms
 
#install semanage
yum -y install policycoreutils-python
 
#allow httpd read/write access to /home/gitms
chcon -R -t httpd_sys_rw_content_t /home/gitms
 
#allow httpd read/write access to /home/gitms
chcon -R -t httpd_sys_rw_content_t /opt/wandisco/git-multisite/replicator/content_delivery
 
#allow the update script make network connection to the Java service
setsebool -P httpd_can_network_connect on
 
setsebool -P git_system_enable_homedirs on
 
#configure selinux for suexec on git-http-backend
semanage fcontext -a -t httpd_sys_script_exec_t /var/www/bin
restorecon /var/www/bin
Final step

After you've finished updating your configuration, the last step is to restart apache:

service httpd restart

SSH Authentication

SSH authentication is available for Git and supported by Git MultiSite. It is simple to set up and is attached to a service which is often already enabled. Occasionally, firewall rules may block clients and is not as popular with Windows users.

Requirements

SSH authentication is done using the SSH daemon and public/private keypairs. Requirements are as follows:

Authorized keys

Git MultiSite requires use of the 'command' keyword to be attached to each key in the authorized_keys file. This associates a username with the key used to login by using it as an argument to the gitms_shell script. By default, the script is found in /opt/wandisco/git-multisite/bin/gitms_shell, but this can vary between installations.

An example entry is as follows:

command="/opt/wandisco/git-multisite/bin/gitms_shell user1" ssh-rsa <SSH_KEY>

If your script is in a different location, just update this line accordingly.

SSH and Gitolite
If you're running both Gitolite and Git MultiSite over SSH, both applications may attempt to use the same system account for SSH, this would introduce the risk of conflicts. We therefore recommend that you set up separate system accounts for Git MultiSite and Gitolite.

5.7.2 Authorization

Git MultiSite configuration

Authorization isn't enabled by default. To change this, and any other authorization settings, edit the application.properties file. The location is /opt/wandisco/git-multisite/replicator/properties/ for default installations.


# enable/disable authz module
# Default:
#gitms.authz.enabled=false
gitms.authz.enabled=true
 
# Set the file location of the authz file
# Default: /opt/wandisco/git-multisite/replicator/properties/auth.authz
#gitms.authz.file=/opt/wandisco/git-multisite/replicator/properties/auth.authz
gitms.authz.file=<filepath>
 
# Set the default permissions policy to DENY or ACCEPT
# for requests without a specific rule
# Default:
#gitms.authz.policy=DENY;
gitms.authz.policy=<policy>
 
# Set the polling period to detect changes in the AuthZ file
# NOTE: The trailing L is important, as it indicates this
# number is a Long value
# Default:
#gitms.authz.poll.timer=50L
gitms.authz.poll.timer=<numberInMilliseconds>L

AuthZ File Format

The AuthZ format is similar to the formatting for WANdisco's SVN MultiSite products.

Begin with the following header:

# Git AuthZ Version:1.0

To define teams, add entries such as:

[groups]
team1 = user1, user2
team2 = user3, user4

To define rules against individual repositories, the repository path is used as a unique identifier, which is added in square brackets. For instance:

[/home/gitms/repos/Repo1.git]
user1 = R+W+
@team1 = R+W+C+D+

AuthZ rules are applied to users and teams using the following tokens:

Note: In order to create a branch, a user must have both Create and Write access, i.e. C+W+

Rules can also be applied at branch level, as follows:

For a branch-level rule to be effective, at least read access is required at the repository level.

[/opt/gitRepos/Repo1.git:BRANCH/secure]
rick = W-

A complete example:


# Git AuthZ Version:1.0
[groups]
team1 = wayne, rick
team2 = wayne, allan
[/home/gitms/repos/Repo1.git]
rick = R+W+
@team2 = R+W+C+D+
[/home/gitms/repos/Repo1.git:BRANCH/secure]
rick = W-
AuthZ Rule Application

The AuthZ rules have 2 hierarchies which are considered in the evaluation of whether a user has a requested level of access. They are the Resource level hierarchy (Repo->Branch|Tag), and the user level hierarchy (User->Team). Rule conflicts within these hierarchies are resolved by picking the rule which is most granular. In our above example, "rick" would have Read and Write access to the whole repo, except for the "secure" branch, on which he wouldn't have write access. The process by which these rules are applied is as follows:

For a branch-level rule to be effective, at least read access is required at the repository level.

If a user called Tom is requesting Write access to branch master on repo Repo1.git, the AuthZ rule resolution goes like this:

  1. Determine that the repo Repo1.git exists on the local node. If not, error out.
  2. Lookup Tom's rules for branch master on Repo1.git
  3. If rules exist for Tom which grant/deny the access he needs, apply them
  4. If rules do not exist for Tom, check each of the teams Tom is a part of.
  5. Tom may be a part of multiple teams which have conflicting rule permissions. One could grant, and one could deny access. In the case where the permissions conflict at the same point on the hierarchy, we always pick the most permissive rule.
  6. If Tom's teams have no rules for the master branch either, we move up the Resource hierarchy and check the permissions assigned against Repo1.git.
  7. If Tom has permissions assigned against Repo1.git, apply them.
  8. If no relevant rules exist for Tom, check again for each of the teams he is a part of.
  9. If no permissions exist at this point, apply the default policy permissions. (gitms.authz.policy)

If read access is provided for a repository, all branches are readable, even if a rule is added at branch level to deny access.

Logging

The Git MultiSite replicator makes several log entries relating to AuthZ acitivities.

Activity Log entry
Detected an AuthZ file change
When an AuthZ file change is detected
INFO: AuthZ: File change detected for <filename>
When there has been an error parsing the new AuthZ file
WARNING: AuthZ: Auth file invalid <errorMessage>
When the new AuthZ file has been successfully parsed
INFO: AuthZ: Time taken for parsing: <timeTaken>
Authorization Request Received
Request received - authorization disabled
DEBUG: AuthZ: Authorization disabled, accepting request
Request received - authorization enabled
INFO: "AuthZ: Request [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>] received"
Request received - authorization configuration error
WARNING: AuthZ: There was an error with the Authorization setup, request declined
Authorization Response
Permissions Specified in AuthZ and applied
INFO: AuthZ: <ACCEPT/DENY> Permissions applied for [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>]
No specific permissions found - using default policy
INFO: AuthZ: No permissions specified for [user: <username, repoPath: <repoPath>, ref:<refName>, accessRequested:<accessRequested>] using default policy: <ACCEPT/DENY>
No matching user found - using default policy
AuthZ: Request received for non-existent user: <username> applying default policy: <ACCEPT/DENY>

Note: Logging descriptions are subject to change between versions and patches

Troubleshooting

Problem: Cannot clone via HTTP

Apache configuration can go wrong in a number of places. If you're trying to clone a repository and get an error before you're asked for user credentials, it's likely there's a problem with your apache configuration. Here are some steps to make sure that your setup is valid:

More information about the apache failures can be found in the apache logs directory.

Problem: SSH setup isn't working - being asked for a password

The SSH daemon is quite strict about the permissions for its files and will check the following before allowing automatic login via a public/private key pair:

Problem: Can't write to new empty repository

The first push to a new empty repo actually creates the master branch, so you will need C+ permissions, not just W+.

Problem: A user that should be denied access was allowed

The permissions hierarchy means that some permissions, specified at a more granular level, take precedence over others. Permissions specified at the branch level take precedence over those at the repo level, and user permissions take precedence over team permissions.

However, it is possible to have a conflict of permissions on the same level in the hierarchy - for example, a user can be part of two teams, one of whom has R-, and the other has R+.

In this case, Git MultiSite picks the most permissive permission available. So if you believe a user should be denied access to a resource that they are able to access, check their teams to ensure they're not in another team which grants them those permissions.

5.8 Setttings

Location Setttings

Setting a nodes geographical location

You can set the location of the node's server as part of managing global replication group where the node's local time-zone might determine what type of node it is set as. See Scheduling node changes - follow the sun

Location

Setting node location.

  1. Click on the SETTINGS link on the top menu.
  2. Scroll down the screen to the System Data.
  3. Many settings are editable. To set the location of the server place the location's Latitude in the Location Latitude box and the longitude in the Location Longitude box.
  4. Click the Save button to update git MultiSite with the new location.
Location

Node Locations are displayed on the Nodes screen.

Node Locations

You can check the location of a node on the map on the Nodes screen. You'll see that by default, nodes have a 0,0 location which corresponds to a location in the Gult of Guinea. It is possible to change the location of a node at any time by repeating the above procedure. The changes are replicated to all the other nodes in due course.