WANdisco
 Navigation:  v | Release Notes | Integration | Install | Integration | Administration | Git MultiSite | Glossary |

Installation Guide

Integration Guide
Make sure that you read the Integration Guide before starting your installation.

1. Software requirements

Install Gerrit on first node before installing Gerrit MultiSite
Install vanilla Gerrit only on your first node. Don't install it on the other nodes because the WANdisco installer manages this. At the end of the installation on the first node, you rsync the whole gerrit root directory and its repos to the next node on which you want to install.

Software requirements, assuming that you use MySQL or replace it with an equivalent, are:

Node 1 Node n > 1
Vanilla Gerrit-2.9.4 Git MS 1.5.3
Git MS 1.5.3 MySQL Server (Slave Node)
MySQL Server (Master Node) MySQL Proxy (Configured to read from local and write to Master)

2. Install procedure

Follow this procedure to install Gerrit:

  1. Make the installer file executable if it is not already:
    chmod +x gerritms-installer.sh
  2. Run the installer:
    ./gerritms-installer.sh
    The installer starts up and you see:
      
       ::   ::  ::     #     #   ##    ####  ######   #   #####   #####   #####
      :::: :::: :::    #     #  #  #  ##  ## #     #  #  #     # #     # #     #
     ::::::::::: :::   #  #  # #    # #    # #     #  #  #       #       #     #
    ::::::::::::: :::  # # # # #    # #    # #     #  #   #####  #       #     #
     ::::::::::: :::   # # # # #    # #    # #     #  #        # #       #     #
      :::: :::: :::    ##   ##  #  ## #    # #     #  #  #     # #     # #     #
       ::   ::  ::     #     #   ## # #    # ######   #   #####   #####   ##### 
     
    GerritMS Version:  Installation
     
    Install Documentation:
     
    http://docs.wandisco.com/git/gerrit/1.2/gerrit_install.html
     
    Welcome to the GerritMS installation. Before the install can continue,
    you must:
     
    * Have Gerrit v2.9.4 installed before beginning
    * Have backed up your existing Gerrit database
    * Have a version of GitMS (1.5.2 or higher) installed and running
    * Have a replication group created in GitMS containing all Gerrit nodes
    * Have a valid GitMS admin username/password
    * Have a valid Gerrit admin username/password
    * Stop the Gerrit service on this node
     
    Do you want to continue with the installation? [Y/n]:
    Is this the first node GerritMS will be installed to? [Y/n]:
    
    We recommend that, if you are upgrading, you stop all Gerrit nodes before the upgrade. This prevents changes to the shared database during the upgrade.
  3. Answer whether this node is the first to to be installed to. This enables better targeted post-install advice.
  4. Answer whether you require the database to be upgraded. This question is only asked on the first node. Because Gerrit nodes share a database, it is not necessary to backup the database several times over.
  5. The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:
    Currently installing as user: gitms
    The current user should be the same as the owner of the GitMS service.
    If this is not currently the case, please exit and re-run the installer 
    as the appropriate user.
    Press [Enter] to Continue
    
  6. The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:
    Configuration Information
    Git Multisite root directory [/opt/wandisco/git-multisite]:
    
    The installer collects details from the user for the install. The first question asked is the location of the GitMS service. A default option is provided, which is determined by the following checks:
    • Fetches the gitmsconfig property from ~/.gitconfig and confirms that the application.properties file it points to exists
    • If the gitmsconfig property does not exist, itlooks in the default install location for GitMS (/opt/wandisco/git-multisite)
    If neither of these resolve to a GitMS install, no default option is provided.
  7. After providing the installer with the location of GitMS, the installer reads the application.properties file and uses it to prepare any previously configured values, e.g. for an upgrade:
    Configuration Information
    Git Multisite root directory [/opt/wandisco/git-multisite]:
    Reading GitMS Configuration...
    Gerrit Root Directory:
    
    Where there are properties that are not set in application.properties, the installer prompts you for input. If a property is set in application.properties, then it is re-used. You cannot change previously configured values during installation.
  8. The installer fetches or asks for the following information:
    Gerrit Root Directory:
    The location of the Gerrit install.
    Gerrit Username and Password:
    GitMS communicate with Gerrit over its REST API. To do this, an account on Gerrit, with Administrator permissions, must be configured for GitMS to use.
    Gerrit Repository Directory:
    The location of the git repositories on disk that are managed by Gerrit.
    Gerrit Events Directory:
    GitMS and Gerrit will share information with each other via the filesystem. This directory is used to pass events from one process to another. If the directory does not exist at the time of installation, you are prompted to create it.
    Will this node send Replicated Events to other Gerrit nodes? [Y/n]:
    Gerrit nodes can send events that appear in its event stream to other nodes, to allow for a fully replicated event stream where you can monitor events from all Gerrit nodes by simply connecting to one. This option tells the current Gerrit node to publish its events to other nodes.
    For more information see Configure Gerrit Event Stream.
    Will this node receive Replicated Events from other Gerrit nodes? [Y/n]:
    This option will configure the current Gerrit node whether to show only its events, or all the replicated events it receives from other nodes too.
    Gerrit Replication Group Name:
    The name of the replication group that contains all Gerrit nodes.
    Note: Even if you will run GerritMS with selective replication for the majority of the repositories, at least one Replication Group must contain every Gerrit repository. This is because critical Gerrit configuration settings have been moved from the database to "system repositories" such as All-Projects and All-Users. These repositories will be required across every Gerrit site, and so they must belong to a replication group that covers every Gerrit node.
    GitMS Username and Password:
    Naming the replication group initiates a REST call to the currently-running GitMS installation to fetch the Replication Group ID to match the name. This requires a GitMS admin username and password.
    For example:
    Configuration Information
    Git Multisite root directory [/opt/wandisco/git-multisite]:
    Reading GitMS Configuration...
    Gerrit Root Directory: /home/gitms/gerrit
    Gerrit Username: admin
    Gerrit Admin Password:
    Confirm Gerrit Password:
    Gerrit Repository Directory: /home/gitms/gerrit/git
    Gerrit Events Directory: /home/gitms/gerrit-events
    Will this node send Replicated Events to other Gerrit nodes? [Y/n]:
    Will this node receive Replicated Events from other Gerrit nodes? [Y/n]:
    Gerrit Replication Group Name: Gerrit
    GitMS Username: admin
    GitMS Password:
    Replication Group found with ID: 1e14904f-ce35-11e4-bc7f-881fa11c523c
    
  9. You are queried about the install path of various helper scripts that help manage GerritMS. By default. these are placed in the GERRIT_ROOT/bin directory:
    Helper Scripts
    We provide some optional scripts to aid in installation/administration of
    GerritMS. Where should these scripts be installed?
    Helper Script Install Directory [/home/gitms/gerrit/bin]:
    
    The helper scripts are:
    reindex.sh:
    A review can appear out of sync on one Gerrit UI compared to the review's actual status. For example, very occasionally, on the review listing page, a review might be flagged as Submitted, Merge Pending, but may actually be Merged. This is caused by the Lucene index that Gerrit uses failing to update properly. Fix this by providing the ChangeID of the review to this script. This causes a reindex to occur on that individual change.
    sync_repo.sh:
    Note: This script does not rsync the repositories. Rsync happens at the same time as rsyncing the Gerrit install to the next node. If the repos directory is a subdirectory of gerrit, it is brought in during the rsync. If it is not a gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node.
    Any repositories created in Gerrit after GerritMS is installed, are automatically added to GitMS replication. If, however, you already have many repositories managed by Gerrit before installing GerritMS, the process to add them individually to replication can be tedious. This script iterates over the Gerrit repository root and automatically adds any repositories it finds to GitMS.
  10. A backup of GERRIT_ROOT is taken before any upgrade happens. You are asked where to store this backup. If the location you give does not exist, the installer prompts to create it:
     
    Backup Information
    Taking a backup of the of GitMS + Gerrit configuration. Where should this
    be saved?
    Backup Location:
    
  11. During the backup of the first node only, you are prompted to back up the database. If the underlying version of Gerrit is being upgraded for example, when Gerrit is re-init'd, it may change the database schema so that it becomes incompatible with a previous version. Therefore, we recommend that you create a backup before installation. If you don't create a backup, then you may not be able to roll back:
    
    Backup Information
    Taking a backup of the of GitMS + Gerrit configuration. Where should this
    be saved?
    Backup Location: /tmp
    NOTE: This instance of Gerrit has been configured to use the database reviewdb.
    It is recommended that you create a backup of this database now if one
    does not exist already.
    Creating backup...
    Backup saved to: /tmp/gerrit-backup-20150319163550.tar.gz
    Press [Enter] to Continue
  12. The backup is taken and its location printed to you. We recommended that you now check the backup file to ensure that it was done successfully.
  13. If the underlying versions of Gerrit being upgraded from/to are different, Gerrit may require schema changes to the database prior to running. For example, going from Gerrit 2.9.1 to 2.9.4 requires a schema change that alters some of the primary key settings. You should check the Release Notes for the underlying version of Gerrit and ensure that you have performed all the necessary steps that Gerrit requires for the upgrade.
    Because the option for the first node was set to true earlier in the install, the output at the end describes how to continue the install across multiple sites:
    
    Finalizing Install
    Gerrit Configuration Updated
    GitMS Configuration Updated
    GitMS and Gerrit have now been configured. Please restart the GitMS service on
    this node now to finalize the configuration updates.
    Next Steps:
    * rsync /home/gerrit2/ to all of your GerritMS nodes
    Should your git repositories directory not be located within your Gerrit directory, this rsync command will indicate that you need to provide the corresponding path:
    * rsync /a/path/to/repos to all of your GerritMS nodes
    * Run this installer on all of your other Gerrit nodes
    * On each of your Gerrit nodes, update gerrit.config:
       - change canonicalURL to the correct hostname
       - ensure that database details are correct
    * Run /home/gerrit2/bin/sync_repo.sh on one node to add any existing
      Gerrit repositories to GitMS. Note that even if this is a new install of
      Gerrit with no user added repositories, running sync_repo.sh is still
      required to ensure that All-Projects is properly replicated.
    * When all nodes have been installed, you are ready to start the Gerrit services
      across all nodes.
    
    rsync If your repos directory is a subdirectory of gerrit, it is brought in during the rsync. If it is not a gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node. See messages above.
    The rsync step is not necessary if you already have Gerrit installed on every node that GerritMS will be installed to. In these cases you can re-run the installer on those sites. This does not happen often, because the utility of standard Gerrit slave nodes is much less than that of master/master nodes.

2.1 Run sync_repo.sh script on Node 1

Gerrit is now integrated with Git MultiSite but you still need to modify Gerrit's Git configuration to introduce its repositories to Git MultiSite.

./sync_repo.sh
When the script has completed, open a terminal session to each node and start Gerrit up, e.g:
./gerrit/bin/gerrit.sh
Gerrit is now replicating its review activities to all nodes. You should test that this is the case.

2.2. Test the integration

Before going into production, run through your regular Gerrit workflow. For example:

  1. Clone a repository.
  2. Add a test file.
  3. Commit and push to your Gerrit magic branch.
  4. Check that you get a URL for the review.
  5. Log in to Gerrit on each node and confirm the review has replicated, you should see it if you click List all.
  6. Add a comment, e.g. "Looks good to me". Publish the comment and confirm that it replicates to the other nodes.
  7. Next,the project owner should (from a different node) approve the review. This should trigger Gerrit to merge the change into the master branch and replicate the change across the Git MultiSite nodes.
  8. Check that replication has completed properly by logging into the Git MultiSite admin UI and view the Repositories tab. Here you can run a consistency check for the applicable repository.

The integration is now complete! Additional information for managing Gerrit with Git MultiSite in the Gerrit Administration section.

Use a local LDAP authority
As we run without LDAP account caching there will be a greater burden placed on your LDAP authority as it deals with all account lookups. For this reason we strongly recommend that you ensure that the LDAP authority is hosted locally rather than via a WAN link which would could result in a significant drop in performance.

3. Automate the installation

The new installer is similar to the GitMS installer in that it allows for a non-interactive mode for automated test installs. To enter non-interactive mode, set the following environment variables:

GITMS_ROOT:
The location of the GitMS install, i.e: /opt/wandisco/git-multisite
BACKUP_DIR:
The location to store the backup of GERRIT_ROOT taken during installation. Should be writeable by current user.
SCRIPT_INSTALL_DIR:
Where to install the reindex.sh and sync_repo.sh scripts
GERRIT_ROOT:
The Gerrit root directory
GERRIT_USERNAME:
The username of the gerrit user set up to allow GitMS REST calls to Gerrit
GERRIT_PASSWORD:
The HTTP password of the gerrit gitms user
GERRIT_RPGROUP_ID:
The ID of the Replication Group setup for GerritMS
GERRIT_REPO_HOME:
The path to the repositories deployed by Gerrit
GERRIT_EVENTS_PATH:
The path for Gerrit events to be written to in communication with the GitMS replicator
GERRIT_REPLICATED_EVENTS_SEND:
Whether this node should send replicated events to other nodes, true / false
GERRIT_REPLICATED_EVENTS_RECEIVE:
Whether this node should report on replicated events from other nodes, true / false

4. Roll back

You need to manually complete the rollback procedure for GerritMS. This accounts for the variety of potential site scenarios. The database can be a complication of the rollback due to changes that may have happened to it, either caused by Gerrit's upgrade procedure (altered schema) or by using Gerrit before an upgrade. Make sure that you back up the database backup before upgrading your version of Gerrit. During an install/upgrade of GerritMS, a backup of the Gerrit root directory is taken and you are prompted on where it should be stored.

4.1 Root backup

The backup taken during install has the following format:

gerrit-backup-<timestamp>.tar.gz

This should be extracted into its own folder:

mkdir /tmp/backup
mv gerrit-backup-.tar.gz /tmp/backup
cd /tmp/backup
tar -xvf gerrit-backup-.tar.gz
ls -shlt
    total 118M
    118M -rw-rw-r-- 1 gitms gitms 118M Mar 18 13:39 gerrit-backup-.tar.gz
    4.0K drwxrwxr-x 3 gitms gitms 4.0K Mar 18 13:39 backup

The backup extracts into a folder named backup. Inside the folder:

ls
    total 8.0K
    4.0K -rw-rw-r--  1 gitms gitms 1.4K Mar 18 13:39 application.properties
    4.0K drwxrwxr-x 13 gitms gitms 4.0K Mar 18 13:18 gerrit

There are 2 components to a backup taken during install:

4.2 Rollback procedure

  1. We recommend that you stop all instances of Gerrit before rolling back any node. As they have a shared database (whether that is a single shared master, master/master or master slave), this avoids changes being written to the database during this process.
  2. Compare the current state of application.properties with the backup of application.properties taken during install. A GerritMS install adds several properties to this file prefixed with gerrit.*. In most cases, it is safe to simply replace the current application.properties with the backed up copy. However, if other entries in application.properties have been changed since the upgrade, these will be lost. If this might happen, we recommend that you compare the files to ensure that any additional modifications are mirrored in the backed up copy.
  3. When you are happy that the backup copy is correct, replace the existing application.properties file with it.
    Note: For GitMS to pick up these changes, you need to restart it.
  4. The gerrit folder has the contents of the Gerrit root directory. Similar to application.properties, compare the existing etc/gerrit.config and etc/secure.config files to verify that no additional changes have been made to these since the upgrade. If, for example, the location of the git repository basePath has changed, the backup needs to be updated to show this.
  5. When you are happy that the backup configuration is correct, replace the contents of the existing gerrit installation root directory with those of the backup.
  6. The gerrit root is now restored, but the database may still be an issue. Work on the database depends on potential schema changes that the Gerrit project has made from one version to the next. Schema changes are not guaranteed on every update, but they occur fairly frequently. If you know that there are no schema changes between the backed up version of Gerrit and the one being rolled back from, then you do not have to modify the database. If, however, you have to roll it back, there maybe issues because user-initiated changes made to the database after the upgrade will be lost. Information about these changes are in the Gerrit release notes.
  7. Whether the database has to be rolled back or not, you must now ensure that the Lucene index for the rolled back version of Gerrit is up to date. Trigger full reindex on gerrit by doing the following:
    java -jar gerrit.war reindex
  8. Restart GitMS and Gerrit on that node.
  9. Note: Any steps taken to roll back a shared database only have to be performed once, but do them before any restored Gerrit instance is started.