Installation Guide

1. Software requirements

Software requirements, using the required Percona XtraDB database solution:

1.1 Plugins

Gerrit supports a number of plugins for integrating additional applications:

• Plugins need to be installed in exactly the same way on every node to ensure deterministic behavior or nodes can lose their sync.
• Plugins that use global configuration of key-value pairs, stored in the gerrit.config will replicate without problem providing they are configured the same on all nodes.
• Plugins with Project-level configuration (stored in project.config within refs/meta/config) should replicate without problem.
• We’re still investigating whether plugins that request data directories for storage can be supported with replication.

Currently we have successfully tested the Gerrit plugins for:

• Jenkins
• JIRA
• commit-message-length-validator - validates that commit messages conform to length limits
• download-commands - defines commands for downloading via different network protocols
• reviewnotes - retains review history for a gerrit project under refs/notes/review, which is replicated automatically by GerritMS
• singleusergroup - provides a group per user which is useful if you want to assign access rights directly to a single user

Gerrit plugins which are known not to work - do not use these plugins:

• replication - provides master-slave replication, and therefore should not be used with GerritMS.
• delete project

2. Install procedure

Follow this procedure to install Gerrit:

• Run the single installer file on the command line.
• Answer questions during the installation. For example, for a new installation you are asked:
  • If this install is being done on the first node
  • The root directory of GitMS
  • The root directory of Gerrit
  • The Gerrit admin account username for GitMS to use
  • The Gerrit admin account password for GitMS to use
  • The root directory for repositories deployed via Gerrit
  • The directory to use for publishing Gerrit Events
  • Whether the user wants the node to send replicated events
  • Whether the user wants the node to receive replicated events
  • The name of the default replication group to use for Gerrit in GitMS
  • The GitMS username and password
  • The location of the backup taken of the Gerrit root directory
• Existing configuration options are detected and used as default options, for example, the location of Git Multisite on the node. However, from an install or upgrade you cannot modify existing GerritMS settings. Any settings which already exist (and have been read from an existing application.properties file) will be automatically filled in and will not be prompted for.

1. Make the installer file executable if it is not already:

   chmod +x gerritms-installer.sh

   Workaround if /tmp directory is "noexec"
   Running the installer script will write files to the system's /tmp directory. If the system's /tmp directory is mounted with the "noexec" option then you will need to use the following argument when running the installer:
   --target <someDirectoryWhichCanBeWrittenAndExecuted>
   E.g.

   ./gerritms-installer.sh --target /opt/wandisco/installation/

2. Run the installer:

   ./gerritms-installer.sh

   The installer starts up and you see:

    [gerrit@dger02 gerrit2114-1702]$ ../installer-1702/gerritms-installer.sh
   Verifying archive integrity... All good.
   Uncompressing GerritMS Installer  100%
   
       ::   ::  ::     #     #   ##    ####  ######   #   #####   #####   #####
      :::: :::: :::    #     #  #  #  ##  ## #     #  #  #     # #     # #     #
     ::::::::::: :::   #  #  # #    # #    # #     #  #  #       #       #     #
    ::::::::::::: :::  # # # # #    # #    # #     #  #   #####  #       #     #
     ::::::::::: :::   # # # # #    # #    # #     #  #        # #       #     #
      :::: :::: :::    ##   ##  #  ## #    # #     #  #  #     # #     # #     #
       ::   ::  ::     #     #   ## # #    # ######   #   #####   #####   #####
   
   
   
   
    GerritMS Version: v2.11.4-RP-1.7-DEV-1-g83c5343-dirty Installation
   
    Install Documentation:
   
    http://docs.wandisco.com/git/gerrit/1.7/gerrit_install.html
   
    Welcome to the GerritMS installation. Before the install can continue,
    you must:
   
    * Have Gerrit v2.11.4 installed before beginning
    * Have backed up your existing Gerrit database
    * Have a version of GitMS (1.7 or higher) installed and running
    * Have a replication group created in GitMS containing all Gerrit nodes
    * Have a valid GitMS admin username/password
    * Stop the Gerrit service on this node
   
    Do you want to continue with the installation? [Y/n]:
   
    Is this the first node GerritMS will be installed to? [Y/n]:
   

   We recommend that, if you are upgrading, you stop all Gerrit nodes before the upgrade. This prevents changes to the shared database during the upgrade.
3. Answer whether this node is the first to to be installed to. This enables better targeted post-install advice.
4. Answer whether you require the database to be upgraded. This question is only asked on the first node. Because Gerrit nodes share a database, it is not necessary to backup the database several times over.
5. The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:

    Currently installing as user: gerrit
   
    The current user should be the same as the owner of the GitMS service.
    If this is not currently the case, please exit and re-run the installer
    as the appropriate user.
   
    Press [Enter] to Continue
   

6. The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:

   Configuration Information
   Git Multisite root directory [/opt/wandisco/git-multisite]:
   

   The installer collects details from the user for the install. The first question asked is the location of the GitMS service. A default option is provided, which is determined by the following checks:
   • Fetches the gitmsconfig property from ~/.gitconfig and confirms that the application.properties file it points to exists
   • If the gitmsconfig property does not exist, itlooks in the default install location for GitMS (/opt/wandisco/git-multisite)
   If neither of these resolve to a GitMS install, no default option is provided.
7. After providing the installer with the location of GitMS, the installer reads the application.properties file and uses it to prepare any previously configured values, e.g. for an upgrade:

   Configuration Information
   Git Multisite root directory [/opt/wandisco/git-multisite]:
   Reading GitMS Configuration...
   Gerrit Root Directory:
   

   Where there are properties that are not set in application.properties, the installer prompts you for input. If a property is set in application.properties, then it is re-used. You cannot change previously configured values during installation.
   Using Regex file with GitMultiSite 1.2.1.1
   You need to configure the property gerrit.rpgroup.regex in the application.properties so that it points towards the regex file, in order for Gerrit Project creation to work.
   
   For example, the entry in application.properties might look like:

   gerrit.rpgroup.regex=/opt/wandisco/gerrit/etc/gitms-regex.txt

   The location must be readable and writable by both the Gerrit and Git MultiSite system user.
   See Configure the regex file into GitMS
8. The installer fetches or asks for the following information:

   Gerrit Root Directory:

   The location of the Gerrit install.

   Gerrit Repository Directory:

   The location of the git repositories on disk that are managed by Gerrit.

   Gerrit Events Directory:

   GitMS and Gerrit will share information with each other via the filesystem. This directory is used to pass events from one process to another. If the directory does not exist at the time of installation, you are prompted to create it.

   Will this node send Replicated Events to other Gerrit nodes? [Y/n]:

   Gerrit nodes can send events that appear in its event stream to other nodes, to allow for a fully replicated event stream where you can monitor events from all Gerrit nodes by simply connecting to one. This option tells the current Gerrit node to publish its events to other nodes.
   For more information see Configure Gerrit Event Stream.

   Will this node receive Replicated Events from other Gerrit nodes? [Y/n]:

   This option will configure the current Gerrit node whether to show only its events, or all the replicated events it receives from other nodes too.

   Gerrit Replication Group Name:

   The name of the replication group that contains all Gerrit nodes.
   Note: Even if you will run GerritMS with selective replication for the majority of the repositories, at least one Replication Group must contain every Gerrit repository. This is because critical Gerrit configuration settings have been moved from the database to "system repositories" such as All-Projects and All-Users. These repositories will be required across every Gerrit site, and so they must belong to a replication group that covers every Gerrit node.

   GitMS Username and Password:

   Naming the replication group initiates a REST call to the currently-running GitMS installation to fetch the Replication Group ID to match the name. This requires a GitMS admin username and password.

   For example:

   Reading GitMS Configuration...
   
    Gerrit Root Directory: /home/gerrit/gerrit2114-1702/
    Gerrit Admin Username: admin
    Gerrit Admin Password: ********
    Gerrit Repository Directory: /home/gerrit/gerrit2114-1702/git/
    Gerrit Events Path: /home/gerrit/gerrit_events/
    Gerrit Receive Replicated Events: true
    Gerrit Send Replicated Events: true
    Gerrit Receive Replicated Events as distinct: false
    Gerrit republish local events as distinct: false
    Gerrit prefix for current node distinct events: REPL-
    Gerrit Replicated Cache enabled: true
    Gerrit Replicated Cache exclude reload for: changes,projects
    Gerrit Replication Group ID: 53edcbee-8183-11e5-b9e5-005056a97efe
   

9. You are queried about the install path of various helper scripts that help manage GerritMS. By default. these are placed in the GERRIT_ROOT/bin directory:

    Helper Scripts
   
    We provide some optional scripts to aid in installation/administration of
    GerritMS. Where should these scripts be installed?
   
    Helper Script Install Directory [/home/gerrit/gerrit2114-1702/bin]:
   

   The helper scripts are:
   

   reindex.sh:

   A review can appear out of sync on one Gerrit UI compared to the review's actual status. For example, very occasionally, on the review listing page, a review might be flagged as Submitted, Merge Pending, but may actually be Merged. This is caused by the Lucene index that Gerrit uses failing to update properly. Fix this by providing the ChangeID of the review to this script. This causes a reindex to occur on that individual change.

   sync_repo.sh:

   Note: This script does not rsync the repositories. Rsync happens at the same time as rsyncing the Gerrit install to the next node. If the repos directory is a subdirectory of gerrit, it is brought in during the rsync. If it is not a gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node.
   Any repositories created in Gerrit after GerritMS is installed, are automatically added to GitMS replication. If, however, you already have many repositories managed by Gerrit before installing GerritMS, the process to add them individually to replication can be tedious. This script iterates over the Gerrit repository root and automatically adds any repositories it finds to GitMS.

10. A backup of GERRIT_ROOT is taken before any upgrade happens. You are asked where to store this backup. If the location you give does not exist, the installer prompts to create it:

     
    Backup Information
    Taking a backup of the of GitMS + Gerrit configuration. Where should this
    be saved?
    Backup Location: /tmp
    

11. During the backup of the first node only, you are prompted to back up the database. If the underlying version of Gerrit is being upgraded for example, when Gerrit is re-init'd, it may change the database schema so that it becomes incompatible with a previous version. Therefore, we recommend that you create a backup before installation. If you don't create a backup, then you may not be able to roll back:

    
    NOTE: This instance of Gerrit has been configured to use the database reviewdb.
    It is recommended that you create a backup of this database now if one
    does not exist already.
    Creating backup...
    Backup saved to: /tmp/gerrit-backup-20150319163550.tar.gz
    Press [Enter] to Continue

12. The backup is taken and its location printed to you. We recommended that you now check the backup file to ensure that it was done successfully.
13. If the underlying versions of Gerrit being upgraded from/to are different, Gerrit may require schema changes to the database prior to running. For example, going from Gerrit 2.9.1 to 2.9.4 requires a schema change that alters some of the primary key settings. You should check the Release Notes for the underlying version of Gerrit and ensure that you have performed all the necessary steps that Gerrit requires for the upgrade.
    Because the option for the first node was set to true earlier in the install, the output at the end describes how to continue the install across multiple sites:

    
    Finalizing Install
    Gerrit Configuration Updated
    GitMS Configuration Updated
    
    GitMS and Gerrit have now been configured. Please restart the GitMS service on
    this node now to finalize the configuration updates.
    
    Next Steps:
    
    
    
    * rsync /home/gerrit/gerrit2114-1702/ to all of your GerritMS nodes

    Should your git repositories directory not be located within your Gerrit directory, this rsync command will indicate that you need to provide the corresponding path:

    * rsync /a/path/to/repos to all of your GerritMS nodes

    * Run this installer on all of your other Gerrit nodes
    * On each of your Gerrit nodes, update gerrit.config:
       - change canonicalURL to the correct hostname
       - ensure that database details are correct
    * Run /home/gerrit/gerrit2114-1702/bin/sync_repo.sh on one node to add any existing
      Gerrit repositories to GitMS. Note that even if this is a new install of
      Gerrit with no user added repositories, running sync_repo.sh is still
      required to ensure that All-Projects is properly replicated.
    * When all nodes have been installed, you are ready to start the Gerrit services
      across all nodes.
    

    rsync If your repos directory is a subdirectory of gerrit, it is brought in during the rsync. If it is not a gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node. See messages above.
    The rsync step is not necessary if you already have Gerrit installed on every node that GerritMS will be installed to. In these cases you can re-run the installer on those sites. This does not happen often, because the utility of standard Gerrit slave nodes is much less than that of master/master nodes.
    

2.1 Run sync_repo.sh script on Node 1

Gerrit is now integrated with Git MultiSite but you still need to modify Gerrit's Git configuration to introduce its repositories to Git MultiSite.

./sync_repo.sh

./gerrit/bin/gerrit.sh

2.2. Test the integration

Before going into production, run through your regular Gerrit workflow. For example:

1. Clone a repository.
2. Add a test file.
3. Commit and push to your Gerrit magic branch.
4. Check that you get a URL for the review.
5. Log in to Gerrit on each node and confirm the review has replicated, you should see it if you click List all.
6. Add a comment, e.g. "Looks good to me". Publish the comment and confirm that it replicates to the other nodes.
7. Next,the project owner should (from a different node) approve the review. This should trigger Gerrit to merge the change into the master branch and replicate the change across the Git MultiSite nodes.
8. Check that replication has completed properly by logging into the Git MultiSite admin UI and view the Repositories tab. Here you can run a consistency check for the applicable repository.

Gerrit Caching

The Gerrit Code Review for Git provides a number of caches which are use to make faster the response from Gerrit. Examples of the caches used in Gerrit are accounts, diff, groups, projects, permission_sort and so on.

Gerrit caches must be replicated between the nodes. That means that when some cache becomes outdated on one node, it will also get outdated on the other nodes, so that it is possible to use the advantage of the caches without the problems that those caches could bring when something happens on a remote node. to ensure that the caches are enabled in GerritMS you need to add these properties to the application.properties file:

If you experience problems using Gerrit's cache, you can disable it in the gerrit config file using the following example configuration snippet

...
[cache "accounts"]
	memorylimit = 0
	disklimit = 0
[cache "accounts_byemail"]
	memorylimit = 0
	disklimit = 0
...

You will need to change the cache labels, e.g. [cache ""] to match the specific cache that you want to disable.

The integration is now complete! Additional information for managing Gerrit with Git MultiSite in the Gerrit Administration section.

The caches will not trigger an event on the other nodes also if they are disabled in the gerrit.config file.

A reload event is always executed on the receiving node. When a cache value is evicted (outdated) on a particular node, the other nodes receive a message to evict that very same value, and also to reload that value from the database or the repository, so that the part of the Gerrit application which relies on the values read directly from the caches, will always show up-to-date content.

3. Automate the installation

The new installer is similar to the GitMS installer in that it allows for a non-interactive mode for automated test installs. To enter non-interactive mode, set the following environment variables:

GITMS_ROOT:

The location of the GitMS install, i.e: /opt/wandisco/git-multisite

BACKUP_DIR:

The location to store the backup of GERRIT_ROOT taken during installation. Should be writeable by current user.

SCRIPT_INSTALL_DIR:

Where to install the reindex.sh and sync_repo.sh scripts

GERRIT_ROOT:

The Gerrit root directory

GERRIT_USERNAME:

The username of the gerrit user set up to allow GitMS REST calls to Gerrit

GERRIT_PASSWORD:

The HTTP password of the gerrit gitms user

GERRIT_RPGROUP_ID:

The ID of the Replication Group setup for GerritMS

GERRIT_REPO_HOME:

The path to the repositories deployed by Gerrit

GERRIT_EVENTS_PATH:

The path for Gerrit events to be written to in communication with the GitMS replicator

GERRIT_REPLICATED_EVENTS_SEND:

Whether this node should send replicated events to other nodes, true / false

GERRIT_REPLICATED_EVENTS_RECEIVE:

Whether this node should report on replicated events from other nodes, true / false

4. Roll back

You need to manually complete the rollback procedure for GerritMS. This accounts for the variety of potential site scenarios. The database can be a complication of the rollback due to changes that may have happened to it, either caused by Gerrit's upgrade procedure (altered schema) or by using Gerrit before an upgrade. Make sure that you back up the database backup before upgrading your version of Gerrit. During an install/upgrade of GerritMS, a backup of the Gerrit root directory is taken and you are prompted on where it should be stored.

4.1 Root backup

The backup taken during install has the following format:

gerrit-backup-<timestamp>.tar.gz

This should be extracted into its own folder:

mkdir /tmp/backup
mv gerrit-backup-<timestamp>.tar.gz /tmp/backup
cd /tmp/backup
tar -xvf gerrit-backup-<timestamp>.tar.gz
ls -shlt
    total 118M
    118M -rw-rw-r-- 1 gitms gitms 118M Mar 18 13:39 gerrit-backup-<timestamp>.tar.gz
    4.0K drwxrwxr-x 3 gitms gitms 4.0K Mar 18 13:39 backup

The backup extracts into a folder named backup. Inside the folder:

ls
    total 8.0K
    4.0K -rw-rw-r--  1 gitms gitms 1.4K Mar 18 13:39 application.properties
    4.0K drwxrwxr-x 13 gitms gitms 4.0K Mar 18 13:18 gerrit

There are 2 components to a backup taken during install:

• The contents of the Gerrit root directory, excluding the Git repositories
• The GitMS replicator settings in application.properties

4.2 Rollback procedure

1. We recommend that you stop all instances of Gerrit before rolling back any node. As they have a shared database (whether that is a single shared master, master/master or master slave), this avoids changes being written to the database during this process.
2. Compare the current state of application.properties with the backup of application.properties taken during install. A GerritMS install adds several properties to this file prefixed with gerrit.*. In most cases, it is safe to simply replace the current application.properties with the backed up copy. However, if other entries in application.properties have been changed since the upgrade, these will be lost. If this might happen, we recommend that you compare the files to ensure that any additional modifications are mirrored in the backed up copy.
3. When you are happy that the backup copy is correct, replace the existing application.properties file with it.
   Note: For GitMS to pick up these changes, you need to restart it.
4. The gerrit folder has the contents of the Gerrit root directory. Similar to application.properties, compare the existing etc/gerrit.config and etc/secure.config files to verify that no additional changes have been made to these since the upgrade. If, for example, the location of the git repository basePath has changed, the backup needs to be updated to show this.
5. When you are happy that the backup configuration is correct, replace the contents of the existing gerrit installation root directory with those of the backup.
6. The gerrit root is now restored, but the database may still be an issue. Work on the database depends on potential schema changes that the Gerrit project has made from one version to the next. Schema changes are not guaranteed on every update, but they occur fairly frequently. If you know that there are no schema changes between the backed up version of Gerrit and the one being rolled back from, then you do not have to modify the database. If, however, you have to roll it back, there maybe issues because user-initiated changes made to the database after the upgrade will be lost. Information about these changes are in the Gerrit release notes.
7. Whether the database has to be rolled back or not, you must now ensure that the Lucene index for the rolled back version of Gerrit is up to date. Trigger full reindex on gerrit by doing the following:

   java -jar gerrit.war reindex

8. Restart GitMS and Gerrit on that node.

Note: Any steps taken to roll back a shared database only have to be performed once, but do them before any restored Gerrit instance is started.

Upgrade Gerrit MultiSite

This section deals with the installation of new versions of Gerrit MultiSite onto existing deployments. For fresh installations see the main Install Guide.

IMPORTANT
Currently, Gerrit MultiSite only supports upgrades to the next version in sequence, it is not possible to skip to an upgrade to a later version, you must complete each available upgrade in turn.

Using Selective Replication?

If you are using the Selective Replication feature to target repositories at different replication groups using Repository name matching regexes, then you need to check those rules will match against "All-Users". This is a core Gerrit repository, similar to All-Projects, and must be made available to every Gerrit node.

Ensure all nodes relevant nodes are online
All the Git MultiSite nodes that are part of your Gerrit replication group must be online during the upgrade. You can identify these nodes from the property gerrit.rpgroupid in the nodes application.properties file. This is a critical requirement because upgrading to 2.10 from 2.9 involves the creation of a new, All-Users repository. This repository will be automatically deployed to GitMS and replicated to others nodes, but it has the same restraints as a normal repository deployment in GerritMS - all nodes in the targetted replication group must be available.
1. Stop all Gerrit services on nodes.
2. Following this run the Gerrit MultiSite installer. Follow the prompts, confirm that proposed changes to the application.properties are acceptable.
3. Run gerritms-installer.sh on all other nodes.
4. Restart both gitMS and Gerrit on all nodes
5. Following the restart all nodes are set to live.
6. Original reviews are present and new ones are replicated as well as submit rules.
7. Perform further testing to ensure that Gerrit is running properly and that changes are properly replicated.