Make sure that you read the Integration Guide before starting your installation.
1. Software requirements
Install vanilla Gerrit only on your first node. Don't install it on the other nodes because the WANdisco installer manages this. At the end of the installation on the first node, you rsync the whole gerrit root directory and its repos to the next node on which you want to install.
Software requirements, using the required Percona XtraDB database solution:
|Node 1||Node n > 1|
|Vanilla Gerrit-2.11.7||Git MultiSite 1.7.1|
|Git MultiSite 1.7.1||Percona XtraDB Cluster 5.6.28-76.1|
|Database (master-master)||Percona XtraDB Cluster 5.6.28-76.1|
Gerrit supports a number of plugins for integrating additional applications:
- Plugins need to be installed in exactly the same way on every node to ensure deterministic behavior or nodes can lose their sync.
- Plugins that use global configuration of key-value pairs, stored in the gerrit.config will replicate without problem providing they are configured the same on all nodes.
- Plugins with Project-level configuration (stored in project.config within refs/meta/config) should replicate without problem.
- We’re still investigating whether plugins that request data directories for storage can be supported with replication.
Currently we have successfully tested the Gerrit plugins for:
- commit-message-length-validator - validates that commit messages conform to length limits
- download-commands - defines commands for downloading via different network protocols
- reviewnotes - retains review history for a gerrit project under refs/notes/review, which is replicated automatically by GerritMS
- singleusergroup - provides a group per user which is useful if you want to assign access rights directly to a single user
Gerrit plugins which are known not to work - do not use these plugins:
- replication - provides master-slave replication, and therefore should not be used with GerritMS.
- delete project
2. Install procedure
Follow this procedure to install Gerrit:
- Run the single installer file on the command line.
- Answer questions during the installation. For example, for a new installation you are asked:
- If this install is being done on the first node
- The root directory of GitMS
- The root directory of Gerrit
- The Gerrit admin account username for GitMS to use
- The Gerrit admin account password for GitMS to use
- The root directory for repositories deployed via Gerrit
- The directory to use for publishing Gerrit Events
- Whether the user wants the node to send replicated events
- Whether the user wants the node to receive replicated events
- The name of the default replication group to use for Gerrit in GitMS
- The GitMS username and password
- The location of the backup taken of the Gerrit root directory
- Existing configuration options are detected and used as default options, for example, the location of Git Multisite on the node. However, from an install or upgrade you cannot modify existing GerritMS settings. Any settings which already exist (and have been read from an existing application.properties file) will be automatically filled in and will not be prompted for.
- Make the installer file executable if it is not already:
chmod +x gerritms-installer.shWorkaround if /tmp directory is "noexec"
Running the installer script will write files to the system's
/tmpdirectory. If the system's
/tmpdirectory is mounted with the "noexec" option then you will need to use the following argument when running the installer:
./gerritms-installer.sh --target /opt/wandisco/installation/
- Run the installer:
./gerritms-installer.shThe installer starts up and you see:
[gerrit@dger02 gerrit2114-1702]$ ../installer-1702/gerritms-installer.sh Verifying archive integrity... All good. Uncompressing GerritMS Installer 100% :: :: :: # # ## #### ###### # ##### ##### ##### :::: :::: ::: # # # # ## ## # # # # # # # # # ::::::::::: ::: # # # # # # # # # # # # # # ::::::::::::: ::: # # # # # # # # # # # ##### # # # ::::::::::: ::: # # # # # # # # # # # # # # # :::: :::: ::: ## ## # ## # # # # # # # # # # # :: :: :: # # ## # # # ###### # ##### ##### ##### GerritMS Version: v2.11.4-RP-1.7-DEV-1-g83c5343-dirty Installation Install Documentation: http://docs.wandisco.com/git/gerrit/1.7/gerrit_install.html Welcome to the GerritMS installation. Before the install can continue, you must: * Have Gerrit v2.11.4 installed before beginning * Have backed up your existing Gerrit database * Have a version of GitMS (1.7 or higher) installed and running * Have a replication group created in GitMS containing all Gerrit nodes * Have a valid GitMS admin username/password * Stop the Gerrit service on this node Do you want to continue with the installation? [Y/n]: Is this the first node GerritMS will be installed to? [Y/n]:We recommend that, if you are upgrading, you stop all Gerrit nodes before the upgrade. This prevents changes to the shared database during the upgrade.
- Answer whether this node is the first to to be installed to. This enables better targeted post-install advice.
- Answer whether you require the database to be upgraded. This question is only asked on the first node. Because Gerrit nodes share a database, it is not necessary to backup the database several times over.
- The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:
Currently installing as user: gerrit The current user should be the same as the owner of the GitMS service. If this is not currently the case, please exit and re-run the installer as the appropriate user. Press [Enter] to Continue
- The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:
Configuration Information Git Multisite root directory [/opt/wandisco/git-multisite]:The installer collects details from the user for the install. The first question asked is the location of the GitMS service. A default option is provided, which is determined by the following checks:
- Fetches the gitmsconfig property from ~/.gitconfig and confirms that the application.properties file it points to exists
- If the gitmsconfig property does not exist, itlooks in the default install location for GitMS (/opt/wandisco/git-multisite)
- After providing the installer with the location of GitMS, the installer reads the application.properties file and uses it to prepare any previously configured values, e.g. for an upgrade:
Configuration Information Git Multisite root directory [/opt/wandisco/git-multisite]: Reading GitMS Configuration... Gerrit Root Directory:Where there are properties that are not set in application.properties, the installer prompts you for input. If a property is set in application.properties, then it is re-used. You cannot change previously configured values during installation.Using Regex file with GitMultiSite 184.108.40.206
You need to configure the property
application.propertiesso that it points towards the regex file, in order for Gerrit Project creation to work.
For example, the entry in
application.propertiesmight look like:
The location must be readable and writable by both the Gerrit and Git MultiSite system user.
See Configure the regex file into GitMS
- The installer fetches or asks for the following information:
- Gerrit Root Directory:
- The location of the Gerrit install.
- Gerrit Repository Directory:
- The location of the git repositories on disk that are managed by Gerrit.
- Gerrit Events Directory:
- GitMS and Gerrit will share information with each other via the filesystem. This directory is used to pass events from one process to another. If the directory does not exist at the time of installation, you are prompted to create it.
- Will this node send Replicated Events to other Gerrit nodes? [Y/n]:
- Gerrit nodes can send events that appear in its event stream to other nodes, to allow for a fully replicated event stream where you can monitor events from all Gerrit nodes by simply connecting to one. This option tells the current Gerrit node to publish its events to other nodes.
For more information see Configure Gerrit Event Stream.
- Will this node receive Replicated Events from other Gerrit nodes? [Y/n]:
- This option will configure the current Gerrit node whether to show only its events, or all the replicated events it receives from other nodes too.
- Gerrit Replication Group Name:
- The name of the replication group that contains all Gerrit nodes.
Note: Even if you will run GerritMS with selective replication for the majority of the repositories, at least one Replication Group must contain every Gerrit repository. This is because critical Gerrit configuration settings have been moved from the database to "system repositories" such as All-Projects and All-Users. These repositories will be required across every Gerrit site, and so they must belong to a replication group that covers every Gerrit node.
- GitMS Username and Password:
- Naming the replication group initiates a REST call to the currently-running GitMS installation to fetch the Replication Group ID to match the name. This requires a GitMS admin username and password.
Reading GitMS Configuration... Gerrit Root Directory: /home/gerrit/gerrit2114-1702/ Gerrit Admin Username: admin Gerrit Admin Password: ******** Gerrit Repository Directory: /home/gerrit/gerrit2114-1702/git/ Gerrit Events Path: /home/gerrit/gerrit_events/ Gerrit Receive Replicated Events: true Gerrit Send Replicated Events: true Gerrit Receive Replicated Events as distinct: false Gerrit republish local events as distinct: false Gerrit prefix for current node distinct events: REPL- Gerrit Replicated Cache enabled: true Gerrit Replicated Cache exclude reload for: changes,projects Gerrit Replication Group ID: 53edcbee-8183-11e5-b9e5-005056a97efe
- You are queried about the install path of various helper scripts that help manage GerritMS. By default. these are placed in the GERRIT_ROOT/bin directory:
Helper Scripts We provide some optional scripts to aid in installation/administration of GerritMS. Where should these scripts be installed? Helper Script Install Directory [/home/gerrit/gerrit2114-1702/bin]:The helper scripts are:
- A review can appear out of sync on one Gerrit UI compared to the review's actual status. For example, very occasionally, on the review listing page, a review might be flagged as Submitted, Merge Pending, but may actually be Merged. This is caused by the Lucene index that Gerrit uses failing to update properly. Fix this by providing the ChangeID of the review to this script. This causes a reindex to occur on that individual change.
- Note: This script does not rsync the repositories. Rsync happens at the same time as rsyncing the Gerrit install to the next node. If the repos directory is a subdirectory of gerrit, it is brought in during the rsync. If it is not a gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node.
Any repositories created in Gerrit after GerritMS is installed, are automatically added to GitMS replication. If, however, you already have many repositories managed by Gerrit before installing GerritMS, the process to add them individually to replication can be tedious. This script iterates over the Gerrit repository root and automatically adds any repositories it finds to GitMS.
- A backup of GERRIT_ROOT is taken before any upgrade happens. You are asked where to store this backup. If the location you give does not exist, the installer prompts to create it:
Backup Information Taking a backup of the of GitMS + Gerrit configuration. Where should this be saved? Backup Location: /tmp
- During the backup of the first node only, you are prompted to back up the database. If the underlying version of Gerrit is being upgraded for example, when Gerrit is re-init'd, it may change the database schema so that it becomes incompatible with a previous version. Therefore, we recommend that you create a backup before installation. If you don't create a backup, then you may not be able to roll back:
NOTE: This instance of Gerrit has been configured to use the database reviewdb. It is recommended that you create a backup of this database now if one does not exist already. Creating backup... Backup saved to: /tmp/gerrit-backup-20150319163550.tar.gz Press [Enter] to Continue
- The backup is taken and its location printed to you. We recommended that you now check the backup file to ensure that it was done successfully.
- If the underlying versions of Gerrit being upgraded from/to are different, Gerrit may require schema changes to the database prior to running. For example, going from Gerrit 2.9.1 to 2.9.4 requires a schema change that alters some of the primary key settings. You should check the Release Notes for the underlying version of Gerrit and ensure that you have performed all the necessary steps that Gerrit requires for the upgrade.
Because the option for the first node was set to true earlier in the install, the output at the end describes how to continue the install across multiple sites:
Should your git repositories directory not be located within your Gerrit directory, this rsync command will indicate that you need to provide the corresponding path:
Finalizing Install Gerrit Configuration Updated GitMS Configuration Updated GitMS and Gerrit have now been configured. Please restart the GitMS service on this node now to finalize the configuration updates. Next Steps: * rsync /home/gerrit/gerrit2114-1702/ to all of your GerritMS nodes
* rsync /a/path/to/repos to all of your GerritMS nodes
* Run this installer on all of your other Gerrit nodes * On each of your Gerrit nodes, update gerrit.config: - change canonicalURL to the correct hostname - ensure that database details are correct * Run /home/gerrit/gerrit2114-1702/bin/sync_repo.sh on one node to add any existing Gerrit repositories to GitMS. Note that even if this is a new install of Gerrit with no user added repositories, running sync_repo.sh is still required to ensure that All-Projects is properly replicated. * When all nodes have been installed, you are ready to start the Gerrit services across all nodes.rsync If your repos directory is a subdirectory of gerrit, it is brought in during the rsync. If it is not a gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node. See messages above.
The rsync step is not necessary if you already have Gerrit installed on every node that GerritMS will be installed to. In these cases you can re-run the installer on those sites. This does not happen often, because the utility of standard Gerrit slave nodes is much less than that of master/master nodes.
2.1 Run sync_repo.sh script on Node 1
Gerrit is now integrated with Git MultiSite but you still need to modify Gerrit's Git configuration to introduce its repositories to Git MultiSite.
./sync_repo.shWhen the script has completed, open a terminal session to each node and start Gerrit up, e.g:
./gerrit/bin/gerrit.shGerrit is now replicating its review activities to all nodes. You should test that this is the case.
2.2. Test the integration
Before going into production, run through your regular Gerrit workflow. For example:
- Clone a repository.
- Add a test file.
- Commit and push to your Gerrit magic branch.
- Check that you get a URL for the review.
- Log in to Gerrit on each node and confirm the review has replicated, you should see it if you click List all.
- Add a comment, e.g. "Looks good to me". Publish the comment and confirm that it replicates to the other nodes.
- Next,the project owner should (from a different node) approve the review. This should trigger Gerrit to merge the change into the master branch and replicate the change across the Git MultiSite nodes.
- Check that replication has completed properly by logging into the Git MultiSite admin UI and view the Repositories tab. Here you can run a consistency check for the applicable repository.
The Gerrit Code Review for Git provides a number of caches which are use to make faster the response from Gerrit. Examples of the caches used in Gerrit are accounts, diff, groups, projects, permission_sort and so on.
Gerrit caches must be replicated between the nodes. That means that when some cache becomes outdated on one node, it will also get outdated on the other nodes, so that it is possible to use the advantage of the caches without the problems that those caches could bring when something happens on a remote node. to ensure that the caches are enabled in GerritMS you need to add these properties to the application.properties file:
|Property name||Default value||Description|
||the current node will send its own cache events to the other nodes|
the name of the caches not trigger a reload event on the receiving node.
If you experience problems using Gerrit's cache, you can disable it in the gerrit config file using the following example configuration snippet
... [cache "accounts"] memorylimit = 0 disklimit = 0 [cache "accounts_byemail"] memorylimit = 0 disklimit = 0 ...
You will need to change the cache labels, e.g.
[cache ""] to match the specific cache that you want to disable.
The integration is now complete! Additional information for managing Gerrit with Git MultiSite in the Gerrit Administration section.
Use a local LDAP authority
As we run without LDAP account caching there will be a greater burden placed on your LDAP authority as it deals with all account lookups. For this reason we strongly recommend that you ensure that the LDAP authority is hosted locally rather than via a WAN link which would could result in a significant drop in performance.
The caches will not trigger an event on the other nodes also if they are disabled in the gerrit.config file.
A reload event is always executed on the receiving node. When a cache value is evicted (outdated) on a particular node, the other nodes receive a message to evict that very same value, and also to reload that value from the database or the repository, so that the part of the Gerrit application which relies on the values read directly from the caches, will always show up-to-date content.
3. Automate the installation
The new installer is similar to the GitMS installer in that it allows for a non-interactive mode for automated test installs. To enter non-interactive mode, set the following environment variables:
- The location of the GitMS install, i.e: /opt/wandisco/git-multisite
- The location to store the backup of GERRIT_ROOT taken during installation. Should be writeable by current user.
- Where to install the reindex.sh and sync_repo.sh scripts
- The Gerrit root directory
- The username of the gerrit user set up to allow GitMS REST calls to Gerrit
- The HTTP password of the gerrit gitms user
- The ID of the Replication Group setup for GerritMS
- The path to the repositories deployed by Gerrit
- The path for Gerrit events to be written to in communication with the GitMS replicator
- Whether this node should send replicated events to other nodes, true / false
- Whether this node should report on replicated events from other nodes, true / false
4. Roll back
You need to manually complete the rollback procedure for GerritMS. This accounts for the variety of potential site scenarios. The database can be a complication of the rollback due to changes that may have happened to it, either caused by Gerrit's upgrade procedure (altered schema) or by using Gerrit before an upgrade. Make sure that you back up the database backup before upgrading your version of Gerrit. During an install/upgrade of GerritMS, a backup of the Gerrit root directory is taken and you are prompted on where it should be stored.
4.1 Root backup
The backup taken during install has the following format:
This should be extracted into its own folder:
mkdir /tmp/backup mv gerrit-backup-<timestamp>.tar.gz /tmp/backup cd /tmp/backup tar -xvf gerrit-backup-<timestamp>.tar.gz ls -shlt total 118M 118M -rw-rw-r-- 1 gitms gitms 118M Mar 18 13:39 gerrit-backup-<timestamp>.tar.gz 4.0K drwxrwxr-x 3 gitms gitms 4.0K Mar 18 13:39 backup
The backup extracts into a folder named backup. Inside the folder:
ls total 8.0K 4.0K -rw-rw-r-- 1 gitms gitms 1.4K Mar 18 13:39 application.properties 4.0K drwxrwxr-x 13 gitms gitms 4.0K Mar 18 13:18 gerrit
There are 2 components to a backup taken during install:
- The contents of the Gerrit root directory, excluding the Git repositories
- The GitMS replicator settings in application.properties
4.2 Rollback procedure
- We recommend that you stop all instances of Gerrit before rolling back any node. As they have a shared database (whether that is a single shared master, master/master or master slave), this avoids changes being written to the database during this process.
- Compare the current state of application.properties with the backup of application.properties taken during install. A GerritMS install adds several properties to this file prefixed with gerrit.*. In most cases, it is safe to simply replace the current application.properties with the backed up copy. However, if other entries in application.properties have been changed since the upgrade, these will be lost. If this might happen, we recommend that you compare the files to ensure that any additional modifications are mirrored in the backed up copy.
- When you are happy that the backup copy is correct, replace the existing application.properties file with it.
Note: For GitMS to pick up these changes, you need to restart it.
- The gerrit folder has the contents of the Gerrit root directory. Similar to application.properties, compare the existing etc/gerrit.config and etc/secure.config files to verify that no additional changes have been made to these since the upgrade. If, for example, the location of the git repository basePath has changed, the backup needs to be updated to show this.
- When you are happy that the backup configuration is correct, replace the contents of the existing gerrit installation root directory with those of the backup.
- The gerrit root is now restored, but the database may still be an issue. Work on the database depends on potential schema changes that the Gerrit project has made from one version to the next. Schema changes are not guaranteed on every update, but they occur fairly frequently. If you know that there are no schema changes between the backed up version of Gerrit and the one being rolled back from, then you do not have to modify the database. If, however, you have to roll it back, there maybe issues because user-initiated changes made to the database after the upgrade will be lost. Information about these changes are in the Gerrit release notes.
- Whether the database has to be rolled back or not, you must now ensure that the Lucene index for the rolled back version of Gerrit is up to date. Trigger full reindex on gerrit by doing the following:
java -jar gerrit.war reindex
- Restart GitMS and Gerrit on that node.
- Stop all Gerrit services on nodes.
- Following this run the Gerrit MultiSite installer. Follow the prompts, confirm that proposed changes to the
gerritms-installer.shon all other nodes.
- Restart both gitMS and Gerrit on all nodes
- Following the restart all nodes are set to live.
- Original reviews are present and new ones are replicated as well as submit rules.
- Perform further testing to ensure that Gerrit is running properly and that changes are properly replicated.
Note: Any steps taken to roll back a shared database only have to be performed once, but do them before any restored Gerrit instance is started.
Upgrade Gerrit MultiSite
This section deals with the installation of new versions of Gerrit MultiSite onto existing deployments. For fresh installations see the main Install Guide.
Currently, Gerrit MultiSite only supports upgrades to the next version in sequence, it is not possible to skip to an upgrade to a later version, you must complete each available upgrade in turn.
Using Selective Replication?
If you are using the Selective Replication feature to target repositories at different replication groups using Repository name matching regexes, then you need to check those rules will match against "All-Users". This is a core Gerrit repository, similar to All-Projects, and must be made available to every Gerrit node.
All the Git MultiSite nodes that are part of your Gerrit replication group must be online during the upgrade. You can identify these nodes from the property
gerrit.rpgroupidin the nodes
application.propertiesfile. This is a critical requirement because upgrading to 2.10 from 2.9 involves the creation of a new, All-Users repository. This repository will be automatically deployed to GitMS and replicated to others nodes, but it has the same restraints as a normal repository deployment in GerritMS - all nodes in the targetted replication group must be available.