1. Introduction
Welcome to the User Guide for WANdisco’s Gerrit MultiSite 1.10.2. Find User Guides for earlier versions in the Archive.
Gerrit is an open-source code review tool that works with Git. Together, Git and Gerrit give development teams a solid workflow for centralized Git management that reduces unexpected errors and catches code issues before they cause problems.
Gerrit MultiSite (GerritMS) uses WANdisco’s Git MultiSite (GitMS) to enable distributed teams to collaborate using a distributed Gerrit deployment with no downtime, no disruption and a consistent security policy enforcement across all locations.
1.1. Get support
See our online knowledge base which contains updates and more information.
If you need more help, raise a case on our community support website.
We use terms like node and replication group. For explanations, see the Glossary. |
If you find an error or if you think that some information needs improving, raise a case or email docs@wandisco.com.
1.2. Symbols in the documentation
In this document we highlight types of information using the following boxes:
Alert
The alert symbol highlights important information.
|
Tip
Tips are principles or practices that you’ll benefit from knowing or using.
|
Stop
The STOP symbol cautions you against doing something.
|
Knowledgebase
The i symbol shows where you can find more information in our online Knowledgebase.
|
1.3. Release Notes
View the Release Notes. These provide the latest information about the current release, including lists of new functionality, fixes, and known issues.
2. Installation Guide
This installation section assumed that you are installing a brand new deployment. If you don’t already have WANdisco Git MultiSite running, carry on to Software requirements.
If you are installing into an existing Git MultiSite deployment, you should first read the Integration Guide.
2.1. Software requirements
Read through this section to ensure that you have the correct software and correct versions for running GerritMS 1.10. See the Release Notes for the latest information about which software versions you will need.
2.1.1. Ensure that you are running a compatible version of Gerrit
Ensure you are running a version of Gerrit that is compatible with GerritMS 1.10.
-
Gerrit - 2.16.27
-
Git MultiSite - 1.10.2
For the latest information, see Gerrit - GerritMS version pairing. |
2.1.2. Install Gerrit only on the first node
Install Vanilla Gerrit only on your first node. Don’t install Gerrit on the other nodes, the WANdisco installer will handle this. At the end of the installation, on the first node, you copy (using rsync) the whole Gerrit root directory/repositories to the next node.
2.1.3. Database Requirements
The WANdisco GerritMS 1.10.0 release (running Gerrit 2.16.x) does not support the use of SQL (Percona) for handling Gerrit metadata. A later release will support the online migration of Gerrit metadata from ReviewDB (SQL) to NoteDB (Git).
Current customers who can do the offline ReviewDB to NoteDB migration as part of their upgrade maintenance time can move forward with this release. All other customers will need to wait for the later release.
Using H2
If your deployment is using H2 then you need to add the following NoteDB configuration, prior to starting the GerritMS installation:
[noteDb "changes"] disableReviewDb = true primaryStorage = note db read = true sequence = true write = true
2.1.4. Replication requirements
The following limits apply to this version of GerritMS:
-
Gerrit currently integrates with a single Replication Group:
Using multiple replication groups with Gerrit is an advanced operation. Before proceeding, contact WANdisco Support. -
All nodes in your Gerrit replication group must be Active or Active-Voters:
Any Gerrit node could also be a Tiebreaker. Passive and Voter-only nodes are not supported.
2.1.5. Authentication
OpenID not compatible: It’s not possible to use Google’s OpenID authentication. If you are planning to use HTTP then you need to ensure that you have an Apache web server running in front of Gerrit.
2.1.6. Caching
Gerrit caches can now be enabled. Cache updates and invalidation happens appropriately based on replicated operations and events. To make sure that caches are properly enabled in GerritMS you need to add properties to the GitMS application.properties
file.
2.1.7. System resources
Protect the server against resource exhaustion
The integration of Gerrit into a GitMS deployment will increase the demands on server resources. Take note of GitMS’s requirements for setting high File descriptor / User process limits. While these requirements are not changed by the addition of Gerrit, it does make resource management even more important.
Gerrit garbage collection
The system administrator should configure Gerrit to run a scheduled garbage collection. This can help ensure that the server doesn’t experience errors or performance downgrade as as result of system resources running out.
Turning off automatic garbage collection (autogc)
The Git feature for automatic garbage collection (autogc) can result in poor performance if applied to large repositories, where every repository change could be followed by a delay as garbage collection is continuously triggered.
Alert
It’s recommended that you turn off autogc , follow the Prevent JGit from running triggered repository garbage collection steps in the Repository Garbage Collection Knowledge base article.
|
For more information, see the GitMS product documentation - Running Garbage Collection. |
2.1.8. Integration with third-party applications
Many Gerrit deployments are integrated with one or more third-party applications. While there are no hard and fast rules for how moving to a replicated environment will affect their operation, the following information may be useful:
Git Hooks
Gerrit does not run standard git hooks in the repositories on which it works. Instead, it has a selection of its own hook plugins.
-
See the list of Gerrit’s own documentation for Supported Hooks.
-
Consult with Gerrit’s official documentation for Gerrit Hook Configuration.
Gerrit event stream
The event stream only publishes events that occur directly on a node.
-
If you have integrations that rely on seeing every event from every node, then you will need to make changes to the configuration.
-
See GerritMS Event Stream for recommended configuration options.
2.1.9. Plugins
Plugins for 3rd-party solutions
These plugins apply to various 3rd-party software, enabling integration with Gerrit. Read the following guidelines:
-
Plugins need to be installed in exactly the same way on every node to ensure deterministic behavior or nodes can lose their sync.
-
Plugins that use global configuration of key-value pairs, stored in the
gerrit.config
will replicate without problem providing they are configured the same on all nodes. -
Plugins with Project-level configuration (stored in
project.config
withinrefs/meta/config
) should replicate without problem. -
We’re still investigating whether plugins that request data directories for storage can be supported with replication - please contact WANdisco support for more information.
Currently we have successfully tested the Gerrit plugins for:
-
commit-message-length-validator - validates that commit messages conform to length limits
-
delete project - deletes or cleans up a project. This is a WANdisco-specific version of the Gerrit plugin, modified to work with GerritMS. see Delete projects.
-
download-commands - defines commands for downloading via different network protocols.
Do not use white space in project namesIt is a known limitation of the
download-commands
plugin that it does not handle projects with white space in their names. It is generally advisable to not use white space in Gerrit project names but if you need to then you can use the following example workaround:If the
download-commands
generates the command:git clone ssh://admin@gerrit.wandisco.com:29418/Project Nightmare
then you need to change it to be:
git clone ssh://admin@gerrit.wandisco.com:29418/"Project Nightmare" ProjectNightmare
-
reviewnotes - retains review history for a Gerrit project under
refs/notes/review
, which is replicated automatically by GerritMS -
singleusergroup - provides a group per user which is useful if you want to assign access rights directly to a single user
Gerrit plugins that are incompatible with GerritMS
-
delete project - the standard version of Gerrit’s "delete project" plugin is not supported. Use the WANdisco-specific "delete project" plugin.
-
replication - provides master-slave replication, and therefore should not be used with GerritMS.
-
rename-project - allows projects to be renamed from Gerrit via an SSH command.
Do not use these plugins with GerritMS 1.10
Plugins for Gerrit
-
its-jira - in addition to the Gerrit plugin for JIRA described above, there is a JIRA plugin for Gerrit. In order for this plugin to work it needs to be configured according to Gerrit’s documentation. In addition to these configurations the following needs to be done:
-
Create a new directory called its in the location <install dir>/gerrit/etc/its/.
-
In this directory create an actions-@Plugin@.config: file e.g. actions-its-jira.config. View a sample config file here - actions-its-jira.config
If Gerrit is installed on multiple sites, then the configuration needs to be repeated on all sites.
-
2.2. Install procedure
The following overview lists the steps taken to install GerritMS using the installer script. For a non-interactive installation, see Non-interactive Installation.
-
Run the installer file on the command line.
-
Answer the For example, for a new installation you are asked:
-
Confirm if this installation is being done on the first node
-
The root directory of GitMS
-
The root directory of Gerrit
-
The Gerrit admin account username and password for GitMS to use
-
The root directory for repositories deployed via Gerrit
-
The directory to use for publishing Gerrit Events
-
Whether the user wants the node to send replicated events
-
Whether the user wants the node to receive replicated events
-
The name of the default replication group to use for Gerrit in GitMS
-
The GitMS username and password
-
The location for archiving deleted repositories
Create and specify this directory to use the Gerrit delete project plugin.
-
The location of the backup taken of the Gerrit root directory
-
2.2.1. Existing configuration
Existing configuration options are detected and used as default options, for example, the location of GitMS on the node. However, from an install or upgrade you cannot modify existing GerritMS settings. Any settings which already exist (and have been read from an existing application.properties file) will be automatically filled in and will not be prompted for.
2.2.2. Installation Steps
Follow the steps below to install:
-
Make the installer file executable if it is not already:
chmod +x gerritms-installer.sh
Workaround if /tmp directory is "noexec"Running the installer script will write files to the system’s
/tmp
directory. If the system’s/tmp
directory is mounted with the "noexec" option then you will need to use the following argument when running the installer:--target <someDirectoryWhichCanBeWrittenAndExecuted>
E.g../gerritms-installer.sh --target /opt/wandisco/installation/
-
Run the installer:
./gerritms-installer.sh
The installer starts up and you see:
$ ./gerritms-latest-installer.sh Verifying archive integrity... All good. Uncompressing GerritMS Installer 100% :: :: :: # # ## #### ###### # ##### ##### ##### :::: :::: ::: # # # # ## ## # # # # # # # # # ::::::::::: ::: # # # # # # # # # # # # # # ::::::::::::: ::: # # # # # # # # # # # ##### # # # ::::::::::: ::: # # # # # # # # # # # # # # # :::: :::: ::: ## ## # ## # # # # # # # # # # # :: :: :: # # ## # # # ###### # ##### ##### ##### Configuration Information Gerrit Root Directory: /opt/wandisco/gerrit Checking Java version and JAVA_HOME is set... GerritMS Version: 2.16.21-RP-1.10.0.1-TC2 Installation Install Documentation: http://docs.wandisco.com/gerrit/1.10/#doc_gerritinstall Welcome to the GerritMS installation. Before the install can continue, you must: * Have one of the following gerrit versions installed before beginning: - Gerrit: 2.16.21 - Gerrit MS: v2.11.7-RP-1.7.1.4 - Gerrit MS: v2.11.9-RP-1.7.2.1 - Gerrit MS: v2.11.9-RP-1.7.2.2 - Gerrit MS: v2.13.9-RP-1.9.1.2 - Gerrit MS: v2.13.9-RP-1.9.2.2 - Gerrit MS: v2.13.9-RP-1.9.3.5 - Gerrit MS: v2.13.11-RP-1.9.4.1 - Gerrit MS: v2.13.12-RP-1.9.5.1 - Gerrit MS: v2.13.12-RP-1.9.6.1 - Gerrit MS: v2.13.12-RP-1.9.6.3 * Have backed up your existing Gerrit database * Have a version of GitMS (1.9.4 or higher) installed and running * Have a replication group created in GitMS containing all Gerrit nodes * Have a valid GitMS admin username/password * Stop the Gerrit service on this node Do you want to continue with the installation? [Y/n]: y
If upgrading, stop all Gerrit nodes before continuing the upgrade. This will ensure that the shared database is not changed during the upgrade. For the <requiredGerritVersion> please see the release notes.
-
There are 5 environment variables that will affect the use of
curl
. If any of these variables are set then the installer will output the variables and their values.The following environment variables are set and will affect the use of 'curl': * HTTP_PROXY=12345 Do you want to continue with the installation? [Y/n]:
Continuing with these variables setWe advise against continuing if you have the following variables set: HTTP_PROXY, HTTPS_PROXY, FTP_PROXY, ALL_PROXY or NO_PROXY. They may cause curl commands to redirect incorrectly and therefore prevent successful installation
If you have any questions, please contact WANdisco support. -
Confirm if this is the first node or not. The installer will only install Gerrit on the first node. This installation is copied to the other nodes in a later step.
Is this the first node GerritMS will be installed to? [Y/n]:
-
The installer confirm the system user that is being used to install GerritMS.
Currently installing as user: wandisco The current user should be the same as the owner of the GitMS service. If this is not currently the case, please exit and re-run the installer as the appropriate user. Press [Enter] to Continue
-
Answer whether you require the database to be upgraded. This question is only asked on the first node. Because Gerrit nodes share a database, it is not necessary to backup the database several times over.
-
The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:
Currently installing as user: gerrit The current user should be the same as the owner of the GitMS service. If this is not currently the case, please exit and re-run the installer as the appropriate user. Press [Enter] to Continue
-
The installer prints the currently running user and asks you to confirm that this user matches the owner of the GitMS/Gerrit services:
Configuration Information Git Multisite root directory [/opt/wandisco/git-multisite]:
The installer collects details from the user for the install. The first question asked is the location of the GitMS service. A default option is provided, which is determined by the following checks:
-
Fetches the gitmsconfig property from ~/.gitconfig and confirms that the
application.properties
file it points to exists. -
If the gitmsconfig property does not exist, it looks in the default install location for GitMS (/opt/wandisco/git-multisite) If neither of these resolve to a GitMS install, no default option is provided.
-
-
After providing the installer with the location of GitMS, the installer reads the application.properties file and uses it to prepare any previously configured values, e.g. for an upgrade:
Configuration Information Git Multisite root directory [/opt/wandisco/git-multisite]: Reading GitMS Configuration... Gerrit Root Directory:
Where there are properties that are not set in application.properties, the installer prompts you for input. If a property is set in application.properties, then it is re-used. You cannot change previously configured values during installation.
Using Regex file with GitMultiSiteYou need to configure the property
gerrit.rpgroup.regex
in theapplication.properties
so that it points towards the regex file, in order for Gerrit Project creation to work.For example, the entry in
application.properties
might look like:gerrit.rpgroup.regex=/opt/wandisco/gerrit/etc/gitms-regex.txt
The location must be readable and writable by both the Gerrit and GitMS system user.
See Configure the regex file into GitMS -
The installer fetches or asks for the following information:
- Gerrit Root Directory
-
The location of the Gerrit install.
- Gerrit Repository Directory
-
The location of the git repositories on disk that are managed by Gerrit.
- Gerrit Events Directory
-
GitMS and Gerrit will share information with each other via the filesystem. This directory is used to pass events from one process to another. If the directory does not exist at the time of installation, you are prompted to create it.
- Deleted Repositories Directory
-
When using the delete project plugin you can choose to archive and remove repositories. The Deleted Repositories Directory is the location for the deleted repositories to be archived to.
This directory needs to be able to store all the deleted repositories until they can be reviewed and removed. - Will this node send Replicated Events to other Gerrit nodes? [Y/n]
-
Gerrit nodes can send events that appear in its event stream to other nodes, to allow for a fully replicated event stream where you can monitor events from all Gerrit nodes by simply connecting to one. This option tells the current Gerrit node to publish its events to other nodes.
For more information see Configure Gerrit Event Stream. - Will this node receive Replicated Events from other Gerrit nodes? [Y/n]
-
This option will configure the current Gerrit node whether to show only its events, or all the replicated events it receives from other nodes too.
- Gerrit Replication Group Name
-
The name of the replication group that contains all Gerrit nodes.
Note: Even if you will run GerritMS with selective replication for the majority of the repositories, there must be one Replication Group which has every node as a member. This is because critical Gerrit configuration settings have been moved from the database to "system repositories" such as All-Projects and All-Users. These repositories will be required across every Gerrit site, and so they must belong to a replication group that covers every Gerrit node. - GitMS Username and Password
-
Naming the replication group initiates a REST call to the currently-running GitMS installation to fetch the Replication Group ID to match the name. This requires a GitMS admin username and password.
For example:
Reading GitMS Configuration... Gerrit Root Directory: /home/gerrit/gerrit2114-1702/ Gerrit Admin Username: admin Gerrit Admin Password: ******** Gerrit Repository Directory: /home/gerrit/gerrit2114-1702/git/ Gerrit Events Path: /home/gerrit/gerrit_events/ Gerrit Receive Replicated Events: true Gerrit Send Replicated Events: true Gerrit Replicated Cache enabled: true Gerrit Replicated Cache exclude reload for: changes,projects Gerrit Replication Group ID: 53edcbee-8183-11e5-b9e5-005056a97efe
Deleted Repositories Directory The Deleted Repositories Directory is only needed if you are using the Gerrit Delete Project Plugin. Remember that you should periodically review the repositories that are in the Deleted Repositories Directory and physically remove those that are no longer needed. You will need this directory to be capable of storing all deleted project's repositories until they can be reviewed and removed. Location for deleted repositories to be moved to : /home/wandisco/gerrit/git/archiveOfDeletedGitRepositories
-
You are queried about the install path of various helper scripts that help manage GerritMS. By default these are placed in the GERRIT_ROOT/bin directory:
Helper Scripts We provide some optional scripts to aid in installation/administration of GerritMS. Where should these scripts be installed? Helper Script Install Directory [/home/gerrit/gerrit2114-1702/bin]:
The helper scripts are:
- reindex.sh
-
A review can appear out of sync on one Gerrit UI compared to the review’s actual status. For example, very occasionally, on the review listing page, a review might be flagged as Submitted, Merge Pending, but may actually be Merged. This is caused by the Lucene index that Gerrit uses failing to update properly. Fix this by providing the ChangeID of the review to this script. This causes a reindex to occur on that individual change.
- sync_repo.sh
-
Note: This script does not rsync the repositories. Rsync happens at the same time as rsyncing the Gerrit install to the next node. If the repos directory is a subdirectory of Gerrit, it is brought in during the rsync. If it is not a Gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node.
Any repositories created in Gerrit after GerritMS is installed, are automatically added to GitMS replication. If, however, you already have many repositories managed by Gerrit before installing GerritMS, the process to add them individually to replication can be tedious. This script iterates over the Gerrit repository root and automatically adds any repositories it finds to GitMS.
-
A backup of GERRIT_ROOT is taken before any upgrade happens. You are asked where to store this backup. If the location you give does not exist, the installer prompts to create it:
Backup Information Taking a backup of the GitMS + Gerrit configuration. Where should this be saved? Backup Location: /tmp
-
During the backup of the first node only, you are prompted to back up the database. If the underlying version of Gerrit is being upgraded for example, when Gerrit is re-init’d, it may change the database schema so that it becomes incompatible with a previous version. Therefore, we recommend that you create a backup before installation. If you don’t create a backup, then you may not be able to roll back:
NOTE: This instance of Gerrit has been configured to use the database reviewdb. It is recommended that you create a backup of this database now if one does not exist already. Creating backup... Backup saved to: /tmp/gerrit-backup-20150319163550.tar.gz Press [Enter] to Continue
-
The backup is taken and its location printed to you. We recommended that you now check the backup file to ensure that it was done successfully.
-
You now need to edit the
<gerrit.home>/etc/gerrit.config
file to contain the following:[container] startupTimeout = 200
This value needs to be increased from the default of 90 to ensure a restart of Gerrit will be successful. Note that if you have a very large implementation this value may need to be higher. -
If the underlying versions of Gerrit being upgraded from/to are different, Gerrit may require schema changes to the database prior to running. For example, going from Gerrit 2.9.1 to 2.9.4 requires a schema change that alters some of the primary key settings. You should check the Release Notes for the underlying version of Gerrit and ensure that you have performed all the necessary steps that Gerrit requires for the upgrade.
Because the option for the first node was set to true earlier in the install, the output at the end describes how to continue the install across multiple sites:Finalizing Install Gerrit Configuration Updated GitMS Configuration Updated GitMS and Gerrit have now been configured. Please restart the GitMS service on this node now to finalize the configuration updates. Next Steps: * rsync /home/gerrit/gerrit2114-1702/ to all of your GerritMS nodes
Should your git repositories directory not be located within your Gerrit directory, this rsync command will indicate that you need to provide the corresponding path:
* rsync /a/path/to/repos to all of your GerritMS nodes
* Run this installer on all of your other Gerrit nodes * On each of your Gerrit nodes, update gerrit.config: - change canonicalURL to the correct hostname - ensure that database details are correct * Run /home/gerrit/gerrit2114-1702/bin/sync_repo.sh on one node to add any existing Gerrit repositories to GitMS. Note that even if this is a new install of Gerrit with no user added repositories, running sync_repo.sh is still required to ensure that All-Projects is properly replicated. * When all nodes have been installed, you are ready to start the Gerrit services across all nodes.
rsyncIf your repos directory is a subdirectory of Gerrit, it is brought in during the rsync. If it is not a Gerrit subdirectory, you are prompted to rsync it at the end of the install on the first node.
2.2.3. Run sync_repo.sh script on Node 1
Gerrit is now integrated with GitMS but you still need to modify Gerrit’s Git configuration to introduce its repositories to GitMS.
./sync_repo.sh
When the script has completed, open a terminal session to each node and start Gerrit up, e.g:
./gerrit/bin/gerrit.sh
Gerrit is now replicating its review activities to all nodes. You should test that this is the case.
2.2.4. Test the integration
Before going into production, run through your regular Gerrit workflow. For example:
-
Clone a repository.
-
Add a test file.
-
Commit and push to your Gerrit magic branch.
-
Check that you get a URL for the review.
-
Log in to Gerrit on each node and confirm the review has replicated, you should see it if you click List all.
-
Add a comment, e.g. "Looks good to me". Publish the comment and confirm that it replicates to the other nodes.
-
Next,the project owner should (from a different node) approve the review. This should trigger Gerrit to merge the change into the master branch and replicate the change across the GitMS nodes.
-
Check that replication has completed properly by logging into the GitMS admin UI and view the Repositories tab. Here you can run a consistency check for the applicable repository.
2.2.5. Gerrit Caching
The Gerrit Code Review for Git provides a number of caches which are use to speed up the response from Gerrit. Examples of the caches used in Gerrit are accounts, diff, groups, projects, permission_sort and so on.
Gerrit caches must be replicated between the nodes.
That means that when some cache becomes outdated on one node, it will also get outdated on the other nodes, so that it is possible to use the advantage of the caches without the problems that those caches could bring when something happens on a remote node.
To ensure that the caches are enabled in GerritMS you need to add these properties to the GitMS application.properties
file:
Property name | Default value | Description |
---|---|---|
|
|
The current node will send its own cache events to the other nodes |
|
|
The name of the caches that will not trigger a reload event on the receiving node. |
Restart required
You need to restart GitMS for any changes to the Gerrit replicated events properties in the GitMS application properties file to take effect.
|
If you experience problems using Gerrit’s cache, you can disable it in the Gerrit config file using the following example configuration snippet
... [cache "accounts"] memorylimit = 0 disklimit = 0 [cache "accounts_byemail"] memorylimit = 0 disklimit = 0 ...
You will need to change the cache labels, e.g. [cache ""]
to match the specific cache that you want to disable.
The integration is now complete! Additional information for managing GerritMS in the Admin section.
Use a local LDAP authority
As we run without LDAP account caching there will be a greater burden placed on your LDAP authority as it deals with all account lookups.
For this reason we strongly recommend that you ensure that the LDAP authority is hosted locally rather than via a WAN link which would could result in a significant drop in performance.
|
A reload event is always executed on the receiving node.
When a cache value is evicted (outdated) on a particular node, the other nodes receive a message to evict that very same value, and also to reload that value from the database or the repository, so that the part of the Gerrit application which relies on the values read directly from the caches, will always show up-to-date content.
Conversely, if a Gerrit cache is disabled in the gerrit.config
file then that cache will not communicate the eviction.
Therefore, if you disable a Gerrit cache on one node you should disable that cache on all nodes.
2.3. Non-interactive installation
You can also install GerritMS with an unattended (scripted) install. Set the following environment variables (defaults are shown):
Variable | Description |
---|---|
|
The location of the Gerrit backups |
|
Permit enironmental variables in curl commands. |
|
Database backup. |
|
Directory where the deleted repos will be archived. |
|
Confirms if Gerrit will be installed. Set "true" to include Gerrit, "false" to not install Gerrit. |
|
Path to where replicated events will be stored. |
|
Set the path for the Gerrit Helper script. |
|
Whether this node should receive replicated events from other nodes |
|
Directory where the deleted repos will be archived. |
|
Path to Gerrit’s Git directory |
|
Path to Gerrit installation |
|
The UUID of the replication group to be used by Gerrit to the deploy repositories to. |
|
The location of the GitMS install |
export REMOVE_PLUGIN="false" |
|
---|---|
|
Whether to run the Gerrit Initialization. |
|
This needs to set if you will use the Gerrit Delete Project Plugin.
If you intend to use the Gerrit Delete Project Plugin, ensure that the deletedRepositories directory is set |
If this deletedRepositories directory does not exist and cannot be created, abort the install (if you want to use the Gerrit Delete Project Plugin).
2.3.1. Example Non-interactive installation
$ echo $GERRIT_MYSQL
$ /opt/wandisco/gerritms-1.10.x.y-installer.sh
Verifying archive integrity... All good.
Uncompressing GerritMS Installer 100%
Starting non-interactive install of GerritMS...
Using settings:
GITMS_ROOT: /opt/wandisco/git-multisite
GERRIT_ROOT: /opt/wandisco/gerrit/
GERRIT_RPGROUP_ID: 4d45537b-a643-11ea-90d8-080027ead692
GERRIT_REPO_HOME: /opt/repos/git/
DELETED_REPO_DIRECTORY: /opt/repos/git/archiveOfDeletedGitRepositories
GERRIT_EVENTS_PATH: /opt/wandisco/gerrit/events/
GERRIT_REPLICATED_EVENTS_SEND: true
GERRIT_REPLICATED_EVENTS_RECEIVE_ORIGINAL: true
GERRIT_REPLICATED_EVENTS_RECEIVE_DISTINCT: false
GERRIT_REPLICATED_EVENTS_LOCAL_REPUBLISH_DISTINCT: false
GERRIT_REPLICATED_EVENTS_DISTINCT_PREFIX: REPL-
GERRIT_REPLICATED_CACHE_ENABLED: true
GERRIT_REPLICATED_CACHE_NAMES_NOT_TO_RELOAD: changes,projects
BACKUP_DIR: /opt/wandisco/gerrit/backups/
GERRIT_HELPER_SCRIPT_INSTALL_DIR: /opt/wandisco/gerrit/bin/
FIRST_NODE: true
CURL_ENVVARS_APPROVED: true
Directory is /opt/repos/git/archiveOfDeletedGitRepositories
Non-interactive install completed
2.4. Roll back
If, for any reason, your upgraded Gerrit is not working and you determine that you need to roll back, then you will need to manually complete the rollback procedure for GerritMS. This accounts for the variety of potential site scenarios. The database can be a complication of the rollback due to changes that may have happened to it, either caused by Gerrit’s upgrade procedure (altered schema) or by using Gerrit before an upgrade. Make sure that you back up the database backup before upgrading your version of Gerrit. During an install/upgrade of GerritMS, a backup of the Gerrit root directory is taken and you are prompted on where it should be stored.
2.4.1. Root backup
The backup taken during install has the following format:
gerrit-backup-<timestamp>.tar.gz
This should be extracted into its own folder:
mkdir /tmp/backup mv gerrit-backup-<timestamp>.tar.gz /tmp/backup cd /tmp/backup tar -xvf gerrit-backup-<timestamp>.tar.gz ls -shlt total 118M 118M -rw-rw-r-- 1 gitms gitms 118M Mar 18 13:39 gerrit-backup-<timestamp>.tar.gz 4.0K drwxrwxr-x 3 gitms gitms 4.0K Mar 18 13:39 backup
The backup extracts into a folder named backup. Inside the folder:
ls total 8.0K 4.0K -rw-rw-r-- 1 gitms gitms 1.4K Mar 18 13:39 application.properties 4.0K drwxrwxr-x 13 gitms gitms 4.0K Mar 18 13:18 gerrit
There are 2 components to a backup taken during install:
-
The contents of the Gerrit root directory, excluding the Git repositories
-
The GitMS replicator settings in application.properties
2.4.2. Rollback procedure
You must stop all instances of Gerrit before rolling back any node. As they have a shared database (whether that is a single shared master, master/master or master slave), this avoids changes being written to the database during this process.
Compare the current state of application.properties with the backup of application.properties taken during install.
A GerritMS install adds several properties to this file prefixed with Gerrit. In most cases, it is safe to simply replace the current application.properties with the backed up copy. However, if other entries in application.properties have been changed since the upgrade, these will be lost. If this might happen, we recommend that you compare the files to ensure that any additional modifications are mirrored in the backed up copy.
When you are happy that the backup copy is correct, replace the existing application.properties file with it.
Note: For GitMS to pick up these changes, you need to restart it.
The Gerrit folder has the contents of the Gerrit root directory. Similar to application.properties, compare the existing etc/gerrit.config and etc/secure.config files to verify that no additional changes have been made to these since the upgrade. If, for example, the location of the git repository basePath has changed, the backup needs to be updated to show this.
When you are happy that the backup configuration is correct, replace the contents of the existing Gerrit installation root directory with those of the backup.
The Gerrit root is now restored, but the database may still be an issue. Work on the database depends on potential schema changes that the Gerrit project has made from one version to the next. Schema changes are not guaranteed on every update, but they occur fairly frequently. If you know that there are no schema changes between the backed up version of Gerrit and the one being rolled back from, then you do not have to modify the database. If, however, you have to roll it back, there maybe issues because user-initiated changes made to the database after the upgrade will be lost. Information about these changes are in the Gerrit release notes.
Whether the database has to be rolled back or not, you must now ensure that the Lucene index for the rolled back version of Gerrit is up to date.
Trigger full reindex on Gerrit by doing the following:
java -jar gerrit.war reindex
When the above steps are complete, restart GitMS and Gerrit on that node.
Note: Any steps taken to roll back a shared database only have to be performed once, but do them before any restored Gerrit instance is started.
2.5. Upgrade Gerrit MultiSite
This section deals with the installation of new versions of GerritMS onto existing deployments. For fresh installations see the main Installation Guide.
Before you begin upgrade you need to make the following updates as outlined in the Release notes.
These will enable a successful restart after they upgrade:
-
If you have any redundant slashes, you need to remove them before upgrade is attempted. Redundant slashes include any duplication of slashes or a slash at the end of the string.
The following steps need to be performed for any GerritMS repositories that have redundant slashes in their GitMS file path. The example uses the example repositoryAll-Users.git
:-
Remove the
All-Users.git
repository from GitMS -
Add the
All-Users.git
repository back into GitMS, this time without redundant slashes -
Edit the
application.properties
file and remove the redundant slashes from any path specified therein (especially,gerrit.root
,gerrit.repo.home
)
-
-
You need to edit the
<gerrit.home>/etc/gerrit.config
file to contain the following:[container] startupTimeout = 200
And set the following properties, also in the
gerrit.config
file:[cache "diff"] timeout = 15000 [cache "diff_intraline"] timeout = 15000
The values given are those recommended for a moderate size Gerrit implementation, but they may need to be much larger if you have a large Gerrit deployment. You will need to evaluate the possible timeout values for your size of deployment. For more information on how to determine your cache size see the Gerrit’s documentation.
Please contact WANdisco support if you have any questions.
Altering the configuration file
If you do any work which involves altering the configuration file you need to ensure that the gerrit.serverId configuration value is the value it was first created with.
This was most likely auto-generated during installation or upgrade to 2.13 or above.
The value must be the same at all sites.
|
2.5.1. Upgrade
The upgrade of GerritMS requires that the following steps be taken.
Please read carefully, all the way to the end of this section. Contact WANdisco support if you have any questions about the process. Additionally, please ensure that you read the latest changes in the Gerrit Release Notes. |
-
Stop Gerrit everywhere.
-
Prevent system restart from starting Gerrit everywhere.
-
-
Upgrade GitMS install everywhere (See GitMS Upgrade Procedure)
-
You may receive warnings from GitMS for Gerrit properties that are needed for the new version and are not yet in place. Ignore these warnings until GerritMS is upgraded.
-
-
Select a single node to be "the first" to upgrade (call it "First").
-
Iterate from the current Gerrit release to the Gerrit release supported by GerritMS:
-
Upgrade First to the next Gerrit vanilla release "latest" (e.g. go from 2.11.9 to 2.12.latest).
-
Run the vanilla Gerrit init: java -jar gerrit.war init -d /path/to/gerrit
-
-
In Gerrit 2.16 the init command automatically triggers a reindex of projects , which may unexpectedly impact operation.
See Release Notes.
|
-
Iterate.
-
On First run the Gerrit reindex, using the installed vanilla Gerrit war file :
java specialTunables -jar gerrit.war reindex -d /path/to/gerrit
Tunables: need all java command line arguments (e.g. "-Xmx")
always set -Xms = -Xmx
-
When the reindex is finished, install GerritMS on the First node.
-
-
When the process asks "Where is GitMS?", it will read properties from the previous GerritMS install and only prompt for newly required properties.
-
It will ask "Is this the 1st node?" - Answer "y".
-
Finish the GerritMS installation.
-
When installation is finished:
-
-
Copy (rsync) Gerrit install directory - skipping/excluding "git" directory (repos) - to all other sites.
-
Edit remote site copies to make sure that the following are correct for the site:
-
canonicalWebURL
-
sshd.host
-
httpd.host
-
smtp settings - if configured
-
-
Copy (rsync) the "All-*" repos to all other sites.
-
Sanity check the "All-*" repos by doing a Consistency Check in GitMS.
-
Run GerritMS installer on all other nodes in parallel.
-
-
-
When asked if 1st node say "No" (this will fix up the local application.properties).
-
After GerritMS installs are finished, restart GitMS on all sites.
-
Start Gerrit on all sites.
-
Post upgrade: Using Gerrit specific features
If you want specific Gerrit features that had previously been used, then after the upgrade you need to run some additional commands, depending on the type of feature. Here are two examples:
|
2.5.2. Using Selective Replication?
If you are using the Selective Replication feature to target repositories at different replication groups using Repository name matching regexes, then you need to check those rules will match against "All-Users". This is a core Gerrit repository, similar to All-Projects, and must be made available to every Gerrit node.
Ensure all nodes relevant nodes are online
All the GitMS nodes that are part of your Gerrit replication group must be online during the upgrade.
You can identify these nodes from the property gerrit.rpgroupid in the nodes application.properties file.
This is a critical requirement because upgrading to 2.10 from 2.9 involves the creation of a new, All-Users repository.
This repository will be automatically deployed to GitMS and replicated to others nodes, but it has the same constraints as a normal repository deployment in GerritMS - all nodes in the targeted replication group must be available.
|
-
Stop all Gerrit services on all GerritMS nodes.
-
Following this run the GerritMS installer. Follow the prompts, confirm that proposed changes to the
application.properties
are acceptable. -
Run
gerritms-installer.sh
on all other nodes. -
Restart both GitMS and Gerrit on all nodes
-
Following the restart all nodes are set to live.
-
Original reviews are present and new ones are replicated as well as submit rules.
-
Perform further testing to ensure that Gerrit is running properly and that changes are properly replicated.
2.6. Uninstall GerritMS
Use the following steps should you need to remove Gerrit MultiSite.
2.6.1. Return Standard Gerrit sequencing
To return a deployment to "vanilla" Gerrit you must remove GerritMS sequencing.
Only perform this step if you are removing GerritMS from your Gerrit deployment. |
Verify that all sequences are using the GerritMS form
-
Create a new user account.
-
Create a new group.
-
Create a new code review.
-
Now check the sequences in All-Projects/All-users repositories using the following commands:
cd ~/gerrit/git/All-Projects; git show refs/sequences/changes cd ~/gerrit/git/All-Users; git show refs/sequences/accounts; git show refs/sequences/groups;
The sequences should print in the following format.
bbc929e3-237a-4c23-bc80-ca11a2e7e5ea:123
Verify that all sequences are using the GerritMS form
-
Run the uninstall command, taking note of the following pre-requisites:
-
Ensure that you are running at a site where ALL repositories are present and removed from replication.
-
GitMS has been shut down.
java -jar gerrit/bin/gerrit.war UninstallWD -d ~/gerrit
[wandisco@gitnode1 All-Projects.git]$ java -jar /home/wandisco/gerrit/bin/gerrit.war UninstallWD -d /home/wandisco/gerrit +================================================================+ | Uninstalling WANdisco GerritMS | | | | You are now reverting GerritMS back to a vanilla Gerrit | | installation. | | | | | | Please ensure you are running at a site with ALL repositories | | present and removed from replication & that GitMS has been | | shut down. | +================================================================+ Please confirm you have removed ALL repositories from replication, including All-Projects and All-Users and you wish to continue? [y/N]? y Please remove WANdisco replicated plugins [lfs, delete-project] from /home/wandisco/gerrit/plugins +================================================================+ | Uninstall of GerritMS is now complete | +================================================================+ -> Next steps: - Uninstall replicated Git binaries and replace with vanilla; - Uninstall GitMS; - Reinstall vanilla Gerrit to continue.
-
-
Check the sequence numbers from the previous section. The sequence should now be just the previous integer, rolled forward by one.
For example:
GerritMS sequence
bbc929e3-237a-4c23-bc80-ca11a2e7e5ea:123
Would now be:
124
-
Uninstall replicated binaries.
yum erase git
-
Uninstall GitMS. On each server, run the following script on all replicated servers:
#This script removes the install of git-multisite to be able to install cleanly service git-multisite stop echo "Removing Git-Multisite RPMS" yum remove -y git-multisite git-multisite-gui git-multisite-hook git-multisite-all echo "Removing Git-Multisite Directory" rm -rf /opt/wandisco/git-multisite cd /tmp rm -rf * cd
-
Reinstall Vanilla Git. Follow the guidance provided by your preferred vendor.
3. Integration Guide
This section describes how to integrate an existing Gerrit installation with WANdisco’s GitMS. Please make sure that the selected version of WANdisco’s GitMS matches your desired version of Gerrit (check with the GerritMS release notes).
Integration includes:
-
Prerequisites: Check that your deployment meets all the requirements for running GitMS with Gerrit.
-
Install GitMS on each of your nodes.
-
Induct your nodes so that they can talk to each other.
-
Create a replication group for your Gerrit projects.
-
Add Gerrit-controlled repositories to GitMS.
-
Run sync_repo.sh script on Node 1 to add Gerrit’s repositories to GitMS.
-
Test the integration: Run through your Gerrit workflow project and confirm that everything is replicated to all your nodes.
3.1. Prerequisites
Assumption: You’re already using Gerrit
We assume that you have an established Gerrit installation, along with the required database and authentication mechanism.
To avoid any problems it is advised that all servers should be in the same timezone.
Installing Gerrit
If you need to install Gerrit see Gerrit’s own documentation.
system user: Gerrit installation instructions set the system user as gerrit2 by default.
We recommend that, instead, you create an account called gitms.
Ensure that, whatever user you set up for running Gerrit, this user works through the following procedures and is used for both Gerrit and GitMS.
Note: If you don’t use the same account, then you will probably have permission problems that will stop the integration from working properly.
Avoid port conflict
GitMS will try to use the same port, 8080, that Gerrit uses, by default, for web access.
You MUST configure GitMS to use a different port, e.g. 7070.
|
Auto-start
Gerrit automatically starts when installation is complete. Either:
-
Run the WAR file with the
--no-auto-start
switch. -
Shut down Gerrit before installing GitMS for Gerrit. E.g:
gerrit2@host:~$ ~/gerrit/bin/gerrit.sh stop Stopping Gerrit Code Review: OK gerrit2@host:~$
Note: Check that your deployment meets the requirements.
3.2. Install GitMS
You need to install GitMS. This is a detailed procedure so this document refers to relevant sections of the GitMS documentation.
-
Remembering the Prerequisites, follow the installation procedure.
-
Complete the installation on each of your nodes.
-
Return to these instructions before you create the first replication group.
3.3. Induct your nodes
GitMS is now installed on all your nodes but they’re not yet connected or able to sync changes. You must follow the instructions in the GitMS User Guide. See Node Induction.
3.4. Create a replication group
Follow this procedure to create a replication group. These instructions are carried out on GitMS’s admin UI.
Current limitations
Note the following limitations that apply to this version of GerritMS:
|
-
When you have created and inducted all nodes, log in to the admin console and click the Replication Groups tab. Then click the Create Replication Group button.
Create Replication Group -
Enter a name for the group, if you’re using multiple groups you may want to indicate that this one is for Gerrit repositories, e.g. name it Gerrit-Repositories. Then click the drop-down selector on the Add Nodes field and select each of the other Gerrit/GitMS nodes that you want to replicate between. The local node will automatically be added as you can’t create a replication group remotely. Note the warnings that may appear if the combination of nodes is incorrect.
Enter a name and add some nodes -
Click each node label to set its node type. New nodes are added as Active Voters, denoted by "AV". You should leave this node type in most cases. When used with Gerrit, GitMS only supports Active or Active-Voter node types. For more information, see the GitMS User Guide, #doc_nodetypes[Guide to node types] section.
Don’t change node typeWhen you have added all nodes and confirmed their type, click Create Replication Group to see a confirmation of the replication group’s details.
-
Newly created replication groups appear on the Replication Group tab, but only on the admin UI of nodes that are themselves members of the new group.
Groups boxes, click View view your options
4. Admin Guide
4.1. Running GerritMS as a service
A service unit template is provided to enable GerritMS to run as a sevice. In a default installation, the file can be found here:
/opt/wandisco/gerrit/bin/gerrit-rp.service
To set up the template for running GerritMS as a service, use the following steps.
-
Copy the SystemD service file for GerritMS into the Gerrit "bin" directory.
-
Ensure that it is named "gerrit-rp.service".
-
To make is usable,also copy it to the SystemD location, most commonly this is "/usr/lib/systemd/system".
-
Set the start-up sequence as follows:
systemctl daemon-reload systemctl enable gerrit-rp.service systemctl start gerrit-rp.service
This is not done by default so as to allow system administrators to set how they want to manage the GerritMS service.
# This service unit template file is used to bootstrap user space and manage processes.
# It contains information about how to manage the gerrit server application
# or service, including how to start or stop the service and when it should
# be automatically started.
# If you want to use this template service unit file do not directly edit
# file in the directory. Instead, you should copy the file to the standard location
# for "systemd" unit files on your operating system distro, edit the copy, then enable
# the service file.
[Unit]
Description=Gerrit Code Review
Requires=network.target nss-lookup.target time-sync.target local-fs.target
After=network.target nss-lookup.target time-sync.target local-fs.target
[Service]
Type=forking
User=root
WorkingDirectory=/home/wandisco/gerrit
ExecStart=/home/wandisco/gerrit/bin/gerrit.sh start
ExecStop=/home/wandisco/gerrit/bin/gerrit.sh stop
ExecReload=/home/wandisco/gerrit/bin/gerrit.sh stop; /home/wandisco/gerrit/bin/gerrit.sh start
RemainAfterExit=yes
SyslogIdentifier=GerritCodeReview
[Install]
WantedBy=multi-user.target
4.2. Starting Gerrit in a non-replicated environment
From Gerrit 1.10, GerritMS can be started without enabling any replication functionality. When set to true
, GerritMultiSite will operate like a regular Gerrit installation that will run without GerritMS’s replication capabilities.
This functionality is of use in only a few specific cases. Starting GerritMS in a non-replicated mode, if done improperly, may result in a loss of sync between sites. Contact WANdisco support before using this functionality. |
4.2.1. System Property
You can initialize Gerrit but set it to not run using the following system property:
java -Dgerritms_replication_disabled=true -jar <GerritMS war file location> init -d <install-directory>
- -Dgerritms_replication_disabled
-
When set to "true", GerritMS starts in a non-replicated mode.
4.2.2. Environmental Variable
For cases where you want to start non-replicated GerritMS without making direct changes through running an GerritMS init, you can instead set up the following environmental variable.
GERRITMS_REPLICATION_DISABLED="true"
- GERRITMS_REPLICATION_DISABLED
-
Adding this variable with a value of
true
to the system on which GerritMS will run allows the service to operate in a non-replicated mode. The specifics of how to set this global environmental variable may differ between supported operating systems. Refer to the documentation of the target OS when setting this configuration.-
Using the variable from a command line:
$ export GERRITMS_REPLICATION_DISABLED="true"
Note that used in this way, the variable is only set for the individual terminal session.
-
Removing the variable:
-
Close the terminal or
-
Run the
unset
command, for example:unset GERRITMS_REPLICATION_DISABLED
-
-
4.3. Gerrit administration
Altering the configuration file
If you do any work which involves altering the configuration file you need to ensure that the gerrit.serverId configuration value is the value it was first created with.
This was most likely auto-generated during installation or upgrade to 2.13 or above.
The value must be the same at all sites.
|
Gerrit User licensing
In GerritMS 1.9.4 onwards, active account information transfers from GerritMS to GitMS.
You therefore need to delete any unused accounts, or contact WANdisco support to get the correct number of licenses.
See this Knowledge base article on Gerrit User Licensing in GitMS for more information.
|
4.3.1. Gerrit client authentication
GitMS will default to calling Gerrit as an anonymous user. If the user is anonymous then, due to Gerrit issues, not all accounts may be visible and there will be a limit of 500 items returned per query. For accurate license reporting in GitMS you will need to configure a daemon account with sufficient access to be able to see all of the GerritMS account information.
To configure GitMS to use this account, see the Gerrit client authentication section in the GitMS User Manual. Note that the configuration will require a restart of GitMS.
4.3.2. Add projects
The functionality for adding new projects remains the same as when Gerrit is used outside of a MultiSite deployment. See Gerrit’s own documentation for this works.
4.3.3. Add existing repository
Normally you should create new Gerrit repositories by creating a new Gerrit project. By doing so a newly initialized Git repository will be created and replicated by GitMS. The Replication Group that it is inserted into will be determined by how you have configured Selective Replication.
However, you can also add an existing repository to GitMS directly and, if the path where it is located is within Gerrit’s repository tree then you will be able to manage that repository using Gerrit (by creating appropriate project permissions, etc.). You do so by following the procedure below. To add a repository to GitMS, see the GitMS User Guide #_add_repositories[Add Repository procedure^].
Ensure the repository is in place on all nodes, it must be in exactly the same state.
Before adding a repository, consider running git fsck to ensure its integrity.
You may also wish to run a git gc (See Git Garbage Collection in the Git MultiSite User Guide) before your git fsck for performance reasons.
|
-
Log into GitMS’s admin UI.
-
Click the Repositories tab. Click on the Add button.
Repositories > Add -
Enter the following details:
Repositories > Enter details then click Add Repo- Repo Name
-
Choose a descriptive name. While this doesn’t need to be the folder name (it can be anything you like), it is best to use a consistent naming convention that includes sufficient information to be able to determine the repository folder name. It really is therefore simplest just to use the folder name.
- FS Path
-
The local file system path to the repository. This needs to be the same across all nodes.
To control the repository through Gerrit, ensure that this path is for Gerrit’s repository directory, e.g. <install-path>/gerrit/git/repository1.git
- Replication Group
-
The replication group in which the repository is replicated. It is the replication group that determines which nodes host repository replicas, and what role each replica plays.
- Deny NFF
-
If you would like to allow non-fast-forward changes on the repository, untick this box.
- Global Read-only
-
Check box that lets you add a repository that will be globally read-only. You can deselect this later. In this state GerritMS continues to communicate system changes, such as repository roles and scheduling, however, no repository changes will be accepted, either locally or through proposals that might come in from other nodes.
- Create New Repository
-
If the repository already exists it must be tested before you place it under the control of GitMS. If it doesn’t already exist then tick the Create New Repository box to create it at the same time as adding.
-
Click Add Repo to add the repository for replication.
These operations should not be performed on Gerrit repositories
See Selective Replication for more information on how to integrate Gerrit with only a subset of your replicated repositories. |
4.3.4. Add repository outside of Gerrit’s control
-
Copy the new repository into GitMS’s repository directory (on ALL NODES), making sure the GitMS repository directory is not Gerrit’s repository directory.
Repositories added in this way will not be seen from Gerrit. You will need to enable repository access using either Apache or "git+ssh". -
Add the repository to GitMS using the #_add_repositories[Add Repository procedure].
Adding or removing repositories outside of Gerrit is not recommended.
Repositories created or removed via the filesystem or through GitMS will not be seen in the Gerrit’s project listing, unless the Gerrit project cache is cleared.
We strongly recommend that you always manage Gerrit-based repositories through Gerrit.
|
4.3.5. Delete project
The Delete-project plugin does not work with GerritMS 1.10.0 Please do not use the plugin. A fix for the issue will be available in the next release. See the Release Notes. |
Gerrit stores project information both on disk and in the database. A plugin called delete project has been created to help wipe this data. You will need to use this if you want to delete a project from Gerrit.
There are several reasons you may want to delete a project including:
-
The project has reached the end of its life and needs to be completely removed, but a backup is required for reference/future use.
-
The project has become hard to manage, there are too many outdated/abandoned reviews and it needs to be cleaned up.
Install the delete project plugin
The Delete-project plugin does not work with GerritMS 1.10.0 Please do not use the plugin. A fix for the issue will be available in the next release. See the Release Notes. |
Once you have installed GerritMS (version 1.9.1 or above) on all nodes, to install the delete project plugin you simply run the following command. The plugin has to be installed on nodes which already has GerritMS installed.
java -jar gerrit.war init -d <SITE_PATH> --install-plugin=gerrit-delete-project-plugin --batch
In Gerrit 2.16 the init command automatically triggers a reindex of projects , which may unexpectedly impact operation.
See Release Notes.
|
Run this command on all nodes on which you want to be able to run the delete project plugin. Then restart GerritMS to complete the plugin installation.
Using delete project plugin
The plugin acts differently depending on whether you are deleting a replicated or non replicated project.
-
A Replicated Project will be present on disk on all nodes of the replication group it is a member of, in both the GitMS UI and the Gerrit UI.
If you are deleting a Replicated Project there a 2 options, preserve and no preserve.Gerrit UI appearance - Replicated projects - Clean Up button- Remove Gerrit project data (leave repository replicating in GitMS)
-
If you select this option all associated changes, reviews etc will be removed, the project will remain in the project list on Gerrit, on disk and will continue to replicate within GitMS. It is a clean up of the project rather than complete removal. You will still be able to checkout the repo and commit changes.
- Remove Gerrit project data, remove from replication and archive repository
-
If you select this option you will completely remove the repository. The project and all associated changes, reviews etc will be removed from Gerrit, the repo will be removed from replication in GitMS and will no longer exist on disk.
However, before removal a zipped version of the repo will be archived to the directory specified during install.
-
A Non-Replicated Project is not replicated across a set of nodes and so will likely only appear on one node, both on disk and in the Gerrit UI, but NOT within GitMS.
If you are deleting a Non-Replicated Project, the functionality is the same as the original delete project plugin.Gerrit UI appearance - Non-Replicated projects - Delete button
Node goes down
If a node goes down during the removal process, then when it comes back up and reloads the project cache, the project may still be listed on the GerritMS UI.
If this happens then just flush the project cache.
|
Deleted repositories directory
To delete projects you will need to have created an appropriate directory during install.
The default location is /home/wandisco/gerrit/git/archiveOfDeletedGitRepositories
.
You should periodically review the repositories that are in the directory and physically remove those that are no longer needed.
If deletion fails
If, for example a node is down when you use the delete project plugin, then deletion may fail. If this does happen you will need to perform a manual cleanup.
You will get a 500 server error in Gerrit, and on the GitMS dashboard a failed task will appear (this will only be on the node you used the plugin from), to alert you that deletion has failed.
Deletion failure creates a file in the repository (WD_GitMS_CRITICAL_README) and an archived zip file. To cleanup follow these steps:
-
Remove the WD_GitMS_CRITICAL_README file from the repository you tried to delete and the archived zip file from the deleted repositories directory. This needs to be done on all nodes in the replication group.
-
Take the project out of GRO.
-
Ensure the issue preventing the initial failure is fixed, for example all nodes are up and running without problems.
-
Run the delete project plugin again.
Reintroducing a deleted project
The following steps are used to reintroduce a repository that has been deleted by the Project-Delete plugin
Requirements
-
GerritMS/GitMS installed on 3 Nodes
-
SSH Key added to admin profile on Gerrit
-
Delete plugin installed on all nodes
-
Deleted GerritMSrepository using the delete-project plugin
Steps
Note, all steps are performed on Node1.
-
Create a new directory in the Gerrit repository directory with the name of the deleted repository (In this guide. the name of the deleted repository is assumed to be RestoreRepo):
mkdir /opt/wandisco/gerrit/git/RestoreRepo.git
-
Copy the archived repository (generated by deleting the repository via the delete project plugin) to the newly created RestoreRepo.git
cp /home/wandisco/gerrit/git/archiveOfGitRepositories/RestoreRepo_*.zip /opt/wandisco/gerrit/git/RestoreRepo.git
-
Copy the newly created repository using rsync, or another suitable application to all other nodes:
rsync -a /home/wandisco/gerrit/git/RestoreRepo.git wandisco@192.168.77.191:/opt/wandisco/gerrit/git/ rsync -a /home/wandisco/gerrit/git/RestoreRepo.git wandisco@192.168.77.192:/opt/wandisco/gerrit/git/
-
Run the sync_repo.sh script on ONLY ONE NODE (in this case node 1):
# ./sync_repo.sh Git Multisite root directory (Default: "none"): /opt/wandisco/git-multisite Using GitMS Root: /opt/wandisco/git-multisite GitMS Username: admin GitMS Password: Scanning for repositories in /opt/repository 20201208T005005 Adding "All-Projects": Added All-Projects 20201208T005011 Adding "All-Users": Added All-Users Repository deployment completed There were 2 repositories processed There were 0 already in GitMS There were 2 repositories added There were 4 paths skipped (turn on verbose to see them) There were 0 errors seen
-
Clear the project cache using the following SSH command:
ssh -p 29418 admin@localhost gerrit flush-caches --cache project_list --cache projects
-
Run the online reindex:
ssh -p 29418 admin@localhost gerrit index start projects --force Reindexer started
If the repository is deleted with open changes/reviews, then these will not be restored when the repository itself is restored.
4.3.6. Manually add new Gerrit projects
In a non-replicated Gerrit it is possible to add new projects just by dropping the <repository-name>.git
file into Gerrit’s repository path (gerrit.basePath
), that is, the local file system directory in which Gerrit stores the repositories that it knows about.
Note that:
-
You may need to restart Gerrit or force a cache refresh in order to see the repository.
-
You may find that the repository has an incorrect refs/meta/config entry which could confuse Gerrit.
When running GitMS there are some additional requirements for using this method for adding projects:
Copy the repository to all nodes
You need to ensure that the repository is copied to the same place on all nodes. The safest way to do this is to use rsync, ensuring that you use the following flags which preserve necessary properties (such as owner and group permissions). e.g.
rsync -rvlHtogpc /path/to/local/repo remoteHost:/path/to/remote/repo
Add the repository to GitMS
For changes to the repository to be replicated, including Gerrit tasks, you need to add the repository’s information into GitMS.
This only needs to be done on a single node as the details will be replicated to the other nodes in the replication group.
This can be done in one of two ways:
-
By using the
sync_repo.sh
script (see Install procedure). -
By adding the repository via GitMS directly - see the GitMS User Guide.
In either case the same 2 issues mentioned above can still occur - this time on all replicated nodes.
It’s possible to script/automate the addition of repositories to GitMS using the REST API.
4.3.7. Manage projects in subfolders
Gerrit allows the grouping of repositories under folders just by adding a path into the project name. For example, you can create a project named 'sandboxes/abc'. This will create a repository called 'abc.git' under a folder called 'sandboxes'. You will see this naming convention carry through to GitMS.
4.3.8. Configure for Gerrit event stream
Gerrit Code Review for Git enables SSH clients to receive a stream of events that happen in the Gerrit node where the client is connected to. GerritMS (Gerrit + GitMS) enables those SSH clients to receive, not only the events from that node, but also the events coming from other nodes in the same replication groups.
See Gerrit event stream for more information on how to enable and disable the stream. To see how the stream of events works normally, refer to the Gerrit documentation. Also refer to information on adding repositories.
4.4. Add new node
Follow this procedure if you need to expand your Git/Gerrit deployment to a new location. We assume that you have already completed the initial installation and setup of your Gerrit and GitMS applications.
Overview
4.4.1. Prepare new server
When bringing a new node up, it’s vital that you make sure that it meets all the deployment requirements set in the #_deployment_overview[deployment checklist]. It’s often been considered good practice to create the new server from the image of an existing server so that software and settings are automatically matched.
4.4.2. Install GitMS
Follow our instructions to install GitMS on your new node. See Install GitMS. During setup, you are asked for the license key and the users.properties file. Take these from your first node and copy them to the corresponding locations on your new node.
4.4.3. Induct new node into replication system
When the installation of GitMS has been completed on your new node, you need to add it to your replication ecosystem. This is done by navigating to one of your existing nodes. Log in to its admin UI and click on the Nodes tab. Click on Connect to Node.
- Node Node ID
-
The name of your new node - you would have entered this during the installation of your new node.
- Node Location ID
-
The unique reference code that is used to define the new node’s location - you can verify this from the NODE ID entry on the new node’s Settings tab.
- Node IP Address
-
The IP address of the new node.
- Node Port No
-
The DConE Port number (6444 by default), again this is confirmed from on the new node’s Settings tab.
When these details are entered, click the Send Connection Request button. The new node will receive the connection request and will be added to the current membership. You will need to refresh your browser to see that this has happened. The new node should now appear on the map of on the Nodes screen.
If you run into problems there is more information about Node Induction in the #_node_induction[GitMS User Guide].
4.4.4. Add new node to Gerrit replication group
Now that the new node is in place we need to a join it to our Gerrit replication group, this will tell GitMS to replicate Git and Gerrit data between the existing member nodes and the new node.
-
Log in to an existing node that is a member of the Gerrit replication group, click on the Replication Groups tab button.
-
Click on View at the bottom of your Gerrit replication group’s box.
-
Click the Add Nodes button. You will see the existing membership along with a Select a Node drop-down button.
-
Click the button and select the new node.
-
Click Add Nodes.
Node Role
You can leave the new node’s role as default unless by adding it you will end up with an even number of voter nodes.
If this is the case then either the new node or an existing node must be assigned as an Active TieBreaker to ensure that it’s not possible for a split vote to occur
which would result in a deadlock of the replication system.
See more about Node Types.
|
4.4.5. Place existing node in "Helper mode"
In the next step we need to use rysnc to copy the Gerrit directory from an existing node over to the new node. During the process we need to ensure that the existing node is not replicating or being written to as this could put it out of sync and corrupt the data we’ll be copying. For this reason we select one of the existing nodes to become a helper. While it takes on the role of helper replication to it will halt. Git users who use the node will not be able to make changes during this time.
Select a node and click Start Sync. Take note of the warning about not closing the browser or logging out during this process.
4.4.6. Use rsync to copy Gerrit directory to new node
Opening a terminal window on the Helper node and use rsync to copy the entire Gerrit repository over to the new node.
Instead of copying one or more Git repositories we need to copy the Gerrit folder (in which Gerrit’s repositories are normally stored). If you’re using a non-standard installation location then you’ll need to adapt this step to account for both Gerrit and the git repositories that it controls.
4.4.7. Put helper node back into operation
Once the Gerrit directory has been copied over and verified, click the Complete All button on the GitMS screen. Both helper and new node will now come out of read-only mode. They’ll now be updated with any Git/Gerrit activity that occurred during the procedure.
4.4.8. Start Gerrit on new node
We need to update Gerrit to account for GitMS. Run the Gerrit integration installer in the Gerrit
./installer.sh
The installation will proceed in the say manner as with the original Gerrit integration. However, we’ve already completed the sync of the repository data so the net steps differ slightly from what is described on screen. You do not have to run the sync_repo.sh script. The repositories are already known to Gerrit because you copied that config data in the previous step.
Open a terminal window to the new node and start the Gerrit service, e.g.
~/gerrit/bin/gerrit start
The new node should be up and running. Open the Gerrit UI and verify that all replicated repositories are present in the 'Projects' list.
4.5. Gerrit LFS
From GerritMS 1.9.4, Gerrit Large File Storage (LFS) is supported. Gerrit LFS is based upon Git LFS; see the Github docs for more information on Git LFS.
GerritMS 1.10.0 does not support Git LFS locking. Support will be added in a future release. |
Git stores data as a directed acyclic graph (DAG), a collection of objects that contain references to other objects within your repository. Branches and tags refer to a commit object, with commits referring to their parent commits. There is also a tree object which represents a snapshot of your repository at a time of the commit which contains a blob representing the files in your repository, and nested trees representing the subdirectories. Git creates a new tree object for every commit and new blobs and trees for every file that is modified in your repository. All of this means that Git repositories can become large quickly if large files are frequently modified.
Git LFS is a Git extension which solves the problem of storing large files. It works by replacing large files in your repository with lightweight pointers which are only a few bytes each. The pointers contain a reference to your large files which are stored alongside your repository but outside the DAG. As a user you never see these pointer files on disk as Git LFS transparently swaps them out when you stage your changes or edit an existing commit.
The Gerrit LFS plugin is based on Git LFS, and WANdisco has created a replicated version which allows selected replication across nodes, based on the replication group of the selected repository. The installation process is outlined below and has few changes from the standard Gerrit LFS installation.
4.5.1. Pre-requisites
-
Git LFS v2.4.0 or higher must be installed on all nodes
-
git-LFS
should be installed on all GerritMS servers -
The LFS client package must be installed on both the client and server
-
-
We recommend Git v2.10 or above to have correct functionality
-
In
gerrit.config
,gitBasicAuth
needs to be set to true on all nodes
4.5.2. Gerrit LFS installation
The installation process is outlined below and has few changes from the standard Gerrit LFS installation. See the LFS plugin docs for more information on the standard installation. The main difference is that installation has to be carried out on all nodes.
All custom server configurations must be the same on all nodes. e.g. in gerrit_root/etc/lfs.config
|
-
Install the LFS plugin using the following command. Run this on all nodes.
java -jar /opt/wandisco/gerrit/bin/gerrit.war init -d /opt/wandisco/gerrit --install-plugin=lfs --batch
In Gerrit 2.16 the init command automatically triggers a reindex of projects
, which may unexpectedly impact operation. See Release Notes.After running this, you can check that
plugin=lfs
has been added togerrit.config
in/opt/wandisco/gerrit/etc
. -
Restart Gerrit on all nodes. You can now check on the Gerrit UI that the plugin is installed.
-
Edit the All Projects repository on Gerrit and add to the file
lfs.config
:[lfs "^lfs-test.*"] enabled = true maxObjectSize = 100m
The LFS namespace determines which repositories are LFS enabled, as well as other information about the repositories. By default, no projects are LFS enabled. The namespace can be an exact project name, a pattern or regular expression, e.g.
plugins/mylfs
,sandbox/
or^test-./.*
. See the LFS plugin docs for more information on this.Now commit this to the
refs/meta/config
branch. You only have to edit this on one node as it will be committed to GitMS and replicated. -
Now configure the global plugin settings. this has to be done on all nodes. To do this, create a file called
lfs.config
in<PathToGerritInstall>/etc
. To this file add:[storage] backend = fs
We only support
fs
backend storage, notS3
. -
Restart GitMS.
-
The LFS plugin can now be used in the standard way, the main difference is that LFS repositories must be in a replication group.
Gerrit LFS will not work for a non-replicated repository. The work-around is to create a single-node replication group which contains the repository. This deficiency will be fixed in the next version of GerritMS. |
In order to push large files (>50MiB), the activitytimeout
property in the gitconfig
file needs to be increased from the default (30 seconds).
To change this run the following command:
git config --global lfs.activitytimeout <time in seconds>
The time in seconds value needs to be suitable for your network environment and file sizes. For instance if you upload 1GiB files and have a latency of 333ms then the value needs to be 3660 or higher.
As this is set globally you do not need to repeat this on each repository clone.
Optional configuration
The following properties can be added to the GitMS application.properties
file.
These are not required properties but can be added if configuration is wanted.
Please consult with WANdisco support if you would like to change these tunables from their default values.
-
gerrit.lfs.cache.population.threadcount
: The number of threads used to populate the LFS object cache. Default is 4. -
gerrit.lfs.content.validation
: LFS content validation is to be performed using SHA256 for uploaded content files. Default is true.
4.6. Selective replication
Selective replication enables you to choose certain repositories to exist at some server nodes and not others:
-
Choose a GitMS replication group where all repositories that are not classified should go (unless you want an error).
-
Create and maintain a configuration file that specifies a series of wildcard, replication group UUID pairs.
Note: To get the replication group UUIDs, you can go to the GitMS UI and click the Replication Groups tab, then click Show UUI on a node. With a simple select copy, you can then create the regex configuration file.
Gerrit requires certain repositories, such as All-Projects, in order to function correctly.
If selective replication is enabled, you must ensure that the replication group the All-Projects repo belongs to spans all Gerrit nodes.
Any Gerrit node without a copy of All-Projects will fail to function.
|
4.6.1. Catch-all configuration
In large deployments you might want to integrate Gerrit with only a subset of your replicated repositories, or maybe for separate Gerrit instances to manage different groups of repositories.
Selective replication requires additional configuration.
Use this procedure, explicitly using gerrit.rpgroupid
, if the regex file does not have a wildcard pattern that matches the repository you want to create.
Note: Do not configure this if you want a failure to occur if the regex file does not match the repository/project to be created.
-
Follow the installation procedure to create a replication group.
-
After adding all nodes to the "all projects" replication group, create additional replication groups that can be used for replicating certain repositories between different locations.
Example:
In a 5-node deployment:-
Create replication group GroupA, adding nodes 1, 2, and 3.
-
Create replication group GroupB, adding nodes 3, 4, and 5.
This creates two separate groups of nodes that replicate repositories and integrate with Gerrit independently. Note that Node 3 is a member of both groups and therefore hosts both sets of replicated repositories and will have visibility of all Gerrit reviews that are created.
-
-
Having created and populated your two additional Replication Groups, capture their UUIDs. Either:
-
Via the GitMS UI, click Replication Groups and click Show UUID on a node.
RG tab -
Use the API on one of the nodes in each of the new replication groups:
curl -u admin:pass http://<IP-ADDRESS>:8082/api/replication-groups
-
-
On each node, create a backup copy and then edit the
application.properties
file so that the propertygerrit.rpgroupid
matches with the appropriate replication group. In the example deployment nodes 1, 2 and 3 would have thegerrit.rpgroupid
for GroupA. -
When the
gerrit.rpgroupid
is set on all nodes, restart GitMS on all nodes. -
Following the restart:
-
Each replication group will replicate separately.
-
New repositories will automatically added only to the nodes in the local replication group (e.g. a repository on node1 will be added to GroupA).
-
Node 3, being in both replication groups will replicate everything. New repositories created on node 3 will be added to the replication group that is declared in the node’s application.properties file.
-
When you create a new repository in Gerrit, the name of the repository is taken as a parameter to decide in what replication group it should live. A list of regular expressions is matched against the name and the resulting rule is used to put the new repository in the right replication group.
-
4.6.2. Regex file configuration
The regex file is used to specify a wildcard to be matched against a Gerrit project path as it is being created, in order to select the replication group where it is automatically placed. The matching starts from the top of the file and proceeds downwards until a match is made. If no match is made, either:
-
The catch-all is selected if configured as described.
-
An error message is generated and the project creation fails.
These regex rules are only applied to repository deployments through the Gerrit-specific deploy endpoint.
Therefore, they are not applied to repositories that are deployed through the standard GitMS mechanisms.
|
The regex format contains two wildcard matchers:
-
* is a wildcard for any valid sequence of characters that can be part of a file path. This wildcard does not pass directory boundaries however, so only matches within the current directory.
-
** is similar to * except that the directory boundary restriction does not apply.
Note: All matching is case sensitive.
Examples of wildcard matchers use:
-
*
-
Matches any single directory entry
-
Does not match past a / in either direction
-
-
*<suffix>
-
Matches any single directory entry ending in exactly *<suffix>
-
Does not match past a single / in either direction
-
-
<prefix>*
-
Matches any single directory entry starting in exactly <prefix>*
-
Does not match past a single / in either direction
-
-
<prefix>*<suffix>
-
Matches any single directory entry starting in exactly <prefix> and ending in exactly <suffix> with zero or more characters between them
-
Does not match past a single / in either direction
-
-
**
-
Matches zero or more directory entries
-
Matches past one or more / in either direction
-
You can use wildcard atoms freely in combination with each other, separated by a single / character. The entire sequence of atoms must match or there is no match.
Note these additional constraints:
-
Matching is done from top to bottom of the list configured by the administrator.
-
First match takes precedence.
-
Used alone, ** matches everything, so if used, it should be the last entry in the list. If the ** wildcard is used then the catch all configuration describe above is unnecessary.
The specification should be done via a config file. This is an example regex configuration file:
# Lines starting with '#' are treated as comments # RG1: 0000-0000-0000-0001 # RG2: 0000-0000-0000-0002 # RG3 0000-0000-0000-0003 # RG4: 0000-0000-0000-0004 team1/* => 0000-0000-0000-0001 team2/* => 0000-0000-0000-0002 * => 0000-0000-0000-0003 ** => 0000-0000-0000-0004
In this example, repositories are treated as follows:
team1/repo1 - deployed to RG1 team2/repo2 - deployed to RG2 team2/subdir/repo3 - deployed to RG4. This is because 'team2/*' does not pass the directory boundary repo4 - deployed to RG3. subdir/another_subdir/test/repo5 - deployed to RG4
4.6.3. Configure the regex file into GitMS
You can put the regex configuration file anywhere on the system, as long as it is readable by the user running the GitMS service.
To enable the regex matching, you must update the application.properties
file with the property gerrit.rpgroup.regex
set to the full path of the regex file.
After you change this property to point to a new file location, you must restart the GitMS service on that node.
Changes to the regex file are detected automatically by the replicator so modifications to its contents should not require a restart of GitMS or Gerrit.
However, updates should be done atomically. To accomplish the atomicity the next version of the file should be renamed into place over the current version.
This means that the file is always complete whenever GitMS decides to re-read it.
Regex file monitoring messages
When the GitMS system is initialized with GerritMS enabled, and the gerrit.rpgroup.regex
property is configured, then a File Monitor is started.
This detects changes to the file specified against the gerrit.rpgroup.regex
property in the application.properties
file.
You can see the File Monitor messages in two main places:
-
GitMS log files
-
GitMS dashboard (for errors)
File Monitor initialization messages are:
WARNING: The property gerrit.rpgroup.regex has been defined with the path: " + regexFilePath + ", but this file does not exist
-
File Not Found. The file monitor has not been started.
WARNING: The property gerrit.rpgroup.regex has been defined with the path: " + regexFilePath + ", but cannot be used as it is a directory.
-
File is a Directory. The file monitor has not been started.
WARNING: The property gerrit.rpgroup.regex has been defined with the path: " + regexFilePath + ", but this file cannot be read.
-
File not Readable for GitMS User. The file monitor is started. Therefore, to recover you can correct the file permissions and modify the file. The changes are then picked up by the file monitor.
WARNING: Failed to start selective replication configuration file monitor" + stackTrace
-
Exception Occurs While Starting File Monitor.
INFO: Starting selective replication configuration file monitor.
-
Message Just Prior to Starting File Monitor.
Where the replication group file is updated, changes to the file should be detected automatically by a dedicated Regex File Listener within 10 seconds of the change occurring. Change events come in for Created, Deleted, and Changed. All these cause the replication group expressions to be reloaded.
INFO: RegexFileListener-1:[CREATED - Selective replication configuration created: " + regexFilePath + "]
-
File monitor created.
WARNING: RegexFileListener-1:[DELETED - Selective replication configuration removed: " + regexFilePath + "]" "WARNING: RegexFileListener-1:[The property gerrit.rpgroup.regex has been defined with the path: " + regexFilePath + ", but this file cannot be read.]
-
File monitor deleted. By deleting the selective replication regex file, the system will be reset to use the default replication group ID. In the case where the file has been deleted incorrectly, the file can be recreated, using the same file name, and the file monitor will recognize that the expected file is present again.
INFO: RegexFileListener-1:[CHANGED - Selective replication configuration updated: " + regexFilePath + "]
-
File changed.
File loading messages are:
WARNING: Invalid regex defined: " + line
-
Invalid Regex detected (invalid line)
WARNING: Failed to compile pattern: " + pattern + ", skipped adding regex for Selective Replication.
-
Invalid Regex Detected (Pattern Compilation Error)
WARNING: Attempt to define the same regex multiple times, skipping second occurrence: " + regex
-
Duplicate Patterns Detected
ERROR: Error in reading regex file: " + regexFilePath + "." + stackTrace
-
IO Exception while Reading File (File system issues or unforeseeable errors)
4.7. Technical Reference
4.7.1. Integration architecture
This diagram shows how Gerrit and GitMS interact in a replicated deployment.
Architecture description
-
Gerrit runs much the same as it does in a stand-alone configuration, being a front-end for Git, intercepting push requests, holding them back until the review workflow is completed.
-
The Gerrit database is shared between all nodes via NoteDB.
-
The Lucene index is local to each node. Re-indexing at each node will be triggered based on Gerrit Events arriving at each node.
-
Git changes are fed through Gerrit’s JGit implementation as modified by WANdisco.
-
When the pack file is in place on a sufficient number of nodes (based on the Content Distribution policy - see here) then the proposal for the change is sent to all nodes over the Coordination stream (port 6444).
-
Both GitMS and Gerrit run their own browser-based admin UIs.
-
The ports shown in the diagram above are the default or recommended ports, all of which can be changed if this is necessary for your set up.
5. Troubleshooting
5.1. Logs
When dealing with errors or similar issues, you should view both Gerrit’s and GitMS’s log files.
5.1.1. Gerrit logs
Error and warning messages from the server are automatically written to the log file under <install-dir>/gerrit/etc/
.
This log file is automatically rotated daily (default: 12:00 midnight GMT).
An external log cleaning service (such as logrotate) is required to manage historical log files.
- audit.log
-
audit events
- system.log
-
INFO-level logging
- sshd.log
-
logs connections to Gerrit’s SSH port (not system shell SSH)
- *.gc.log
-
information on garbage collector usage
Gerrit Documentation
For more information about error messages that Gerrit can send, see Gerrit’s own documentation.
|
5.1.2. Git MultiSite logs
GitMS stores logs in 2 locations:
-
Application logs -
/opt/wandisco/git-multisite/
-
Replicator logs -
/opt/wandisco/git-multisite/replicator/logs
For more detailed information on GitMS logs, see the GitMS User Guide.
5.2. Setting your timezone
If your timezone is set incorrectly you may experience Gerrit Event and Indexing replication issues.
If you are using CentOS6 then you can set the timezone by changing the file /etc/localtime
.
However if using CentOS 7, RHEL 7 or SLES 12 you must use the timedatectl
command.
See the Knowledgebase article Setting the timezone on Linux for more information.
5.3. Error messages in a replicated environment
Running Gerrit with GitMS introduces a number of potential errors that are very unlikely to be seen on a single-server deployment.
Below are outlined various instances in which you may get errors, what the cause may be, and the recommended action to take. Errors include 400 invalid project name error, 400 Invalid revision "HEAD", 500 internal server error and command line errors.
Check GitMS is running properly
If you experience any errors when using GerritMS a good first step is to check that GitMS is running without problems (all nodes are up, there are no LRO or GRO repositories, etc).
Errors in GitMS are a common cause of error messages when using Gerrit.
|
5.3.1. User gets a 400 invalid project name error when attempting to view Gerrit
The following issues can cause users to see the above 400 invalid project name error:
Create Project in Gerrit
-
Local GitMS Node Down
Recommended action: Contact the person or organization responsible for administering GitMS and ask them to ensure GitMS nodes are functioning properly. -
Remote GitMS Node Down
GitMS log: Error - Failed to deploy repository: com.wandisco.gitms.api.dto.GitRepositoryDTO@<NUMBER>
Recommended action: The error message "Failed to deploy repository" suggests you should contact the person or organization responsible for administering GitMS and ask them to ensure GitMS nodes are functioning properly. If any nodes are down they need to be brought back up. Network issues may also give this error message. -
Timeout
GitMS log: View sample log message
Recommended action: The error message "Failed to deploy repository" suggests you should contact the person or organization responsible for administering GitMS and ask them to ensure GitMS nodes are functioning properly. If any nodes are down they need to be brought back up. Network issues may also give this error message.
5.3.2. User gets a 400 Invalid revision "HEAD"
-
This error is displayed if you try to create a branch from a GitMS repository which has not had any commits made to it. When creating a branch through the Gerrit UI there will be no initial commit from which to branch off.
Recommended action: The error will disappear after the first commit has been made.
5.3.3. User gets a 500 internal server error when attempting to view Gerrit
The following issues can cause users to see the above 500 internal server error:
Merge Review Error
-
Local GitMS Node Down
Gerrit log: View sample log message
Recommended action: The error message "Failure to replicate update" suggests that either the node you are currently trying to push to, or another node in the replication group is down, causing a replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly. -
No Quorum
Gerrit log: View sample log message
GitMS log:View sample log message
Recommended action: The error message "Minimum required learners are not available" suggests that nodes in the replication group are down, causing a lack of quorum and therefore replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly.
Create Branch
-
Local GitMS Node Down
Gerrit log:View sample log message
Recommended action: The error message "Failure to replicate update" suggests that either the node you are currently trying to push to, or another node in the replication group is down, causing a replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly. -
No Quorum
Gerrit log: View sample log message
GitMS log:View sample log message
Recommended action: The error message "Minimum required learners are not available" suggests that nodes in the replication group are down, causing a lack of quorum and therefore replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly.
Delete Branch
-
Local GitMS Node Down
Gerrit log:View sample log message
Recommended action: The error message "Failure to replicate update" suggests that either the node you are currently trying to push to, or another node in the replication group is down, causing a replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly. -
No Quorum
Gerrit log: View sample log message
GitMS log:View sample log message
Recommended action: The error message "Quorum is not available" suggests that nodes in the replication group are down. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly.
5.3.4. User gets errors in the command line
The following issues can occur in the Git command line when attempting to push up to a Gerrit controlled project.
Git Push to Gerrit-controlled repository
-
Local GitMS Node Down
Git log output:$ git push origin master Password for 'http://admin@192.168.62.191:7070': Counting objects: 3, done. Writing objects: 100% (3/3), 288 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: Processing changes: refs: 1, done To http://admin@192.168.62.191:7070/a/TestRepo ! [remote rejected] master -> master (lock error: Failure to replicate update. ) error: failed to push some refs to 'http://admin@192.168.62.191:7070/a/TestRepo'
Recommended action: The error message "Failure to replicate update" suggests that either the node you are currently trying to push to, or another node in the replication group is down, causing a replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly.
-
Content Distribution Fail
Git log output:$ git push origin master Counting objects: 3, done. Writing objects: 100% (3/3), 289 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: Processing changes: refs: 1, done To ssh://admin@192.168.62.190:29418/TestRepo ! [remote rejected] master -> master (lock error: Failure to replicate update. GitMS - minimum number of learners not available ) error: failed to push some refs to 'ssh://admin@192.168.62.190:29418/TestRepo'
GitMS log: View sample log message
Recommended action: The error message "Minimum required learners are not available" suggests that nodes in the replication group are down, causing a lack of quorum and therefore replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly. -
No Quorum
Git log output:$ git push origin master Password for 'http://admin@192.168.62.191:7070': Counting objects: 3, done. Writing objects: 100% (3/3), 288 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: Processing changes: refs: 1, done To http://admin@192.168.62.191:7070/a/TestRepo ! [remote rejected] master -> master (lock error: Failure to replicate update. GitMS - minimum number of learners not available ) error: failed to push some refs to 'http://admin@192.168.62.191:7070/a/TestRepo'
GitMS log: View sample log message
Recommended action: The error message "Minimum required learners are not available" suggests that nodes in the replication group are down, causing a lack of quorum and therefore replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly.
Create Review
-
Local GitMS Node Down
Git log output:$ git push origin HEAD:refs/for/master Password for 'http://admin@192.168.62.191:7070': Counting objects: 3, done. Writing objects: 100% (3/3), 288 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: Processing changes: refs: 2, done To http://admin@192.168.62.191:7070/a/TestRepo ! [remote rejected] HEAD -> refs/for/master (Unable to create changes: REJECTED_OTHER_REASON lock error: Failure to replicate update.) error: failed to push some refs to 'http://admin@192.168.62.191:7070/a/TestRepo'
Gerrit log: View sample log message
Recommended action: The error message "Failure to replicate update" suggests that either the node you are currently trying to push to, or another node in the replication group is down, causing a replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly. -
No Quorum
Git log output:
$ git push origin HEAD:refs/for/master Password for 'http://admin@192.168.62.191:7070': Counting objects: 3, done. Writing objects: 100% (3/3), 288 bytes | 0 bytes/s, done. Total 3 (delta 0), reused 0 (delta 0) remote: Processing changes: refs: 2, done To http://admin@192.168.62.191:7070/a/TestRepo ! [remote rejected] HEAD -> refs/for/master (Unable to create changes: REJECTED_OTHER_REASON lock error: Failure to replicate update. GitMS - minimum number of learners not available) error: failed to push some refs to 'http://admin@192.168.62.191:7070/a/TestRepo'
Gerrit log: View sample log message
GitMS log: View sample log message
Recommended action: The error message "Minimum required learners are not available" suggests that nodes in the replication group are down, causing a lack of quorum and therefore replication failure. You should contact the person or organization responsible for administering GitMS and ask them to ensure all nodes are running properly.
5.3.5. Understanding push/change failures
Push operations that fail in a replicated environment can be difficult to understand. The cause may be because an atomic commit operation has failed because it failed on a single node. This error may appear as an error containing the following messaging:
remote: Processing changes: (\)error: Failure to replicate batch update. Output is: GitMS - error replicating update. {"statusCode":"UNKNOWN","errorMessage":"Error updating repository: Received 1 failures, while performing revision checks on 3 updates.","updateResults":[{"statusCode":"UNKNOWN","errorMessage":"","updateResultCode":"NOT_ATTEMPTED"},{"statusCode":"CLIENT_ERROR","errorMessage":"Reason : Client sent an update for ref: [ refs/heads/Common01 ], from oldrev: [ 0000000000000000000000000000000000000000 ] to newRev: [ 1dde7923fb97ff90f4703635359df6370c24c351 ] but this ref has a HEAD of: [ 0e1d4737e6e3ba2dc875de171b7d4c905d16718a ]. Client should update to latest revision. ","updateResultCode":"REJECTED_OTHER_REASON"},{"statusCode":"UNKNOWN","errorMessage":"","updateResultCode":"NOT_ATTEMPTED"}]}
In this example the error can be explained by the appearance of the following snippet, that shows that the push was behind in one of their refs:
{"statusCode":"CLIENT_ERROR","errorMessage":"Reason : Client sent an update for ref: [ refs/heads/Common01 ], from oldrev: [ 0000000000000000000000000000000000000000 ] to newRev: [ 1dde7923fb97ff90f4703635359df6370c24c351 ] but this ref has a HEAD of: [ 0e1d4737e6e3ba2dc875de171b7d4c905d16718a ]. Client should update to latest revision. ","updateResultCode":"REJECTED_OTHER_REASON"}
Ensure that you check the logs carefully as these log messages appear on a single line, so can therefore be easily missed.
5.4. Gerrit ACL rules and potential issues with ordering
Gerrit has the concept of storing the ACL rules in both parent repositories and in meta references within a repository itself. WANdisco’s replication system guarantees ordered writes on a repository by repository level, so when ACL rules for a particular repository is defined in a "parent" repository, the ordering of operations between repositories can’t be guaranteed, as each repository is its own distributed state machine.
Example: You could change a rule on a parent repository that impacts a child repository at the time when someone attempts to push something that would be rejected by that rule change. In a scenario where there is moderate network latency it is possible that the push would go through in spite of the just-added proscriptive rule.
So, rule updates on remotes are not applied immediately when applied to parent repositories. If a rule has to be changed on a repository and MUST apply immediately, the correct approach is to edit that individual repository ACL instead of the parent.
5.5. Garbage collection
Gerrit uses JGit to do repository Garbage Collection. There is a bug in JGit’s implementation until JGit 4.6 (Gerrit 2.14) where an object could be lost during the GC operation (see discussion here and actual bugfix here). Therefore, we suggest you use the coordinated GC capabilities built into GitMS. See the Git Garbage Collection for more information. |
Gerrit has its own garbage collection command that can be used to free up system resources by removing objects that are no longer required by Gerrit. However, you should not use the Gerrit garbage collection for a number of reasons. It will:
-
greatly inflate the size of the Gerrit process (perhaps exceeding Java HEAP).
-
require a huge amount of Java GC when it is done. This could make the Gerrit server process freeze for a period of time.
-
not replicate to the other members of the family and could cause divergence.
5.5.1. GerritMS 1.10 has increased demand for Repository Garbage Collection
GerritMS 1.10 now stores its data in the project repository, potentially resulting in a significant increase in the server-side repository size and more replication traffic. Ensure that this increase doesn’t cause an impact on performance due to the resulting increase in server-side processing for Git remote requests.
Guidelines for when to perform repository GC
Git repositories managed by Gerrit/JGit should be repository garbage collected if there are more than 5000 loose objects or more than 200 pack files.
These are advisory GC thresholds. Base your thresholds on your server size, specification (for example, storage I/O, CPU speed, filesystem type), and peak concurrent loading. Using in-production monitoring to schedule your repo RC will ensure that it is effective.
Given the very high level of activity placed on the All-projects and All-users repositories, it is very likely that you will need to increase your garbage collection frequency. |
5.5.2. Setting up garbage collection
It’s good practice to incorporate an automatically scheduled garbage collection. How frequently you need to run repository garbage collection on any specific repository depends on the frequency of updates to that repository. A large accumulation of pack files will slow Gerrit down. Consider a repository GC after 100 updates have been made; definitely after 200 updates. See more about Git Garbage Collection in the Git MultiSite User Guide.
5.6. Unauthorized proposal logging caused by password conflict
The following unauthorized proposal error doesn’t offer many clues to its cause.
2014-11-17 18:53:06 INFO [RestCaller:handle] - Thread-151:[com.wandisco.gitms.repository.handlers.GerritUpdateIndexProposalHandler: output=Unauthorized] 2014-11-17 18:53:29 INFO [NodeEventHandler:handle] - 993753a9-6e7b-11e4-b352-080027776fdc:[Received node event, Unknown macro: {Node}
Cause: GitMS stores a default username and password in its application.properties
file.
e.g.
gerrit.enabled=true
gerrit.rpgroupid=0683d2fc-6e7c-11e4-9956-080027a5ec28
gerrit.repo.home=/home/wandisco/gerrit/git
gerrit.username=admin
gerrit.password=pass
gerrit.events.basepath=/home/wandisco/gerrit_events
gerrit.root=/home/wandisco/gerrit
Changing the user password via Gerrit’s settings screen (shown below) can result in the stated error. Ensure that passwords are matched to fix the problem. Note that
5.7. Reindex process
Below is a diagram that shows how Gerrit’s reindexing system runs. In most cases this system will run without intervention, however, in the event of a persistent reindexing failure, the administrator may need to get involved in order to fix an underlying problem or to trigger a retry.
5.7.1. Description
-
Gerrit detects a change and an index file is created.
-
The file is read and change-id is gathered.
-
The local instance of GitMS sends a proposal for the index file to be replicated to the other nodes.
-
Now, an index request is sent, where the reply is OK, or an error can occur. If an error does occur then the process will retry the sending of the proposal 3 times.
-
After the 3rd failed retry the proposal is stored in a
retry_failed
directory, by default you’ll find the directory in the gerrit-events directory, located (by default) in the same location as the Gerrit installation, usually /opt/wandisco/gerrit/. Change-ids stored in this directory are retried every 30 seconds for the next 24 hours. -
Should a change-id not succeed during the 24 hours it will then be moved to the
definitely_failed
directory. -
The client may elect to push these definitely failed change-ids back out for a retry.
-
A failed change-id that eventually succeeds is deleted from the failed_retry directory. As and when the proposal is accepted, the reindexing will proceed.
5.7.2. Restore Gerrit after temporary loss of replication
Should replication be lost on a deployment, Gerrit will not allow you to make changes on a repository. On GitMS’s dashboard a warning that replication to the lost nodes has failed. The usual GitMS procedure for repository recovery can be used, after which the replication group will catch up.
If a repository replica is lost or becomes unrecoverable then it becomes necessary to perform a manual recovery by using rsync to replace the corrupted/lost replica with a copy of the remaining good replicas.
5.7.3. Force a reindex
If a change index is out of date upon an active review, most often the next event that occurs on it will also pick up the new data that was missed in the previous reindex. This may not be always sufficient however, as generating false content to trigger reindexes is not desired. If a change needs to be updated on a node, the reindex.sh script can be used to update the index.
reindex.sh
The reindex.sh script is shipped with the gerrit-installer, and has the following requirements:
-
Should be run on the node that is to be reindexed.
-
Should have read access to the Gerrit config file.
-
Curl and Git must be installed.
Running the script
./reindex.sh <argument> <argument> <argument> <argument>
The script takes 4 arguments:
- -u: Gerrit account username
-
-
this should be the username of an Administrator account.
-
- -p: Gerrit account HTTP password
-
-
note, this is the password shown in the Gerrit UI, not necessarily the same password that user uses for logging into the Gerrit UI.
-
- -i: ChangeID that must be reindexed
-
-
This ID can be retrieved from the URL of the change.
For example:http://dger03.qava.wandisco.com:8080/#/c/309633/
has a changeID of309633
.
-
- -c: gerrit.config file
-
-
specify location of the Gerrit config file.
-
5.8. Gerrit 2.15/2.16 Strict label validation
From Gerrit 2.15, posting a review including a vote on any labels that do not exist or are outside of the permitted range based on the ACLs can now result in a failure of the entire operation. This feature is currently set to "off" by default.
To enable Strict label validation:
-
Open the
gerrit.config
file. -
Ensure the following parameter and value are included.
[change] strictLabels = true
-
Save the file. Restart Gerrit to apply the change to the node.
Make the edit on a node by node basis as each node handles and enforces the request individually.
5.9. Gerrit event stream
Gerrit Code Review for Git provides ssh clients with the possibility to receive a stream of events that happen in the Gerrit node where the client is connected to. GerritMS enables those ssh clients to receive not only the events from that node, but also the events coming from other nodes in the same replication groups.
To see how the stream of events works normally refer to the standard Gerrit documentation. An example of how to attach to an ssh connection, and the related output is:
$ ssh -p 29418 review.example.com gerrit stream-events {"type":"comment-added",change:{"project":"tools/gerrit", ...}, ...} {"type":"comment-added",change:{"project":"tools/gerrit", ...}, ...}
5.9.1. Enable/disable event stream replication
All the events that happen on a Gerrit node can be shared with other nodes, and a node can receive events coming from the other nodes.
The replication of these events (send and receive) is on by default, i.e. every node will receive by default all the events happening on the other nodes which are on the same replication groups, and every node will send its events to the other nodes.
To modify or disable the replication of these events you need to modify the application.properties
file and change the properties below.
Restart required
You need to restart both Gerrit and GitMS for any changes to the Gerrit replicated events properties in the GitMS application properties file to take effect.
|
Property name | Default value | Description |
---|---|---|
|
|
The current node sends its own events to the other nodes |
|
|
The current node receives events coming from other nodes |
|
|
The current node receives events coming from other nodes - this setting overrides |
Setting send
and/or receive
to false
will make the node stop sending and/or stop receiving events from the other nodes.
Other configurable properties for the events are:
In v1.9.3, the property previously called gerrit.wait.on.event.receive.and.publish changed it’s name to gerrit.max.secs.to.wait.on.poll.and.read .
|
Property name | Default value | Description |
---|---|---|
|
N/A |
Can be used to point Gerrit and the replicator to use a specific directory to exchange the temporary events files. If this is not set, events will be stored temporarily in the |
|
2 |
Maximum number of seconds to wait on the sending event queue before processing the next incoming event. Do not set this value to 0, but it can be set to e.g. 0.5 if faster replication is needed in the sub-second range. |
|
5 |
Maximum number of seconds to queue events in a single file before releasing that file as ready to be sent. Do not set this value to 0, but it can be set to e.g. 0.5 if faster replication is needed in the sub-second range. |
|
30 events |
Maximum number of events to be packaged in one single proposal before (gzipping and) sending to the other nodes.
If the number of events exceeds this value then the events file is set as ready for replication. All the events need to be associated with a single project so this is unlikely. |
|
600L |
The sleep period between reading the files listed in the outgoing events directory. This configurable is a GitMS configurable, if there are no event files in the directory, this is the period that GitMS will wait again before checking. |
|
false |
When true, forces both Gerrit and the replicator to sync the temporary files with the filesystem before proceeding. This could be an additional precaution not to lose any events in the case of an operating system crash. |
|
true |
When true, the replicator will compress the file containing the events before sending it to the other nodes as a proposal |
|
1 second |
The maximum wait time between checks for events to be received or published. If an event is received or published then no wait will occur before the next check. |
|
20 seconds |
How long the queue of unique replicated index events builds for before sending. The delay avoids sending duplicates and prevents the maximum throughput of GitMS being reached. |
|
60 mins |
Used to calculate the time when a change was last indexed. This timeout feature enables indexing to take place without the expected required state changes. |
Example
Example settings of some of the properties described above being set to speed up the delivery of events:
gerrit.replicated.events.secs.on.queue=0.2 gerrit.replicated.events.secs.before.proposing=1 gerrit.replicated.events.ms.between.reads=500L gerrit.max.secs.to.wait.on.poll.and.read=0.5 gerrit.replicated.events.max.append.before.proposing=25 gerrit.replicated.index.unique.changes.queue.wait.time=5
How to enable/disable the Gerrit event stream replication
A dashboard message will be displayed if the number of files in the incoming/outgoing replicated events directory tree exceeds a specified value. The directories are checked periodically (the period is also specifiable). Here is how you configure these checks:
-
Files that Gerrit shares with WANdisco’s replicator are stored in two directories, one for incoming data, the other for outgoing data. i.e.
$ find /home/gerrit/gerrit_events /home/gerrit/gerrit_events/replicated_events/outgoing /home/gerrit/gerrit_events/replicated_events/incoming /home/gerrit/gerrit_events/replicated_events/index_events
There’s a watchdog process that monitors how many files are accumulated in these directories and a warning is sent to the dashboard (and log) when that number reaches a level that you can specify in the configuration:
-
Open application.properties and edit the following lines:
gerrit.replicated.events.incoming.outgoing.threshold.email.notification=200 gerrit.replicated.events.incoming.outgoing.time.email.notification=120000L
- gerrit.replicated.events.incoming.outgoing.threshold.email.notification
-
The combined number of files contained in the Gerrit replicated events incoming and outgoing folders. Default: 200
- gerrit.replicated.events.incoming.outgoing.time.email.notification
-
Period of time to wait between each check on number of files in the incoming/outgoing directories. This is to avoid spam notifications every time a file is added to these directories after the threshold is met. Default 120000L
Note that the "L" at the end of the above parameter is a requirement.
-
Save the file and restart the replicator for the changes to take effect.
How to distinguish lines related to the replication of events in the GitMS log
In the log files of Gerrit and the replicator, the information related to the replicated events is marked with the RE
string to signify Replicated Event.
For example, currently only the standard Gerrit events are supported for the replication. If a custom Gerrit plugin tries to publish an event which is not supported, in the Gerrit log the following line will appear:
RE Event is not supported! Will not be replicated.
To apply a modification to the application.properties
file you need to restart both Gerrit and the replicator.