Ensure you are running a version of Gerrit that is compatible with GerritMS 1.10.

Install Vanilla Gerrit only on your first node. Don’t install Gerrit on the other nodes, the WANdisco installer will handle this. At the end of the installation, on the first node, you copy (using rsync) the whole Gerrit root directory/repositories to the next node.

The WANdisco GerritMS 1.10.0 release (running Gerrit 2.16.x) does not support the use of SQL (Percona) for handling Gerrit metadata. A later release will support the online migration of Gerrit metadata from ReviewDB (SQL) to NoteDB (Git).

Current customers who can do the offline ReviewDB to NoteDB migration as part of their upgrade maintenance time can move forward with this release. All other customers will need to wait for the later release.

The following limits apply to this version of GerritMS:

OpenID not compatible: It’s not possible to use Google’s OpenID authentication. If you are planning to use HTTP then you need to ensure that you have an Apache web server running in front of Gerrit.

Gerrit caches can now be enabled. Cache updates and invalidation happens appropriately based on replicated operations and events. To make sure that caches are properly enabled in GerritMS you need to add properties to the GitMS application.properties file.

See Enable Gerrit Caching.

Many Gerrit deployments are integrated with one or more third-party applications. While there are no hard and fast rules for how moving to a replicated environment will affect their operation, the following information may be useful:

Existing configuration options are detected and used as default options, for example, the location of GitMS on the node. However, from an install or upgrade you cannot modify existing GerritMS settings. Any settings which already exist (and have been read from an existing application.properties file) will be automatically filled in and will not be prompted for.

Follow the steps below to install:

Gerrit is now integrated with GitMS but you still need to modify Gerrit’s Git configuration to introduce its repositories to GitMS.

When the script has completed, open a terminal session to each node and start Gerrit up, e.g:

Gerrit is now replicating its review activities to all nodes. You should test that this is the case.

Before going into production, run through your regular Gerrit workflow. For example:

The Gerrit Code Review for Git provides a number of caches which are use to speed up the response from Gerrit. Examples of the caches used in Gerrit are accounts, diff, groups, projects, permission_sort and so on.

Gerrit caches must be replicated between the nodes. That means that when some cache becomes outdated on one node, it will also get outdated on the other nodes, so that it is possible to use the advantage of the caches without the problems that those caches could bring when something happens on a remote node. To ensure that the caches are enabled in GerritMS you need to add these properties to the GitMS application.properties file:

If you experience problems using Gerrit’s cache, you can disable it in the Gerrit config file using the following example configuration snippet

You will need to change the cache labels, e.g. [cache ""] to match the specific cache that you want to disable.

The integration is now complete! Additional information for managing GerritMS in the Admin section.

A reload event is always executed on the receiving node. When a cache value is evicted (outdated) on a particular node, the other nodes receive a message to evict that very same value, and also to reload that value from the database or the repository, so that the part of the Gerrit application which relies on the values read directly from the caches, will always show up-to-date content. Conversely, if a Gerrit cache is disabled in the gerrit.config file then that cache will not communicate the eviction. Therefore, if you disable a Gerrit cache on one node you should disable that cache on all nodes.

Back to Contents - Back to beginning of section

The backup taken during install has the following format:

This should be extracted into its own folder:

The backup extracts into a folder named backup. Inside the folder:

There are 2 components to a backup taken during install:

You must stop all instances of Gerrit before rolling back any node. As they have a shared database (whether that is a single shared master, master/master or master slave), this avoids changes being written to the database during this process.

Compare the current state of application.properties with the backup of application.properties taken during install.

A GerritMS install adds several properties to this file prefixed with Gerrit. In most cases, it is safe to simply replace the current application.properties with the backed up copy. However, if other entries in application.properties have been changed since the upgrade, these will be lost. If this might happen, we recommend that you compare the files to ensure that any additional modifications are mirrored in the backed up copy.

When you are happy that the backup copy is correct, replace the existing application.properties file with it.
Note: For GitMS to pick up these changes, you need to restart it.

The Gerrit folder has the contents of the Gerrit root directory. Similar to application.properties, compare the existing etc/gerrit.config and etc/secure.config files to verify that no additional changes have been made to these since the upgrade. If, for example, the location of the git repository basePath has changed, the backup needs to be updated to show this.

When you are happy that the backup configuration is correct, replace the contents of the existing Gerrit installation root directory with those of the backup.

The Gerrit root is now restored, but the database may still be an issue. Work on the database depends on potential schema changes that the Gerrit project has made from one version to the next. Schema changes are not guaranteed on every update, but they occur fairly frequently. If you know that there are no schema changes between the backed up version of Gerrit and the one being rolled back from, then you do not have to modify the database. If, however, you have to roll it back, there maybe issues because user-initiated changes made to the database after the upgrade will be lost. Information about these changes are in the Gerrit release notes.

Whether the database has to be rolled back or not, you must now ensure that the Lucene index for the rolled back version of Gerrit is up to date.

Trigger full reindex on Gerrit by doing the following:

When the above steps are complete, restart GitMS and Gerrit on that node.

Note: Any steps taken to roll back a shared database only have to be performed once, but do them before any restored Gerrit instance is started.

Back to Contents - Back to beginning of section

The upgrade of GerritMS requires that the following steps be taken.

If you are using the Selective Replication feature to target repositories at different replication groups using Repository name matching regexes, then you need to check those rules will match against "All-Users". This is a core Gerrit repository, similar to All-Projects, and must be made available to every Gerrit node.

Back to Contents - Back to beginning of section

To return a deployment to "vanilla" Gerrit you must remove GerritMS sequencing.

Verify that all sequences are using the GerritMS form

You can initialize Gerrit but set it to not run using the following system property:

For cases where you want to start non-replicated GerritMS without making direct changes through running an GerritMS init, you can instead set up the following environmental variable.

GitMS will default to calling Gerrit as an anonymous user. If the user is anonymous then, due to Gerrit issues, not all accounts may be visible and there will be a limit of 500 items returned per query. For accurate license reporting in GitMS you will need to configure a daemon account with sufficient access to be able to see all of the GerritMS account information.

To configure GitMS to use this account, see the Gerrit client authentication section in the GitMS User Manual. Note that the configuration will require a restart of GitMS.

The functionality for adding new projects remains the same as when Gerrit is used outside of a MultiSite deployment. See Gerrit’s own documentation for this works.

Normally you should create new Gerrit repositories by creating a new Gerrit project. By doing so a newly initialized Git repository will be created and replicated by GitMS. The Replication Group that it is inserted into will be determined by how you have configured Selective Replication.

However, you can also add an existing repository to GitMS directly and, if the path where it is located is within Gerrit’s repository tree then you will be able to manage that repository using Gerrit (by creating appropriate project permissions, etc.). You do so by following the procedure below. To add a repository to GitMS, see the GitMS User Guide #_add_repositories[Add Repository procedure^].

Gerrit stores project information both on disk and in the database. A plugin called delete project has been created to help wipe this data. You will need to use this if you want to delete a project from Gerrit.

There are several reasons you may want to delete a project including:

In a non-replicated Gerrit it is possible to add new projects just by dropping the <repository-name>.git file into Gerrit’s repository path (gerrit.basePath), that is, the local file system directory in which Gerrit stores the repositories that it knows about.
Note that:

When running GitMS there are some additional requirements for using this method for adding projects:

Gerrit allows the grouping of repositories under folders just by adding a path into the project name. For example, you can create a project named 'sandboxes/abc'. This will create a repository called 'abc.git' under a folder called 'sandboxes'. You will see this naming convention carry through to GitMS.

Gerrit Code Review for Git enables SSH clients to receive a stream of events that happen in the Gerrit node where the client is connected to. GerritMS (Gerrit + GitMS) enables those SSH clients to receive, not only the events from that node, but also the events coming from other nodes in the same replication groups.

See Gerrit event stream for more information on how to enable and disable the stream. To see how the stream of events works normally, refer to the Gerrit documentation. Also refer to information on adding repositories.

Back to Contents - Back to beginning of section

When bringing a new node up, it’s vital that you make sure that it meets all the deployment requirements set in the #_deployment_overview[deployment checklist]. It’s often been considered good practice to create the new server from the image of an existing server so that software and settings are automatically matched.

Follow our instructions to install GitMS on your new node. See Install GitMS. During setup, you are asked for the license key and the users.properties file. Take these from your first node and copy them to the corresponding locations on your new node.

When the installation of GitMS has been completed on your new node, you need to add it to your replication ecosystem. This is done by navigating to one of your existing nodes. Log in to its admin UI and click on the Nodes tab. Click on Connect to Node.

When these details are entered, click the Send Connection Request button. The new node will receive the connection request and will be added to the current membership. You will need to refresh your browser to see that this has happened. The new node should now appear on the map of on the Nodes screen.

If you run into problems there is more information about Node Induction in the #_node_induction[GitMS User Guide].

Now that the new node is in place we need to a join it to our Gerrit replication group, this will tell GitMS to replicate Git and Gerrit data between the existing member nodes and the new node.

In the next step we need to use rysnc to copy the Gerrit directory from an existing node over to the new node. During the process we need to ensure that the existing node is not replicating or being written to as this could put it out of sync and corrupt the data we’ll be copying. For this reason we select one of the existing nodes to become a helper. While it takes on the role of helper replication to it will halt. Git users who use the node will not be able to make changes during this time.

Select a node and click Start Sync. Take note of the warning about not closing the browser or logging out during this process.

Opening a terminal window on the Helper node and use rsync to copy the entire Gerrit repository over to the new node.

Instead of copying one or more Git repositories we need to copy the Gerrit folder (in which Gerrit’s repositories are normally stored). If you’re using a non-standard installation location then you’ll need to adapt this step to account for both Gerrit and the git repositories that it controls.

Once the Gerrit directory has been copied over and verified, click the Complete All button on the GitMS screen. Both helper and new node will now come out of read-only mode. They’ll now be updated with any Git/Gerrit activity that occurred during the procedure.

We need to update Gerrit to account for GitMS. Run the Gerrit integration installer in the Gerrit

The installation will proceed in the say manner as with the original Gerrit integration. However, we’ve already completed the sync of the repository data so the net steps differ slightly from what is described on screen. You do not have to run the sync_repo.sh script. The repositories are already known to Gerrit because you copied that config data in the previous step.

Open a terminal window to the new node and start the Gerrit service, e.g.

The new node should be up and running. Open the Gerrit UI and verify that all replicated repositories are present in the 'Projects' list.

Back to Contents - Back to beginning of section

The installation process is outlined below and has few changes from the standard Gerrit LFS installation. See the LFS plugin docs for more information on the standard installation. The main difference is that installation has to be carried out on all nodes.

In order to push large files (>50MiB), the activitytimeout property in the gitconfig file needs to be increased from the default (30 seconds). To change this run the following command:

The time in seconds value needs to be suitable for your network environment and file sizes. For instance if you upload 1GiB files and have a latency of 333ms then the value needs to be 3660 or higher.

As this is set globally you do not need to repeat this on each repository clone.

If you need to repair one or more repositories with Gerrit LFS enabled then contact WANdisco contact support for help with repairing LFS objects.

Back to Contents - Back to beginning of section

In large deployments you might want to integrate Gerrit with only a subset of your replicated repositories, or maybe for separate Gerrit instances to manage different groups of repositories. Selective replication requires additional configuration. Use this procedure, explicitly using gerrit.rpgroupid, if the regex file does not have a wildcard pattern that matches the repository you want to create.

Note: Do not configure this if you want a failure to occur if the regex file does not match the repository/project to be created.

The regex file is used to specify a wildcard to be matched against a Gerrit project path as it is being created, in order to select the replication group where it is automatically placed. The matching starts from the top of the file and proceeds downwards until a match is made. If no match is made, either:

The regex format contains two wildcard matchers:

Note: All matching is case sensitive.

Examples of wildcard matchers use:

You can use wildcard atoms freely in combination with each other, separated by a single / character. The entire sequence of atoms must match or there is no match.

Note these additional constraints:

The specification should be done via a config file. This is an example regex configuration file:

In this example, repositories are treated as follows:

You can put the regex configuration file anywhere on the system, as long as it is readable by the user running the GitMS service.

To enable the regex matching, you must update the application.properties file with the property gerrit.rpgroup.regex set to the full path of the regex file. After you change this property to point to a new file location, you must restart the GitMS service on that node. Changes to the regex file are detected automatically by the replicator so modifications to its contents should not require a restart of GitMS or Gerrit. However, updates should be done atomically. To accomplish the atomicity the next version of the file should be renamed into place over the current version. This means that the file is always complete whenever GitMS decides to re-read it.

This diagram shows how Gerrit and GitMS interact in a replicated deployment.

Architecture description

Back to Contents - Back to beginning of section

Error and warning messages from the server are automatically written to the log file under <install-dir>/gerrit/etc/. This log file is automatically rotated daily (default: 12:00 midnight GMT). An external log cleaning service (such as logrotate) is required to manage historical log files.

GitMS stores logs in 2 locations:

For more detailed information on GitMS logs, see the GitMS User Guide.

The following issues can cause users to see the above 400 invalid project name error:

The following issues can cause users to see the above 500 internal server error:

The following issues can occur in the Git command line when attempting to push up to a Gerrit controlled project.

Push operations that fail in a replicated environment can be difficult to understand. The cause may be because an atomic commit operation has failed because it failed on a single node. This error may appear as an error containing the following messaging:

In this example the error can be explained by the appearance of the following snippet, that shows that the push was behind in one of their refs:

Ensure that you check the logs carefully as these log messages appear on a single line, so can therefore be easily missed.

GerritMS 1.10 now stores its data in the project repository, potentially resulting in a significant increase in the server-side repository size and more replication traffic. Ensure that this increase doesn’t cause an impact on performance due to the resulting increase in server-side processing for Git remote requests.

It’s good practice to incorporate an automatically scheduled garbage collection. How frequently you need to run repository garbage collection on any specific repository depends on the frequency of updates to that repository. A large accumulation of pack files will slow Gerrit down. Consider a repository GC after 100 updates have been made; definitely after 200 updates. See more about Git Garbage Collection in the Git MultiSite User Guide.

Should replication be lost on a deployment, Gerrit will not allow you to make changes on a repository. On GitMS’s dashboard a warning that replication to the lost nodes has failed. The usual GitMS procedure for repository recovery can be used, after which the replication group will catch up.

If a repository replica is lost or becomes unrecoverable then it becomes necessary to perform a manual recovery by using rsync to replace the corrupted/lost replica with a copy of the remaining good replicas.

If a change index is out of date upon an active review, most often the next event that occurs on it will also pick up the new data that was missed in the previous reindex. This may not be always sufficient however, as generating false content to trigger reindexes is not desired. If a change needs to be updated on a node, the reindex.sh script can be used to update the index.

All the events that happen on a Gerrit node can be shared with other nodes, and a node can receive events coming from the other nodes.

The replication of these events (send and receive) is on by default, i.e. every node will receive by default all the events happening on the other nodes which are on the same replication groups, and every node will send its events to the other nodes.

To modify or disable the replication of these events you need to modify the application.properties file and change the properties below.

Setting send and/or receive to false will make the node stop sending and/or stop receiving events from the other nodes.

Other configurable properties for the events are:

Example
Example settings of some of the properties described above being set to speed up the delivery of events: