1. Technical overview

1.1 Integration architecture

This diagram below shows how Gerrit and Git MultiSite interact in a replicated deployment.

Gerrit - Git MultiSite integration

Architecture description

• Gerrit runs much the same as it does in a stand-alone configuration, being a front-end for Git, intercepting push requests, holding them back until the review workflow is completed.
• The Gerrit database is shared between all nodes. Lucene indexing is shared between nodes.
• Git changes are fed through Gerrit's JGit library where the rp-git-update hook script receives push requests. The replication system then creates a pack file that contains the actual repository changes and replicates the file to all the other nodes over the Content Distribution stream.
• When the pack file is in place on all nodes then the proposal for the change is sent to all nodes over the Coordination stream (port 6444). Both git MultiSite and Gerrit run their own browser-based admin UIs.

2. Gerrit Administration

2.2 Add Projects

The functionality for adding new Projects remains the same as when Gerrit is used outside of a MultiSite deployment. See Gerrit's own documentation for this works.

2.3 Adding an existing repository

You can create new repositories from Gerrit by creating a new project. It's also possible to take an existing Git repository and add it into Gerrit, so that it will come under Gerrit's control.

1. Log into Git MultiSite's admin UI.
2. Click the Repositories tab. Click on the Add button.
   
   

   Repositories > Add

3. Enter the following details:
   
   

   Repositories > Enter details then click ADD REPO

   • Repo Name - a name for the repository
   • FS Path - the file system path (full path to the repository)

     To control the repository through Gerrit, ensure that this path is for Gerrit's repository directory, e.g. <install-path>/gerrit/git/repository1.git

   • Choose the Gerrit replication group from the drop-down. You can set the repository to be Read-only by ticking the Global Ready-only.
     You can re-enable write permission later by unticking the checkbox.
   • Click ADD REPO to add the repository for replication.

   IMPORTANT - these operations that should not be performed on Gerrit repositories

   • Creating new replication groups - Git MultiSite for Gerrit currently only integrates with a single replication group.
   • Moving repositories to other replication groups - Gerrit integration will break for the repository.
   • Adding a new repository outside of Gerrit - It will not itegrate with Gerrit.
   • Removing a repository in any way other than through Gerrit (project deletion) - Gerrit integration will break.

   2.3 Adding and removing Git repositories outside of Gerrit's control

   You can add repositories that are not for control by Gerrit using the procedure detailed in Git MultiSites's admin guide. See 3.1 Add a repository

   2.4 Adding reposities through the file system

   It's possible to to add additional repositories to the Gerrit Git MultiSite deployment without going through the Gerrit UI. This method relies on Gerrit behavior that isn't necessarily supported where Gerrit will automatically pick up repositories that are dropped into Gerrit's repository folder.

   2.5 Adding a repository for Gerrit control

   • Copy the new repository into Gerrit's repository directory (on ALL NODES), matching the Gerrit configuration.
   • Add the repository to Git MultiSite using the Add Repository procedure.

   2.6 Adding a repository outside of Gerrit's control

   • Copy the new repository into Git MultiSite's repository directory (on ALL NODES).
   • Add the repository to Git MultiSite using the Add Repository procedure.

   2.7 Delete Projects

   There are two approaches that you can use to delete a project from Gerrit:

   Delete from under Gerrit

   This approach pulls the repository out from under Gerrit by removing it from replication and from the Git Node.

   1. Remove the respository to you plan to delete from its Git MultiSite replication group. See the Git MultiSite administration guide for how to Remove a repository.
      
   2. Remove repository from disk on each node.
   3. This results in the project disappearing from the Gerrit UI, although the project will still be known to Gerrit.

   The following approach is provided as an alternative:

   1. Gerrit stores project information both on disk and in the database. A plugin has been created to help wipe this data. It can be built from source, though I found it worked well to download it from.
   2. Install the delete-project plugin with this command on each Gerrit node:

      ssh -p 29418 localhost gerrit plugin install -n delete-project.jar /full/path/on/server/to/delete-project-2.9.jar
      

   3. Remove the project from replication through Git MultiSite's admin console. It will still appear in the Gerrit UI because it the repo remains on disk and in the database, which is expected.
   4. Visit the Gerrit project UI on each Gerrit node. A Delete button appears on the project details screen. Delete the project from each node. This cleans up files on disk as well as the database.

   2.8 Manually adding new Gerrit projects

   It's possible to add new projects to Gerrit just by dropping the <repository-name>.git file into Gerrit's repository path (gerrit.basePath), that is, the local file system directory in which Gerrit stores the repositories that it knows about. Note that you may need to restart Gerrit in order to see the new project listed.

   When running Git MultiSite there are some additional requirements for using this method for adding projects:

   Copy the repository to all nodes

   You need to ensure that the repository is copied to the same place on all nodes. The safest way to do this is to use rsync, ensuring that you use the following flags which preserve neccessary properties (such as owner and group permissions). e.g.

   rsync -rvlHtogpc /path/to/local/repo  remoteHost:/path/to/remote/repo

   Read more about how to Synchronize repositories using rsync.

   Add the repository to Git MultiSite

   For changes to the repository, including gerrit tasks you need to add the repository's information into Git MultiSite. This only needs to be done on a single node as the details will be replicated to the other nodes in the replication group. For information about how you do this, see the Git MultiSite User Guide - 4.3 Add repositories

   It's possible to script/automate the addition of repositories to Git MultiSite using the REST API.

   2.9 Managing projects in subfolders

   Gerrit allows the grouping of repositories under folders just by adding a path into the project name. For example, you can create a project named 'sandboxes/abc'. This will create a repository called 'abc.git' under a folder called 'sandboxes'. You will see this naming convention carry through to the Git MultiSite.

   3. Adding a new node

   Follow this procedure if you need to expand your Git/Gerrit deployment to a new location. We assume that you have already completed the inital installation and setup of your Gerrit and Git MultiSite applications.

   Overview

   1. Prepare your new server
   2. Install Git and then Git MultiSite
   3. Induct the new node into your replication system
   4. Add the new node to your Gerrit replication group
   5. Place an existing node in "Helper mode"
   6. Use rsync to copy the Gerrit directory to the new node
   7. Put the helper node back into operation
   8. Start Gerrit on the new node

   3.1 Prepare your new server

   When bringing a new node up, it's vital that you make sure that it meets all the deployment requirements set in the deployment checklist. It's often been considered good practice to create the new server from the image of an existing server so that software and settings are automatically matched.

   3.2 Install Git MultiSite

   Follow the instructions for installing Git MultiSite on your new node. See 2. Install Git MultiSite. During the set up you'll be asked for the license key and the users.properties file. You can take these from your first node and copy them to the corresponding locations on your new node.

   3.3 Induct the new node into your replication system

   When the installation of Git MultiSite has been completed on your new node, you need to add it to your replication 'ecosystem'. This is done by navigating to one of your existing nodes. Log in to its admin UI and click on the Nodes tab. Click on Connect to Node.
   

   Node induction

   Node Node ID *

   The name of your new node - you would have entered this during the installation of your new node.

   Node Location ID *

   The unique reference code that is used to define the new node's location - you can verify this from the NODE ID entry on the new node's SETTINGS tab.

   Node IP Address *

   The IP address of the new node.

   Node Port No *

   The DConE Port number (6444 by default), again this is confirmed from on the new node's SETTINGS tab.

   When these details are entered, click the SEND CONNECTION REQUEST button. The new node will receive the connection request and will be added to the current membership. You will need to refresh your browser to see that this has happened. The new node should now appear on the map of on the Nodes screen.

   If you run into problems there is more information about Node Inductions in the installation section.

   3.4 Add the new node to your Gerrit replication group

   Now that the new node is in place we need to a join it to our Gerrit replication group, this will tell Git MultiSite to replicate Git and Gerrit data between the existing member nodes and the new node. Log in to an existing node that is a member of the Gerrit replication group, click on the Replication Groups tab button. Click on View at the bottom of your Gerrit replication group's box. Click the Add Nodes button. You will see the existing membership along with a Select a Node... drop-down button. Click the button and select the new node. Click Add Nodes.

   
   

   Adding the new node to the Gerrit replication group

   Node Role
   You can leave the new node's role as default unless by adding it you will end up with an even number of voter nodes. If this is the case then either the new node or an existing node must be assigned as an Active TieBreaker to ensure that it's not possible for a split vote to occur which would result in a deadlock of the replication system. See more about Node Types.

   3.5 Place an existing node in "Helper mode"

   In the next step we need to use rysnc to copy the Gerrit directory from an existing node over to the new node. During the process we need to ensure that the existing node is not replicating or being written to as this could put it out of sync and corrupt the data we'll be copying. For this reason we select one of the existing nodes to become a helper. While it takes on the role of helper replication to it will halt. Git users who use the node will not be able to access repositories or interact with Gerrit.
   

   Helper Node

   Select a node and click Start Sync. Take note of the warning about not closing the browser or logging out during this process.

   3.6 Use rsync to copy the Gerrit directory to the new node

   Instead of using Git MultiSite's repair option we'll be opening a terminal window on the Helper node and using rsync to copy the entire Gerrit repository over to the new node. For information about the best approach for performing the rsync see 4. Synchronize repositories using rsync.

   Instead of copying a git repository we need to copy the Gerrit folder (in which Gerrit's repositories are normally stored). If you're using a non-standard installation location then you'll need to adapt this step to account for both Gerrit and the git repositories that it controls.

   3.7 Put the helper node back into operation

   Once the Gerrit directory has been copied over and verified, click the Complete All button on the Git MultiSite screen. Both helper and new node will now come out of read-only mode. They'll now being to resync with any git/gerrit activity that occured during the procedure.

   
   

   Helper Node - Complete All

   3.8 Start Gerrit on the new node

   We need to update Gerrit to account for Git MultiSite. Run the Gerrit integration installer in the Gerrit

    ./installer.sh

   The installation will proceed in the say manner as with the original Gerrit integration. However, we've already completed the sync of the repository data so the net steps differ slighly from what is described on screen.There's no need to run the sync_repo.sh script, the repositories are already known to Gerrit because we copied that config data in the previous step.

   Open a terminal window to the new node and start the gerrit service, e.g.

   ~/gerrit/bin/gerrit start

   The new node should be up and running. Open the Gerrit UI and verify that all replicated repositories are present in the 'Projects' list.

   4. Troubleshooting

   Logs

   When dealing with errors or similar issues, you should view both Gerrits and Git MultiSite's log files.

   Gerrit logs

   Error and warning messages from the server are automatically written to the log file under <install-dir>/gerrit/etc/. This log file is automatically rotated daily (default: 12:00 midnight GMT). An external log cleaning service is required to manage historical log files.

   audit.log

   audit events

   system.log

   INFO-level logging

   sshd.log

   logs connections to Gerrit's SSH port (not system shell SSH)

   *.gc.log

   information on garbage collector usage

   Gerrit Documentation
   For more information about error messages that Gerrit can send, Gerrit Code Review - Error Messages.

   4.2 Error Messages in a replicated environment

   Running Gerrit with Git MultiSite introduces a number of potential errors that are very unlikely to be seen on a single-server deployment:

   User gets a 500 internal server error when attempting to view Gerrit

   
   

   500 internal server error

   The following issues can cause users to see the above 500 internal server error:

   4.3 Create Project in Gerrit

   • Local Node Down:
     Gerrit Log file: View sample log message
     Git MultiSite Log: No error is logged in the Git MultiSite Log file.
     Recommended action:
   
   
   • Remote Node Down:
     Gerrit Log file: View sample log message
     Git MultiSite Log: No error is logged in the Git MultiSite Log file.
     Recommended action:
   
   
   • Timeout
     Gerrit log: View sample log message
     Git MultiSite log: View sample log message
     Recommended action:

   4.4 Git Push to Gerrit-controlled repository

   • Local Node Down
     Git output:

     gitms ~/jgit-update-service/TestRepo $ git push origin master
     Counting objects: 3, done.
     Writing objects: 100% (3/3), 238 bytes | 0 bytes/s, done.
     Total 3 (delta 0), reused 0 (delta 0)
     remote: Processing changes: refs: 1, done 
     To http://adminUser@localhost:4007/TestRepo
     ! [remote rejected] master -> master (lock error: Failure to replicate update. Output is:
     )
     error: failed to push some refs to 'http://adminUser@localhost:4007/TestRepo'

     Gerrit log: No error message in log
     Git MultiSite Log: No error message in log
     Recommended action:
   
   
   • Content Distribution Fail
     Git output:

     gitms ~/jgit-update-service/TestRepo $ git push origin master
     Counting objects: 3, done.
     Writing objects: 100% (3/3), 238 bytes | 0 bytes/s, done.
     Total 3 (delta 0), reused 0 (delta 0)
     remote: Processing changes: refs: 1, done 
     To http://adminUser@localhost:4007/TestRepo
     ! [remote rejected] master -> master (lock error: Failure to replicate update. Output is:
     GitMS - minimum number of learners not available
     )
     error: failed to push some refs to 'http://adminUser@localhost:4007/TestRepo'        
     

     
     Gerrit log: No error message in log
     Git MultiSite log: View sample log message
     Recommended action:
   
   
   • No Quorum
     Git log:

     gitms ~/jgit-update-service/TestRepo $ git push origin master
     Counting objects: 5, done.
     Delta compression using up to 2 threads.
     Compressing objects: 100% (3/3), done.
     Writing objects: 100% (5/5), 444 bytes | 0 bytes/s, done.
     Total 5 (delta 0), reused 0 (delta 0)
     remote: Processing changes: refs: 1, done 
     To http://adminUser@localhost:4007/TestRepo
     ! [remote rejected] master -> master (lock error: Failure to replicate update. Output is:
     GitMS Error: Quorum not available.
     )
     error: failed to push some refs to 'http://adminUser@localhost:4007/TestRepo'    
     

     Gerrit log: no error message in log
     Git MultiSite log: View sample log message
     Recommended action:

   Create Review

   • Local Node Down
     Git output:

     gitms ~/jgit-update-service/TestRepo $ git push origin HEAD:refs/for/master
     Counting objects: 2, done.
     Delta compression using up to 2 threads.
     Compressing objects: 100% (2/2), done.
     Writing objects: 100% (2/2), 277 bytes | 0 bytes/s, done.
     Total 2 (delta 0), reused 0 (delta 0)
     remote: Processing changes: refs: 2, done 
     To http://adminUser@localhost:4007/TestRepo
     ! [remote rejected] HEAD -> refs/for/master (Unable to create changes: REJECTED_OTHER_REASON lock error: Failure to replicate update. Output is:)
     error: failed to push some refs to 'http://adminUser@localhost:4007/TestRepo'    
     

     Gerrit log: View sample log message
     Git MiultiSite log: no log output
     Recommended action:
   
   
   • No Quorum
     
     Git output:

     gitms ~/jgit-update-service/TestRepo $ git push origin HEAD:refs/for/master
     Counting objects: 2, done.
     Delta compression using up to 2 threads.
     Compressing objects: 100% (2/2), done.
     Writing objects: 100% (2/2), 277 bytes | 0 bytes/s, done.
     Total 2 (delta 0), reused 0 (delta 0)
     remote: Processing changes: refs: 2, done 
     To http://adminUser@localhost:4007/TestRepo
     ! [remote rejected] HEAD -> refs/for/master (Unable to create changes: REJECTED_OTHER_REASON lock error: Failure to replicate update. Output is:
     GitMS - minimum number of learners not available)
     error: failed to push some refs to 'http://adminUser@localhost:4007/TestRepo'    
     

     Gerrit log: View sample log message
     Git MultiSite log: View sample log message
     Recommended action:
   
   

   User gets a Merge Review Error

   
   

   Merge Review Error

   • Local Node Down
     
     Gerrit log: View sample log message
     Git MultiSite log: no log output
     Recommended action:
   
   
   • No Quorum
     
     

     Merge Review Error - no quorum

     Gerrit log: View sample log message
     Git MultiSite log:View sample log message
     Recommended action:

   Create Branch

   • Local Node Down
     Gerrit log:View sample log message
     Git MultiSite log: no log message
     Recommended action:
   
   
   • No Quorum
     Gerrit log: View sample log message
     Git MiultiSite log:View sample log message
     Recommended action:

   Delete Branch

4. Local Node Down
   Gerrit log:View sample log message
   Git MultiSite log: no log message
   Recommended action:


5. No Quorum
   Gerrit log: View sample log message
   Git MiultiSite log:View sample log message
   Recommended action:

4.3 Gerrit ACL rules and potential issues with ordering

Gerrit has the concept of storing the ACL rules in both parent repositories and a meta references within a repository itself. WANdisco's replication system guarantees ordered writes on a repository by repository level, so when ACL rules for a particular repository is defined in a "parent" repository, the ordering of operations between repositories can't be guarenteed, as each repository is its own distributed state machine.

Example: You could change a rule on a parent repository that impacts a child repository at the time when someone attempts to push something that would be rejected by that rule change. In a scenario where there is moderate network latency it is possible that the push would go through in spite of the just-added proscriptive rule.

So, rule updates on remotes are not applied immediately when applied to parent repositories. If a rule has to be changed on a repository and MUST apply immediately, the correct approach is to edit that individual repository ACL instead of the parent.

4.4 Gerrit Garbage Collection

Gerrit has its own garbage collection command to free up system resources by removing objects that are no longer required by Gerrit.

Setting up garbage collection

It's good practice to incorporate an automatically scheduled garbage collection using this command. How frequently you need to run garbage collection will depend on several factors including the size of the repositories and the load on the system. Gerrit lets you have default and project-level gc parameters, so you can tune garbage collection on a per-repository basis.

Running gc through SSH

This is a way to run gerrit garbage collection through the SSH interface. e.g.:

ssh -p 29418 adminUser@dger01.qava.wandisco.com 'gerrit gc' --all --show-progress

A password will be requested. This will be applied to all repositories. You can leaving off the --show-progress to perform the command without an output.

Read more about Gerrit's Garbage Collection command in the Gerrit's own documentation. See the gerrit gc page - https://gerrit-review.googlesource.com/Documentation/cmd-gc.html

Unauthorized proposal logging caused by password conflict

The following unauthorized proposal error doesn't offer many clues to its cause.

2014-11-17 18:53:06 INFO [RestCaller:handle] - Thread-151:[com.wandisco.gitms.repository.handlers.GerritUpdateIndexProposalHandler:
output=Unauthorized]
2014-11-17 18:53:29 INFO [NodeEventHandler:handle] - 993753a9-6e7b-11e4-b352-080027776fdc:[Received node event,
Unknown macro: {Node}
    

Cause: Git MultiSite stores a default username and password in its application.properties file.
e.g.

gerrit.enabled=true
gerrit.rpgroupid=0683d2fc-6e7c-11e4-9956-080027a5ec28
gerrit.repo.home=/home/wandisco/gerrit/git
gerrit.username=admin
gerrit.password=pass
gerrit.events.basepath=/home/wandisco/gerrit_events
gerrit.root=/home/wandisco/gerrit	
    

Changing the user password via Gerrit's settings screen (shown below) can result in the stated error. Ensure that passwords are matched to fix the problem. Note that



Changing the gerrit password in settings can conflict with Git MultiSite's stored password.

4.6 Code Review Error - 400 Invalid revision "HEAD"

If an attempt is made to create a branch of a Git MultiSite repository which has not had any commits made to it then you will see the following error message:




No commits on a new branch results in this error.

Cause:

When creating a branch through Gerrit UI there will be no initial commit from which to branch off.

Recommended Action:

The error will disapear after the first commit has been made.

4.7 Gerrit reindex process

Below is a diagram that shows how Gerrit's reindexing system runs. In most cases this system will run without intervention, however, in the event of a persistent reindexing failure, the administrator may need to get involved in order to fix an underlying problem or to trigger a retry.




Gerrit's reindexing sequence

Description

1. Gerrit detects a change and an index file is created.
2. The file is read and change-id is gathered.
3. The local instance of Git MultiSite sends a proposal for the index file to be replicated to the other nodes.
4. Now, an index request is sent, where the reply is OK, or an error can occur. If an error does occur then the process will retry the sending of the proposal 3 times.
5. After the 3rd failed retry the proposal is stored in a retry_failed directory, by default you'll find the directory in the gerrit-events directory, located (by default) in the same location as the Gerrit installation, e.g.

       [root@daily-gerrit-static-1 wandisco]# ls -l
       total 28
       drwxr-xr-x 13 wandisco wandisco 4096 Dec  8 07:31 gerrit
       drwxr-xr-x  5 wandisco wandisco 4096 Dec  8 07:42 gerrit_events
   	drwxr-xr-x 2 wandisco wandisco 4096 Dec  8 07:42 failed_definitely
   	drwxr-xr-x 2 wandisco wandisco 4096 Dec  8 07:42 failed_retry
   	drwxr-xr-x 2 wandisco wandisco 4096 Dec  8 07:42 gen_events
       drwxr-xr-x  2 wandisco wandisco 4096 Dec  8 07:40 gerrit-ms
       drwxrwxr-x  2 wandisco wandisco 4096 Dec  8 07:29 git
       drwxrwxr-x  4 wandisco wandisco 4096 Dec  8 07:29 keys
       drwxrwxr-x  2 wandisco wandisco 4096 Dec  8 07:29 misc
       drwxrwxr-x  2 wandisco wandisco 4096 Dec  8 07:29 tools    
       

   Change-ids stored in this directory are retried every 30 seconds for the next 24 hours.
6. Should a change-id not succeed during the 24 hours it will then be moved to the definitely_failed directory.
7. The client may elect to push these definitely failed change-ids back out for a retry.
8. A failed change-id that eventually succeeds is deleted from the failed_retry diretory. As and when the proposal is accepted, the reindexing will proceed.