JIRA MultiSite v4.0w1 Admin Guide

1 Introduction

Welcome to the WANdisco world of replication. JIRA is designed to run as a central server to which multiple JIRA clients connect. WANdisco's replication technology makes it possible to have multiple active replicas of a JIRA database that are in synch. The JIRA replicas can be anywhere on a WAN - distributed throughout a company's campus or throughout the world.

WANdisco users experience the performance of a local JIRA database, with the semantics of a single shared JIRA database. We call this "active replication with one-copy-equivalence".

A stand-alone JIRA instance is hosted on a J2EE-compliant application server. Users access the application with a standard web browser and HTTP protocol (via the HTTP port configured for JIRA). WANdisco JIRA MultiSite configures an additional port for replication communications (known as DConE) using the DConENet protocol. See the following illustration, which shows a sample JIRA MultiSite configuration of five sites.

1.1 Release Notes for JIRA MultiSite v4.0w1 (v4.0.0.1 build 12594)

WANdisco JIRA MultiSite now supports JIRA version 4.0. Included in the release are the following fixes:

A reset script is now available in the jira-multisite/bin directory, running a reset is now correctly reported in the logs. (JIRA-227)

Fixed a problem where re-indexing could result in a duplicate key exception when using the Charting plug-in. (JIRA-251)

Fixed problem with sequence values going out of sync due to database instances having different schema names.
(JIRA-226)

We've upped the default buffersize from 5000 to 50000 as some customers have reported problems when using the smaller buffer size. (JIRA-214)

A new WANdisco updated version of the GreenHopper plug-in has now been made available which provides cache invalidation logic for the local caches maintained with GreenHopper itself. Any JIRA (MultiSite/Clustering) customer deploying the GreenHopper plug-in should download this version along with the JIRA (MultiSite/Clustering) release from WANdisco and deploy this version of the plug-in on their WANdisco JIRA nodes. (JIRA-250)

Read Atlassian's Release Notes for JIRA 4.0

WANdisco JIRA MultiSite

See the Appendix for more details about WANdisco's replication technology works.

1.2 About JIRA

You may be new to JIRA, or you may have used it extensively. In any case, you need to be familiar with a few issues that are important in dealing with JIRA MultiSite.

1.2.1 JIRA Application Server

JIRA comes packaged with Apache Tomcat, so WANdisco also packages Tomcat as the default application server. You can use any supported application server; the Atlassian documentation is very helpful on this topic. This Administration Guide assumes you are using Tomcat, but at certain key points gives instructions for using other supported application server.

1.2.2 JIRA Database

JIRA comes packaged with the HSQL database, "for evaluation purposes only." Atlassian warns that it is not robust enough for a production environment. We agree and strongly recommend that you use an enterprise level database solution. See the Atlassian product documentation.

2 Before Deployment

2.1 Setting up identical JIRA replicas on all servers

All replicas must start out identical, Wandisco's JIRA MultiSite will then ensure they stay in synch.

2.2 Physical Environment

We strongly recommend that you follow these guidelines to ensure the successful installation and use of JIRA MultiSite:

Running servers for each site in the replication group, pre-configured with:

the same operating system
a command line compression utility (Note: WANdisco does not support WinZip 8.1 or WinRAR 3.2)
Java & Perl (see Appendix 8.1 - Installing Java and Perl)
browser with network access to all servers

E-mail from WANdisco containing the tar file link and the attached production licence key file.

2.3 Firewalls and Virus Scanners

2.3.1 Firewalls

By default, users need access to the HTTP port (80) and the Wandisco replication port (6444). Ensure that you configure firewalls to not block or filter these or other ports you specify during installation.

2.3.2 Virus Scanners

If you have a virus scanner running on your network, you must configure it to not filter traffic on the ports you specify during installation.

2.4 Deployment Checklist

You may be familiar with this checklist from an evaluation copy of JIRA MultiSite. It is listed here for reference.

System Setup: All sites must share the same operating system
Supported Operating Systems: Fedora (32 or 64 bit): 6, 7, 8, 9
Red Hat Linux Enterprise Server (32 or 64 bit): 4, 5.2
                                                      Sun Solaris (32 or 64 bit): 9, 10
                                                      Linux: Linux kernel 2.6 or higher
                                                      CentOS-4
                                                      Windows Server, (32 or 64 bit) 2003
JIRA version: WANdisco ships with JIRA version 4.0. Atlassian supports data migration from earlier versions. See Atlassian's documenation.
System Memory - per site: RAM: 2 GB

We consider 2GB to be a minimum amount. On a 32 bit operating system the max heap allowed is 2 GB. With a max available memory of ~3.2GB (unless a special kernel is used) this doesn't give much head room for handling the JIRA database and MultiSite on the same server, or memory hungry plug-ins such as GreenHopper.
Disk Space - per site: JIRA: depends on the number of projects and issues
MultiSite Transaction Journal: Recommended - equivalent of seven days of commit changes
File Descriptor limit: Ensure hard and soft limits are raised to 64000.
Journaling File System: Replicator logs should be on a journaling file system, for example, Veritas, ext3 on Linux. Notes: NTFS is not a journaling file system (OK on Windows): ext4 is a journaling file system, however WANdisco does not support its use because of its deferred writes.
Java: Install JDK 1.6.0. (update 10 or higher if using the Sun JDK)

There should not be any spaces or control characters in the path where Java is installed. For example, c:Program Files\java does not work with WANdisco as a JAVA install directory. See Appendix A - Installing Java and Perl.
Perl: Install version 5.6.1+. See Appendix 8.1 - Installation Guides.
Compression Utility: There are problems with using certain compression archiving utilities to extract JIRA install files. Both Winzip and Winrar have been known to cause problems with extracted files. Total commander 7.01 and 7Zip both work without problem.

Application Server
JIRA: JIRA ships pre-configured with the Apache Tomcat application server, a stable, lightweight and fast performing server.
JIRA MultiSite WAR/EAR: The JIRA MultiSite WAR/EAR can be installed with any supported application servers (subject to compatibility with your OS/JDK).

Database
JIRA: JIRA ships pre-configured with the HSQLDB database, which is suitable for evaluation purposes. You should use an enterprise database for running JIRA in a production environment.
JIRA MultiSite WAR/EAR: The JIRA MultiSite WAR/EAR can be connected to any of the supported databases, provided they are compatible with your application server.
Network Setup: Reserved Port (i.e. 6444, another port for synchronization) JIRA MultiSite needs a dedicated port, which is used for both WANdisco protocol and HTTP. Depending on your database tools, you may need an additional port available for database synchronization in case the databases get out of sync. (You specify port numbers during installation.)
Firewall virus scanner: Notify the firewall virus scanner of the JIRA MultiSite port numbers. (You specify port numbers during installation.)
Persistent Connection Keep Alive: Ensure VPN doesn't reset persistent connections for WANdisco or else ensure no RST bugs
Bandwidth: Understand the available bandwidth for testing, and set user expectations.
DNS Setup: WANdisco recommends using static IP addressing instead of hostnames for servers running JIRA MultiSite. This can deliver better performance and doesn't make your DNS service a dependancy.

2.5 JIRA Plug-in Support

WANdisco's JIRA products (MultiSite and Clustering) come with the same bundled plug-ins as Atlassian's stand-alone JIRA. You're welcome to install additional plug-ins to further enhance JIRA, however, before doing so, you should read our policy on which JIRA plug-ins we're able to support.

2.5.1 JIRA Plug-in Support Policy

This policy applies from Thursday 4th February 2010, and may be subject to change without notice.

The large number of available plug-ins is a testament to JIRA's popularity and flexibility. While some of these plug-ins are written to Atlassian's quality requirements, and are rigorously tested, many others are not.

Some plug-ins may cause JIRA to behave unpredictably, or produce results that are not deterministic, which could break WANdisco's replication technology. As it's not possible to test the compatibility of our products with all plug-ins we've set out this policy, which lists all the plug-ins that have been found to work, and that we're able to support through our own product testing.

Fully Supported Plug-ins

This is the list of JIRA plug-ins that we have confirmed as compatible with our products, and our replication technology.

GreenHopper (We provide a WANdisco specfic version on our file download website)

Provisionally Support Plug-ins

This is the list of plug-ins that are supported on a "best efforts" basis.

Under review for future support

This is a list of plug-ins that have recently been added to our product testing process with a view to providing support for them in future.

Unsupported Plug-ins

All other JIRA plug-ins should be considered unsupported. Plug-ins in this category may be compatible, however, we have not tested their suitability for use with our products. If there's a plug-in that's crucial to your operation, contact our sales team, who will be happy to discuss what options are available for adding additional plug-ins to our supported list. Request support for a JIRA Plug-in: (Pre-exisiting Customers), (Prospective Customers).

2.6 Plug-in Compatibility

To ensure compatibility, JIRA plug-ins should conform to the JIRA plug-in API and interface definition of the OFBiz framework. If a Plug-in maintains its own local data cache, then this will need to be updated by WANdisco's replication engine, in addition to any custom fields used by the plug-in.

2.7 Plug-in Installation Uniformity

The plug-ins that you use must be installed at all nodes within a replication group. Many Plug-ins add custom fields to the JIRA database that must be applied at all nodes to ensure successful replication.

2.8 Plug-ins and JIRA Services

Some plugin enable a JIRA service. There may be times where it is desirable to have that service running at only one node in a replication group (see Section 5.14 - Selectively Disabling Services for more details).

2.9 Common Plug-ins

GreenHopper - http://www.atlassian.com/greenhopper/
GreenHopper is a JIRA plug-in that adds a broad collection of agile project management capabilities to JIRA, and extends JIRA as a powerful platform for agile development teams. GreenHopper simplifies the planning and organisation of tasks, work flows and reporting for agile teams.

Ensure that you use our version of GreenHopper, available on our file download site JIRA MultiSite as this has been modified to provides cache invalidation logic for the local caches maintained by GreenHopper. Also note that GreenHopper can require a lot of memory, ensure that you follow the hardware guidelines in the Deployment Checklist

There's a known problem with GreenHopper's card templates, where modifcations are not replicated to other nodes. Read our Knowledgebase article.

Subversion Plug-in (Atlassian; free) - http://confluence.atlassian.com/display/JIRAEXT/JIRA+Subversion+plugin
The JIRA Subversion plug-in runs a service that scans Subversion commit records for user comments with a JIRA record number then updates corresponding records in JIRA.

Using with JIRA MultiSite AND Subversion MultiSite

Not using Subversion MultiSite

Charting - http://confluence.atlassian.com/display/JIRAEXT/JIRA+Charting+Plugin
A plug-in providing various charts and reports for JIRA. These charts and reports provide a visual representation of a project or a saved filter in different contexts.

Only version 1.5 of the Charting Plug-in is supported by JIRA 4.0.
Also, there are performance problems with the plug-in. A manual re-index takes a lot longer to complete. It's also possible that a crash of the distinguished node can cause the indexes at the other nodes to be corrupted.

3 Installation

You are ready to install JIRA MultiSite. Make sure you print the worksheet in the next section to fill out as you go through the installation.

3.1 Installation Checklist

The following checklist can be used to ensure that you're sufficiently well prepared to begin the installation. Some points on the list can answered during the installation.

WANdisco licenses JIRA MultiSite by customer-supplied IP address. Does WANdisco know all IP addresses to be used in the replication group?

Which site is to be the designated Install Node? This by default is the first site installed.

Which site is the designated distinguished node? (user assigned during installation)

What application server am I using for JIRA? The Tomcat that ships with JIRA is the default.

Which database am I using for JIRA? WANdisco does not recommend using the HSQL database that ships with JIRA.

Do you have an existing stand-alone JIRA? Are you going to use the JIRA export, then import at each new site, or are you going to manually synchronize the database across all sites?

DO NOT log back into JIRA after you upload the backup.xml file during the installation. This may alter the database at that site, thereby creating an out-of-sync situation even before replication begins. This problem only occurs because replication is not yet enabled. As soon as you enable replication, this is no longer an issue.

In either case, you need a new WANdisco-supplied JIRA license (as well as the WANdisco license).

Are you planning to continue plug-ins that you've already got installed in your previous JIRA setup? If so, ensure that you are able to reinstall them immediately after completing the upgrade and before you restore your data.

Are firewalls, if any, configured as required?

Are virus scanners, if any, configured as required?

3.1.1 Have an Existing JIRA License?

If you have a working stand-alone JIRA instance, you need to use the JIRA license that came with the WANdisco license.key file. The Atlassian-provided JIRA stand-alone license file does not work with JIRA MultiSite.

How you treat the new JIRA license depends on how you are treating the JIRA data, since the license is stored in the database.

3.1.1.1 Option 1 - Export your existing JIRA data

You are going to use the JIRA export command, and create a backup.xml file that contains your data and the license file.

a. Go to your existing JIRA installation.
b. Log in as administrator.
c. Go to the Admin tab.
d. Select System from the left side menu.
e. Select License key details.
f. Paste in the JIRA license (specific to JIRA, not be be confused with WANdisco license file).
g. Ensure JIRA is still working properly.
h. Generate a backup.xml for your JIRA instance, which includes the new license information. You use this backup.xml in the JIRA MultiSite installation. Continue with section 3.2.1, Configuring the Replication Group - the Setup Page.

3.1.1.2 Option 2 - Manually synch your existing JIRA data

You are going to manually synchronize your JIRA data to all the new JIRA instances. Update to the new JIRA license now.

a. Go to your existing JIRA.
b. Log in as administrator.
c. Go to the Admin tab.
d. Select System from the left side menu.
e. Select License key details.
f. Paste in the JIRA MultiSite license (specific to JIRA, not be be confused with WANdisco license file).
g. Ensure JIRA is still working properly.
h. Continue on with section 3.2.1, Configuring the Replication Group - the Setup Page, and use the empty backup.xml file that comes with the WANdisco installation. This is described at the appropriate step.

During your manual synchronization, you take a snapshot of the JIRA database, which now includes the new license file.

In the section 3.2.6.3, Importing JIRA Data, manually synchronize your data as described. You are going to load your data into each site, and then continue installation with 3.2.6.4, Validating the Installation.

3.2 Installation Procedure

Perform these steps from the first server, which becomes the install server. The installation prepares a .zip file for each site. The end of this procedure includes installing WANdisco JIRA MultiSite on this site first.

1. Save the jira-multisite.tar.gz file.
2. Uncompress the file, which produces two folders: tomcat and jira-multisite.
WANdisco does not support WinZip 8.1 or WinRAR 3.2.
3. JIRA needs a database driver, ensure it's in place by following checking below:

Using the bundled version of Tomcat and HSQL database:
   the database driver is already in place.

Using another version of tomcat with HSQL:
   copy the JDBC driver jira-multisite/lib/hsqldb-1.8.0.5.jar to the Tomcat /lib directory.

Using a different version of Tomcat with HSQL:
   copy jira-multisite/lib/hsqldb-1.8.0.5.jar to your Tomcat /lib directory.

Using a different database solution:
   you'll need to get the appropriate JDBC file to substitute for /hsqldb-1.8.0.5.jar.
4. Copy the licence.key file to the config folder in the jira-multisite directory.
5. At the command prompt (or editor), go to jira-multisite/bin. Enter the command:
```
perl setup
```
Setup will start, providing a URL for the browser based installation wizard.

6. Copy the URL <serverip>:6444/ into a browser. The JIRA MultiSite Installation page appears.
7. On the welcome screen, click on Continue.

8. Read the User Licence Agreement. Click I Agree to continue.

3.2.1 Replication Group Setup Page

9.Enter your setup details:
Number of Servers: Specify the number of JIRA servers to be replicated, including the install server.
Admin Username: Is hard-coded as "admin" and will be the same for each server.
Admin Password Select a password to associate with the admin login.
Confirm Admin password: Enter your password again to confirm you typed it correctly.
Packaging Dir: Specify where you'd like the packaged install files for your other servers to be saved
Package as: Use the radio button to tell the installer how to handle the packaging of your Web application ARchive files:
- Bundled Tomcat: Package using the version of Apache Tomcat that is provided with JIRA MultiSite.
- Unique WAR files: Create a unique package for each server, allowing for server-specific settings.
- Single WAR files: Use the same package on all sites.
If you have less than three servers specified, a warning prompt appears. The prompt warns that with two servers, there is a single point of failure if the server that acts as the distinguished node were to fail. Read it and click Yes.

3.2.2 Configuring the install server

10. Enter the properties for the Install server (JIRA Server 1):
Name: A useful name that JIRA MultiSite will use to refer to the server.
IP Address: The server's IP address.
MAC Address: The physical address of the server.
To find these addresses for this server, go to that server's command prompt. For Unix, type:
```
ifconfig
```
For Windows, type:
```
ipconfig /all
```
Make sure the IP address can be resolved at all sites (sometimes machines have multiple IP addresses where one only works on the local area network but not over the WAN or VPN). The MAC address is the physical address. For example, on a Windows machine, the output would look like:

Physical Address. . . . . . . . . : 00-1A-A0-36-53-3C
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 192.168.1.184
Subnet Mask . . . . . . . . . . . : 255.244.255.0
Default Gateway . . . . . . . . . : 192.134.1.1
DHCP Server . . . . . . . . . . . : 192.192.1.1
DNS Servers . . . . . . . . . . . : 192.198.1.50
				   64.405.172.26

WANdisco Port:

HTTP Port:

Tomcat Shutdown Port:

JIRA MultiSite Directory:

Tomcat Directory:

JIRA Index Directory:

JIRA Attachments Directory:

WANdisco Logs Directory:

JIRA Logs Directory:

Which JIRA Backup XML:

New Install

Existing JIRA Upgrade

are you currently installing on this server

If your file systems doesn't allow duplicate pathnames on your networks you can use different paths for the Tomcat, Index and Attachments directories. Each site's configuration set up page allows you to enter distinct pathnames for these directories. If you do this, you won't be able to use the 'single WAR package' option in the 'Package as' part of step 9. as this assumes identical directory structure.

3.2.3 Setting JIRA Home

The 'JIRA home directory' contains key data that help define how JIRA works. You'll need to set the JIRA home directory. If you are using JIRA WAR-EAR, you need to set your JIRA home before you build JIRA.

You can specify any location on a disk for your JIRA home directory. Please be sure to specify an absolute path.

Windows users - Please use forward-slashes ("/"), not back-slashes ("\").

set JIRA Home

Please note that you cannot use the same JIRA home directory for multiple instances of JIRA. We recommend that you do not specify your JIRA home directory to be inside your installation directory, to prevent information from being accidentally lost during major operations (e.g. backing up and restoring instances).

3.2.4 Configuring Subsequent Sites

11. Repeat the setup for each server in your replication group. The installer presents you with a page for each site you specified.

While the Installation Directory is still a type-in field, we recommend using identical directory structure across all your servers.

3.2.5 Looking Over the Summary Page

12. After you have defined each site in your replication group, you are presented with the Summary page. Before you click Next, take note of any alerting messages. These messages will be important in guiding you through any changes that will be required to match the installation to your selected configuration.

3.2.5.1 Database Customization

At this point, set the JDBC settings for JIRA. These settings are used by the installer, assuming you have identical configurations at each site, you should only need to set them once.

Atlassian documentation for Database Settings
You will need to notify the installer of the database username, password, driver class name, and URL. In the file
jira-multisite/lib/template.tomcat.server.xml,
modify the data source properties for username, password, driverClassName, and url.

Here's a sample file:

<Context path="<%= contextPath %>" docBase="<%= docBase %>" reloadable="false">
<Parameter name='wandisco.prefs.root' value='<%= wandisco.prefs.root %>'/>
<Resource name="jdbc/JiraDS" auth="Container" type="javax.sql.DataSource"
   username="<%= dbUsername %>"
   password="<%= dbPassword %>"
   driverClassName="<%= dbDriverClassName %>"
   url="<%= dburl %>"
.
.
.

For an example of how to complete database configuration - see our example in the Appendix - Example Database Configuration - MySQL

3.2.5.2 JIRA Customizations - Pre Installation

You can customize JIRA before WANdisco makes any installation packages. The JIRA application is built from the jira-multisite/war directory. Say, for example, the Atlassian documentation says to change the file WEB-INF/classes/entityengine.xml. You can find this file in jira-multisite/war/WEB-INF/classes/entityengine.xml.

3.2.5.3 Selecting a Distinguished Node

On the Summary page, in the Distinguished Node field, select the site to be the replication group's distinguished node. The pull-down list includes all of the replication group's sites. WANdisco recommends you choose the most reliable server in the replication group as the distinguished node.

3.2.5.4 Selecting a Quorum Type

In the Quorum Type field, select either the default singleton response quorum, majority response quorum, or unanimous quorum. If you select singleton response quorum, the first site becomes the Distinguished Node. Singleton response quorum means the distinguished node makes decisions on transaction order; majority response quorum means a majority of sites must agree on transaction order; unanimous quorum relies on all sites being up and responsive to make transaction ordering decisions. For a discussion on quorum, see 8.2.2, Singleton, Majority and Unanimous Response Quorum.

3.2.6 Installing the Application Server (Packaged Tomcat)

Use this section if you are using the version of Tomcat that is bundled with JIRA MultiSite. If you're using a different Application Server, go to 3.2.7 Installing the Application Server.

3.2.6.1 First Site Installation and Package Creation

This step installs JIRA MultiSite on Site 1.

All packages include a JIRA backup file, regardless of whether you had a JIRA instance installed. If you did not have JIRA installed, the Backup.XML file is a dummy database file that gets replicated. If you're Manually re-synching your databases after installation you also use this Backup.XML dummy file. If you did have a JIRA instance installed, and you specified the path to the backup in Step 9, it is renamed in this process to Backup.XML and placed in the directory you specified in Step 11.

Server 1 has been configured. The installer creates a package for each of the other sites.

JIRA Setup 6

13. Click Next. The installer has created .zip files for each site, which are saved to the Packaging Directory on the install server (specified in step 9.)

3.2.6.2 Installing Subsequent Sites

Perform these steps for all the other sites in your replication group.

JIRA Setup 7

Copy each .zip file and unzip it.
Go to that site's command prompt. In Unix, type:
```
perl unpackage.sh
```

unpackage

3.2.6.3 Importing JIRA Data

If you backed up your database so that you could synchronize it, do so now. All sites must have an identical copy of the database.

If you are new to JIRA, or if you exported a backup to the file specified in Step 10, you must perform these steps.

WARNING: DO NOT log back into JIRA after completing step 16., where you import the backup.xml file. The act of logging in to JIRA at a site may alter the database, thereby putting the databases out of sync before you even begin using MultiSite. This problem occurs only because you have not yet enabled replication. You don't have to worry about this with replication enabled.

14. On each site, start Tomcat. In tomcat/bin, type:
```
perl watchdog
```

15. Go to the JIRA interface at each site, for example, at http://<server name>:8080.
Click import your existing data at each site. The Import Existing Data page displays.

large repositories4

16. Type in the path to the Backup.XML file. For example, JIRA/wandisco/backup.xml.
17. Enter in the same path for the JIRA index as you entered in step 9.
18. Click Import. DO NOT log back into JIRA.
19. Repeat steps 14 to 18 at each site.

If you are planning to use the "blank" Backup.xml provided with JIRA MultiSite you'll need to provide your own license key, or generate a new evaluation key. See our Knowledgebase article - How to create a new evaluation key for JIRA MultiSite

3.2.6.4 Validating the Installation

All the databases are identical.
All database user privileges are identical at all sites.

3.2.6.5 Turning On Replication

WARNING: Once you turn replication on, you should not turn it off unless you are in consultation with WANdisco support.

You must turn replication on for each server concurrently.

20. At each site, stop Tomcat. In tomcat/bin, type:
```
shutdown.sh
```
For Windows, in tomcat/bin, type:
```
shutdown.bat
```
21. Turn replication on at each site. Open the file tomcat/jira/WEB-INF/classes/entityengine.xml.
22. In entityengine.xml change the datasource element attribute replication-enabled to true. (By default, it is set to false.)
Note that you are now editing the post-installation entityengine.xml file, which is located in the tomcat directory.

WARNING: Turn replication on AT EACH SITE before proceeding. Failure to do so causes the databases to go out of sync.

23. Start tomcat. At each site, in tomcat/bin, type:
```
perl watchdog
```
For Windows, at tomcat\bin, type:
```
perl watchdog
```
This starts Tomcat server with JIRA MultiSite.
24. Go to the Admin Console at each site. Type:
```
http://<server name>:6444/
```
25. Verify that replication is enabled.

large repositories4

If replication is not enabled, verify that you edited the correct entityengine.xml file at each site. Note that you should be editing the post-installation entityengine.xml file, which resides in the tomcat directory. If replication is still not enabled, go to http://support.wandisco.com.

3.2.7 Installing With a WAR File

Use the steps outlined in this section if you are using your own application server. This Administration Guide does not discuss application servers other than the bundled Tomcat, but the Atlassian documentation is very helpful. See the WAR/EAR discussion for JIRA on http://confluence.atlassian.com/display/JIRA/JIRA+WAR-EAR+Configuration+Overview.

3.2.7.1 Generating a WAR/EAR File

You just clicked Next on the Summary page. The install program completes installation on the first site, and produces a WAR for each site (or a single WAR for all sites, depending on what you selected during installation).

3.2.7.2 Installing the Sites

26. Copy the zip files to the appropriate sites. The filename is <site name>.zip.
27. Step 2 Unzip the files at the appropriate sites. This creates a wandisco directory that includes all necessary files.
28. Unpackage the files. In each site's wandisco directory, type:
```
perl unpackage
```

This creates the jira-multisite directory placed in the directory you specified in Step 11.

29. Deploy the war file at each site. For example, for Tomcat, copy jira-multisite/jira.xml to tomcat/conf/Catalina/localhost.
30. Define context parameters, if the WANdisco Summary page instructed you to specify them. This will only be the case if you selected the single WAR packaging option, and have jira-multisite installed to different locations on the sites. For example:
```
wandisco.prefs.root = C:\JIRA\jira-multisite\config
wandisco.prefs.file.name = prefs.xml
```
The WANdisco JIRA installation (wandisco.prefs.root) directory, and its config directory, you specified in step 19.
31. jira-multisite For Tomcat, restart at each site. At tomcat/bin,
In unix, type:
```
perl shutdown.sh
```
then
```
startup.sh
```
In Windows, type:
```
shutdown
```
then
```
startup
```

3.2.7.3 Importing JIRA Data

WARNING: DO NOT log back into JIRA after step 11, where you import the backup.xml file. The act of logging in to JIRA at a site may alter the database, putting the database replicas out of sync.

WARNING: If installing data from one JIRA server to another, ensure that exactly the same plug-ins (and same versions) are installed at the destination server. Otherwise you're likely to get errors relating to missing custom field.

32. Go to the JIRA interface at each site, for example, at
http://<server name>:8080/jira.

large repositories4

33. Click import your existing data at each site. The Import Existing Data page displays.

large repositories5

34. Type in the path to the Backup.XML file. For example, JIRA/wandisco/Backup.XML.
35. Enter in the same path for the JIRA index directory that you specified in step 9.
36. Click Import. Do this at each site, making sure that nobody logs in to JIRA.
37. Undeploy the WAR and clear the cache. For example, for Tomcat, delete the tomcat/conf/Catalina/localhost/jira.xml file and the cache, at tomcat/webapps/jira directory.

3.2.7.4 Turning On Replication

WARNING: Once you turn replication on, you should not turn it off unless you are in consultation with WANdisco support. This command is good for any application server you are using (except the bundled Tomcat). The reptoggle command is located in jira-multisite/bin.

38. To turn replication on at each site, for Unix, type:

perl reptoggle jira-<site name>.war -ron

For Windows, type:

perl reptoggle jira-<site name>.war -ron

3.2.7.5 Redeploying the WAR With Replication Enabled

39. Redeploy the war file at each site. For example, for Tomcat, copy jira-multisite/jira.xml to tomcat/conf/Catalina/localhost.
40. Define context parameters, if the WANdisco Summary page instructed you to specify them.
For example
```
wandisco.prefs.root = C:\JIRA\jira-multisite\config
wandisco.prefs.file.name = prefs.xml
```
41. For Tomcat, restart at each site. At tomcat/bin, type:
```
shutdown
```
When the shutdown has completed, type:
```
startup
```

WARNING: If the Admin Console does not display JIRA Replication enabled, it may be because your application server is caching the WARs. Follow the instructions for your application server about clearing the cache and recheck the display.

3.2.7.6 For Tomcat Users

Tomcat users not using the bundled tomcat can use another way to turn replication on and off. Edit the file tomcat/jira/WEB-INF/classes/entityengine.xml. Note that this is the post-installation entityengine.xml file, which resides in the tomcat directory.
The datasource element has an attribute: replication-enabled. By default, it is set to false.
At each site, edit it to true to turn on replication.

3.2.7.7 Validating the Installation

all the databases are identical
all database user privileges are identical at all sites

3.3 Installing Upgrades

This procedure involves taking the JIRA replication group offline. Please follow your company procedures about notifying JIRA users of down time.
You must either export the existing database to XML or manually synchronize to the new databases.

1. Shut down all sites. At each site, for Unix, type:
```
tomcat/bin/shutdown.sh
```
for Windows, type:
```
tomcat\bin\shutdown
```
2. Change the WANdisco JIRA port from the default 8080. This ensures that no users can access WANdisco while the upgrade is occuring. Where you change the port depends on your configuration. For tomcat, it is in conf/server.xml
```
<Service name="Catalina">
<Connector port="8080">
```

NOTE: Do not change the replicator port 6444.

3. Start tomcat. At each site, go to tomcat/bin/ and run:
For Unix, type:
```
perl watchdog.sh
```
For Windows, at tomcat\bin, type:
```
watchdog
```
4. Watch the Dashboard until all sites have processed their pending transactions.
5. Back up JIRA at each site. Access JIRA through the port specified in step 2. Go to the Administration tab, and use the back up command.
6. Shut down all sites. At each site, for Unix, type:
```
tomcat/bin/shutdown.sh
```
for Windows, type:
```
tomcat\bin\shutdown
```
7. At the install site, copy the WANdisco license.key file (in the jira-multisite/config directory), and your database driver (in the tomcat/common/lib directory). You use these for the installation.
8. Archive the JIRA database back ups and the WANdisco directories: attachments, index, jira-multisite, tomcat, wandisco.
9. Perform the installation procedure. See section 3.2, First Time Installation.

NOTES: The port number you changed in step 2 is overwritten with the new installation.

Your existing JIRA license is backed up when you back up your JIRA database. Just use the WANdisco supplied empty backup.xml and synchronize it when the procedure calls for it.

In the installation procedure, step 23 allows you to select an existing JIRA database to include in the WANdisco installation package. If your JIRA database is not large, you may want to use this option. If your JIRA database is large, just use the New JIRA Install option, which uses the empty backup.xml, and you synchronize it later in section 3.2.6.3, Importing JIRA Data.

3.4 Adding JIRA nodes

To create a replication group with a different number of nodes (adding or removing nodes):

Timestamp problem: JIRA doesn't store dates in UTC format, so transferring a database to a node based at a site within a different timezone will result in reported times being inconsistant between nodes.

We are currently producing a workaround that will allow you update an exported databases to match it's stored times to the local timezone. The fix will be available in the Download section our support website.

Backing up your data

1. Ensure that no users are accessing JIRA. Before proceeding Check that there are no pending transactions on the admin console's dashboard.
2. Shutdown all JIRA instances.
For Unix, type:
```
tomcat/bin/shutdown.sh
```
For Windows, type:
```
tomcat\bin\shutdown
```
3. Take a database dump / export your database via backup.xml from any node.
4. Although all databases should be at exactly the same state, it's worth taking the time to confirm this by checking that they all have the same state.
To do this, you can view use the SEQUENCE_VALUE_ITEM table in JIRA that keeps track of the maximum ID for all tables. Query the database on all nodes, using:
```
Select * from SEQUENCE_VALUE_ITEM order by seq_name; 
```
Compare the output from all sites:
If ALL seq_id are the the same on all nodes, they're all at the same level.
If ALL seq_id are higher at one of the nodes, then it has the most up-to-date database.
If you're unable to determind which copy of the database has the hightest seq_id, don't continue with the procedure, contact our support team for further assistance.

Installing the new replication group

5. Rename your <jira-install> directory to jira_backup etc. If the indexes and attachments directories are not included, ensure these are alsobacked up.
6. Follow the installation instructions, adding any new nodes during the setup.

Reloading your data

7. You'll need to use either the backup.xml file, or the database dump, depending on which method you selected during step 3.

Using Backup.xml

In entityengine.xml, ensure "replication-enabled" is set to false.
Restart all nodes
Load backup.xml to the new node(s). DO NOT login to JIRA, as this will cause your databases to be knocked out of sync.
Shutdown the new node(s).
Restart all nodes, login and run a re-index on all nodes.

Using a database dump

Load database dump to new node(s).
Copy the attachments directory to new node(s).
Set replication-enabled to "true" in entityengine.xml file at all nodes in new replication group.
Restart all nodes, login and run a re-index on all nodes.

4 Using the Admin Console

By now you have installed WANdisco JIRA MultiSite and turned replication on. The Admin Console is a simple interface to monitor your sites. You can use the Admin Console with replication turned on or off.

You can run the Admin Console from any node. The logins and passwords are the same for all nodes, and are set during installation. The login is:

admin

and the password was specified in step 10 of section 3, Installation.

To start the Admin console, all you need is the IP address of any site and the replication port number.

4.1 Starting the Admin Console

In your browser's address bar, type:

http://<IP address>:<WANdisco port number>

The Admin Console's home page appears.

JIRA Setup 6

4.2 JIRA Admin Console

The console is a simple interface, with two main pages, JIRA and Dashboard. Here's a description of what appears on the screens.

Local JIRA Service

Node Name: the name of the current site
JIRA Replication: reports the current state, either enabled or disabled.
WANdisco install: the installation directory pathname.
Tomcat Directory: the Tomcat installation directory
GUID: the site's globally unique identifier, given by JIRA MultiSite at the time of installation

Update Tomcat Directory

local override properties: The pathname for JIRA's index and attachments. Each JIRA database uses the same pathname for its index and attachments. Some customer file systems do not support identical filenames on a network, so WANdisco allows you to override the JIRA path and specify a different pathname for each node during installation.
Node Name: The names of all sites and their WANdisco (DConE) port numbers are listed. You can check which ones to display on the Dashboard.
Show Dashboard and Selected: Use the checkboxes to select which servers you'd like to display on the Dashboard.

Left Side Navigation Menu

Local JIRA Service

Log Viewer: There are a few logs to view. Select a log and click Show Log. The log appears in the Dashboard. From the Dashboard, you can select another log, or return to the Admin Console home page.
Atlassian-jira.log - the current servers JIRA log
Jira-multisite.2008-06-06.log - date-based instance of JIRA MultiSite. Any errors in the log are highlighted in red.
Web.proxy.log - the MultiSite Installation log/

JIRA Setup 6

Change Distinguished Node: use this to change the Distinguished Node. See 5.7, Changing the Distinguished Node.

Log Level: WANdisco uses one log, and the default level is info.

The levels vary from severe, where you get only the most severe warnings, to finest, which logs every action.

Free Memory: This command frees the memory (GB stands for garbage collection) for the current site. The command occurs when you click on this menu selection. The display shows information on the command that was just performed.

JIRA Setup 6

Max Memory used by JVM: the maximum memory that JVM can use on the current site.
Free Memory Before GC: the amount of free memory before you ran this command.
Free Memory After GC: the amount of free memory after you ran this command.
Memory Freed: the total amount of memory that was freed by running the command.

Dashboard: offers another way to get to the Dashboard. The Dashboard is discussed in detail in the next section.

4.3 The Dashboard

The Dashboard shows the JIRA transactions. As soon as JIRA MultiSite receives a transaction request, that transaction joins the replication group's queue. JIRA MultiSite keeps track of which site it emanated from, but otherwise, the transaction joins the one queue for the replication group. By default, the Dashboard displays all sites. If you only want certain sites displayed, check just those sites and click Show Dashboard with Selected.

Here is a sample displaying two sites. The name of the Distinguished Node (Cluster3) displays in green.

4.3.1 Pending Transactions

There are three statuses of transactions before they are committed to the JIRA database.

Not Yet Scheduled - these transactions are in the queue
Scheduled - these transactions are approved by the distinguished node and are waiting to be executed
In Progress - these transactions are in the process of being completed

The Total Transactions Pending lists the total number of transactions in all statuses.

4.3.2 Completed Transactions

Once a transaction is completed, the Dashboard lists it in the transaction list, and that transaction is no longer considered in the In Progress count. The completed transaction joins the count in the Transaction Completed display. This display keeps count of transactions since replication began.

4.3.3 Refreshing the Dashboard Display

The Dashboard by default does not refresh. As JIRA MultiSite is completing many transactions, a display that refreshes often would be very confusing to read. While the order of completed transactions is the same at all sites, the real time of when a transaction is posted may vary from site to site. To refresh the Dashboard, click Update. You can also set the Dashboard to refresh automatically by entering a number in seconds in the field.

5 Procedures

5.1 Finding the Last Committed Transaction

Even though committed transactions are always in the same order for each site, the timing of the commits usually varies from site to site. So unless there are no JIRA users logged in, you probably are going to have timing variations per site for committed transactions.
Go to any site's Dashboard. Type:

http://<IP address>:/dashboard2

You see all the sites on the Dashboard to compare the listed transactions.

5.2 Adding a Site to the Replication Group

You can add a site to an existing JIRA replication group. Notify WANdisco of the IP address of the new site or sites, and WANdisco sends you a new installation file and license key file.

Follow the instructions in 3.3, Installing Upgrades. The instructions are identical; you are just adding more sites to the replication group.

5.3 Deleting a Site from the Replication Group

You can delete a site or sites from an existing JIRA replication group. Follow the instructions in 3.3, Installing Upgrades.

5.4 Identifying the Right prefs.xml File

The preferences files for the entire replication group are located on each site, in jira-multisite/config. The current site's preference file is called prefs.xml. All the replication group's preferences files are also included for reference, with the filename format wandisco.prefs-<site name>.xml. JIRA MultiSite ignores all files but prefs.xml.

The following table gives a clear picture of which preferences file affects which site.

Node	Has All Preferences Files	Relevant File for This Node
Site 1	`WANdisco.prefs-Site1.xml WANdisco.prefs-Site2.xml WANdisco.prefs-Site3.xml prefs.xml`	JIRA MultiSite reads `prefs.xml` only for the Site 1. Make any changes for Site 1 with this `prefs.xml`. (`prefs.xml` is a copy of `WANdisco.prefs-Site1.xml`)
Site 2	`WANdisco.prefs-Site1.xml WANdisco.prefs-Site2.xml WANdisco.prefs-Site3.xml prefs.xml`	JIRA MultiSite reads `prefs.xml` only for the Site 2. Make any changes for Site 2 with this `prefs.xml`. (`prefs.xml` is a copy of `WANdisco.prefs-Site2.xml`)
Site 3	`WANdisco.prefs-Site1.xml WANdisco.prefs-Site2.xml WANdisco.prefs-Site3.xml prefs.xml`	JIRA MultiSite reads prefs.xml only for the Site 3. Make any changes for Site 3 with this prefs.xml. (`prefs.xml` is a copy of `WANdisco.prefs-Site3.xml`)

5.5 Changing the prefs.xml Files

Each site contains all Preferences files for all sites in the replication group, although JIRA MultiSite ignores all files except a specific site's preferences file. See 5.4, Identifying the Right prefs.xml File.

If you make changes that affect more than one site, you must change each site's prefs.xml file. But if your change affects just one site, you can change just that site's preferences file.

This procedure involves taking the JIRA replication group offline momentarily. Please follow your company procedures about notifying JIRA users of down time. Remember that each site contains the preferences files for the entire replication group. Make sure you change the prefs.xml for the current site.

5.5.1 Changing prefs.xml for Multiple servers

Replication can continue if sites in a replication group are shut down and restarted. However, replication does not continue if you shut down the distinguished node.

1. At each site, shutdown Tomcat. For Unix, type:
```
tomcat/bin/shutdown.sh
```
For Windows, type:
```
tomcat\bin\shutdown
```
2. Make any changes in a site's preferences file.
3. Repeat Step 52 for each site affected by the change.
4. At each site, start tomcat. In tomcat/bin, type:
```
perl watchdog
```
The changes take effect on startup. All sites now are using the new preferences files.

5.5.2 Changing One Site's prefs.xml File

1. On that site, shutdown tomcat. For Unix, type:
```
tomcat/bin/shutdown.sh
```
For Windows, type:
```
tomcat\bin\shutdown
```
2. Make any changes in the specified site's preferences file.
3. On that site, start Tomcat. In tomcat/bin, type:
```
perl watchdog
```

The changes take effect on startup. This site is now using the new preferences file.

5.6 Finding the Distinguished Node

The Distinguished Node's name appears in green on the Dashboard. Also, you can click on Change Distinguished Node on the left side menu. The current Distinguished Node's name is displayed.

5.7 Changing the Distinguished Node

1. To change the Distinguished Node, go to the Admin Console.
2. Click on Change Distinguished Node on the left side menu.

3. Select the desired site from the drop down.
4. Click Assign Selected Node. The transaction may take a few moments to take effect. Refresh the page to see the change.

5.8 Rotating the Distinguished Node

For replication groups with a Singleton quorum (or Multiple quorum with a even number of sites), you can schedule the Distinguished Node to change, based on physical location around the world.

For example, say you have JIRA MultiSite in San Jose, CA, in Madrid, and in New Zealand. Since being the Distinguished Node offers the fastest response time, you would want each of those locations to take advantage of the time they could each be the Distinguished Node.

To do that, you would edit the Preferences file at only one location. Use the local time for that location, and figure out the times the Distinguished Node would change.

Leave the exisiting <Quorum> tag alone, and add two more tags, <Quorum> and <Schedule>, under the <AgreementManagerList> tag. For example,

<AgreementManagerList>
 <AgreementManager name="52ec6735-ce20-11d9-8e57-00065be02953">
   <DisplayName>cvs-am</DisplayName>
   <Quorum>
     <Schedule>
      <at name="09:00 AM">
        <!-- San Jose Site --> <DistinguishedNode>fb7723de-ce1e-11d9-ae57-00065be02953
	</DistinguishedNode>
     </at>
     <at name="08:30 PM">
        <!-- Madrid Site --> <DistinguishedNode>659a768d-ce1f-11d9-aeec-00065be02953
	</DistinguishedNode>
     </at>
     <at name="03:00 AM">
        <!-- New Zealand Site --> <DistinguishedNode>3fae40f3-ce20-11d9-8e6a-00065be02953
	</DistinguishedNode>
     </at>
     </Schedule>
   </Quorum>
 </AgreementManager>
</AgreementManagerList>

The time syntax is hh:mm:ss: AM|PM. Seconds are optional.

If you do rotate your distinguished node, the other sites' Preferences files do not reflect the rotation. You must rely on the Admin Console (available at all sites) to display the current distinguished node (the site's name displays in green).

5.9 Finding a Site's GUID

To find a site's GUID, you can go to that site's Admin Console's Home page, where it is listed. The GUIDs for the entire replication group are found in the Preferences file. Find a site's GUID by looking for the Member Name definitions (shown in bold) in the Preferences file. Each site's Preferences file lists all GUIDs for the group, so you can go to any site's Preferences file: jira-multisite/config/wandisco.prefs-<site name>.xml.

<Member name="59cb677b-0a65-11dd-891f-000000000002">
<Name>Site1</Name>
<unlisted>false</unlisted>
<Profiles>
<TransportPolicy>AlwaysDConeNet</TransportPolicy>
<Transport>
<DConeNet>
<ListenerIP>0.0.0.0</ListenerIP>
<ListenerPort>6444</ListenerPort>
</DConeNet>
</Transport>
</Profiles>
</Member>

5.10 Excluding a Failed Distinguished Node

Use this procedure if your Distinguished Node has failed, and you cannot wait for it to be restored.

Perform the procedure 5.3, Deleting a Site from the Replication Group, but during the installation, you must specify a different replication port for all sites. For example, if the WANdisco port was 6444 for all sites in the replication group, now use 6445 for all ports in this smaller group. This makes sure that any sites left out of the new replication group cannot rejoin the group unbeknownst to you.

To add the site when it is up and running, determine if you want to keep the existing Distinguished Node or return the errored site as the Distinguished Node.

5.11 Installing JIRA Plug-ins

Follow these instructions for include plug-ins into either the bundled or JIRA WAR-EAR installation. However, JIRA 4.0 introduces several changes that may break existing plugins.

There are now two different types of plugins. Each type of plugin needs to be installed into a different directory to work.

Version 1 plug-ins - Built with the original framework, they're located in /WEB-INF/lib/

Version 2 plug-ins - New to JIRA 4, these are located in /plugins/installed-plugins/

How do you tell the version of a plug-in?
Check the details of the plugin on the JIRA Plug-in website. Each plug-in has a 'Plugin System' field entry of either 'ONE' (for Version 1) or 'TWO' (for Version 2).

5.11.1 Plugin installation procedure

These are generic steps for installing JIRA plug-ins.

1. Shut down JIRA MultiSite at all sites. At each site, stop Tomcat. In tomcat/bin, type:
```
shutdown.sh
```
For Windows, in tomcat/bin, type:
```
shutdown.bat
```
2. Copy the JIRA plug-in jar file to each site per the plug-in installation instructions.
3. the plug-in license to one site. It gets replicated when you start JIRA MultiSite.
4. JIRA MultiSite at all sites. In tomcat/bin, type:
```
perl watchdog
```
5. After all the sites are restarted, create any custom fields for the plug-in at one site. JIRA MultiSite replicates them to all sites.

5.11.2 Enabling or Disabling Plug-ins

When you enable or disable JIRA plug-ins from the Administration Panel, you must do so at each site where you want the plug-in enabled or disabled.

5.12 Re-indexing JIRA Database

You can selectively re-index only those sites that need re-indexing. You do not have to re-index all the sites, and you do not have to re-index them at the same time.

5.13 Selectively Disabling JIRA Filter Subscriptions

JIRA's filter subscriptions are by default enabled. If you save a filter that is scheduled to run routinely, JIRA MultiSite replicates that schedule. So instead of receiving one result, you receive one from each replicator in your replication group.

You can avoid this scenario by disabling the filter subscription service at all but one site. Set the JAVA system property JIRA.FilterSubscriptions.enabled to false. How you set the JAVA system property depends on your environment and the application server. Please refer to the application server documentation.

For example, at the command line, type (for Unix):

export JAVA_OPTS='-DJIRA.FilterSubscriptions.enabled=false'

For Windows, for example, type:

SET JAVA_OPTS='-DJIRA.FilterSubscriptions.enabled=false'

5.14 Selectively Disabling JIRA Services

If you have enabled JIRA's backup service, JIRA MultiSite replicates the service at each site in the replication group. Our replication technology ensures that each site is a replica of the others, so we recommend either not running JIRA's backup service, or just running the service at one site (and disabling it at all other sites).

If you are going to run any service at only one site, follow this procedure.

1. Create a text file (for example, DisableServices.txt) that contains a list of all service names you want to disable on this site, one service per line. To get the correct service name, look on the JIRA Administration > Services page. Make sure to list the Name and not the Class.

2. Set the Java system property JIRA.services.disabled to point to the text file you created in step 1. How you set the system property depends on your environment and the application server. Please refer to the application server documentation.

For example, for Tomcat, you must set the environment variable JAVA_OPTS before calling startup.sh (.bat for Windows) or before calling catalina.sh (.bat for Windows). This is best accomplished in a file in the [tomcat]/bin directory named setenv.sh (.bat for Windows). The setenv.sh should contain the command below to set the JAVA_OPTS property.

For example, using the example file DisableServices for the text file name, type at the command line (for Unix)
```
export JAVA_OPTS='-DJIRA.services.disabled=<pathname>/DisableServices'
```
For Windows, for example, type:
```
SET JAVA_OPTS='-DJIRA.services.disabled=<pathname>/DisableServices'
```
3. Restart JIRA.
4. Verify that the Java system property is set by going to JIRA Administration > System Info and check the JVM Input Arguments entry.

Please refer to http://www.atlassian.com/software/jira/docs/latest/automated_backups.html.

5.15 Restart/reload databases on multiple servers

1. Shut down JIRA MultiSite at all sites. At each site, stop Tomcat. In tomcat/bin, type:
```
shutdown.sh
```
For Windows, in tomcat/bin, type:
```
shutdown.bat
```
2. Drop the data in the databases for each server.
3. Disable replication at each server by editing entityengine.xml (after installation this resides in the Tomcat directory), changing:
```
replication-enabled="true"
```
to
```
replication-enabled="false"
```
4. Backup and remove the systemdb directory.
5. Restart JIRA MultiSite on all the servers. In tomcat/bin, type:
```
perl watchdog
```
6. Load the data in each node.
7. Shutdown JIRA MultiSite on all servers (same as step 1).
8. Re-enable replication by undoing the change made in step 3.
9. Restart JIRA MultiSite on each node.

5.16 Running the reset script

The reset script has been introduced from version 4.0 to save you having to manually delete the /systemdb directory when forced to restart the replicator. To perform a reset:

1. Go to the svn-replicator/bin/ directory.
2. Run the following command
on Windows:
```
reset
```
on Unix:
```
perl reset
```

3. Heed the warning about running the reset script on all replicators, then follow the on-screen instructions.

6 Troubleshooting

6.1 How Do I Know Replicator is Working?

There are two ways you can check. You can make a minor change in JIRA on one client, wait a minute, and go to another client to ensure the change is reflected.

Another way to check if JIRA MultiSite is replicating, is to verify there are commit transactions posted to the log file tomcat\logs\catalina<today's date>.log.

...
INFO: [reader-1] Connection request to Node Id = 84f15c7f-fb5e-11dc-8024-000000000003,
host = 192.168.1.111, port = 6555, timed out in 500ms
1206566551115 org.nirala.communication.transport.DConeNet.AsyncConnector makeConnection
INFO: [reader-1] Connection request to DFTPEndpoint - Node Id = 192.168.1.1116555,
host = 192.168.1.111, port = 6555, timed out in 500ms
1206566556787 org.apache.coyote.http11.Http11BaseProtocol start
INFO: [main] Starting Coyote HTTP/1.1 on http-8080
1206566557006 org.apache.jk.common.ChannelSocket init
INFO: [main] JK: ajp13 listening on /0.0.0.0:8009
1206566557006 org.apache.jk.server.JkMain start
INFO: [main] Jk running ID=0 time=0/47 config=null
1206566557084 org.apache.catalina.storeconfig.StoreLoader load
INFO: [main] Find registry server-registry.xml at classpath resource
1206566557209 org.apache.catalina.startup.Catalina start
INFO: [main] Server startup in 40781 ms
1206567654006 org.nirala.util.proposals.TransactionalProposal onCommit
INFO: [gsn-1] Committed transaction number 199 file 9fa1b89a-fb7a-11dc-95ac-000000000003_0
1206567654022 org.nirala.util.proposals.TransactionalProposal onCommit
...

6.2 Connection Request Timeout Messages

In the MultiSite logs, you'll sometimes see messages about connection request timeouts. These are informational messages and should be ignored unless it is guaranteed that the connection can be established in a number of milliseconds and happens often. Normally two connections are established between each of the replicated machines, a DConE connection and a DFTP connection. These two connections were established when WANdisco JIRA MultiSite started and are used when required. A keep-alive signal is sent on the DConE port periodically. There is no traffic on DFTP until a file transfer.

Some lesser routers in the path of the two end points will close an established connection if there is no traffic on the connection without notifying the end points. When end points sent data on this stale connection, they hang forever. To deal with these lesser routers, JIRA MultiSite does not keep the DFTP open in its connection pool forever. JIRA MultiSite establishes a DFTP connection from receiver to sender when a file transfer was required. This solved the problem dealing with lesser routers.

Some companies have a corporate policy that network connections can only be established in one direction. To deal with this scenario, the replicated machines establish a DFTP connection to other sites periodically and tear them down if there is no traffic within a known interval. Once a connection is established, any side is free to use the connection regardless of which side initiated the connection. A connection in use is never torn down until it is available as a free connection. This is the current implementation.

It takes between 300 to 400 milliseconds to establish a network connection even on a slow Wide Area Network (WAN). By default, JIRA MultiSite waits for 500 milliseconds before giving up that a connection cannot be established to a peer machine and prints this informational message. What if the establishments of connection always take 501 milliseconds. In this case, a connection is never established. To solve this problem, the timeout value is adjusted in 10% increments of the last timeout, starting at 500 milliseconds, to a maximum of 10 seconds for each timed out connection. Upon establishment of a successful connection, this timeout value is used for subsequent connection establishment unless an adjustment is required for failed attempts.

6.3 How Do I Get WANdisco Support?

Before opening a case or submitting a new issue, always search the Knowledgebase on
http://www.support.wandisco.com.

See our Knowledgebase article on how to raise a case to our support team - How to raise a case to support.

6.4 Recovering a single node from a corrupt database.

Use this procedure if one of your JIRA nodes becomes read-only due to a failure of its database.

Timestamp problem: JIRA doesn't store dates in UTC format, so copying a database to a corrupted node based at a site within a different timezone will result in reported times being inconsistant between nodes.

We have a workaround that will allow you update an exported databases to match its stored times to the local timezone. The fix is available in the Download section of our support website.

Read our knowledgebase article JIRA timestamp offset script.

1. Access the node you intend to recover. On the other nodes, run the shutdown script:
```
tomcat\bin\perl shutdown.sh
```
or on Windows
```
tomcat\bin\shutdown
```
2. On the remaining node, backup the JIRA database using the JIRA Administration screen.
3. Shutdown the remaining node (same as step 1).
4. Disable replication by modifying the /tomcat/jira/WEB-INF/classes/entityengine.xml
Change
```
replication-enabled="true"
```
to
```
replication-enabled="false"
```
5. Restart all nodes. They will continue to replicate each other.
6. On the node where the database is to be restored, re-initialize the database
7. Start tomcat, go to tomcat\bin\ and use the following command:

on Windows
```
startup.bat
```
on Unix
```
perl startup.sh	
```
8. Login to JIRA at the node:
```
http://<node's IP Address>:8080/jira
```
9. Following the instructions to import your existing data.
10. shutdown tomcat: tomcat\bin\shutdown.bat
11. turn replication back on in entityengine.xml (Turning On Replication 3.2.6.5)
12. start tomcat: tomcat\bin\startup.bat

7 Frequently Asked Questions

7.1 Why Are So Many Java Processes Running?

On older versions of Linux, every thread is listed as a process by the ps command. This does not affect the operation of JIRA MultiSite. WANdisco does not support the older versions of Linux.

7.2 Can I Store Logs or Content on NFS?

NFS (Network File System) allows files and directories to be accessed remotely over a network using NFS clients. NFS clients are typically built into the operation system kernel these days. However, some operations, like renaming a file, are not guaranteed to be atomic over NFS. Here is a snippet from the rename function's man page on Linux, for example:

BUGS

On NFS filesystems, you can not assume that if the operation failed the file was
not renamed. If the server does the rename operation and then crashes, the
retransmitted RPC which will be processed when the server is up again causes a
failure. The application is expected to deal with this. See link(2)
for a similar problem.

Bug tracking systems such as JIRA make heavy use of the rename operation to modify the underlying databases. Independent of WANdisco, it is a risky practice to store JIRA database content on NFS. The bug tracking community at large recommends not using NFS for storing repositories. JIRA MultiSite is bundled with a built-in transactional journal and an object database. These are by default stored in the jira-multisite/systemdb and jira-multisite/config directories. These directories should not be mounted on an NFS drive. The Replicator itself may be installed on an NFS drive but the systemdb and config directories should be on direct storage (non-NFS options like RAID, SCSI, SAN, etc). Replicator's transactional integrity can be compromised if writes to an NFS server are lost due to potential NFS client cache crash after the NFS server has indicated IO completion.

7.3 Why is Set Up Configuring IP Addresses as 0.0.0.0?

The address 0.0.0.0 is a special IP address, treated as a wild-card IP address. In other words, on a machine with multiple NICs (Network Interface Cards), it binds to all interfaces. The advantages of using wild-card IP address include:

It avoids binding to a fixed IP address. If the host's IP address changes, (for example, the subnet changes, or the machine is moved to a different location) you don't have to change the wild-card IP in the prefs.xml file to the new IP address.
There is wider bandwidth to TCP clients. Now TCP clients can connect to any NIC, because JIRA MultiSite is listening on multiple NICs.

The disadvantage to using the wild-card IP is that it gives coarser access control at the IP address level, as all address are being listened to at the specified port.

You can always switch from the wildcard IP address to a fixed, static IP address or a DNS hostname, though for the most part, WANdisco recommends you stick with wild-card addressing.

7.4 How are timezone differences between servers handled?

Each server can run in its own local timezone and as the data gets replicated, the times of the events (issue creation, comment addition) appear in the timezone of the server being accessed. Dates get stored in JIRA in UTC format and then get offset that JIRA server by the time zone when that date is retrieved and displayed. .

The time when a transaction is processed is the time of that event when stored in the database for that node. If a node is down, a time could be offset by 2 minutes, 2 hours or 2 days. The timezone differences could then provide an additional offset.

At the time of writing, there is an outstanding issue in Atlassian's own JIRA database for the JIRA product, JRA-9, that requests that users can set a preference to specify their local time zone and have JIRA display data in the user's time zone.

7.5 Does WANdisco Support Dynamic DNS?

Yes, WANdisco supports dynamic DNS, but strongly discourages its use.

If a hostname is specified during the setup process, WANdisco requires that it should be able to connect to a valid DNS and resolve the hostname to valid IP address upon startup. If the hostname cannot be resolved to an IP (either by not being able to connect to DNS, or no entry is found at the given hostname), WANdisco dies gracefully. This has never been a problem during production and with static IPs.

However, if dynamic DNS support is required, please modify the prefs.xml file at each site and set UseDynamicDNS to true in DConeNet element.

<Preferences>
...
<DConeNet>
...
<UseDynamicDNS>true</UseDynamicDNS>
</DConeNet>

In addition, the following Java security properties should be set to different Time-to-live (TTL).

networkaddress.cache.ttl

networkaddress.cache.negative.ttl

Please read InetAddress Caching for more details.

8 Appendix

8.1 Installation Guides

You should have already installed Java and Perl at all the nodes in your replication group for your trial evaluation. Any new node you add to the replication group needs Java and Perl installed as well.

8.1.1 Installing Java

1.Install JDK 1.6 and define the JAVA_HOME environment variable to point to the directory where the JDK is installed. You can download JDK 1.6. from http://java.sun.com/.

2. Add $JAVA_HOME/bin to the path and ensure that no other java (JDK or JRE) is on the path.

$ which java
/usr/bin/java
$export JAVA_HOME="/usr"

$which java
/export/share/apps/jdk/1.6.0/bin/java
$export JAVA_HOME="/export/share/apps/jdk/1.6.0"

3. Ensure the full JDK is installed, not just the JRE. This can be confirmed by running java -server -version. If it generates a not found error, repeat Steps 1 and 2.

If you find package management problems or conflicts with the JDK version you are downloading (for example, rpm download for Linux), you may want to use the self-extracting download file instead of the rpm (on Linux) package. The self-extracting download easily installs in any directory without any dependency checks.

8.1.2 Installing Perl

1. On UNIX or Cygwin, install perl version 5.6 or greater and ensure that the perl executable is on the system path.

2. On Windows, install ActivePerl version 5.8 or greater and ensure that the perl executable is on the system path. You can download the MSI installer for ActivePerl

8.1.3 Example Database Configuration - MySQL

1. Create a database user that JIRA will connect as.
2. Create a database (using UTF8 character set) for JIRA to store issues in called jiradb.
3. Assign the user privileges to connect to the database, then create and populate the tables.
4. Download the mysql JDBC driver and install following Atlassian's MySQL connection guide.
5. Change the template.tomcat.server.xml and entityengine.xml in locations as per the admin guide.
In entityengine.xml:

datasource name="defaultDS" field-type-name="hsql"
schema-name="PUBLIC"
helper-class=.....

6. Change the field-type-name to mysql and remove the schema-name="PUBLIC" line to give the following :

 datasource name="defaultDS" field-type-name="mysql"
helper-class=....

In server.xml :

username="sa"
password=""
driverClassName="org.hsqldb.jdbcDriver"
url="jdbc:hsqldb:${catalina.home}/database/jiradb"
minEvictableIdleTimeMillis="4000"
timeBetweenEvictionRunsMillis="5000"
maxActive="20" />

7. Change the username/password to the one you created in mysql to manage the JIRA database Change the driverClassName and URL.
8. Remove minEvictableIdleTimeMillis and timeBetweenEvictionRunsMillis
Add the following lines to validationQuery:

username="database username"
password="database password"
driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost/jiradb?autoReconnect=true&useUnicode=true&characterEncoding=UTF8"
maxActive="20"
validationQuery="select 1"

8.2 WANdisco MultiSite Concepts

All MultiSite sites are synchronized at all times: each JIRA database is a functional replica of the others. WANdisco replication technology is the concept of one database, multiplied. Because there are multiple synchronized databases, each replicated site is effectively a current hot backup, which makes disaster recovery easy to plan and implement.

8.2.1 How Replication Works

The sites in the replication group are continuously coordinating the JIRA write transactions users are making. The group establishes transaction ordering through the agreement of a quorum of replicas. By default, JIRA MultiSite uses a Singleton Response quorum, which means that only one of the sites decides on the transaction order. The site that decides transaction ordering is called the distinguished node. So, with a Singleton Response quorum, if the replication group has five sites adding transactions to the transaction queue, only one site determines the position of each transaction in the transaction sequence.

8.2.2 Singleton, Majority and Unanimous Response Quorum

Singleton quorum

The Singleton quorum offers the fastest response time for those users working at the distinguished node, because as soon as the distinguished node determines that a transaction can be processed in the correct order, WANdisco sends that transaction to the JIRA server. Any replicator except the distinguished node can go down, but the replication group continues. However, the Singleton quorum also represents a single point of failure, since replication halts if the distinguished node fails.

You can schedule the distinguished node to rotate to different sites around the world, to "follow the sun". Rotating the distinguished node allows other sites to take advantage of the quickest response time when they are most active. See 5.8 Rotating the Distinguished Node. Besides the Singleton quorum, another quorum option is Majority Response.

Majority quorum

The Majority quorum requires that a majority of nodes agree on transaction order before any transaction is committed.

Having a majority quorum ensures that if one site goes down in a replication group, the other sites can continue uninterrupted, as long as a majority of the sites remain available. The replication group replays the missing transactions when that site rejoins the group.

What happens if a majority of nodes stay up (quorum retained):
When connection with one or more nodes is lost, so long as the remaining nodes still form a majority, they'll continue to work without interuption. Users at the downed nodes can only execute read operations (viewing issues, running reports), working with stale data.

When the disconnected node is back online it will automatically sync up with the replication group. First, the local node consults its local recovery journal (similar to a database redo log), and then, if necessary, attempts recovery from any of the quorum nodes.

What happens if a majority of nodes go down (quorum lost):
When there are not sufficient nodes available to form a majority, all nodes will enter a read-only mode.
Replication can only continue after enough nodes are brought back up to form a majority.

Unanimous Response quorum

A Unanimous Response qourum requires agreement from all nodes. As a result, all nodes will remain completely synchronised. However, this ideal mechanism for transaction agreement comes at the cost of both performance and reliability.

8.2.3 Replication Example

Here is an overview of what occurs when a write transaction is received by any replicator in the replication group.

The originating client sends the transaction to JIRA MultiSite, which passes it along through the replication group.
Transaction data is successfully received by the quorum (the distinguished node for Singleton quorum). The quorum assigns it a Global Sequence Number (GSN).
After receiving the transaction, each JIRA MultiSite passes the transaction data to its local JIRA server.
Each local JIRA processes the transaction.
JIRA MultiSite waits for JIRA to complete the transaction. JIRA MultiSite only marks the transaction complete when JIRA returns a completion status.

If for some reason replication goes down during this process (the replicator crashes, is stopped by an admin or the server it's running on shuts down), JIRA MultiSite marks the transaction as not complete, and it gets reprocessed upon replication restart. This allows the recovery logic to work properly.

8.2.4 Establishing a Replication Baseline

When you deploy JIRA MultiSite, you must ensure that all the replicas start out in sync. Once JIRA MultiSite is deployed, WANdisco's replication technology ensures they remain in sync.

You start with one database, referred to as a replication baseline. To create this baseline, you typically export a JIRA database to an XML file. If your baseline is an existing stand-alone JIRA instance, export that instance to an XML file. Otherwise, JIRA MultiSite comes with an empty JIRA database exported to a file,backup.xml, which you use as a baseline.

For extremely large JIRA databases, JIRA's export and import functions may be too slow to be practical. In this case, you may consider using your database vendor's dump and restore tools instead.

8.3 WANdisco Glossary

replica: A JIRA instance that is an exact equivalent or copy of another JIRA instance. In WANdisco's MultiSite product, a replica is also called a site. Each site is a replica of the other sites.
replicator: Each replica has an associated replicator. The replicator coordinates with other replicators (on the DConE port) to ensure that distributed JIRA databases stay in sync with each other.
replication group: A collection of replicators that work together to keep JIRA database replicas in sync.
one copy equivalence: All replicas are functionally equivalent to a single JIRA instance
installDir: this is the installation directory for WANdisco JIRA MultiSite
GUID: Globally Unique Identifier. WANdisco JIRA MultiSite assigns each site a GUID on installation. The sites identify each other by their GUIDs.
site: a server on which is installed a replicator and a replica. The sites comprise the replication group.
distinguished node: The distinguished node defines the quorum. Any site can go down, but replication continues as long as the distinguished node is up.
Quorum: A set of sites that can reach an agreement without participation from the remaining sites. JIRA MultiSite by default has a Singleton quorum, meaning the distinguished node alone sets the order of transactions.
Prefs.xml: The preferences file is a configuration file. There is one for each site. Each file contains configuration information for each site in the replication group, but the files are specific to each site. The preference files are located in jira-multisite/config.
DConE: WANdisco's Distributed Coordinated Engine, the software engine underlying replication.
Install server: the server on which you install JIRA MultiSite. By default this is the first site in the replication group. We used to call this the Install site, although this could cause confusion for customers who use more than one server on a single site.

JIRA MultiSite v4.0 w1 Admin Guide

1 Introduction

1.1 Release Notes for JIRA MultiSite v4.0w1 (v4.0.0.1 build 12594)

WANdisco JIRA MultiSite

1.2 About JIRA

1.2.1 JIRA Application Server

1.2.2 JIRA Database

2 Before Deployment

2.1 Setting up identical JIRA replicas on all servers

2.2 Physical Environment

2.3 Firewalls and Virus Scanners

2.3.1 Firewalls

2.3.2 Virus Scanners

2.4 Deployment Checklist

2.5 JIRA Plug-in Support

2.5.1 JIRA Plug-in Support Policy

Fully Supported Plug-ins

Provisionally Support Plug-ins

Under review for future support

Unsupported Plug-ins

2.6 Plug-in Compatibility

2.7 Plug-in Installation Uniformity

2.8 Plug-ins and JIRA Services

2.9 Common Plug-ins

3 Installation

3.1 Installation Checklist

3.1.1 Have an Existing JIRA License?

3.1.1.1 Option 1 - Export your existing JIRA data

3.1.1.2 Option 2 - Manually synch your existing JIRA data

3.2 Installation Procedure

3.2.1 Replication Group Setup Page

3.2.2 Configuring the install server

3.2.3 Setting JIRA Home

3.2.4 Configuring Subsequent Sites

3.2.5 Looking Over the Summary Page

3.2.5.1 Database Customization

3.2.5.2 JIRA Customizations - Pre Installation

3.2.5.3 Selecting a Distinguished Node

3.2.5.4 Selecting a Quorum Type

3.2.6 Installing the Application Server (Packaged Tomcat)

3.2.6.1 First Site Installation and Package Creation

3.2.6.2 Installing Subsequent Sites

3.2.6.3 Importing JIRA Data

3.2.6.4 Validating the Installation

3.2.6.5 Turning On Replication

3.2.7 Installing With a WAR File

3.2.7.1 Generating a WAR/EAR File

3.2.7.2 Installing the Sites

3.2.7.3 Importing JIRA Data

3.2.7.4 Turning On Replication

3.2.7.5 Redeploying the WAR With Replication Enabled

3.2.7.6 For Tomcat Users

3.2.7.7 Validating the Installation

3.3 Installing Upgrades

3.4 Adding JIRA nodes

Backing up your data

Installing the new replication group

Reloading your data

4 Using the Admin Console

4.1 Starting the Admin Console

4.2 JIRA Admin Console

Left Side Navigation Menu

4.3 The Dashboard

4.3.1 Pending Transactions

4.3.2 Completed Transactions

4.3.3 Refreshing the Dashboard Display

5 Procedures

5.1 Finding the Last Committed Transaction

5.2 Adding a Site to the Replication Group

5.3 Deleting a Site from the Replication Group

5.4 Identifying the Right prefs.xml File

5.5 Changing the prefs.xml Files

5.5.1 Changing prefs.xml for Multiple servers

5.5.2 Changing One Site's prefs.xml File

5.6 Finding the Distinguished Node

5.7 Changing the Distinguished Node

5.8 Rotating the Distinguished Node

5.9 Finding a Site's GUID

5.10 Excluding a Failed Distinguished Node

5.11 Installing JIRA Plug-ins