Create your storage account that provides Azure Data Lake Storage Gen2 capabilities.

Once created, navigate to the storage account in the Azure console, access the “Access keys” item under “Settings”, and take note of the Key value for either key1 or key2, as this will be used in later configuration. You will also need to recall the name you used for the storage account itself.

Create an HDInsight cluster with Azure Storage as the primary storage type, referencing your newly-created storage account for ADLS Gen2. You must add a VNet and subnet in the “Virtual Network Settings” step. Select Hadoop 2.7 on Linux (HDI 3.6) as the cluster type on creation.

After confirming that your cluster is operations, access the Ambari interface to add appropriate configuration for use of the ADLS Gen2 storage as the primary storage type:

Restart all affected cluster services, and wait on Ambari showing healthy cluster state, then validate the you can access ADLS Gen2 as the primary file system, e.g.

Where the <CONTAINER_NAME> portion is the container in the storage account that was created for the cluster, <STORAGE_ACCOUNT_NAME> should be replaced with your storage account details.

Install the HDI 3.6 version of WANdisco Fusion onto a cluster edge or head node as the root user, choosing the “hdi-3.6” option, not the “hdi 3.6 Data Lake Store” option.

Choose the default hdfs/hadoop user and group.

You may need to setup a port redirection using SSH to gain easy access to the Fusion UI to complete installation.

On the Node Details step (5), provide the storage account access key for the “KEY1 Access Key” value, and the value of the cluster’s fs.defaultFS property for the WASB Storage URI, e.g.

This should validate correctly.

You can complete server installation without following the clients step for now.

Then modify the /etc/wandisco/fusion/server/core-site.xml to refer to abfs rather than _wasb as the underlying file system, e.g.:

Restart the Fusion server after making this change.

You can deploy the WANdisco Fusion client from the HDInsight Marketplace.

Go to your cluster in the Azure portal and select “Application” under Configuration, then click on “+ Add”, and select the correct WANdisco Fusion HDI App item.

Note the in the field labelled “License key” you actually need to provide the hostname of the instance where the Fusion server is running.

Fusion client application deployment will take some time. You can monitor progress in Ambari and the Script Actions section of your cluster. After completion, modify the ambari configuration properties to refer the client to the correct underlying file system, changing the configuration property for fs.fusion.underlyingFs from _wasb://_something to the same setting as for the Fusion server, e.g.:

and add the fs.fusion.underlyingFsClass=org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem configuration property to the custom core-site.

Restart all affected cluster services.

At this stage, you should be able to verify the correct operation of either cluster when accessing their default file systems (ADLS Gen1 and ADLS Gen2 respectively.)

Gen1 Cluster

Gen2 Cluster

With WANdisco Fusion installed for each cluster, you are now free to establish replication rules that govern which data will be made consistent among the environments.

Configure these as described in the WANdisco Fusion product documentation to create specific directories for which any new or changed data will be replicated between the ADLS Gen1 and ADLS Gen2 storage accounts.

Use the repair feature to perform an initial transfer of pre-existing content from ADLS Gen1 to ADLS Gen2 if required. This allows you to bring data sets to your ADLS Gen2 environment. A key benefit of WANdisco Fusion is that application operation in either the Gen1 or Gen2 clusters can continue while data are replicated, because WANdisco Fusion maintains data consistency.

WANdisco Fusion is designed for workloads and data volumes at massive scale, but will benefit from configuration based on the specific demands of you upgrade needs. Choices such as the number of WANdisco Fusion server instances, replication rule configuration and other finer-grained configuration options will be affected by your specific upgrade needs. Options also exist for the replication of ancillary information such as Hive metadata, Ranger policies, and more. WANdisco can be consulted for assistance and support with a paid subscription to the WANdisco Fusion product.

WANdisco Fusion uses the network to replicate data and maintain consistency between the ADLS Gen1 and ADLS Gen2 storage accounts. If there is insufficient bandwidth to accommodate the rate of change of data, you may need to reduce the speed with which new data are created or existing data are modified.

Once your environments are consistent WANdisco Fusion will need to remain operational if you intend to keep data available to either storage platform. This will allow you to continue to operate on your data in either Gen1 or Gen2, which may need to extend beyond the initial period of upgrade to ADLS Gen2.