Before you start
Make sure you have the following setup and resources to successfully install and use LiveData Platform for Azure.
Supported operating systems#
LiveData Migrator for Azure supports the following:
- Ubuntu 16 and 18
- CentOS 6 and 7
- Red Hat Enterprise Linux 6 and 7
Supported filesystem versions#
Filesystems supported as sources:
- HDP 2.6.3, 2.6.5, and 3.1.0
- CDH 5.13, 5.15, 5.16, 6.2, and 6.3
- CDP 7.1.4
note
All HDFS versions above Hadoop 2.6 should work. However, if you need additional support, contact WANdisco Support.
Filesystems supported as targets:
- ADLS Gen 2
note
LiveData Migrator for Azure does not support ADLS Gen1 or Azure Storage Blob (ASB) as target filesystems.
Supported metastores for metadata migrations:
- Azure HDI 3.6/4.0 Hive External Metastore
- Azure SQL Database
Required Azure CLI versions:
- 2.26.0 or higher
note
We recommend you use the highest available version of the Azure CLI at all times.
Azure HDInsight version requirements:
- HDI 4.0
HDI Enterprise Security Package (ESP) is not supported.
Source environment#
Provide your own Hadoop cluster or use the HDFS Sandbox if you want to test LiveData Platform for Azure. We also support Hortonworks Data Platform (HDP) sandboxes.
info
See Getting Started with the Trial Sandbox to learn how to use the HDFS Sandbox with LiveData Migrator for Azure.
To use LiveData Migrator with your own Hadoop cluster, you need the following:
- Root access to the Hadoop cluster.
- All of the network requirements.
- For example, ports 5671 and 443 must be open for outbound traffic.
The edge node on your on-premises cluster needs the following:
- Hadoop client libraries installed.
- Hadoop client available within the systemd environment.
- Java 1.8+
- If Kerberos is enabled on your Hadoop cluster, a valid keytab containing a suitable principal for the HDFS superuser must be available on the edge node.
- If you want to migrate Hive metadata from your Hadoop cluster, the edge node must also have a keytab containing a suitable principal for the Hive service.
Azure resources#
Make sure your selected region is supported and consistent across all your resources.
- An Azure subscription.
- Resource group to contain the Azure zone resources.
- Owner or Contributor level access to the resource group.
- Create an ADLS Gen2 storage account and container with hierarchical namespace enabled.
Resources for LiveData Migrator metadata migrations#
To migrate metadata, you will require the following:
- An Azure SQL Database in the same Azure resource group as LiveData Migrator.
Supported regions#
The following regions are supported.
- Australia East
- Canada Central*
- East Asia
- East Japan
- East US 2
- East US
- Germany West Central*
- Korea Central*
- North Europe
- Southeast Asia
- South Central US
- UK South
- West Europe
- West Central US
- West Japan
- West US
- West US 2
- West US 3
Support for regions marked with an * is offered as a feature preview, and will be improved in the future.
note
If you can't see or select some of the above regions in the Azure Portal, you're most likely using an old resource provider. Re-register the resource provider to fix this.
Next steps#
As an additional prerequisite to use LiveData Migrator for Azure, register the LiveData resource provider.