Connect metastores for metadata migrations

LiveData Migrator replicates metadata between Databricks (target only), Apache Hive, AWS Glue and Azure SQL.

Ready to migrate metadata? First, connect to your Metastores by adding Hive agents.

note

Databricks agents are currently available as a preview feature.

info

The source table format must be Parquet to ensure a successful migration to Databricks Delta Lake.

Connect Metastores With The UI

Apache Hive

LiveData Migrator will attempt to auto-discover Apache Hive and create an agent for your source environment. Check whether an existing agent is listed under the Agents panel.

If Kerberos is enabled on your cluster and HDFS is configured as your source filesystem, select to configure the existing agent and provide the Kerberos credentials.

1. Click Connect To Metastore.

2. Provide a Display Name.

3. Select Hive as the Agent type.

4. Provide an Override Default Hadoop Configuration Path.

   caution

   If using a local Hive agent for a target filesystem, then hive-site.xml must be copied from the target cluster to the local cluster into a location specified by the Override Default Hadoop Configuration Path. Alternatively, a remote agent can be used for the target filesystem (not currently supported via the ui).

5. Select the Filesystem.

6. Specify DefaultFs Override (optional).

7. Click Save

AWS Glue Data Catalog

1. Click Connect To Metastore.

2. Select AWS Glue as the Agent type.

3. Provide a Display Name.

4. Select the AWS Catalog Credentials Provider.

5. Enter the AWS Glue Service Endpoint.

6. Enter the AWS Region.

7. Select the Filesystem.

8. Specify DefaultFs Override (optional).

9. Click Save

Azure SQL

1. Click Connect To Metastore.

2. Select Azure SQL DB as the Agent type.

3. Provide a Display Name.

4. Enter the Azure SQL Server Name

5. Enter the ADLS Gen2 Storage Account Name and Container Name.

6. Specify the Root Folder.

7. Select the Authentication Method.

8. Select the HDI version.

9. Select the Filesystem.

10. Specify DefaultFs Override (optional).

11. Click Save

Databricks Delta Lake (Target Only)

Databricks Delta Lake Metastores are supported as a target only. LiveData Migrator can convert tables to Delta format during migration.

note

Version 2.6.25 of the Databricks JDBC driver was released recently. If you download this version, you'll receive an error message stating you haven't downloaded a driver. This is because the latest version isn't compatible with Hive Migrator. Download version 2.6.22 to create your agent.

1. Click Connect To Metastore.

2. Select Databricks as the Agent type.

3. Provide a Display Name.

4. Enter the JDBC Server Hostname, Port and HTTP Path.

5. Enter the Access Token.

6. Enter the FS Mount Point.

7. Select the Filesystem.

8. Specify DefaultFs Override (optional).

9. Click Save

Google Cloud Dataproc

1. Click Connect To Metastore.

2. Select Google Cloud Dataproc as the Agent type.

3. Provide a Display Name.

4. Provide the Hostname or IP Address.

5. Provide the Port.

6. Select the Filesystem.

7. Specify DefaultFs Override (optional).

8. Click Save

Connect Metastores With The CLI

Connect To Metastores

Connect To Metastores to connect your source and target Metastores.

Command	Action
hive agent add azure	Add a Hive agent for an Azure SQL connection
hive agent add filesystem	Add a Hive agent for a local filesystem
hive agent add glue	Add a Hive agent for an AWS Glue Data Catalog
hive agent add hive	Add a Hive agent for a local or remote Apache Hive Metastore
hive agent add databricks	Add a Hive agent for a Databricks Delta Lake Metastore
hive agent add dataproc	Add a Hive agent for a Google Cloud Dataproc Metastore

Configure Existing Hive Agents

Command	Action
hive agent configure azure	Change the configuration of an existing Hive agent for the Azure SQL database server
hive agent configure filesystem	Change the configuration of an existing Hive agent for the local filesystem
hive agent configure glue	Change the configuration of an existing Hive agent for the AWS Glue Data Catalog
hive agent configure hive	Change the configuration of an existing Hive agent for the Apache Hive Metastore
hive agent configure databricks	Change the configuration of an existing Hive agent for the Databricks Delta Lake Metastore
hive agent configure datapropc	Change the configuration of an existing Hive agent for the Google Cloud Dataproc Metastore

Manage Hive Agents

Command	Action
hive agent check	Check whether the Hive agent can connect to the Metastore
hive agent delete	Delete a Hive agent
hive agent list	List all configured Hive agents
hive agent show	Show the configuration for a Hive agent
hive agent types	List supported Hive agent types

Next Steps

Connected to your Metastores? Define metadata rules for your metadata migrations.