Skip to main content
Version: 1.15.1

Connect metastores for metadata migrations

LiveData Migrator replicates metadata between Databricks (target only), Apache Hive, AWS Glue and Azure SQL.

Ready to migrate metadata? First, connect to your Metastores by adding Hive agents.

note

Databricks agents are currently available as a preview feature.

info

The source table format must be Parquet to ensure a successful migration to Databricks Delta Lake.

Connect Metastores With The UI#

Apache Hive#

LiveData Migrator will attempt to auto-discover Apache Hive and create an agent for your source environment. Check whether an existing agent is listed under the Agents panel.

If Kerberos is enabled on your cluster and HDFS is configured as your source filesystem, select to configure the existing agent and provide the Kerberos credentials.

  1. Click Connect To Metastore.

  2. Provide a Display Name.

  3. Select Hive as the Agent type.

  4. Provide an Override Default Hadoop Configuration Path.

    caution

    If using a local Hive agent for a target filesystem, then hive-site.xml must be copied from the target cluster to the local cluster into a location specified by the Override Default Hadoop Configuration Path. Alternatively, a remote agent can be used for the target filesystem (not currently supported via the ui).

  5. Select the Filesystem.

  6. Specify DefaultFs Override (optional).

  7. Click Save

AWS Glue Data Catalog#

  1. Click Connect To Metastore.

  2. Select AWS Glue as the Agent type.

  3. Provide a Display Name.

  4. Select the AWS Catalog Credentials Provider.

  5. Enter the AWS Glue Service Endpoint.

  6. Enter the AWS Region.

  7. Select the Filesystem.

  8. Specify DefaultFs Override (optional).

  9. Click Save

Azure SQL#

  1. Click Connect To Metastore.

  2. Select Azure SQL DB as the Agent type.

  3. Provide a Display Name.

  4. Enter the Azure SQL Server Name

  5. Enter the ADLS Gen2 Storage Account Name and Container Name.

  6. Specify the Root Folder.

  7. Select the Authentication Method.

  8. Select the HDI version.

  9. Select the Filesystem.

  10. Specify DefaultFs Override (optional).

  11. Click Save

Databricks Delta Lake (Target Only)#

Databricks Delta Lake Metastores are supported as a target only. LiveData Migrator can convert tables to Delta format during migration.

  1. Click Connect To Metastore.

  2. Select Databricks as the Agent type.

  3. Provide a Display Name.

  4. Enter the JDBC Server Hostname, Port and HTTP Path.

  5. Enter the Access Token.

  6. Enter the FS Mount Point.

  7. Select the Filesystem.

  8. Specify DefaultFs Override (optional).

  9. Click Save

Google Cloud Dataproc#

  1. Click Connect To Metastore.

  2. Select Google Cloud Dataproc as the Agent type.

  3. Provide a Display Name.

  4. Provide the Hostname or IP Address.

  5. Provide the Port.

  6. Select the Filesystem.

  7. Specify DefaultFs Override (optional).

  8. Click Save

Connect Metastores With The CLI#

Connect To Metastores#

Connect To Metastores to connect your source and target Metastores.

CommandAction
hive agent add azureAdd a Hive agent for an Azure SQL connection
hive agent add filesystemAdd a Hive agent for a local filesystem
hive agent add glueAdd a Hive agent for an AWS Glue Data Catalog
hive agent add hiveAdd a Hive agent for a local or remote Apache Hive Metastore
hive agent add databricksAdd a Hive agent for a Databricks Delta Lake Metastore
hive agent add dataprocAdd a Hive agent for a Google Cloud Dataproc Metastore

Configure Existing Hive Agents#

CommandAction
hive agent configure azureChange the configuration of an existing Hive agent for the Azure SQL database server
hive agent configure filesystemChange the configuration of an existing Hive agent for the local filesystem
hive agent configure glueChange the configuration of an existing Hive agent for the AWS Glue Data Catalog
hive agent configure hiveChange the configuration of an existing Hive agent for the Apache Hive Metastore
hive agent configure databricksChange the configuration of an existing Hive agent for the Databricks Delta Lake Metastore
hive agent configure datapropcChange the configuration of an existing Hive agent for the Google Cloud Dataproc Metastore

Manage Hive Agents#

CommandAction
hive agent checkCheck whether the Hive agent can connect to the Metastore
hive agent deleteDelete a Hive agent
hive agent listList all configured Hive agents
hive agent showShow the configuration for a Hive agent
hive agent typesList supported Hive agent types

Next Steps#

Connected to your Metastores? Define metadata rules for your metadata migrations.