Skip to main content
Version: 1.14.0

Connect metastores for metadata migrations

This version of LiveData Migrator includes a feature that migrates metadata. Metadata can be migrated from Apache Hive, AWS Glue, and Azure SQL.

Additionally, Databricks Delta Lake metastores are supported as a target and LiveData Migrator can convert tables to Delta format during migration.

note

Databricks agents are currently available as a preview feature.

info

The source table format must be Parquet to ensure a successful migration to Databricks Delta Lake.

Ready to migrate metadata? First, connect to your metastores by adding Hive agents. Then, define metadata rules before migrating metadata.

Connect metastores with the UI#

Add source agent#

LiveData Migrator will attempt to auto-discover and create an agent for your source environment. Select your LiveData Migrator instance from the Storage panel and check whether an existing agent is listed.

If Kerberos is enabled on your cluster and HDFS is configured as your source storage, select to configure the existing agent and provide the Kerberos credentials.

Alternatively, add a new agent. The source agent can be any of the following distributions:

Add target agents#

Select your LiveData Migrator instance from the Storage panel and add new agents. LiveData Migrator supports target agents of the following distributions:

Connect metastores with the CLI#

Add hive agents#

Add hive agents to connect your source and target metastores.

CommandAction
hive agent add azureAdd a hive agent for an Azure SQL connection
hive agent add filesystemAdd a hive agent for a local filesystem
hive agent add glueAdd a hive agent for an AWS Glue Data Catalog
hive agent add hiveAdd a hive agent for a local or remote Apache Hive metastore
hive agent add databricksAdd a hive agent for a Databricks Delta Lake metastore

Configure existing hive agents#

CommandAction
hive agent configure azureChange the configuration of an existing hive agent for the Azure SQL database server
hive agent configure filesystemChange the configuration of an existing hive agent for the local filesystem
hive agent configure glueChange the configuration of an existing hive agent for the AWS Glue Data Catalog
hive agent configure hiveChange the configuration of an existing hive agent for the Apache Hive metastore
hive agent configure databricksChange the configuration of an existing hive agent for the Databricks Delta Lake metastore

Manage hive agents#

CommandAction
hive agent checkCheck whether the hive agent can connect to the metastore
hive agent deleteDelete a hive agent
hive agent listList all configured hive agents
hive agent showShow the configuration for a hive agent
hive agent typesList supported hive agent types

Next Steps#

Connected to your metastores? Define metadata rules for your metadata migrations.