Skip to main content

LiveData extension commands

This page lists the available commands that the LiveData extension provides for the Azure CLI. You can use the -h option for more info on all commands. For example, az livedata migrator create -h.

caution

All commands require registration of LiveData resource providers.

note

In this preview version, az find is not yet supported.

See the Variables and where to find them article for info on the parameters used in this section.

LiveData Migrator for Azure commands​

LiveData Migrator can be configured and operated from the Azure CLI using commands that begin with the following:

az livedata migrator

Running az livedata migrator -h provides a list of commands for LiveData Migrator for Azure, and adding the -h parameter to any of the sub-commands will provide an explanation of the command and a list of valid arguments. This section details all available commands and their most commonly used arguments.

Migrator​

Create​

Create a migrator resource.

Example
az livedata migrator create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
Required arguments for migrator create​
  • --location <azure_region>: The Azure region for the migrator. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group to contain the migrator.
  • --migrator-name <migrator_name>: The reference name for the migrator.
Optional arguments for migrator create​
  • --no-wait: Do not wait for the long-running operation to finish.
  • --offer <offer_name>: The product name. Default: ldma.
  • --plan <plan_name>: The plan name. Default: metered-v1.

Delete​

Delete a migrator. You must stop all active migrations on a migrator instance before you can delete it.

Example
az livedata migrator delete \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

List​

Display a list of migrators.

Example
az livedata migrator list
  • --resource-group <resource_group>: (Optional) List the migrators inside the specified resource group.

Configure Kerberos​

Enter or update the Kerberos keytab and principal of an existing migrator instance.

Example
az livedata migrator set-kerberos \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--keytab /etc/security/keytabs/hdfs.keytab \
--principal hdfs@REALM.COM
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.
  • --keytab <file_path>: The location of the keytab file to be used for Kerberos authentication.
  • --principal <keytab_principal>: The Kerberos principal to be used for Kerberos authentication. This principal must be contained in the specified keytab.

Configure Kerberos for metadata​

note

You must configure Kerberos for LiveData Migrator before configuring Kerberos for metadata.

Enter or update the Kerberos keytab and principal for the metadata storage of an existing migrator instance.

Example
az livedata migrator set-meta-kerberos \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--keytab /etc/security/keytabs/hive.keytab \
--principal hive/myhostname@REALM.COM
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.
  • --keytab <file_path>: The location of the keytab file to be used for Kerberos authentication.
  • --principal <keytab_principal>: The Kerberos principal to be used for Kerberos authentication. This principal must be contained in the specified keytab.

Reset bandwidth limit​

Reset LiveData Migrator for Azure's bandwidth limit to the default settings, which will use all available bandwidth on the host.

Example
az livedata reset-bandwidth-policy \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Set bandwidth limit​

Set the bandwidth limit for all migrations managed by the migrator (if not set, the default will be all available bandwidth on the host).

note

Bandwidth limits only apply if LiveData Migrator for Azure handles the transfer of data from source. Data transfer agents don't apply bandwidth limits.

Example
az livedata set-bandwidth-policy \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--limit nbunit
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.
  • --limit <bandwidth_limit>: Define the number of byte units per second to restrict the bandwidth to. For example: 10MB.

The following bandwidth limit values are supported:

UnitDescriptionWritten as
B/sBytes per second.B
KB/sKilobytes per second.KB
MB/sMegabytes per second.MB
GB/sGigabytes per second.GB
PB/sPetabytes per second.PB

Add the corresponding 'written as' value to the end of the numerical value you enter. For example: --limit 10MB for a bandwidth limit of ten megabytes per second.

Show​

Display information on a specific migrator.

Example
az livedata migrator show \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Get install key​

Get the install key for the specified Migrator resource.

Example
az livedata migrator get-install-key \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Wait​

Place the CLI in a waiting state until the migrator reaches a specified state.

Example
az livedata migrator wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments​
  • --created: Causes the wait period to end when the migrator is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the migrator is deleted.
  • --exists: Causes the wait period to end when the migrator is detected, even if not yet fully deployed as with the Create argument.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the migrator is updated with a provisioningState of "Succeeded".

Set metastore agent mode​

Set agent preferred operation mode (SCANNING or LISTENING) for the source metastore

Example
az livedata migrator set-meta-agent-mode \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--meta-agent-mode mode
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.
  • --meta-agent-mode <mode>: Specify the agent operation mode either SCANNING or LISTENING.

Migrator target​

Create​

Create a migrator target.

Example with Access Key
az livedata migrator target create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage \
--storage-account  mystorageaccountname\
--storage-container mycontainer \
--storage-access-key HJUhgtnACCESSoUrqlJuil+e6rWO2ku8yhKEYDH4sFMKAHjJg5JkCG8tEXAMPLEXvBwtYfoj5nZaCf5pLo0iuj==
Example with Managed Identity
az livedata migrator target create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage \
--storage-account  mystorageaccountname\
--storage-container mycontainer \
--oauth-endpoint https://login.microsoftonline.com/78u098ex-ampl-e498-8bce-ndpoint5f2e5/oauth2/v2.0/token \
--oauth-id b67f67ex-ampl-e2eb-bd6d-client9385id \
--oauth-secret 2IPO8*secretk-9OPs8n*TexampleHJ=
Required arguments for target create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name for the target storage.
  • --storage-container <storage_container>: The ADLS Gen2 storage container name for the target.
  • --storage-account <storage_account_name>: The name of the ADLS Gen2 storage account for the target.
Required arguments if using an access key​
  • --storage-access-key <access_key>: The access key for the ADLS Gen2 storage account.
Required arguments if using a managed identity (service principal)​
note

The service principal that you want to use must have the Storage Blob Data Owner role assigned to the ADLS Gen2 storage account. See the Microsoft documentation for further guidance.

  • --oauth-endpoint <service_principal_endpoint>: The client endpoint for the Azure service principal. This will often take the form of https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token where {tenant} is the directory ID for the Azure service principal.
  • --oauth-id <service_principal_id>: The client ID (also known as application ID) for your Azure service principal.
  • --oauth-secret <service_principal_secret>: The client secret (also known as application secret) for the Azure service principal.

Delete​

Delete a migrator target.

Example
az livedata migrator target delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name of the target storage.

List​

Display a list of migrator targets from the specified migrator deployment.

Example
az livedata migrator target list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show​

Display information on a specific migrator target.

Example
az livedata migrator target show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name of the target storage.

Wait​

Place the CLI in a waiting state until the migrator target reaches a specified state.

Example
az livedata migrator target wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name of the target storage.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments​
  • --created: Causes the wait period to end when the migrator target is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the migrator target is deleted.
  • --exists: Causes the wait period to end when the migrator target is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the migrator target is updated with a provisioningState of "Succeeded".

Update authentication​

note

If you update your authentication settings while your target is part of an active migration, the migration may experience failed paths. Delete and recreate the migration to ensure data consistency.

Update the storage access key, service principal client ID, or service principal client secret for your migrator target.

Update storage access key
az livedata migrator target update-authentication -resource-group myResourceGroup --migrator-name myMigrator --migrator-target-name myMigratorTarget --storage-access-key storageAccessKey
Update service principal client ID
az livedata migrator target update-authentication -resource-group myResourceGroup --migrator-name myMigrator --migrator-target-name myMigratorTarget --oauth-id servicePrincipalClientID
Update service principal client secret
az livedata migrator target update-authentication -resource-group myResourceGroup --migrator-name myMigrator --migrator-target-name myMigratorTarget --oauth-secret servicePrincipalClientSecret
Required arguments for target update-authentication​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name for the target storage.
Optional arguments for target update-authentication​
  • --storage-access-key <access_key>: The access key for the ADLS Gen2 storage account.
  • --oauth-id <service_principal_id>: The client ID (also known as application ID) for your Azure service principal.
  • --oauth-secret <service_principal_secret>: The client secret (also known as application secret) for your Azure service principal.
info

After successfully updating your authentication settings, restart the LiveData Migrator service. When the restart completes, check your data migrations and manually start any in a Paused state.

Path mapping​

Create​

Create a path mapping between the source and target filesystems.

Example
az livedata migrator path-mapping create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--description "Path A to Path B" \
--path-mapping-name mypathmappingatob \
--source-path /my/migration/path/A \
--target-id mytargetstorage \
--target-path /my/migration/path/B
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --description <description>: A description of what the path mapping does.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping.
  • --source-path <source_migration_path>: The source filesystem path to migrate from.
  • --target-id <target_filesystem>: The reference name of the target filesystem (value of --migrator-target-name when running az livedata migrator target create).
  • --target-path <target_migration_path>: The target filesystem path to migrate to.

Delete​

Delete a path mapping.

Example
az livedata migrator path-mapping delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--path-mapping-name mypathmappingatob
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping to delete.

List​

Display a list of path mappings from the specified migrator deployment.

Example
az livedata migrator path-mapping list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show​

Display information on a specific path mapping.

Example
az livedata migrator path-mapping show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--path-mapping-name mypathmappingatob
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping to show.

Wait​

Place the CLI in a waiting state until the path mapping reaches a specified state.

Example
az livedata migrator path-mapping wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--path-mapping-name mypathmappingatob \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments​
  • --created: Causes the wait period to end when the path mapping is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the path mapping is deleted.
  • --exists: Causes the wait period to end when the path mapping is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the path mapping is updated with a provisioningState of "Succeeded".

Migration​

Create​

Create a data migration.

Example
az livedata migrator migration create \
--resource-group myresourcegroup \
--auto-start \
--migrator-name mymigrator \
--migration-name mymigration1 \
--path /my/migration/path \
--recurring-migration \
--recurring-period \
--resource-group myresourcegroup \
--scan-only \
--target-action-policy \
--target-name mytargetstorage
--priority priority_level \
Required arguments for migration create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migration-name <migration_name>: The reference name to give to the migration.
  • --path <source_migration_path>: The source filesystem path to migrate from.
  • --target-name <target_filesystem>: The reference name of the target filesystem (value of --migrator-target-name when running az livedata migrator target create).
Optional arguments for migration create​
  • --auto-start: Specify this parameter to start the migration automatically once created.
  • --recurring-migration: Add this flag to enable periodic rescanning of the migration.
  • --recurring-period: Use with --recurring-migration to specify the time period between scheduling of recurring migration scan iterations. Use the one of the following time units H(hours) or M(minutes).
  • --scan-only: Use this flag to create a one-time migration or with --recurring-migration and --recurring-period to create a recurring migration.
  • --target-action-policy <action_policy>: The policy to use when replacing files on the target.
    There are two options available:
    1. Overwrite (default policy)
      Every file is replaced, even if the file size is identical on the target storage. This option is incompatible with the --recurring-migration option. Use the SkipIfSizeMatchActionPolicy parameter instead.
    2. SkipIfSizeMatch
      If the file size is identical between the source and target, the file is skipped. If it’s a different size, the whole file is replaced.
  • --priority: Set the priority of this migration, with a value of either HIGH, NORMAL or LOW.

Delete​

Delete a data migration.

Example
az livedata migrator migration delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration.

List​

Display a list of data migrations from the specified migrator deployment.

Example
az livedata migrator migration list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Reset​

Reset a data migration.

Example
az livedata migrator migration reset \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--reset-path-mapping \
--target-action-policy policyType \
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration to add exclusions to.
Optional arguments for migration reset​
  • --reset-path-mapping: Reload the latest path mappings when resetting the migration.
  • --target-action-policy <policyType>: The policy to use when replacing files on the target.
    Allowed values: Overwrite, SkipIfSizeMatch
    Default: Overwrite

Show​

Display information on a specific data migration.

Example
az livedata migrator migration show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration.

Start​

Start a data migration.

Example
az livedata migrator migration start \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration to start.

Stop​

Stop a data migration.

Example
az livedata migrator migration stop \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration to stop.

Wait​

Place the CLI in a waiting state until the data migration reaches a specified state.

Example
az livedata migrator migration wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments​
  • --created: Causes the wait period to end when the migration is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the migration is deleted.
  • --exists: Causes the wait period to end when the migration is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the migration is updated with a provisioningState of "Succeeded".

Add exclusions​

Add exclusions to a migration.

Example
az livedata migrator migration add-exclusions \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--exclusions myExclusion1,myExclusion2
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration to add exclusions to.
  • --exclusions <exclusion_template_names>: The exclusion template names to add to the migration (a comma-separated list if more than one).

Remove exclusions​

Remove exclusions from a migration.

Example
az livedata migrator migration remove-exclusions \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--exclusions myExclusion1,myExclusion2
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration to add exclusions to.
  • --exclusions <exclusion_template_names>: The exclusion template names to remove from the migration (a comma-separated list if more than one).

Set priority​

Set the priority of an existing migration.

Example
az livedata migrator migration set-priority \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--priority HIGH \
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration to add exclusions to.
  • --priority <priority_level>: Set the priority level or either HIGH, NORMAL or LOW for this migration.

Exclusion templates​

Create​

Create an exclusion template.

Example for a date exclusion
az livedata migrator exclusion-template create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name mydateexclusion \
--description "Exclude files that were last modified before April 2021" \
--exclude-older-than 2021-04-01T00:00:00.000Z
Example for a file size exclusion
az livedata migrator exclusion-template create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myfilesizeexclusion \
--description "Exclude files over 10TB in size" \
--exclude-files-over 10TB
Example for a pattern exclusion
az livedata migrator exclusion-template create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name mypatternexclusion \
--description "Exclude files and directories that begin with test" \
--exclude-pattern /**/test*
Required arguments for exclusion-template create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --exclusion-template-name <exclusion_name>: A name for the exclusion.
  • --description <description>: A description of what the exclusion does.
Exclusion type arguments​

Choose one of these arguments depending on what type of exclusion you want to create:

  • --exclude-older-than <iso_date>: An ISO formatted date/time that will exclude all files and directories modified before this date.
  • --exclude-files-over <file_size>: A value and unit for the file size exclusion. Valid units are B for bytes, KB for kilobytes, MB for megabytes, GB for gigabytes, TB for terabytes, and PB for petabytes.
  • --exclude-pattern <glob_pattern>: A glob pattern that will exclude any files and directories that are matched by the pattern.

Delete​

Delete an exclusion template.

Example
az livedata migrator exclusion-template delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myexclusion
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --exclusion-template-name <exclusion_name>: The name of the exclusion to delete.

List​

Display a list of exclusions from the specified migrator deployment.

Example
az livedata migrator exclusion-template list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show​

Display information on a specific exclusion template.

Example
az livedata migrator exclusion-template show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myexclusion
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --exclusion-template-name <exclusion_name>: The name of the exclusion to display information on.

Wait​

Place the CLI in a waiting state until the exclusion template reaches a specified state.

Example
az livedata migrator exclusion-template wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myexclusion \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --exclusion-template-name <exclusion_name>: The name of the exclusion to display information on.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments​
  • --created: Causes the wait period to end when the exclusion template is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the exclusion template is deleted.
  • --exists: Causes the wait period to end when the exclusion template is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the exclusion template is updated with a provisioningState of "Succeeded".

Verification​

Create​

Create a data verification.

Example
az livedata migrator verification create \
--resource-group myresourcegroup \
--migrator-name migratorName \
--migration-name migration1 \
--verification-name verification_name \ 
--verification-depth depth \
--original-paths path \
--ignore-after-timestamp date \
Required arguments for verification create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migration-name <migration_name>: The reference name to give to the migration.
  • --verification-name <verification_name>: The name you want to give this verification.
  • --verification-depth <depth>: Integer value for how deep in the directory you want to run the verification check. '0' means there's no limit to the verification depth.
  • --original-paths <path>: The path or paths from your source or target filesystem you want to verify using the format /path/to
  • --ignore-after-timestamp <date>: Date and time as a verification cutoff point. Checks files modified before this cutoff. Use the format 2023-01-09T10:05:31.000Z.
Example​

See the example below showing a migration verification create with the Azure CLI. 

az livedata migrator verification create -g resourceGroupName --migrator-name migratorName --migration-name migration1 --verification-name verification --verification-depth 0 --original-paths "/repl1" --ignore-after-timestamp 2023-01-09T10:05:31.000Z

List​

Returns list of Migration Verifications

Example
az livedata migrator verification list \
--resource-group myresourcegroup \
--migrator-name migratorName \
Required arguments for verification list​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
Example​

See the example below showing a verification list with the Azure CLI.

az livedata migrator verification list -g resourceGroupName --migrator-name migratorName

Show​

Show a data verification.

Example
az livedata migrator verification show \
--resource-group myresourcegroup \
--migrator-name migratorName \
--verification-name verification_name \

Required arguments for verification show​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --verification-name <verification_name>: The name of the verification to show.
Example​

Migration verification show with the Azure CLI example.

az livedata migrator verification show -g resourceGroupName --migrator-name migratorName --verification-name verification

Delete​

Delete a migration verification.

Example
az livedata migrator verification delete \
--resource-group myresourcegroup \
--migrator-name migratorName \
--verification-name verification_name \
Required arguments for verification delete​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --verification-name <verification_name>: The name of the verification to delete.
Example​

See the example below showing a migration verification delete with the Azure CLI.

az livedata migrator verification delete -g resourceGroupName --migrator-name migratorName --verification-name verification

Cancel​

Cancel an existing migration verification.

Example
az livedata migrator verification cancel \
--resource-group myresourcegroup \
--migrator-name migratorName \
--verification-name verification_name \
Required arguments for verification cancel​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --verification-name <verification_name>: The name of verification to cancel.
Example​

See the example below showing a migration verification cancel with the Azure CLI.

az livedata migrator verification cancel -g resourceGroupName --migrator-name migratorName --verification-name verification

Wait​

Place the CLI in a waiting state until a condition of the verification is met.

Example
az livedata migrator verification wait \
--resource-group myresourcegroup \
--migrator-name migratorName \
--verification-name verification_name \ 
--<condition> \

Where --<condition> may be substituted with any Wait condition argument.

Wait condition arguments​
  • --created: Wait until created with 'provisioningState' at 'Succeeded'.
  • --custom <JMESPath_query>: Wait until the condition satisfies a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress'
  • --deleted: Wait until deleted.
  • --exists: Wait until the resource exists.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: Maximum wait in seconds. Default: 3600.
  • --updated: Wait until updated with provisioningState at 'Succeeded'.
Required arguments for verification wait​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --verification-name <verification_name>: The name you want to give this verification.
  • --<condition>: The condition that must be met.
Example​

See the example below showing a migration verification wait with the Azure CLI.

az livedata migrator verification wait -g resourceGroupName --migrator-name migratorName --verification-name verification --created
az livedata migrator verification wait -g resourceGroupName --migrator-name migratorName --verification-name verification --custom provisioningState!='InProgress'

Metadata migration target​

Create an Azure SQL metadata target​

Create an Azure SQL metadata target resource.

Example
az livedata migrator metadata-target azure-sql create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget \
--sql-server-id myazuresqlserver \
--sql-database mydb \
--sql-user username \
--sql-password password \
--hdi-version 4.0 \
--data-target mytargetresource
Required arguments for migrator metadata-target azure-sql create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment to attach the metadata target to.
  • --metadata-target-name <migration_name>: The reference name to assign to the metadata target.
  • --sql-server-id <sql_server_id>: The resource ID of the SQL server in Microsoft Azure.
  • --sql-database <sql_database_name>: The name of the SQL database to connect to.
  • --sql-user <sql_username>: The username to use when accessing the SQL database.
  • --sql-password <sql_user_password>: The password to use when accessing the SQL database.
  • --hdi-version <version_number>: Sets the HDI version. The only supported version is currently 4.0.

Additionally, use only one of the following parameters:

  • --data-target <target_resource>: The name of the filesystem that will be associated with this agent. For example, myazurestorage.
  • --default-fs-override: Enter an override for the default filesystem URI instead of a filesystem name. For example, abfss://mycontainer@mystorageaccount.dfs.core.windows.net.

Create a Databricks metadata target​

Create a Databricks metadata target resource.

Example
az livedata migrator metadata-target databricks-agent create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget \
--jdbc-server-hostname myhostname \
--jdbc-http-path http://myhostname \
--jdbc-port 6060 \
--access-token s8Fjs823JdkeXaMpLeKeYWoSd82WjD23kSd8 \
--fs-mount-point /tmp \
--catalog myUnityCatalog \
--data-target mytargetresource \
--default-fs-override dbfs:/tmp
Required arguments for migrator metadata-target databricks-agent create​
  • --access-token: The access token to use when connecting to your Databricks cluster.
  • --jdbc-http-path: The JDBC HTTP path of your Databricks cluster.
  • --jdbc-port: The JDBC port to use when connecting to your Databricks cluster.
  • --jdbc-server-hostname: The JDBC hostname for your Databricks cluster.
  • --metadata-target-name: The name to give the created metadata target resource.
  • --migrator-name: The name of the LiveData Migrator resource to attach this Databricks target resource to.
  • --fs-mount-point: The ADLS location in your Databricks cluster where you've mounted your cloud filesystem. For example, /mnt/mybucketname.
  • --default-fs-override: Enter an override for the default filesystem URI in the format dfbs:<mount_point> instead of a filesystem name. If --convert-to-delta-lake is not enabled, this must be set to dbfs:<your_fs_mount_point>, even if the mount point is blank. If --convert-to-delta-lake is enabled, set this to the path where your Delta Lake tables are stored. For example, dbfs:<path/to/delta/tables>.
Optional arguments for migrator metadata-target databricks-agent create​
  • --convert-to-delta-lake: When you add this parameter, metadata sent to the target will be converted to the Delta Lake format.
  • --delete-after-convert: When you add this parameter, the original metadata on the target will be deleted after it's converted to the Delta Lake format. Requires --convert-to-delta-lake.
  • --catalog Enter the name of your Databricks Unity Catalog.

Update authentication for an Azure SQL metadata target​

note

Before updating authentication settings, stop any migrations to the metadata target. After you've updated your credentials, restart the migrations.

Update the authentication settings for your Azure SQL metadata target.

Update authentication settings for Azure SQL metadata target
az livedata migrator metadata-target azure-sql update-authentication -resource-group myResourceGroup
        --migrator-name myMigrator --metadata-target-name myMetadataTarget \
        --sql-user myUser \
        --sql-password letmein
Required arguments for metadata-target azure-sql update-authentication​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The name identifier of the migrator resource.
  • --metadata-target-name <migration_name>: The reference name of the metadata target.
  • --sql-user <sql_username>: The username to use when accessing the SQL database.
  • --sql-password <sql_user_password>: The password to use when accessing the SQL database.
info

If you're unable to restart your metadata migrations after successfully updating your authentication settings, restart the LiveData Migrator and Hive Migrator services.

Create a Snowflake metadata target​

Create a Snowflake metadata target resource.

Example
az livedata migrator metadata-target snowflake create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget \
--data-target mytargetresource \
--stage mystagename \
--stage-database mystagedatabase \
--stage-schema myschema \
--account-identifier myorganization-myaccount \
--warehouse mywarehouse \
--auth-type Basic \
--user username \
--password password123
Required arguments for migrator metadata-target snowflake create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment to attach the metadata target to.
  • --metadata-target-name <migration_name>: The reference name to assign to the metadata target.
  • --stage <stage_name>: The name of the Snowflake stage you want to store metadata in.
  • --stage-database: The name of the database containing your stage.
  • --stage-schema: The schema to use for data stored by metadata migrations to this target resource.
  • --account-identifier: The identifier for your Snowflake account. This consists of your Snowflake account name and organization name in the format myorganization-myaccount.
  • --warehouse: The name of your Snowflake warehouse.
  • --auth-type: Your choice of basic authentication or private key authentication. This can be either Basic or PrivateKey.
  • --user: The username to connect to the Snowflake platform with.
  • --password: Your password for basic authentication to Snowflake. If you've selected private key authentication, this is the password (if any) for your private key file./
  • --private-key-location: The path on your LiveData Migrator host machine to the file in which you store the private key for connection to Snowflake. For example, /etc/wandisco/my-private-key.pem. This is only necessary for private key authentication.

Additionally, use only one of the following parameters:

  • --data-target <target_resource>: The name of the filesystem that will be associated with this agent (for example: myazurestorage).
  • --default-fs-override: Enter an override for the default filesystem URI instead of a filesystem name (for example: snowflake://myusername:mypassword@myaccountname-myorganization).
note

Don't include @snowflakecomputing.com domain name as part of your account identifier in --default-fs-override. Snowflake automatically appends it.

Delete​

Delete a metadata target resource.

Example
az livedata migrator metadata-target delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget
Required arguments for migrator metadata-target delete​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment containing the metadata target to be deleted.
  • --metadata-target-name <migration_name>: The reference name of the metadata target to delete.
Optional arguments for migrator metadata-target delete​
  • --no-wait: When supplied, skips the waiting period of the operation before accepting new commands.

List​

List all metadata target resources available in a specified migrator deployment.

Example
az livedata migrator metadata-target list \
--resource-group myresourcegroup \
--migrator-name mymigrator
Required arguments for migrator metadata-target list​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment containing the metadata targets to be listed.

Show​

Show details about a metadata target resource.

Example
az livedata migrator metadata-target show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget
Required arguments for migrator metadata-target delete​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment containing the metadata target to be shown.
  • --metadata-target-name <migration_name>: The reference name of the metadata target to show.

Metadata migration​

Create​

Create a metadata migration.

Example
az livedata migrator metadata-migration create \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymetamigration1 \
--metadata-target mymetadatatarget \
--db-name-pattern "*" \
--table-name-pattern "*"
Required arguments for metadata-migration create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migration-name <migration_name>: The reference name to give to the metadata migration.
  • --metadata-target <metadata-target>: The reference name of the metadata target to migrate metadata to (value of --metadata-target-name when running az livedata migrator metadata-target create).
Optional arguments for metadata-migration create​
  • --db-name-pattern <ddl_db_pattern>: A database query pattern based on Hive DDL. Databases with names following this pattern will be transferred from the source filesystem. For example: --db-name-pattern test* will match any database with "test" at the beginning of its name, such as test01 and test02.
  • --table-name-pattern <ddl_table_pattern>: A database table query pattern based on Hive DDL. Tables in the database with names following this pattern will be transferred from the source filesystem. For example: --table-name-pattern test* will match any table with "test" at the beginning of its name, such as test01 and test02.
  • --auto-start: Specify this parameter to start the metadata migration automatically once created.
note

You can use asterisk values enclosed in quotation marks for --db-name-pattern and --table-name-pattern to migrate all databases or tables. For example, --table-name-pattern "*".

Delete​

Delete a metadata migration.

Example
az livedata migrator metadata-migration delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymetamigration1 \
Required arguments for metadata-migration create​
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment that contains the metadata migration.
  • --migration-name <migration_name>: The metadata migration to remove.

Show​

Show a metadata migration.

Example
az livedata metadata-migrator show \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Start​

Start a metadata migration.

Example
az livedata migrator metadata-migration start \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the metadata migration to start.

Stop​

Stop a metadata migration.

Example
az livedata migrator metadata-migration stop \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the metadata migration to stop.

Wait​

Place the CLI in a waiting state until the metadata migration reaches a specified state.

az livedata migrator metadata-migration wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the metadata migration.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments​
  • --created: Causes the wait period to end when the metadata migration is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the metadata migration is deleted.
  • --exists: Causes the wait period to end when the metadata migration is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the metadata migration is updated with a provisioningState of "Succeeded".

Data transfer agent​

Create a Data Transfer Agent.

Create​

az livedata migrator data-transfer-agent create \
--resource-group myresourcegroup \
--migrator-name migrator \
--data-transfer-agent-name myDta \
--host myhost.com \
--token YmVnaW4gbGFyZ2UgYW1vdW5 \
--limit 2Gb

Required arguments for migrator data-transfer-agent create​

  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --data-transfer-agent-name <dta_name>: The name of the data transfer agent.

Optional arguments for migrator data-transfer-agent create​

  • --host <hostname>: The transfer agent hostname
  • --token <token>: Access token for data transfer agent. Use local path to a token file with --token @./mytoken.txt or token value.
  • --limit <bandwidth_limit>: The application bandwidth limit to apply to the data transfer agent.

Delete​

Delete data transfer agent.

az livedata migrator data-transfer-agent delete \
--resource-group myresourcegroup \
--migrator-name migrator \
--data-transfer-agent-name myDta \

Required arguments for migrator data-transfer-agent delete​

  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --data-transfer-agent-name <dta_name>: The name of the data transfer agent.

List​

Returns a list of data transfer agents for the migrator specified.

az livedata migrator data-transfer-agent list \
--resource-group myresourcegroup \
--migrator-name migrator \

Required arguments for migrator data-transfer-agent list​

  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show​

Check the status of a data transfer agent.

az livedata migrator data-transfer-agent show \
--resource-group myresourcegroup \
--migrator-name migrator \
--data-transfer-agent-name myDta \

Required arguments for migrator data-transfer-agent show​

  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --data-transfer-agent-name <dta_name>: The name of the data transfer agent.

Set bandwidth policy​

Apply a bandwidth limit to a data transfer agent.

az livedata migrator data-transfer-agent set-bandwidth-policy \
--resource-group myresourcegroup \
--migrator-name migrator \
--data-transfer-agent-name myDta \
--limit 2GB

Required arguments for migrator data-transfer-agent set-bandwidth-policy​

  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --data-transfer-agent-name <dta_name>: The name of the data transfer agent.
  • --limit <bandwidth_limit>: The application bandwidth limit to apply to the data transfer agent.

Reset bandwidth policy​

Reset a bandwidth limit to a data transfer agent.

az livedata migrator data-transfer-agent reset-bandwidth-policy \
--resource-group myresourcegroup \
--migrator-name migrator \
--data-transfer-agent-name myDta \

Required arguments for migrator data-transfer-agent reset-bandwidth-policy​

  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --data-transfer-agent-name <dta_name>: The name of the data transfer agent.

Wait​

Place the CLI in a waiting state until a condition of the data transfer agent is met.

az livedata migrator data-transfer-agent wait \
--resource-group MyResourceGroup \
--migrator-name migrator \
--data-transfer-agent-name myAgentName \
--created

Required arguments for migrator data-transfer-agent wait​

  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --data-transfer-agent-name <dta_name>: The name of the data transfer agent.

Conditional arguments for migrator data-transfer-agent wait​

  • --created: The name of the data transfer agent.
  • --deleted: Wait until deleted..
  • --exists: Wait until the resource exists.
  • --interval <seconds>: Polling interval in seconds. Default: 30
  • --timeout <seconds>: Maximum wait in seconds. Default: 3600.
  • --updated: Wait until updated.

Global arguments​

Debug​

Increase logging verbosity to show all debug logs.

--debug

Help​

Get more info on a particular command or subcommand and exit.

--help -h

Only show errors​

Only show errors, suppressing warnings.

--only-show-errors

Output​

Output format. Allowed values (default is json):

  • json
  • jsonc
  • none
  • table
  • tsv
  • yaml
  • yamlc

--output -o myOutputFormat

Query​

JMESPath query string. See http://jmespath.org/ for more information and examples.

--query myQueryType

Subscription​

Name or ID of Azure subscription to use with this command. You can configure the default subscription using az account set -s mySubNameOrID.

--subscription mySubNameOrID

Verbose​

Increase logging verbosity. Use --debug for full debug logging.

--verbose