Azure CLI LiveData extension

This article provides a list of the available commands that the LiveData extension provides for the Azure CLI. You can use the -h option for more info on all commands. For example, az livedata migrator create -h.

caution

All commands require registration of LiveData resource providers.

note

In this preview version, az find is not yet supported.

See the Variables and where to find them article for info on the parameters used in this section.

LiveData Migrator for Azure commands#

LiveData Migrator can be configured and operated from the Azure CLI using commands that begin with the following:

az livedata migrator

Running az livedata migrator -h provides a list of commands for LiveData Migrator for Azure, and adding the -h parameter to any of the sub-commands will provide an explanation of the command and a list of valid arguments. This section details all available commands and their most commonly used arguments.

Migrator#

Create#

Create a migrator resource.

Example
az livedata migrator create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--kerberos \
--keytab /etc/security/keytabs/hdfs.keytab \
--principal hdfs@REALM.COM
Required arguments for migrator create#
  • --location <azure_region>: The Azure region for the migrator. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group to contain the migrator.
  • --migrator-name <migrator_name>: The reference name for the migrator.
Optional arguments for migrator create#
  • --deployment-type <file_type>: The package type for the migrator installation. Choose DEB or RPM depending on the operating system of your host machine. Default is RPM.
  • --install-directory <install_path>: The local filesystem path for the core LiveData Migrator installation. Default is /opt/wandisco/livedata-migrator.
  • --log-directory <log_path>: The local filesystem path for the LiveData Migrator logs. Default is /var/log/wandisco/livedata-migrator.
  • --user -u <username>: The local system user to run the LiveData Migrator service. Default is hdfs.
  • --auto-start: Configure the migration to start immediately upon creation.
  • --scan-only: Specify the migration as a one-time migration.
Kerberos arguments for migrator create#
  • --kerberos: Provide this argument if Kerberos authentication is enabled on your source environment.
    Provide the Keytab and Principal arguments when using this parameter.
    • --keytab <file_path>: The location of the keytab file to be used for Kerberos authentication.
    • --principal <keytab_principal>: The Kerberos principal to be used for Kerberos authentication. This principal must be contained in the specified keytab.

Delete#

Delete a migrator. You must stop all active migrations on a migrator instance before you can delete it.

Example
az livedata migrator delete \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Generate upgrade#

Generate the packaging for the latest version of LiveData Migrator (if a new one is available).

Example
az livedata migrator generate-upgrade \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

List#

Display a list of migrators.

Example
az livedata migrator list
  • --resource-group <resource_group>: (Optional) List the migrators inside the specified resource group.

Configure Kerberos#

Supply or update the Kerberos keytab and principal of an existing migrator instance.

Example
az livedata migrator set-kerberos \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--keytab /etc/security/keytabs/hdfs.keytab \
--principal hdfs@REALM.COM
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.
  • --keytab <file_path>: The location of the keytab file to be used for Kerberos authentication.
  • --principal <keytab_principal>: The Kerberos principal to be used for Kerberos authentication. This principal must be contained in the specified keytab.

Configure Kerberos for metadata#

note

You must configure Kerberos for LiveData Migrator before configuring Kerberos for metadata.

Supply or update the Kerberos keytab and principal for the metadata storage of an existing migrator instance.

Example
az livedata migrator set-meta-kerberos \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--keytab /etc/security/keytabs/hive.service.keytab \
--principal hive/myhostname@REALM.COM
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.
  • --keytab <file_path>: The location of the keytab file to be used for Kerberos authentication.
  • --principal <keytab_principal>: The Kerberos principal to be used for Kerberos authentication. This principal must be contained in the specified keytab.

Reset bandwidth limit#

Reset LiveData Migrator for Azure's bandwidth limit to the default settings, which will use all available bandwidth on the host.

Example
az livedata reset-bandwidth-policy \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Set bandwidth limit#

Set the bandwidth limit for all migrations managed by the migrator (if not set, the default will be all available bandwidth on the host).

Example
az livedata set-bandwidth-policy \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--limit nbunit
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.
  • --limit <bandwidth_limit>: Define the number of byte units per second to restrict the bandwidth to. For example: 10MB.

The following bandwidth limit values are supported:

UnitDescriptionWritten as
B/sBytes per second.B
KB/sKilobytes per second.KB
MB/sMegabytes per second.MB
GB/sGigabytes per second.GB
PB/sPetabytes per second.PB

Add the corresponding 'written as' value to the end of the numerical value you enter. For example: --limit 10MB for a bandwidth limit of ten megabytes per second.

Show#

Display information on a specific migrator.

Example
az livedata migrator show \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Update download location#

Update the download location for a migrator. Use this when the original download URL has expired, and you want to generate a new link.

Example
az livedata migrator update-download-location \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Get install key#

Get the install key for the specified Migrator resource.

Example
az livedata migrator get-install-key \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Wait#

Place the CLI in a waiting state until the migrator reaches a specified state.

Example
az livedata migrator wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator.
  • --migrator-name <migrator_name>: The name of the migrator.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments#
  • --created: Causes the wait period to end when the migrator is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the migrator is deleted.
  • --exists: Causes the wait period to end when the migrator is detected, even if not yet fully deployed as with the Create argument.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the migrator is updated with a provisioningState of "Succeeded".

Migrator target#

Create#

Create a migrator target.

Example with Access Key
az livedata migrator target create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage \
--storage-account-id /subscriptions/mySubscriptionId/resourceGroups/myResourceGroup/providers/Microsoft.Storage/storageAccounts/myStorageAccount \
--storage-container mycontainer \
--storage-access-key HJUhgtnACCESSoUrqlJuil+e6rWO2ku8yhKEYDH4sFMKAHjJg5JkCG8tEXAMPLEXvBwtYfoj5nZaCf5pLo0iuj==
Example with Managed Identity
az livedata migrator target create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage \
--storage-account-id /subscriptions/mySubscriptionId/resourceGroups/myResourceGroup/providers/Microsoft.Storage/storageAccounts/myStorageAccount \
--storage-container mycontainer \
--oauth-endpoint https://login.microsoftonline.com/78u098ex-ampl-e498-8bce-ndpoint5f2e5/oauth2/v2.0/token \
--oauth-id b67f67ex-ampl-e2eb-bd6d-client9385id \
--oauth-secret 2IPO8*secretk-9OPs8n*TexampleHJ=
Required arguments for target create#
  • --location <azure_region>: The Azure region for the migrator deployment. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name for the target storage.
  • --storage-account-id <storage_account>: The ADLS Gen2 storage account ID for the target.
  • --storage-container <storage_container>: The ADLS Gen2 storage container name for the target.
Required arguments if using an access key#
  • --storage-access-key <access_key>: The access key for the ADLS Gen2 storage account.
Required arguments if using a managed identity (service principal)#
note

The service principal that you want to use must have the Storage Blob Data Owner role assigned to the ADLS Gen2 storage account. See the Microsoft documentation for further guidance.

  • --oauth-endpoint <service_principal_endpoint>: The client endpoint for the Azure service principal. This will often take the form of https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token where {tenant} is the directory ID for the Azure service principal.
  • --oauth-id <service_principal_id>: The client ID (also known as application ID) for your Azure service principal.
  • --oauth-secret <service_principal_secret>: The client secret (also known as application secret) for the Azure service principal.

Delete#

Delete a migrator target.

Example
az livedata migrator target delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name of the target storage.

List#

Display a list of migrator targets from the specified migrator deployment.

Example
az livedata migrator target list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show#

Display information on a specific migrator target.

Example
az livedata migrator target show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name of the target storage.

Wait#

Place the CLI in a waiting state until the migrator target reaches a specified state.

Example
az livedata migrator target wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migrator-target-name mytargetstorage \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migrator-target-name <target_storage_name>: The reference name of the target storage.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments#
  • --created: Causes the wait period to end when the migrator target is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the migrator target is deleted.
  • --exists: Causes the wait period to end when the migrator target is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the migrator target is updated with a provisioningState of "Succeeded".

Path mapping#

Create#

Create a path mapping between the source and target filesystems.

Example
az livedata migrator path-mapping create \
--resource-group myresourcegroup \
--location westus \
--migrator-name mymigrator \
--description "Path A to Path B" \
--path-mapping-name mypathmappingatob \
--source-path /my/migration/path/A \
--target-id mytargetstorage \
--target-path /my/migration/path/B
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --location <azure_region>: The Azure region of the migrator deployment. Use the az account list-locations command to get a list of valid location values.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --description <description>: A description of what the path mapping does.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping.
  • --source-path <source_migration_path>: The source filesystem path to migrate from.
  • --target-id <target_filesystem>: The reference name of the target filesystem (value of --migrator-target-name when running az livedata migrator target create).
  • --target-path <target_migration_path>: The target filesystem path to migrate to.

Delete#

Delete a path mapping.

Example
az livedata migrator path-mapping delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--path-mapping-name mypathmappingatob
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping to delete.

List#

Display a list of path mappings from the specified migrator deployment.

Example
az livedata migrator path-mapping list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show#

Display information on a specific path mapping.

Example
az livedata migrator path-mapping show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--path-mapping-name mypathmappingatob
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping to show.

Wait#

Place the CLI in a waiting state until the path mapping reaches a specified state.

Example
az livedata migrator path-mapping wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--path-mapping-name mypathmappingatob \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --path-mapping-name <path_mapping_name>: The reference name for the path mapping.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments#
  • --created: Causes the wait period to end when the path mapping is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the path mapping is deleted.
  • --exists: Causes the wait period to end when the path mapping is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the path mapping is updated with a provisioningState of "Succeeded".

Migration#

Create#

Create a migration.

Example
az livedata migrator migration create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--path /my/migration/path \
--target-name mytargetstorage
Required arguments for migration create#
  • --location <azure_region>: The Azure region of the migrator. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migration-name <migration_name>: The reference name to give to the migration.
  • --path <source_migration_path>: The source filesystem path to migrate from.
  • --target-name <target_filesystem>: The reference name of the target filesystem (value of --migrator-target-name when running az livedata migrator target create).
Optional arguments for migration create#
  • --auto-start: Specify this parameter to start the migration automatically once created.
  • --target-action-policy <action_policy>: The policy to use when replacing files on the target.
    Allowed values: Overwrite, SkipIfSizeMatch
    Default: Overwrite

Delete#

Delete a migration or metadata migration.

Example
az livedata migrator migration delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration.

List#

Display a list of migrations and metadata migrations from the specified migrator deployment.

Example
az livedata migrator migration list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show#

Display information on a specific migration or metadata migration.

Example
az livedata migrator migration show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration.

Start#

Start a migration or metadata migration.

Example
az livedata migrator migration start \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration to start.

Stop#

Stop a migration or metadata migration.

Example
az livedata migrator migration stop \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration to stop.

Wait#

Place the CLI in a waiting state until the migration or metadata migration reaches a specified state.

Example
az livedata migrator migration wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration or metadata migration.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments#
  • --created: Causes the wait period to end when the migration is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the migration is deleted.
  • --exists: Causes the wait period to end when the migration is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the migration is updated with a provisioningState of "Succeeded".

Add exclusions#

Add exclusions to a migration.

Example
az livedata migrator migration add-exclusions \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--exclusions myExclusion1,myExclusion2
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration to add exclusions to.
  • --exclusions <exclusion_template_names>: The exclusion template names to add to the migration (a comma-separated list if more than one).

Remove exclusions#

Remove exclusions from a migration.

Example
az livedata migrator migration remove-exclusions \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--exclusions myExclusion1,myExclusion2
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the migration to add exclusions to.
  • --exclusions <exclusion_template_names>: The exclusion template names to remove from the migration (a comma-separated list if more than one).

Exclusion templates#

Create#

Create an exclusion template.

Example for a date exclusion
az livedata migrator exclusion-template create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name mydateexclusion \
--description "Exclude files that were last modified before April 2021" \
--exclude-older-than 2021-04-01T00:00:00.000Z
Example for a file size exclusion
az livedata migrator exclusion-template create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myfilesizeexclusion \
--description "Exclude files over 10TB in size" \
--exclude-files-over 10TB
Example for a pattern exclusion
az livedata migrator exclusion-template create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name mypatternexclusion \
--description "Exclude files and directories that begin with test" \
--exclude-pattern /**/test*
Required arguments for exclusion-template create#
  • --location <azure_region>: The Azure region for the migrator deployment. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --exclusion-template-name <exclusion_name>: A name for the exclusion.
  • --description <description>: A description of what the exclusion does.
Exclusion type arguments#

Choose one of these arguments depending on what type of exclusion you want to create:

  • --exclude-older-than <iso_date>: An ISO formatted date/time that will exclude all files and directories modified before this date.
  • --exclude-files-over <file_size>: A value and unit for the file size exclusion. Valid units are B for bytes, KB for kilobytes, MB for megabytes, GB for gigabytes, TB for terabytes, and PB for petabytes.
  • --exclude-pattern <glob_pattern>: A glob pattern that will exclude any files and directories that are matched by the pattern.

Delete#

Delete an exclusion template.

Example
az livedata migrator exclusion-template delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myexclusion
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --exclusion-template-name <exclusion_name>: The name of the exclusion to delete.

List#

Display a list of exclusions from the specified migrator deployment.

Example
az livedata migrator exclusion-template list \
--resource-group myresourcegroup \
--migrator-name mymigrator
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.

Show#

Display information on a specific exclusion template.

Example
az livedata migrator exclusion-template show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myexclusion
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --exclusion-template-name <exclusion_name>: The name of the exclusion to display information on.

Wait#

Place the CLI in a waiting state until the exclusion template reaches a specified state.

Example
az livedata migrator exclusion-template wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--exclusion-template-name myexclusion \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --exclusion-template-name <exclusion_name>: The name of the exclusion to display information on.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments#
  • --created: Causes the wait period to end when the exclusion template is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the exclusion template is deleted.
  • --exists: Causes the wait period to end when the exclusion template is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the exclusion template is updated with a provisioningState of "Succeeded".

Metadata migration target#

Create an Azure SQL metadata target#

Create an Azure SQL metadata target resource.

Example
az livedata migrator metadata-target azure-sql create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget \
--sql-server-id myazuresqlserver \
--sql-database mydb \
--sql-user username \
--sql-password password \
--hdi-version 4.0 \
--data-target mytargetresource
Required arguments for migrator metadata-target azure-sql create#
  • --location <azure_region>: The Azure region for the migrator deployment. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment to attach the metadata target to.
  • --metadata-target-name <migration_name>: The reference name to assign to the metadata target.
  • --sql-server-id <sql_server_id>: The resource ID of the SQL server in Microsoft Azure.
  • --sql-database <sql_database_name>: The name of the SQL database to connect to.
  • --sql-user <sql_username>: The username to use when accessing the SQL database.
  • --sql-password <sql_user_password>: The password to use when accessing the SQL database.
  • --hdi-version <version_number>: Sets the HDI version. Valid options are 3.6 and 4.0. Choose the value matching the version of your HDInsight cluster.

Additionally, use only one of the following parameters:

  • --data-target <target_resource>: The name of the filesystem that will be associated with this agent. For example, myazurestorage.
  • --default-fs-override: Enter an override for the default filesystem URI instead of a filesystem name. For example, abfss://mycontainer@mystorageaccount.dfs.core.windows.net.

Create a Databricks metadata target#

Create a Databricks metadata target resource.

Example
az livedata migrator metadata-target databricks-agent create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget \
--jdbc-server-hostname myhostname \
--jdbc-http-path http://myhostname \
--jdbc-port 6060 \
--access-token s8Fjs823JdkeXaMpLeKeYWoSd82WjD23kSd8 \
--fs-mount-point /tmp \`
--data-target mytargetresource
Required arguments for migrator metadata-target databricks-agent create#
  • --access-token: The access token to use when connecting to your Databricks cluster.
  • --jdbc-http-path: The JDBC HTTP path of your Databricks cluster.
  • --jdbc-port: The JDBC port to use when connecting to your Databricks cluster.
  • --jdbc-server-hostname: The JDBC hostname for your Databricks cluster.
  • --metadata-target-name: The name to give the created metadata target resource.
  • --migrator-name: The name of the LiveData Migrator resource to attach this Databricks target resource to.
Optional arguments for migrator metadata-target databricks-agent create#
  • --fs-mount-point: The ADLS location in the Databricks filesystem for containing migrations. For example, /mnt/mybucketname.

Create a Snowflake metadata target#

Create a Snowflake metadata target resource.

Example
az livedata migrator metadata-target snowflake create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget \
--data-target mytargetresource \
--stage mystagename \
--stage-database mystagedatabase \
--stage-schema myschema \
--account-identifier myorganization-myaccount \
--warehouse mywarehouse \
--auth-type Basic \
--user username \
--password password123
Required arguments for migrator metadata-target snowflake create#
  • --location <azure_region>: The Azure region for the migrator deployment. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment to attach the metadata target to.
  • --metadata-target-name <migration_name>: The reference name to assign to the metadata target.
  • --stage <stage_name>: The name of the Snowflake stage you want to store metadata in.
  • --stage-database: The name of the database containing your stage.
  • --stage-schema: The schema to use for data stored by metadata migrations to this target resource.
  • --account-identifier: The identifier for your Snowflake account. This consists of your Snowflake account name and organization name in the format myorganization-myaccount.
  • --warehouse: The name of your Snowflake warehouse.
  • --auth-type: Your choice of basic authentication or private key authentication. This can be either Basic or PrivateKey.
  • --user: The username to connect to the Snowflake platform with.
  • --password: Your password for basic authentication to Snowflake. If you've selected private key authentication, this is the password (if any) for your private key file./
  • --private-key-location: The path on your LiveData Migrator host machine to the file in which you store the private key for connection to Snowflake. For example, /etc/wandisco/my-private-key.pem. This is only necessary for private key authentication.

Additionally, use only one of the following parameters:

  • --data-target <target_resource>: The name of the filesystem that will be associated with this agent (for example: myazurestorage).
  • --default-fs-override: Provide an override for the default filesystem URI instead of a filesystem name (for example: snowflake://myusername:mypassword@myaccountname-myorganization).
note

Don't include @snowflakecomputing.com domain name as part of your account identifier in --default-fs-override. Snowflake automatically appends it.

Delete#

Delete a metadata target resource.

Example
az livedata migrator metadata-target delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget
Required arguments for migrator metadata-target delete#
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment containing the metadata target to be deleted.
  • --metadata-target-name <migration_name>: The reference name of the metadata target to delete.
Optional arguments for migrator metadata-target delete#
  • --no-wait: When supplied, skips the waiting period of the operation before accepting new commands.

List#

List all metadata target resources available in a specified migrator deployment.

Example
az livedata migrator metadata-target list \
--resource-group myresourcegroup \
--migrator-name mymigrator
Required arguments for migrator metadata-target list#
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment containing the metadata targets to be listed.

Show#

Show details about a metadata target resource.

Example
az livedata migrator metadata-target show \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--metadata-target-name mymetadatatarget
Required arguments for migrator metadata-target delete#
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment containing the metadata target to be shown.
  • --metadata-target-name <migration_name>: The reference name of the metadata target to show.

Metadata migration#

Create#

Create a metadata migration.

Example
az livedata migrator metadata-migration create \
--location westus \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymetamigration1 \
--metadata-target mymetadatatarget \
--db-name-pattern "*" \
--table-name-pattern "*"
Required arguments for metadata-migration create#
  • --location <azure_region>: The Azure region for the migrator deployment. Use the az account list-locations command to get a list of valid location values.
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment.
  • --migration-name <migration_name>: The reference name to give to the metadata migration.
  • --metadata-target <metadata-target>: The reference name of the metadata target to migrate metadata to (value of --metadata-target-name when running az livedata migrator metadata-target create).
Optional arguments for metadata-migration create#
  • --db-name-pattern <ddl_db_pattern>: A database query pattern based on Hive DDL. Databases with names following this pattern will be transferred from the source file system. For example: --db-name-pattern test* will match any database with "test" at the beginning of its name, such as test01, test02 etc.
  • --table-name-pattern <ddl_table_pattern>: A database table query pattern based on Hive DDL. Tables in the database with names following this pattern will be transferred from the source file system. For example: --table-name-pattern test* will match any table with "test" at the beginning of its name, such as test01, test02 etc.
  • --auto-start: Specify this parameter to start the metadata migration automatically once created.

Delete#

Delete a metadata migration.

Example
az livedata migrator metadata-migration delete \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymetamigration1 \
Required arguments for metadata-migration create#
  • --resource-group <resource_group>: The name of the resource group that contains the migrator deployment.
  • --migrator-name <migrator_name>: The reference name for the migrator deployment that contains the metadata migration.
  • --migration-name <migration_name>: The metadata migration to remove.

Wait#

Place the CLI in a waiting state until the metadata migration reaches a specified state.

az livedata migrator metadata-migration wait \
--resource-group myresourcegroup \
--migrator-name mymigrator \
--migration-name mymigration1 \
--created
  • --resource-group <resource_group>: The name of the resource group containing the migrator deployment.
  • --migrator-name <migrator_name>: The name of the migrator deployment.
  • --migration-name <migration_name>: The name of the metadata migration.

Where --created may be substituted with any Wait Condition argument.

Wait condition arguments#
  • --created: Causes the wait period to end when the metadata migration is created.
  • --custom <JMESPath_query>: Causes the wait period to end based on a custom JMESPath query.
    Example
    --custom provisioningState!='InProgress',instanceView.statuses[?code=='PowerState/running']
  • --deleted: Causes the wait period to end when the metadata migration is deleted.
  • --exists: Causes the wait period to end when the metadata migration is detected.
  • --interval <duration>: Causes the wait period to end after a specified number of seconds. Default is 30.
  • --timeout <duration>: A maximum wait period, in seconds, for the wait command to endure before timing out if none of the specified conditions are met. Default is 3600 (one hour).
  • --updated: Causes the wait period to end once the metadata migration is updated with a provisioningState of "Succeeded".

Global arguments#

Debug#

Increase logging verbosity to show all debug logs.

--debug

Help#

Get more info on a particular command or subcommand and exit.

--help -h

Only show errors#

Only show errors, suppressing warnings.

--only-show-errors

Output#

Output format. Allowed values (default is json):

  • json
  • jsonc
  • none
  • table
  • tsv
  • yaml
  • yamlc

--output -o myOutputFormat

Query#

JMESPath query string. See http://jmespath.org/ for more information and examples.

--query myQueryType

Subscription#

Name or ID of Azure subscription to use with this command. You can configure the default subscription using az account set -s mySubNameOrID.

--subscription mySubNameOrID

Verbose#

Increase logging verbosity. Use --debug for full debug logging.

--verbose