Migrate your data

Once you have installed LiveData Migrator for Azure and created a target, you're ready to migrate data.

info

Use exclusions to prevent certain file sizes or file/directory names (defined using GLOB patterns) from being migrated to your target storage. You can also exclude files and directories by their last modified date.

Migrate data with the Azure Portal

1. In the Azure Portal, navigate to the LiveData Migrator resource page.
2. From the LiveData Migrator menu on the left, select Migrations.
3. Select Add.
4. Complete the Basic details:
   • Target storage: your previously defined ADLS Gen2 target storage.
   • Path: the directory on your source filesystem to migrate data from.
5. Select Next: Migration settings.
6. Complete the Migration settings:
   • Target Action Policy: forcibly migrate everything, overwriting even existing files of the same file size as the source files ("Overwrite") or only overwrite content that differs in filesize ("Skip if size match").
   • Automatically Start Migration: start the migration right away, or start it manually later.
   • Enable Live Migration: create a live migration, which will continue to migrate data from the source when it receives changes ('Yes'), or a one-time migration, which will only migrate all data once ('No').
   • Exclusions: select the exclusions that you want to include for this migration.
7. Select Review and create.
8. Select Create.

View the migration

1. In the Azure Portal, navigate to the LiveData Migrator resource page.
2. From the LiveData Migrator menu on the left, select Migrations.
3. Select the migration to view it.

Start and stop migrations

1. In the Azure Portal, navigate to the LiveData Migrator resource page.
2. From the LiveData Migrator menu on the left, select Migrations.
3. Select the migration and select Start or Stop as required.

Migrate data with the CLI

Create a migration

The following command will create a migration using the target you created. You can have multiple migrations per migrator. During the creation of a migration you can start a migration right away, or start it manually later.

Defining exclusions (--exclusions) are optional when creating a migration.

Run the following command to create a migration to start right away:

az livedata migrator migration create -g <RESOURCE_GROUP> --migrator-name <MIGRATOR_NAME> --target-name <TARGET_NAME> --migration-name <NEW_MIGRATION_NAME> --path <PATH_TO_MIGRATE> --exclusions <exclusion_name_1>,<exclusion_name_2> --auto-start --location <LOCATION>

Run the following command to create a migration but start it later. Use the start migration command to start the migration.

az livedata migrator migration create -g <RESOURCE_GROUP> --migrator-name <MIGRATOR_NAME> --target-name <TARGET_NAME> --migration-name <NEW_MIGRATION_NAME> --path <PATH_TO_MIGRATE> --exclusions <exclusion_name_1>,<exclusion_name_2> --location <LOCATION>

View a migration

If you want to see how a migration is progressing, you can check its status by running the following command. See the migrationStatus property in the response for the status of the migration.

az livedata migrator migration show -g <Resource Group> --migrator-name <migrator name> --migration-name <migration name>

Start a migration

Run the following command to start a migration that has been created but not started automatically. You can also use this command on a stopped migration.

az livedata migrator migration start --resource-group <resource_group> --migrator-name <migrator_name> --migration-name <migration_name>

Stop a migration

Run the following command to stop a migration.

az livedata migrator migration stop --resource-group <resource_group> --migrator-name <migrator_name> --migration-name <migration_name>

Resume a migration again by starting it.

Add exclusions to an existing migration

Run the following command to add an exclusion or multiple exclusions to an existing migration:

az livedata migrator migration add-exclusions -g <resource_group> --migrator-name <migrator_name> \

--migration-name <migration_name> \

--exclusions <exclusion_name_1>,<exclusion_name_2>

Adding exclusions to an existing migration will apply immediately to the migration, but will not affect previously migrated content.

Remove exclusions from an existing migration

Run the following command to remove an exclusion or multiple exclusions from an existing migration:

az livedata migrator migration remove-exclusions -g resource_group> --migrator-name <migrator_name> \

--migration-name <migration_name> \

--exclusions <exclusion_name_1>,<exclusion_name_2>

Removing exclusions from an existing migration will apply immediately to the migration, but will not affect previously migrated content.

Data migration states

Migrations can be in one of six states:

• SCHEDULED: A scheduled migration is a migration currently starting.
• NONSCHEDULED: A non-scheduled migration is a migration that has been created but isn't going to start automatically. Create a migration in this state by excluding the --auto-start parameter on creation.
• RUNNING: A running migration is scanning through source content and transferring content to the target, as well as responding to change notifications from the source if applicable.
• LIVE: A live migration has completed its initial scan through source content. It continues to respond to change notifications from the source, and will transfer content to and make changes in the target as required.
• COMPLETED: A completed migration is a one-time migration section that has finished transferring all applicable data from the source filesystem to the target, and won't migrate any further data changes.
• PAUSED: A paused migration has been instructed to stop transfer, and is not transferring content or making other changes to the target. This state is applied by stopping migrations.

Create a one-time migration

One-time migrations do not track activity in your data during a migration. Create a one-time migration if you do not want LiveData Migrator for Azure to scan your source filesystem for updates to the data and update the files in your target filesystem as the migration progresses.

A one-time migration does not write a marker file to the source storage. Once the scan of the source completes (to determine which files and directories are to be migrated), the migration will enter a COMPLETED data migration state and won't perform any further scans.

Create a one-time migration with the Azure Portal

During creation of a migration, change the value of the Enable Live Migration option from the default 'Yes' to 'No'.

Your created one-time migration will display the parameter Live Migration: No in the migration overview.

Create a one-time migration with the CLI

Specify the scan-only flag when you create the migration:

az livedata migrator migration create -g <RESOURCE_GROUP> --migrator-name <MIGRATOR_NAME> --target-name <TARGET_NAME>--migration-name <NEW_MIGRATION_NAME> --path <PATH_TO_MIGRATE> --location <LOCATION> --scan-only

Next steps

If you want to migrate your metadata (for example: Hive databases and tables), create a metadata migration target next.

Troubleshooting

If you are having problems migrating data, you can find solutions in the troubleshooting guide.