Skip to main content

Migrate your data

Once you have installed LiveData Migrator for Azure and created a target, you're ready to migrate data.

info

Use exclusions to prevent certain file sizes or file/directory names (defined using GLOB patterns) from being migrated to your target storage. You can also exclude files and directories by their last modified date.

Create a migration​

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Migrations.
  3. Select Add.
  4. Complete the details under the Basics section:
    • Migration Name: enter a reference name to use for the migration.
    • Target storage: your previously defined ADLS Gen2 target storage.
    • Path: the directory on your source filesystem to migrate data from.
  5. Select Next: Migration settings.
  6. Complete the Migration settings:
    • Target Action Policy: forcibly migrate everything, overwriting even existing files of the same file size as the source files ("Overwrite") or only overwrite content that differs in filesize ("Skip if size match").
    • Automatically Start Migration: start the migration right away, or start it manually later.
    • Migration Type: create a live migration, which will continue to migrate data from the source when it receives changes ('Live'), or a Recurring migration, which will continuously scan for changing data ('Recurring'), or a one-time migration, which will migrate all data once ('NonLive').
    • Exclusions: select the exclusions that you want to include for this migration.
    • Migration Priority: Assign a priority of High, Normal, or Low. The default priority is Normal.
  7. Select Review and create.
  8. Select Create.

View the migration​

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Migrations.
  3. Select the migration to view it.

Start and stop migrations​

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Migrations.
  3. Select the migration and select Start or Stop as required.

Reset a migration​

Reset the migration to re-scan the source filesystem. Stop the migration and ensure it is in a PAUSED state before you perform a reset. You can't reset a recurring migration.

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Migrations.
  3. Select the migration and select Reset.
  4. Optionally, adjust the details under the Live Data Migration Reset section:
    • Target Action Policy: forcibly migrate everything, overwriting even existing files of the same file size as the source files ("Overwrite") or only overwrite content that differs in filesize ("Skip if size match").
    • Reset Path Mappings: select to reload the latest path mappings when resetting the migration.
  5. Select Submit to reset to this migration.
note

After a migration reset, the migration will be in a NONSCHEDULED state. Start the migration to resume. See Data migration states for more info on states.

caution

If a migration is deleted, and re-created with the same name, the Azure portal will identify this migration as the original migration, historic metrics of the original migration are saved and still visible.

Data migration states​

Migrations can be in one of the following states:

  • SCHEDULED: A scheduled migration is a migration currently starting.
  • NONSCHEDULED: A non-scheduled migration is a migration that has been created but isn't going to start automatically. Create a migration in this state by excluding the --auto-start parameter on creation.
  • RUNNING: A running migration is scanning through source content and transferring content to the target, as well as responding to change notifications from the source if applicable.
  • LIVE: A live migration has completed its initial scan through source content. It continues to respond to change notifications from the source, and will transfer content to and make changes in the target as required.
  • COMPLETED: A completed migration is a one-time migration section that has finished transferring all applicable data from the source filesystem to the target, and won't migrate any further data changes.
  • PAUSED: A paused migration has been instructed to stop transfer, and is not transferring content or making other changes to the target. This state is applied by stopping migrations.
  • RECURRENCE_SCHEDULED: A recurring migration is scheduled to be run.
  • WAITING_FOR_NEXT_RECURRENCE: A recurring migration has completed a scan and is now waiting for its next recurrance.

Create a one-time migration​

One-time(Nonlive) migrations don't track activity in your data during a migration. Create a one-time migration if you don't want LiveData Migrator for Azure to scan your source filesystem for updates to the data and update the files in your target filesystem as the migration progresses.

A one-time(Nonlive) migration does not write a marker file to the source storage. Once the scan of the source completes (to determine which files and directories are to be migrated), the migration will enter a COMPLETED data migration state and won't perform any further scans.

You can only create one-time(Nonlive) data migrations. You can't create one-time metadata migrations.

Create a one-time migration with the Azure Portal​

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Migrations.
  3. Select Add.
  4. Complete the details under the Basics section:
    • Migration Name: enter a reference name to use for the migration.
    • Target storage: your previously defined ADLS Gen2 target storage.
    • Path: the directory on your source filesystem to migrate data from.
  5. Select Next: Migration settings.
  6. Complete the Migration settings:
    • Target Action Policy: forcibly migrate everything, overwriting even existing files of the same file size as the source files ("Overwrite") or only overwrite content that differs in filesize ("Skip if size match").
    • Automatically Start Migration: start the migration right away, or start it manually later.
    • Migration Type: Select NonLive.
    • Exclusions: select the exclusions that you want to include for this migration.
    • Migration Priority: Assign a priority of High, Normal, or Low. The default priority is Normal.
  7. Select Review and create.
  8. Select Create.

Your created one-time migration will display the parameter Migration Type: One-time in the migration overview.

Create a one-time migration with the CLI​

Specify the scan-only flag when you create the migration:

az livedata migrator migration create  -g <resource_group> --migrator-name <migrator_name> --target-name <target_name> --migration-name <new_migration_name> --path <path_to_migrate> --scan-only

Create a recurring migration​

Recurring migrations let you continuously scan for changing data if your source doesn't support or isn’t configured to support notification-based live changes. Existing data on the source will be replicated to the target, then the migration scan will be repeated to discover new changes.

tip

Dangling files.

Recurring migrations repeatedly scan source and copy to target, files subsequently deleted from source will therefore remain present (dangling) at target.

Create a recurring migration with the Azure Portal​

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.
  2. From the LiveData Migrator menu on the left, select Migrations.
  3. Select Add.
  4. Complete the details under the Basics section:
    • Migration Name: enter a reference name to use for the migration.
    • Target storage: your previously defined ADLS Gen2 target storage.
    • Path: the directory on your source filesystem to migrate data from.
  5. Select Next: Migration settings.
  6. Complete the Migration settings:
    • Target Action Policy: For a recurring migration, select the Skip if size match option.
    • Automatically Start Migration: start the migration right away, or start it manually later.
    • Migration Type: Select Recurring.
    • Recurring Period: Enter the duration that must pass between recurring re-scans. See Recurring Period for more info on this setting.
    • Exclusions: select the exclusions that you want to include for this migration.
    • Migration Priority: Assign a priority of High, Normal, or Low. The default priority is Normal.
  7. Select Review and create.
  8. Select Create.

Your newly created recurring migration will display the parameter Migration Type: Recurring in the migration overview.

Create a recurring migration with the CLI​

Specify both the scan-only and recurring-migration flags and set a recurring-period when you create the migration:

az livedata migrator migration create  -g <resource_group> --migrator-name <migrator_name> --target-name <target_name>--migration-name <new_migration_name> --path <path_to_migrate> --target-action-policy SkipIfSizeMatch --scan-only --recurring-migration --recurring-period 10
note

Use the SkipIfSizeMatch as the target-action-policy for your recurring migration. A Recurring Migration with no target-action-policy will default to SkipIfSizeMatch. The Overwrite target-action-policy isn't supported with recurring migrations.

Recurring period​

Recurring migrations scan continuously by default, which can have a performance impact on the source.

Set a Recurring Period to reduce the performance impact by selecting the minimum number of minutes, hours or days that must pass from the start of the first scan to the start of the next scan.

Only one scan can run at once for a migration.

Examples

  • If you set a Recurring Period of 12 hours and your scan takes 1 hour, the next one won't start for another 11 hours.
  • If you set a Recurring Period of 24 hours and your scan takes 25 hours, the next one will start immediately.

If a period isn't specified, it defaults to 0, and the scan will immediately recur after the last scan completes. note

You can amend the Recurring Period at any time for an existing migration.

Check or amend the current Recurring Period. From the LiveData Migrator menu, select Migrations, select your migration, then select Recurring Scan Limit. The current Recurring Period is shown under the migration's Properties tab.

Next steps​

If you want to migrate your metadata (for example: Hive databases and tables), create a metadata migration target next.

Troubleshooting​

If you are having problems migrating data, you can find solutions in the troubleshooting guide.