Skip to main content
Version: 1.15.1

Create a migration

Create new migrations with LiveData Migrator using either the UI or CLI.

Migrations transfer existing data from the defined source to a target. LiveData Migrator migrates any changes made to the source data while it is being migrated and ensures that the target is up to date with those changes. It does this while continuing to perform the migration.

You will typically create multiple migrations so that you can select specific content from your source filesystem by Path. You can also migrate to multiple independent filesystems at the same time by defining multiple migration resources.

caution

Do not write to target filesystem paths when a migration is underway. This could interfere with LiveData Migrator functionality and lead to undetermined behavior.

Use different filesystem paths when writing to the target filesystem directly (and not through LiveData Migrator).

Create a new migration with the UI#

  1. Provide a name for the migration.

  2. Choose a Source and Target from your filesystems.

  3. Choose the Path on your source filesystem that you want to migrate.

    Use the folder browser and click the path name you want to migrate. Click the grey folder next to a path name to go inside of it and view its subdirectories.

    Alternatively, enter the path manually.

    note

    ADLS Gen2 has a filesystem restriction of 60 segments. If you are migrating to an ADLS Gen2 storage, your path must have less than 60 segments.

  4. Enable the Auto-start migration option if you want to start the migration right away. (You can start it manually when viewing it later if you prefer.)

  5. Apply any exclusions to reduce the scope within this Path.

  6. Select the Overwrite or Skip if Size Match setting for the migration.

    • Skip if Size Match - If the file size is identical between the source and target, the file is skipped. If it’s a different size, the whole file is replaced.
    • Overwrite - Everything is replaced, even if the file size is identical.

If you've already migrated some data from the same source to the same target, you can choose whether to overwrite all the content (Overwrite) or only migrate new content that isn't already there (Skip if Size Match).

Assign exclusions to a new migration#

Assign exclusions to your migration to exclude specific file sizes or file names from the migration. If you want to exclude file sizes or names from your migration but haven't defined any exclusion templates yet, see the define exclusions topic to learn how.

  1. In the new migration page, click Add new exclusion
  2. Select the appropriate exclusion template from the dropdown.

The exclusion appears in the list, and can be removed before the migration is started.

Stop a migration with the UI#

In the migration status page, click Stop

Resume a migration with the UI#

In the migration status page, click Resume. The migration will continue from where it was stopped.

Reset a migration with the UI#

In the migration status page, click Reset. The migration will reset.

Bulk reset migrations with the UI#

  1. In the dashboard page, click on the product from the Products pane.

  2. Click on migrations from the left hand menu.

  3. Select reset under Bulk Action.

  4. Click Reset.

Create a new migration with the CLI#

Migrate data from your source file system to a target defined using the migration command. Migrations will transfer existing data, as well as any subsequent changes made to the source data (in its scope), while LiveData Migrator remains in operation.

You will typically create multiple migrations so that you can select specific content from your source file system by path/directory. It is also possible to migrate to multiple independent file systems at the same time by defining multiple migration resources.

Follow the command links to learn how to set the parameters and see examples.

  1. Create a new migration:

    migration add

    Apply the --auto-start parameter if you would like the migration to start right away.

  2. Assign exclusions to the migration:

    migration exclusion add

  3. If you don't have auto-start enabled, manually start the migration:

    migration run

Create a one-time migration#

Create a one-time migration if you do not want LiveData Migrator to scan for changes to your data during a migration. These migrations do not require you to have write access to the source filesystem, or operate the migration as the hdfs user.