Create a migration

Create new migrations with Data Migrator using either the WANdisco UI or CLI.

Migrations transfer existing data from the defined source to a target. Data Migrator migrates any changes made to the source data while it is being migrated and ensures that the target is up to date with those changes. It does this while continuing to perform the migration.

You will typically create multiple migrations so that you can select specific content from your source filesystem by Path. You can also migrate to multiple independent filesystems at the same time by defining multiple migration resources.

caution

Do not remove or change the target filesystem path after the migration has been created, and do not write to target filesystem paths when a migration is underway.

This could interfere with Data Migrator functionality and lead to undetermined behavior.

Use different filesystem paths when writing to the target filesystem directly (and not through Data Migrator).

Create a new migration with the UI

1. Enter a name for the migration.

2. Choose a Source and Target from your filesystems.

3. Choose the path on your source filesystem that you want to migrate.

   Use the folder browser and select the path name you want to migrate. Select the grey folder next to a path name to go inside of it and view its subdirectories.

   Alternatively, enter the path manually.

note

• You can't create multiple migrations from the same path.
• If you're migrating to Azure Data Lake Storage (ADLS) Gen2, your path must have fewer than 60 segments due to the ADLS Gen2 storage limit.

1. Enable the Auto-start migration option if you want to start the migration right away. (You can start it manually when viewing it later if you prefer.)
2. Apply any exclusions to reduce the scope within this Path.
3. Select the Overwrite or Skip if Size Match setting for the migration.
   • Skip if Size Match - If the file size is identical between the source and target, the file is skipped. If it’s a different size, the whole file is replaced.
   • Overwrite - Everything is replaced, even if the file size is identical.

If you've already migrated some data from the same source to the same target, you can choose whether to overwrite all the content (Overwrite) or only migrate new content that isn't already there (Skip if Size Match).

Assign exclusions to a new migration

Assign exclusions to your migration to exclude specific file sizes or file names from the migration. If you want to exclude file sizes or names from your migration but haven't defined any exclusion templates yet, see the define exclusions topic to learn how.

1. In the new migration page, select Add new exclusion
2. Select the appropriate exclusion template from the dropdown.

The exclusion appears in the list, and can be removed before the migration is started.

Manage a migration with the UI

You can Stop, Resume, or Reset a migration in the migration status page. Learn more.

note

When you reset a migration, Data Migrator:

• Reinitializes the migration statistics to a count of zero (0) for the migration that you reset.
• Assigns a new file identifier to the migration audit log file.

See Reset a migration.

Bulk actions

You can apply Add exclusions, Reset, Resume, Start, and Stop actions to multiple migrations at once with the UI. See Bulk actions for more details.

Create a new migration with the CLI

Migrate data from your source filesystem to a target defined using the migration command. Migrations will transfer existing data, as well as any subsequent changes made to the source data (in its scope), while Data Migrator remains in operation.

You will typically create multiple migrations so that you can select specific content from your source filesystem by path/directory. It is also possible to migrate to multiple independent filesystems at the same time by defining multiple migration resources.

Follow the command links to learn how to set the parameters and see examples.

1. Create a new migration:

   migration add

   Apply the --auto-start parameter if you would like the migration to start right away.

2. Assign exclusions to the migration:

   migration exclusion add

3. If you don't have auto-start enabled, manually start the migration:

   migration run

Create a one-time migration

Create a one-time migration if you do not want Data Migrator to scan for changes to your data during a migration. These migrations do not require you to have write access to the source filesystem, or operate the migration as the hdfs user.