Skip to main content
Version: 1.19.1

Manage migrations

Manage your migrations with LiveData Migrator using either the UI or CLI.

caution

Don't write to target filesystem paths when a migration is underway. This could interfere with LiveData Migrator functionality and lead to undetermined behavior.

Use different filesystem paths when writing to the target filesystem directly (and not through LiveData Migrator).

Manage migrations with the UI#

The Dashboard displays an overview of migrations and their status, showing what pre-existing data has been moved and data added since the migration started.

Select View migration to see more detail.

note

You must stop a migration before you can delete it.

Migration metrics#

Use metrics to view details about your migrations to better understand progress, including their bandwidth usage, migration events, and remaining scanned paths.

  1. In the Dashboard, select a migration.
  2. You will be taken to the Status of the migration tab by default.

Under Migration Metrics, observe the following:

  • Events yet to be processed: this graph displays the size of the event manager queue across a time period selected in the Filter by dropdown list.
  • Events yet to be migrated: this graph shows the size of the scheduled actions queue across a time period selected in the Filter by dropdown list.
  • Paths to be scanned: this graph displays the size of the queue for paths yet to be scanned across a time period selected in the Filter by dropdown list.

Assign exclusions to an existing migration#

Adding exclusions to an existing migration will change the future actions performed for that migration, but will not affect previously migrated content.

  1. In the Dashboard, select a migration.
  2. Under the sidebar that appears, select Exclusions.
  3. Select Add and the appropriate exclusion template from the dropdown list.

Remove exclusions from an existing migration#

  1. In the Dashboard, select a migration.
  2. Under the sidebar that appears, select Exclusions.
  3. Remove any of the exclusions one at a time.

Start a migration#

To start a migration and initiate data transfer, select Start.

Stop a migration#

When you start a migration, the Start button will be replaced with a Stop button. Select it to stop the migration and suspend all current file transfer.

Resume a migration#

When you stop a migration, the Stop button will be replaced with a Resume button. Select it to resume the migration, which will continue from where it left off.

Delete a migration#

note

Migrations must be stopped before they can be deleted.

When viewing a migration, select on the Details panel. Select the option to Delete the migration.

Reset a migration#

Reset a migration to set it to the state it was in before it started. This deletes the migration and creates a new migration in its place with the same settings as the previous.

When you stop a migration, a Reset button becomes available next to the Resume button. Select it to open the Reset Migration panel. Configure the following:

  • Reset Path Mappings: Select this box to erase the migration's path mapping configuration and automatically use your most recent path mapping configuration for LiveData Migrator. Clear it to continue using the same path mapping configuration from before the reset.
  • Skip or Overwrite Settings: Select Overwrite if you want the migration to re-migrate any and all files that had already been migrated before the reset, regardless of whether the file sizes are the same at the source and target filesystems. Select Skip If Size Match to skip migrating any files that already exist in the target and have the same file size as the source.

Select Reset to confirm your selection and reset the migration.

Pending regions#

LiveData Migrator uses pending regions to keep your directories up to date if they change during data migrations. If directories are updated while being migrated, the changed paths on the source filesystem are tracked by the migration so they can be re-scanned for updates when the migration finishes.

LiveData Migrator collects pending regions automatically during a migration, but you can manually add them if you want the directories to be re-scanned after further updates. You can also re-run the entire migration by making the root directory the pending region.

Add pending regions in the UI#

To add a pending region in the UI:

  • Select the migration you want to add a pending region to in the Dashboard.
  • Under the sidebar that appears, select Rescan.
  • Type the path to the directory in Add rescan directories.
  • Choose which action policy to use (Overwrite or Skip If Size Match).
  • Select +Add.

Add pending regions in the CLI#

Add a pending region to a migration by running the migration pending-region add command.

Example
migration pending-region add --migration-id myFirstMigration --path /dir1/userA --action-policy com.wandisco.livemigrator2.migration.SkipIfSizeMatchActionPolicy

Data migration states#

Migrations can be in one of the following states:

NONSCHEDULED : A non-scheduled migration has been defined but not yet started. Create a migration in this state by not entering the --auto-start parameter on creation.

RUNNING : A running migration is scanning through source data and transferring data to the target, as well as responding to change notifications from the source, if applicable.

STOPPED : A stopped migration has been stopped by a user manually and is not transferring content or making other changes to the target.

LIVE : A live migration has completed scanning through source data, continues to respond to change notifications from the source, and transfers data and makes changes to the target as required.

COMPLETED : A completed migration has scanned through all source data and finished transferring all applicable data to the target filesystem, and doesn't migrate changes to source data. See One time migrations.

Monitor and recover from failed operations#

When Data Migrator encounters a problem with a file or directory, it retries 180 times over a short 3-minute period. If Data Migrator still encounters the problem, it marks the path as a failed path and doesn't attempt the action again.

When the reason for failure is fixed, you can rescan the failed paths.

note

The rescan function for failed paths is necessary to recover from failed operations. For more information about how to rescan failed paths, see Pending regions. See the knowledge base article about Failed paths in Data Migrator for more detailed information on error handling.

To view failed paths, go to the Failed Operations panel on the Migration page in the UI. To view the latest failures, select View details. This opens a list of failures that includes the following information for each failure:

  • Date and time: Date and time of the failure.

    Example
    2020-01-07 13:30:40 AM UTC
  • Path: Path that failed to migrate.

    Example
    /path/to/migration/files/000000_1
  • Failure: Error message generated by the failure.

    Example
    java.io.IOException: Error rescanning [/path/to/migration/files/part-1234] [not found]

Check your migration#

Check the status of a live migration by going to the migration's screen and selecting Migration Check from the Migration menu.

The Migration Check panel provides a Migration Check Report that lists failed migration operations. The report table lists each failed operation with columns for the Timestamp, Operation type, and Path and Failure, which provides the path to the failed element, along with a description of the failure.

Filter and search for failed operations#

You can filter or search for failed operations by using the Operation type dropdown on the Migration Check Report panel. The filter contains the following operations:

OperationDescription
CreateCreate operations set up a new data migration. For more information, see Create a migration.
Migration (Transfer file)Migration operations handle the transfer of data to be migrated.
Unlink (Delete path)Unlink operations remove files under migration.
Rescan (retry)Rescan operations restart an earlier source filesystem scan.
RenameRename operation handles name changes in files or directories under replication.
Metadata (Change permissions)Metadata operations change file metadata, such as file ownership.
Marker (Internal to LiveData Migrator)Marker operations are part of LiveData Migrator's internal migration mechanics. Marker file operation failures may be evidence of a serious problem with your LiveData Migrator deployment. Contact WANdisco Support with details of the errors.

Advanced search#

Select Advanced search ^ to display Filter Rules for additional search options. Use the following entry fields to create an advanced search rule:

  • Field: Select Path to search all available migration paths or Failure to search only failed migration paths.

  • Operator: Select Contains or Does not contain to apply to your search.

  • Enter Value: Enter the value you want your search to match, according to your chosen operator. If you select the "Does not contain" operator, your search will match with paths or failure paths that don't include the value.

Select + Add Rule to add additional search criteria. Select Apply to view the search results. You can continue to add rules to your current search, but you must select Apply to update the search results.

note

Historical failure information may be garbage collected, so it may not be possible to show all failures. In this case, a warning appears:

"We're unable to show detailed results of failed operations. We only have detailed results from the most recent [number of available failures] results"

Manage migrations with the CLI#

CommandAction
migration stopStop a migration
migration resetReset a migration
migration resumeResume a stopped migration
migration deleteDelete a migration
migration exclusion addAdd an exclusion to a migration
migration exclusion deleteRemove an exclusion from a migration
migration listList running and active migrations
migration startStart a migration
migration showGet migration details
migration pending-region addAdd a pending region to a migration
statusGet migration status