Skip to main content
Version: 1.16.0

Manage migrations

Manage your migrations with LiveData Migrator using either the UI or CLI.

caution

Do not write to target filesystem paths when a migration is underway. This could interfere with LiveData Migrator functionality and lead to undetermined behavior.

Use different filesystem paths when writing to the target filesystem directly (and not through LiveData Migrator).

Manage migrations with the UI#

The Dashboard displays an overview of migrations and their status, showing what pre-existing data has been moved and data added since the migration started.

Click View migration to see more detail.

note

A migration must be stopped or aborted before it can be deleted.

Migration metrics#

Use metrics to view details about your migrations to better understand progress, including their bandwidth usage, migration events, and remaining scanned paths.

  1. In the dashboard, select a migration.
  2. You will be taken to the Status of the migration tab by default.

Under Migration Metrics, observe the following:

  • Oldest event metrics: this table shows the age of the oldest event that hasn't yet been migrated, names the migration with this event and displays the path the migration event is happening in. This lets you better understand LiveData Migrator's time spent processing certain events in your migration.
  • Events yet to be migrated: this graph shows the size of the scheduled actions queue across a time period selected in the Filter by dropdown menu.
  • Paths to be scanned: this graph displays the size of the queue for paths yet to be scanned across a time period selected in the Filter by dropdown menu.

Assign exclusions to an existing migration#

Adding exclusions to an existing migration will change the future actions performed for that migration, but will not affect previously migrated content.

  1. In the dashboard, select a migration.
  2. Under the sidebar that appears, select Exclusions.
  3. Click Add and select the appropriate exclusion template from the drop-down list.

Remove exclusions from an existing migration#

  1. In the dashboard, select a migration.
  2. Under the sidebar that appears, select Exclusions.
  3. Remove any of the exclusions one at a time.

Start a migration#

To start a migration and initiate data transfer, click the Start button.

Stop a migration#

When you start a migration, the Start button will be replaced with a Stop button. Click it to stop the migration and suspend all current file transfer.

Resume a migration#

When you stop a migration, the Stop button will be replaced with a Resume button. Click it to resume the migration, which will continue from where it left off.

Delete a migration#

note

Migrations must be stopped before they can be deleted.

When viewing a migration, click on the Details panel. Click the option to Delete the migration.

Reset a migration#

Reset a migration to set it to the state it was in before it started. This deletes the migration and creates a new migration in its place with the same settings as the previous.

When you stop a migration, a Reset button becomes available next to the Resume button. Click it to open the Reset Migration panel. Configure the following:

  • Reset Path Mappings: Select this box to erase the migration's path mapping configuration and automatically use your most recent path mapping configuration for LiveData Migrator. Clear it to continue using the same path mapping configuration from before the reset.
  • Skip or Overwrite Settings: Select Overwrite if you want the migration to re-migrate any and all files that had already been migrated before the reset, regardless of whether the file sizes are the same at the source and target filesystems. Select Skip If Size Match to skip migrating any files that already exist in the target and have the same file size as the source.

Click Reset to confirm your selection and reset the migration.

Manage migrations with the CLI#

CommandAction
migration stopStop a migration
migration resumeResume a stopped migration
migration deleteDelete a migration
migration exclusion addAdd an exclusion to a migration
migration exclusion deleteRemove an exclusion from a migration
migration listList running and active migrations
migration startStart a migration
migration showGet migration details
migration pending-region addAdd a pending region to a migration
statusGet migration status

Pending regions#

LiveData Migrator uses pending regions to keep your directories up to date if they change during data migrations. If directories are updated while being migrated, the changed paths on the source filesystem are tracked by the migration so they can be re-scanned for updates when the migration finishes.

LiveData Migrator collects pending regions automatically during a migration, but you can manually add them if you want the directories to be re-scanned after further updates. You can also re-run the entire migration by making the root directory the pending region.

Add pending regions in the UI#

To add a pending region in the UI:

  • Click on the migration you want to add a pending region to in the dashboard.
  • Under the sidebar that appears, select Rescan.
  • Type the path to the directory in Add rescan directories.
  • Choose Overwrite or Skip If Size Match.
  • Click +Add.

Add pending regions in the CLI#

Add a pending region to a migration by running the migration pending-region add command.

Example
migration pending-region add --migration-id myFirstMigration --path /dir1/userA

Data migration states#

Migrations can be in one of eight states:

NONSCHEDULED : A non-scheduled migration has been defined but not yet started. Create a migration in this state by not specifying the --auto-start parameter on creation.

SCHEDULED : A scheduled migration will start when directed to run.

STARTING : A starting migration is being started and will soon begin transferring content to the target.

RUNNING : A running migration is scanning through source content and transferring content to the target, as well as responding to change notifications from the source if applicable.

LIVE : A live migration has completed scanning through source content, continues to respond to change notifications from the source, and will transfer content to and make changes in the target as required.

PAUSING : A pausing migration has been instructed to pause transfer, but is temporarily continuing to make changes to the target.

PAUSED : A paused migration has been instructed to pause transfer, and is not transferring content or making other changes to the target.

ABORTED : An aborted migration will not make any changes to the target and cannot be run again.

COMPLETED : A completed migration has scanned through all source content and finished transferring all applicable data to the target filesystem, and has ignored any updates to the source data.

Monitoring failed operations#

The Failed Operations panel on LiveData UI's Migration screen provides a counter of any failed operations, including paths that failed to migrate. For a breakdown of the latest failures, click View details. This opens a list of failures that includes the following information for each failure:

  • Date and time: Date and time of the failure.

    Example
    2020-01-07 13:30:40 AM UTC
  • Path: Path that failed to migrate.

    Example
    /path/to/migration/files/000000_1
  • Failure: Error message generated by the failure.

    Example
    java.io.IOException: Error rescanning [/path/to/migration/files/part-1234] [not found]

Check your migration#

Check the status of a live migration by going to the migration's screen and selecting Migration Check from the Migration menu.

The Migration Check panel provides a Migration Check Report that lists failed migration operations. The report table lists each failed operation with columns for the timestamp, Operation type, and Path and Failure, which provides the path to the failed element, along with a description of the failure.

Filter and search for failed operations#

You can filter or search for failed operations by using the Operation type dropdown on the Migration Check Report panel. The filter contains the following operations:

OperationDescription
CreateCreate operations set up a new data migration. For more information, see Create.
Migration (Transfer file)Migration operations handle the transfer of data to be migrated.
Unlink (Delete path)Unlink operations remove files under migration.
Rescan (retry)Rescan operations restart an earlier source filesystem scan.
RenameRename operation handles name changes in files or directories under replication.
Metadata (Change permissions)Metadata operations change file metadata, such as file ownership.
Marker (Internal to LDM)Marker operations are part of LDM's internal migration mechanics. Marker file operation failures may be evidence of a serious problem with your LDM deployment. Contact WANdisco Support with details of the errors.

Advanced Search#

Click on Advanced search ^ to display Filter Rules for additional search options. Use the following entry fields to create an advanced search rule:

  • Field: Select Path to search all available migration paths or Failure to search only failed migration paths.

  • Operator: Select Contains or Does not contain to apply to your search.

  • Enter Value: Enter the value you want your search to match, according to your chosen operator. If you select the "Does not contain" operator, your search will match with paths or failure paths that don't include the value.

Click on the + Add Rule button to add additional search criteria. Click Apply to view the search results. You can continue to add rules to your current search, but you must click Apply to update the search results.

note

Historical failure information may be garbage collected, so it may not be possible to show all failures. In this case a warning will appear that states' "We're unable to show detailed results of failed operations. We only have detailed results from the most recent [number of available failures] results"

Reset a migration in the CLI#

Run the migration reset command to reset a migration in the CLI.