Skip to main content
Version: 2.0

Verify migrations

Data Migrator migrates data from a source to a target filesystem, including rapidly changing transactional data. Occasionally, a migration can run into problems, such as:

  • Service failures or interruptions.
  • Software errors or individual file migration failures.

Use migration verification to scan the source and target, and check data on the source is also on the target. The verification compares the source and target for any discrepancies to allow you to check your migration status.

Discrepancies​

IncludeDon't include
Data that exists on one side and not on the other.Changes made after a specified time (see Verification Cutoff Point below).
File size mismatches.Exclusions.
info

Adding exclusions to an active migration may cause discrepancies.

Files migrated to the target before adding the exclusion are listed as discrepancies because they exist on both the source and target.

  • An excluded file that exists on both the source and target (FOUND_ON_BOTH) is a discrepancy.
  • An excluded file that exists on the source (FOUND_ON_SOURCE) but not the target isn't a discrepancy.
note

Full verification reports list all excluded files.

Limitations and considerations​

  • Only one verification can be active at a time for each data migration.
  • You can run verifications on different migrations at the same time.
  • Data Migrator can run a maximum of 10 verifications at the same time and queues any additional verifications until older ones are completed.
  • If you find a discrepancy in the full report that was a modification to the source or target and was made after the verification start time, it isn't a true discrepancy.
  • Data Migrator doesn't check any files and directories that are excluded from the migration for discrepancies. The summary report shows the number of files and directories excluded on the source.
  • Use the CLI to verify migrations if you don't want to set a verification cutoff point.
  • Use the UI to verify migrations if you want to set a verification cutoff point.
  • Verifications with path mappings are supported, where the root of the migration is mapped. Path mappings within the migration aren't included.
info

Don't use migration verification for migrations with auto source cleanup enabled.

Using migration verification and auto source clean up on the same migration will cause the verification report to list files intentionally deleted during auto source cleanup as discrepancies.

note

If your migration has S3 as a source or target filesystem, the time a directory is reported as having changed is always the same as the time that the directory was queried by the verification.

This is because of how S3 stores metadata about directories.

If you set a verification cutoff point, discrepant S3 directories will be ignored as they are considered to be after the cutoff point.

Prerequisites​

Migrations must have one of the following statuses to perform verification scans:

  • Live (real-time event stream notifications for changes to the source that are replicated to the target)
  • Complete (one-time migration without event stream notifications)
  • Stopped (a user has stopped the migration manually)
note

You can’t verify a migration that is in progress. Wait for the migration to finish or stop it and run the verification scan.

Verification application properties​

There are application properties to configure how many threads are available for verifications to run on.

For more information, see verification application properties.

Verify migrations with the UI​

View the migration status​

  1. From the Dashboard, select the migration you want to verify.

  2. Go to the Migration Status page.

  3. On the Migration Verification panel, you can see:

    • View details about the migration verification status.
    • Verification Status - Not Started, In Progress, Complete.
    • Total Inconsistencies - Number of discrepancies between the source and target paths.

Verify a migration​

Use the following options to create a new verification for a migration:

  1. Path to verify
    Enter the path or paths from your source or target filesystem you want to verify using the format /path/to.
    If you want to verify certain files on this path, for example, everything for a specific calendar month, you can enter a path like /path/to/oct_2022_*. This option allows you to specify subsections of large migration root directories for verification scanning.

    You can add paths:

    • One by one, up to a maximum of 100.
    • Using wildcards to match multiple paths.

  2. Verification depth
    Enter a number to specify how deep in the directory you want to run the verification check.
    The number must be equal to or less than the total number of levels in the directory structure of your migration.
    Zero means there's no limit to the verification depth.

    Example
    You enter two.
    Data Migrator scans and verifies the top two levels of your migration.

  3. Verification Cutoff Point
    Select a date and time as a verification cutoff point.

    The verification checks files modified on the source filesystem before the date and time you specify and excludes any changes after that cutoff point.

    Example
    You enter the date and time a migration was completed (one-time migration) or changed to live (live migration with continuous event stream notifications). No changes to the source filesystem are recorded as discrepancies in the verification report.

Cancel a verification​

You can cancel a verification that is queued or in progress. After you select Start verification, you can simply select Cancel check.
You can also delete a verification report for a completed or canceled verification.

View a verification summary report​

After you select Start verification, the Verification Summary Report panel is updated with information from the source and target found during the verification scan. You can view this while the verification is in progress or when it's complete or canceled. The scan may not start immediately as there can be one verification running at a time and new verifications are queued. When the verification starts checking the source and target, you can compare:

  • The number of files and directories found on the source and the target
  • Total amount of data on the source and the target
  • Total number of inconsistencies found, including file size mismatches, missing or extra files and directories on the target
note

If you have more than one verification for a migration, you can view up to a maximum of the last 10 verifications for a migration. If you create more than 10 verifications for a migration, the oldest are garbage collected.

Download a full verification report​

After the verification is complete, you can view and expand reports for completed verifications under Last x Verification Reports. Select Download all files to view, share, and analyze the results of the migration verification. This will give you a full report as a tar archive file containing summary.json and the following compressed files:

  • full-verification.jsonl.gz
  • verification-discrepancy.jsonl.gz
  • verification-discrepancy.csv.gz

You can also download the individual files for up to 10 latest reports by selecting the individual download icon next to the file(s) you want. The maximum number displayed is 10.

Set up email notifications​

You can be notified by email about the status of migration verifications and receive the results by email. Go to the Email Notification page to set up these alerts.

For more information, see Configure email notifications.