Migration verifications

note

Migration verifications are currently available as a preview feature. You must enable this feature before you can use it.

This preview version can only be used through the CLI or REST API.

LiveData Migrator migrates existing data and ongoing changes to data from source filesystems to target filesystems. You may encounter issues such as:

• Service failures or interruptions.
• Software errors or individual file migration failures.

Use this feature to scan the source and check data on the source is also on the target. The verification compares the source and target for any discrepancies to check the status of your migration. If the target has extra files that aren't on the source filesystem this isn't reported as a discrepancy.

How it works

You can verify migrations while they are in progress.

The verification uses a different scanning method to that of a migration scan. The migration path and any client activity will be scanned, but no data will be moved. The verification job will simply compare the source and target filesystems and provide the scanning status.

The verification results are available much faster than the data migration itself.

Limitations and considerations

These are limitations and considerations of this feature:

• A migration verification can only be performed on a live, running, or completed data migration.
• Only one verification can be active at a time for each data migration.
• A verification job doesn't ignore paths that are not yet migrated, so the status report shows missing files for a running migration.
• For one-time migrations (--scan-only), verifications ignore any client changes.
• The verification ignores any files and directories that have been excluded from the migration.
• The verification takes into account any path mappings that apply to the migration.

Run a migration verification

Add a migration verification for a specified migration by using migration verification add.

Example

migration verification add --name myMigration

Override the current verification job

Stop the current verification and start a new one by using the --override parameter.

Example with override

migration verification add --name myMigration --override

Show the verification status

Show the verification status to view an ongoing or completed verification job. The verification status will show the number of missing paths and files on the target filesystem and also the number of file size mismatches between the source and target.

The verification status can be viewed by using:

• migration verification show (for individual verification jobs)
• migration verification list (for all verification jobs)

Example status

Example status of a completed verification

{  "migrationId" : "testmig",  "state" : "COMPLETED",  "verificationId" : "e1aedfbd-b094-4a1b-a294-69cdd5a6030a",  "verificationPath" : "/testdir",  "startTime" : "2021-04-29T13:27:44.278Z",  "completeTime" : "2021-04-29T13:27:45.392Z",  "verificationEdge" : "/testmig/testdir01/testfile01",  "scannerSummary" : {    "progressSummary" : {      "filesScanned" : 177,      "directoriesScanned" : 47,      "bytesScanned" : 1105391944,      "filesExcluded" : 51,      "dirsExcluded" : 0,      "bytesExcluded" : 0,      "baseScanCompletionTime" : "2021-04-29T13:27:45.392Z"    },    "contentSummary" : {      "byteCount" : 1105391944,      "fileCount" : 194,      "directoryCount" : 81    }  },  "verificationProgress" : {    "matchedPathCount" : 224,    "totalFailedPathCount" : 0,    "targetFilesMissing" : 0,    "targetDirectoriesMissing" : 0,    "filesizeMismatches" : 0  }}

Verification status values

The important values to consider are contained within the verificationProgress output:

• matchedPathCount The total number of files and directories that were or have been migrated to the target filesystem.
• totalFailedPathCount The total number of files and directories that are missing on the target filesystem.
• targetFilesMissing The number of files that are missing on the target filesystem.
• targetDirectoriesMissing The number of directories that are missing on the target filesystem.
• filesizeMismatches The number of files that are mismatched in size between the source and target filesystems.

Verification reports

Once a verification job is complete, a verification report will be created in the /var/log/wandisco/livedata-migrator directory in the format of verification-report-{verificationId}-{startTime}.log.

/var/log/wandisco/livedata-migrator/verification-report-e1aedfbd-b094-4a1b-a294-69cdd5a6030a-1619781713672.log

This report will contain more details including any paths that have discrepancies.

Example report

Example report with errors

2021-04-22 22:44:41.783: Verification job b8264981-c70f-4231-b40e-7b2760afd2b4 is STARTING.  Estimated to scan ScannerContentMatrix{byteCount=20101, fileCount=162, directoryCount=17}2021-04-22 22:44:41.817: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/aexcl_0/4","SourceFileLength":100,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481817}2021-04-22 22:44:41.821: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/aexcl_0/8","SourceFileLength":100,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481821}2021-04-22 22:44:41.821: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/aexcl_0/1","SourceFileLength":100,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481821}2021-04-22 22:44:41.821: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/aexcl_1/6","SourceFileLength":100,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481821}2021-04-22 22:44:41.825: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/aexcl_0/7","SourceFileLength":100,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481825}2021-04-22 22:44:41.827: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/aexcl_0/0","SourceFileLength":100,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481826}2021-04-22 22:44:41.827: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/aexcl_0/6","SourceFileLength":100,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481827}2021-04-22 22:44:41.879: Verification failure: File not found on target. {"MigrationId":"b8264981-c70f-4231-b40e-7b2760afd2b4","Path":"/repl_test/C8412698_VerificationExclusionTest.b610650c-3496-435d-b1d0-809fbba538ba/mig/dexcl_3/0","SourceFileLength":200,"TargetFileLength":-1,"VerifyResult":"MISSING_ON_TARGET","VerifyTime":1619131481879}2021-04-22 22:44:41.901: Verification job b8264981-c70f-4231-b40e-7b2760afd2b4 is COMPLETED.  8 files failed verification from 162