Release notes

Product Version	LiveData UI	LiveData Migrator	Hive Migrator	CLI
2.2	11.7.3	1.25.3	1.16.3	1.16.2

For more information on component versions by product release, see this article.

Release Highlights

Databricks Unity Catalog Migration Configuration Support

In this release, Databricks Unity Catalog support has been extended to also be configurable at the migration level. So rather than enabling the use of Unity Catalog when creating the target agent, users can now enable which catalog to be used when creating migrations. This allows more flexibility when creating migrations to Databricks to use different catalogs. More information can be found here.

Scanner Retry

Additional resiliency has been included in this release in the source scanner. In a situation when the connection to the event stream is lost, Data Migrator will retry for a period of time before failing the migration. This allows the user an opportunity to be notified when there is an issue with the migration and to fix the issue for the migration to continue where it left off. This also allows Data Migrator to be resilient during source maintenance, such as cluster upgrades or restarts where there will be a period of downtime. So rather than needing to stop and reset your migration, Data Migrator will retry to continue migrating before failing.

JDK11 Support

In this release and onwards, we provide support for environments with JDK11. All operations and features will be supported on this version of Java.

Google as a Source (Non-live migration only)

Google Cloud storage is now included in the list of support sources. You can find this option via the CLI or on the UI drop-down menu when creating a migration source and can migrate to any supported target. This is currently only supported for non-live. More information on how to set this up can be found here

Talkback Improvements

Additional improvements have been made to the talkback script to allow users to collect not only local Data Migrator logs but also heap dumps and remote agent logs. We’ve also provided the capability to include configuration parameters as flags when running the script. This is useful when working with our support team to provide environmental information for debugging purposes. More information on this can be found here here

Other improvements

Display Notification Type in Email

Emails now explicitly indicate whether it is an “info”, “warn”, or “error” notification rather than just a colored template. Emails will now be sent out with the notification type in the subject line of the email as well as in the email text.

Hostname in email notifications

Emails will now include the hostname of the migrator so that if a user has multiple migrators deployed, they can be differentiated and distinguish which instance the notification is referring to.

Option to keep or remove deletion reports when deleting a migration

When a user decides to remove a migration, the confirmation prompt will now have the option to keep or discard existing deletion reports associated with the migration.

Remove the display of zero-length deletion reports

When there are no deletes, the UI will no longer show or archive zero-length reports.

Resolved Issues

LM2-6663 Failure when pausing then resuming metadata migration with “Migration Gate Not Applicable”

LM2-6708 Metadata migration hanging with “Failed to trigger gate marker injection on gate xxxx-xxxx-xxxx-xxxx-xxxx for migration xxx, no engine found.”

LM2-6706 Data Transfer Agent fail to start after installation with “AccessDeniedException” when loading certs

LM2-7108 Fix logging where MigrationStarter logs that it is starting a migration when it is not

LM2-6927 Edited configuration properties were not being stored in backups

LM2-6763 Backup schedule can be set to invalid values

LM2-6233 Fix issue where symlink needed to be removed before a re-install/upgrade can be successful

LM2-5199 Prevent users to be able to create multiple S3 sources with the same bucket when access point is being used

LM2-7072 Deleting/resetting migrations does not shrink DB

LM2-6134 File size exclusion value is not correct when backed up

LM2-6422 A large migration can fail to stop when stop is executed

LM2-6631 Source auto discovery failure should return error code 400

LM2-7120 Internal count can get corrupted due to PriorityBlockingQueue

LM2-6419 Fix race condition with create/reset/delete migration actions

LM2-6691 Notification of an unhealthy DTA even when its failed to be added

LM2-6559 Add check for email property when adding GCS

LM2-6641 DTAs agents ignores permissions on the target filesystem

LM2-6419 Create and reset migration can cause race condition

LM2-6766 Double slash in output of talkback if output location has trailing slash

LM2-7033 Data migrator service fails to start after upgrade for GCP target if fs.gs.auth.service.account.email is not set

HVM-3803 Migrating from Hive 1 or 2 to Hive 3 location fails

HVM-3603 Table queried on Databricks not returning results

HVM-3727 Fix misleading logging for notification events

HVM-3927 Creating a migration incorrectly logs sourceAgentName in place of targetAgent

HVM-3754 Table migration failed with NoSuchObjectException when source and target have primary key constraints with different name

HVM-3933 Changes to metadata source table after upgrade causes metadata migration to fail

HVM-3766 Alter table properties are not migrated when unity catalog is used

HVM-3759 NPE when metastore becomes unavailable

HVM-3751 Snowflake agent fails to update table when table is recreated

HVM-3750 Dataproc agent fails to deploy with SSL enabled

HVM-3749 SSl keystore password incorrectly populated when initialising the dataproc remote agent

HVM-3739 VIEW comment added by ALTER command fails to migrate with the snowflake agent

HVM-3796 Postgresql link points to a 404 error

HVM-3779 Update /opt/wandisco/hivemigrator to have executable permissions

HVM-3891 Snowflake agent fails to accomodate "_" character which caused error in migration

HVM-3896 Snowflake agent incorrectly constructs SQL

HVM-3945 Hivemigrator DB fails to upgrade on Centos6

HVM-4091 Snowflake has empty rows when column and parquet columns case mismatch

HVM-4053 Fail to alter table name with error "AgentMetaException : Specified defaultFS override doesn't match warehouse location, fix that before migration of managed table."

HVM-4132 Renaming a table duplicated its contents in Databricks target

HVM-4133 Renaming table does not remove original table on Snowflake

ONEUI-6993 Symlinked UI systemd unit file mounted on partition failed to load

ONEUI-7156 NPE when resetting migration

ONEUI-7075 Failed to load resource when DATA_AGENT_NOT_FOUND exception is thrown

ONEUI-7041 When adding a DTA a notification contains "{{newHealthStatus}}"

ONEUI-7038 Kerberos field should not be required

ONEUI-7057 Migration verification page jumps to page end when using datetime selector

ONEUI-7097 Bar above migrations should not move when there is a migration with 0 items in-between other migrations

ONEUI-7079 Remove ellipsis buttons for data transfer agents

ONEUI-7078 Fix naming validation for data transfer agents

ONEUI-7143 Unable to send test email after saving settings