Release notes

Product Version	LiveData UI	LiveData Migrator	Hive Migrator	CLI
2.5.4	13.11.4	1.28.7	2.0.4	1.20.4

Release Highlights

Live GPFS Source

In this release, we are introducing a new supported source, IBM Spectrum Scale(GPFS). This source is supported for both Live and One-time migrations in both CLI and UI. To utilize the Live capabilities, Kafka must be installed and configured to send notifications on the GPFS paths. Users can now migrate from their GPFS sources to all supported Data Migrator targets. More information on how to configure an IBM Spectrum Scale(GPFS) source can be found here.

RBAC for Data Migrator API

We’ve now introduced support for Role Based Access Control (RBAC) for the API. This will be via the already configured LDAP users for the UI and will support the same roles: admin, read-only, and migration manager. In this release, the API will only be supported for data migrations, not metadata migrations. The support for metadata migrations will come in a future release. More information on how to enable API access control can be found here.

Other Improvements

Databricks Improvements

In this release, we’ve made significant improvements to Databricks and Unity Catalog support. When creating the Databricks agent, there is now a choice of metastore type where the user can choose Unity Catalog or Workspace Hive Metastore agent types. We’ve addressed a variety of bugs to make the support of Unity Catalog more stable. We now allow users to choose whether they want their tables to be managed or external when using the Unity agent on the target. Additionally, we support migrating of managed and external tables as long as they're not transactional. We don't support transactional managed tables yet. More information on Databricks agents can be found here.

License Usage Report Download CLI Support

Users are now able to download the license usage report via the Data Migrator CLI. More information on the license report command can be found here.

Resolved Issues

Data Migrator Core

LM2-8111 Session cookie persists when Basic Auth is enabled

LM2-8080 EventListener can leak resources

LM2-8079 Possible to get migration stuck in SCHEDULED

LM2-8071 Disable session cookie when LDAP is enabled

LM2-8070 LM2 delivers replication factor 1 to HDFS target

LM2-8007 Verifications should be cancelled on restart rather than shutdown

LM2-8001 Log spam - Collecting ActionQueueDiagnostic diagnostics

LM2-7889 LM2 Migration Events fail to process

LM2-7844 No health check for ADLS2 target with 0auth2

LM2-7841 log missing from file tracker log file

Hive Migrator

HVM-4869 Missing property and incorrect default in HVM application.properties

HVM-4818 SERDE properties and table KV pairs should be configurable

HVM-4797 HVM/Remote agent upgrade fails to run successfully

HVM-4788 Column comments are not being migrated when converting to delta

HVM-4711 Remote agent GRPC calls droppage caused by unrelated failure

HVM-4696 HVM upgrade failing due to liquibase change

HVM-4689 Incorrectly formed SQL statement in Databricks SHOW SCHEMAS

HVM-4670 Failed to complete the metadata migration while the jdbc connection is blocked and losing data even after the connection is established

HVM-4650 Recurring data migration on a Delta table's data when a recurrence happens cause triggering the HVM to do copy into on that table again which cause the data to be lost on Databricks

HVM-4647 Live Data Events on Delta Source Tables trigger COPY INTOs

HVM-4642 Not able to migrating tables with time stamp type partitioned column

HVM-4640 Migration level override takes no effect for External Location

HVM-4639 Databricks External Location COPY INTO scope and construction issues

HVM-4621 Checksum calculation between equivalent hive table and delta table fails

HVM-4609 Unrecoverable error occurred: java.lang.StackOverflowError

HVM-4471 Copy Into not with force when migration is restarted when convertToDelta is true

HVM-4046 Remote agent InstanceId file being searched for in wrong directory

HVM-3999 /stats/{hour} endpoint time format doesn't seem to work

UI

ONEUI-7854 Error message when checking metadata rule validation has incorrect message

ONEUI-7842 ActionStore and MigrationAndThroughput Diagnostic calls timeout with 1000 long-running migrations

ONEUI-7837 Clicking on an empty field in line with "Learn More" opens the docs

ONEUI-7833 If the user doesn't have HVM installed or added in the UI, we still can see the metadata section in the side menu

ONEUI-7810 Inconsistent behaviour of required parameters on databricks agent form

ONEUI-7807 External location in the Databricks unity catalog agent in the UI should be empty

ONEUI-7802 Change to Migration Type modifies Skip or Overwrite Settings value

ONEUI-7800 Displaying wrong value for convertToDelta on metadata agent view page

ONEUI-7775 Hive Rule names and migration names containing & or ; can't be deleted

ONEUI-7649 LDMDiagnosticsSyncDataProvider times out with 500 long-running migrations

ONEUI-7593 UI client requests bandwidth graph data when UI in read-only mode

ONEUI-7544 LDMFailedPathsSyncDataProvider fails if migration does not exist on LM2

CLI

LDMC-446 Regression adding hive migration in CLI

LDMC-435 Add the missing fields to the usage report in the CLI to make it consistent with the UI

LDMC-372 Create migration should allow you to create with more than one exclusion

LDMC-347 Limited selection of possible email notification types available for autofill