Release notes

Product Version	LiveData UI	LiveData Migrator	Hive Migrator	CLI
2.3	13.2.2	1.26.2	1.17.5	1.17.2

Release Highlights

Rebranding

With the recent rebrand of the business to "Cirata", the product's visual identity has been reimagined to provide a seamless experience as we continue on our mission of cloud and data with no limits.

Match target with source

The Target Match feature allows users to enable scanning of both the source and target file systems at the start of migrations and match the target file system to the source. Migrations with 'Target Match' enabled remove files on the target that are not on the source and migrate all source files, allowing both to be consistent when migrations become "Live." The feature provides users confidence and accuracy that both source and target are the same during migration. Target Match will log any removed files inconsistent between the source and target. For this release, Target Match is configurable in the CLI only. Find more information on how to configure Target Match here.

New exclusion type

A new template is available for file systems without live feeds that can exclude data created more recently than a defined time(age). The age exclusion type supports use cases where there's a requirement to transfer only recently modified content of a specific age. Find more information on exclusion types and use here and here.

DTA Bandwidth limiting

This release introduces the ability to set a bandwidth limit per individual data transfer agent. This gives network managers control over how much of a node's bandwidth a data transfer agent will use. More information on DTA bandwidth limiting can be found here.

Migration Manager Groups

An extension of migration management in LDAP configuration has been added in this release that allows data migration management to be delegated to groups of users that will create and manage a portion of the total migrations within Data Migrator. This allows an organization’s business groups to view and manage the migrations associated with their group. Find more information on Migration Management here.

Other improvements

Data Migrator data transfer improvements

A new data memory management layer to manage memory usage while transferring data has been implemented to lower the configuration burden and reduce the potential for out-of-memory issues. There is additional optimization of transferring small files to ADLS targets.

UI Performance improvements

In this release, we've improved the UI efficiency when there are a large number of migrations. This improvement has drastically reduced response times to user interactions with multiple migrations simultaneously through bulk actions. Users should see smoother UI use when driving large migration loads.

HVM Enhancements

The enhancements applied in this release provide increased performance and resilience simultaneously to Hive Migrator. The main focus point of improvements is for data ingestion to Databricks and Snowflake, allowing users to migrate to these cloud targets at much faster speeds. Review the various improvements below for more detail.

Auto source clean up removal

This feature has now been removed as of this release (Data Migrator 2.3).

Resolved Issues

Data Migrator Core

LM2-7522 Reduce Talkback frequency to 12 hours

LM2-7513 SysInfo configurations can cause talkback to fail

LM2-7487 Remove redundant diagnostics from Talkback

LM2-7473 TalkbackAPIDataWriter is inefficiently using memory

LM2-7454 Creating migration with a file path on a single file causes LDM and UI errors

LM2-7253 Prometheus metrics endpoint returns zero for all metrics

LM2-7231 ADLS smaller buffer optimisation

LM2-7212 Typo in talkback

LM2-7194 Remove need to restart when verifications are stuck

LM2-7194 INotifyEventSourceIsNotAvailableError repeated constantly in notifications

LM2-7178 Verifications report getting stuck and unable to cancel

LM2-7132 Too many migrations in running state over the configured amount

LM2-7132 Scheduled migrations are not started after a restart

LM2-7104 Remove confusing log message “Transferring removal failed”

LM2-7101 Verification should not reference “.livemigrator” folder

LM2-7100 Object reference rather than Object printed in log message

LM2-7012 event-logger.log is created but never written to

LM2-6896 Reduce DTA notifications

Hive Migrator

HVM-4368 Hive migrator failing with Agent Error

HVM-4350 CSV table migrations fail due to malformed SQL requests

HVM-4299 Account timeout reached when using incorrect Snowflake basic auth password

HVM-4298 HVM unable to authenticate with Azure SQL server if password ends in “

HVM-4297 If AzureSQL Server password contains “, it is logged by the AzureUtils class in the hivemigrator log file

HVM-4214 Improve logging in FileTrackingService and Client classes for Snowflake/Databricks

HVM-4212 Resuming migrations leads to potentially unnecessary scans of previously migrated objects

HVM-4208 Migration gets stuck when migration is stopped and resumed while initial scan is still in progress

HVM-4179 Migration hangs after StackOverflowError

HVM-4170 UI failing to startup and creates a temporary hive H2 DB under /tmp

HVM-4149 Partitions are null when data is added while migration is stopped

HVM-4132 Renaming a table duplicates its contents in the Databricks target

HVM-4106 Unable to restore HVM from backup when Databricks is the target

HVM-4106 Snowflake empty rows when column and parquet columns case mismatch

HVM-4051 After enabling basic auth, HVM REST API doc “/swagger-ui” or “/docs” is not accessible

HVM-4040 Migration heavy tables in different databases causes duplication in Snowflake target

HVM-3992 INSERT to a table, HVM log will print out an unexpected event ERROR

HVM-3992 Increase tolerance of HVM during failed license checks

HVM-3973 Migrating CSV formatted table with custom separators/escape/quote characters fails to copy data into Databricks

HVM-3940 HVM does not support sub-directories loaded into Snowflake

UI

ONEUI-7550 LDMStorage sync processor should not attempt to delete storage with migrations

ONEUI-7409 SQL constraint exceptions in logs

ONEUI-7405 NPE in GatewayAccountService

ONEUI-7402 Backup and Restore page is blank

ONEUI-7401 LDM licenses are duplicated when adding an instance

ONEUI-7386 HVMConsumptionDataSyncDataProcessor adds new MigrationPairs instead of updating existing ones

ONEUI-7383 When Databricks metadata migration has no params, a 0 is shown on settings page

ONEUI-7474 SyncQueueService: Sync Queue capacity reached.

CLI

LDMC-362 Cannot delete hive migrations with plus symbol in the name

LDMC-346 Unable to create verification without setting IgnoreAfterTimestamp

LDMC-339 Incorrect masking of jdbc password

LDMC-316 CLI does not gracefully handle EOF of stdin