Snowflake metadata migration

LiveData Migrator supports metadata migration to Snowflake. The feature is automatically enabled, though currently only accessible through the CLI.

Add a Snowflake metadata agent with the CLI

Use the hive agent add snowflake command to set up a Databricks agent in the CLI.

Example with basic authentication

Example with Basic authentication

hive agent add snowflake basic --account-identifier st12345.west-us-2.azure --file-system-id adlsgen2_target --name snowflake_agent --stage snowflake_stage --user user_one --warehouse snowflake_warehouse --password **********

Example with Private Key authentication

hive agent add snowflake privateKey --account-identifier st12345.west-us-2.azure --file-system-id adlsgen2_target --name snowflake_agent --stage snowflake_stage --user user_one --warehouse snowflake_warehouse --private-key-file /Users/user_one/.ssh/id_rsa --private-key-file-pwd **********

Limitations

• CLI only
• Parquet tables only
• Non-transactional tables only
• Non-partitioned tables only
• Public Schema only
• No passwords with special characters

JDBC Driver timeout settings

Occasionally, migrations may fail due to brief network disruption or heavy data processing. Reduce the chance of these failures by setting the following timeout properties in the JDBC connection string:

Property	Description	Recommended
networkTimeout	Number of milliseconds to wait for a response when interacting with the Snowflake service before returning an error. 0 (zero) specifies that no network timeout is set. Default=0	600000
queryTimeout	Number of seconds to wait for a query to complete before returning an error. 0 (zero) specifies that the driver should wait indefinitely. Default=0	0

• The recommended 10-minute (600000ms) networkTimeout ensures that brief disruption to the network won't cause migration failures but migrations will not hang indefinitely.
• Leaving queryTimeout at the default "0" value ensures that long query times will never result in a timeout.