Skip to main content
Version: 1.17.1

Snowflake metadata migration

LiveData Migrator supports metadata migration to Snowflake. The feature is automatically enabled, though currently only accessible through the CLI.

Add a Snowflake metadata agent with the CLI#

Use the hive agent add snowflake command to set up a Databricks agent in the CLI.

Example with basic authentication

Example with Basic authentication
hive agent add snowflake basic --account-identifier st12345.west-us-2.azure --file-system-id adlsgen2_target --name snowflake_agent --stage snowflake_stage --user user_one --warehouse snowflake_warehouse --password **********
Example with Private Key authentication
hive agent add snowflake privateKey --account-identifier st12345.west-us-2.azure --file-system-id adlsgen2_target --name snowflake_agent --stage snowflake_stage --user user_one --warehouse snowflake_warehouse --private-key-file /Users/user_one/.ssh/id_rsa --private-key-file-pwd **********

Limitations#

  • CLI only
  • Parquet tables only
  • Non-transactional tables only
  • Non-partitioned tables only
  • Public Schema only
  • No passwords with special characters

JDBC Driver timeout settings#

Occasionally, migrations may fail due to brief network disruption or heavy data processing. Reduce the chance of these failures by setting the following timeout properties in the JDBC connection string:

PropertyDescriptionRecommended
networkTimeoutNumber of milliseconds to wait for a response when interacting with the Snowflake service before returning an error. 0 (zero) specifies that no network timeout is set. Default=0600000
queryTimeoutNumber of seconds to wait for a query to complete before returning an error. 0 (zero) specifies that the driver should wait indefinitely. Default=00
  • The recommended 10-minute (600000ms) networkTimeout ensures that brief disruption to the network won't cause migration failures but migrations will not hang indefinitely.
  • Leaving queryTimeout at the default "0" value ensures that long query times will never result in a timeout.