Snowflake metadata migration
LiveData Migrator supports metadata migration to Snowflake. The feature is automatically enabled, though currently only accessible through the CLI.
#
Add a Snowflake metadata agent with the CLIUse the hive agent add snowflake
command to set up a Databricks agent in the CLI.
Example with basic authentication
Example with Basic authentication
hive agent add snowflake basic --account-identifier st12345.west-us-2.azure --file-system-id adlsgen2_target --name snowflake_agent --stage snowflake_stage --user user_one --warehouse snowflake_warehouse --password **********
Example with Private Key authentication
hive agent add snowflake privateKey --account-identifier st12345.west-us-2.azure --file-system-id adlsgen2_target --name snowflake_agent --stage snowflake_stage --user user_one --warehouse snowflake_warehouse --private-key-file /Users/user_one/.ssh/id_rsa --private-key-file-pwd **********
#
Limitations- CLI only
- Parquet tables only
- Non-transactional tables only
- Non-partitioned tables only
- Public Schema only
- No passwords with special characters
#
JDBC Driver timeout settingsOccasionally, migrations may fail due to brief network disruption or heavy data processing. Reduce the chance of these failures by setting the following timeout properties in the JDBC connection string:
Property | Description | Recommended |
---|---|---|
networkTimeout | Number of milliseconds to wait for a response when interacting with the Snowflake service before returning an error. 0 (zero) specifies that no network timeout is set. Default=0 | 600000 |
queryTimeout | Number of seconds to wait for a query to complete before returning an error. 0 (zero) specifies that the driver should wait indefinitely. Default=0 | 0 |
- The recommended 10-minute (600000ms)
networkTimeout
ensures that brief disruption to the network won't cause migration failures but migrations will not hang indefinitely. - Leaving
queryTimeout
at the default "0" value ensures that long query times will never result in a timeout.