Skip to main content
Version: 1.19.1

Configure target filesystems

Configure one or more target filesystems to start migrating data.

Add target filesystems with the UI#

To add a target filesystem from your LiveData Migrator dashboard, select the following:

  1. The relevant instance from the Products panel.

  2. Add target filesystem in the Filesystem Configuration page.

  3. You can create a target from one of the following supported targets:

    • Azure Data Lake Storage (ADLS) Gen2 (preview) - You can override the default value (dfs.core.windows.net) for Data Lake Storage Endpoint by replacing it with a custom or private endpoint.
      Select one of the following under Authentication Type:

    • Amazon S3/IBM Cloud Object Storage (preview)

      info

      If you have HDFS as your source with Kerberos enabled, you must enter a value for dfs.namenode Do this by adding a key/value pair in the UI under S3A Properties when configuring S3 as a target.

      note

      You must not include dots (.) in the names of S3 buckets you want to use with LiveData Migrator. See the recommendation in the Amazon bucket naming rules guide.

    • JCEKS Keystore hadoop.security.credential.provider.path

      This authentication method uses an access key and a secret key for Amazon S3 contained in a Java Cryptography Extension KeyStore (JCEKS).

      Enter the path containing the JCEKS keystore. For example, jceks://hdfs@active-namenode-host:8020/credentials/aws/aws.jceks.

      The keystore needs to contain values for the access key and the secret key.

      The access key and secret key are already in the keystore properties file so you don't need to enter them once you've saved the path.

      If you use a JCEKS file stored on a Hadoop Distributed File System (HDFS), ensure you add that HDFS as a source or target filesystem, otherwise, LiveData Migrator won't be able to find the file.

    • Oracle Object Storage (preview) - you must enter an Oracle Cloud endpoint to configure Oracle Cloud as an S3 target. When you enter an Oracle Cloud endpoint, all the other fields are filled automatically. For more information about Oracle endpoints and availability domains, go to the Oracle Cloud documentation.

    • Google Cloud Storage (preview)

note

To add a Google Cloud Storage target, you must first add a Service Account with at least Storage Admin role permissions to your Google Cloud Storage bucket.

For more information, see Creating and managing service accounts.

Update target filesystem configurations#

Update a target filesystem's configuration in its Filesystem Configuration page by selecting it in the Filesystems & Agents list in the LiveData Migrator dashboard.

note

To update existing filesystems, first stop all migrations associated with them. After saving updates to your configuration, you'll need to restart the LiveData Migrator service for your updates to take effect. In most supported Linux distributions, run the command service livedata-migrator restart.

Update the filesystem details and select Save.

The following details can be reconfigured:

Filesystem typeDetails
Google Cloud Storage (preview)The credentials file type can be changed. You can't change the bucket name.
Azure Data Lake Storage (ADLS) Gen2 (preview)Authentication fields can be changed, but not authentication type or container name.
Hadoop Distributed File System (HDFS) targetOnly configuration files can be changed.
Amazon S3 (preview)Authentication fields can be changed, but not the authentication type.
IBM Cloud Object Storage (preview)Authentication fields and configuration files can be changed. The endpoint can't be changed.
Oracle Object Storage (preview)Authentication fields, type, and configuration files can be changed.

Delete target filesystems#

Delete a target filesystem from the Filesystems panel. Select Delete Filesystem. Before you delete a target filesystem, first delete any associated migrations.

Add and manage target filesystems with the CLI#

Add filesystems to provide LiveData Migrator with information required to read content from your source and to migrate content to your target.

A range of different filesystem types are supported as targets, including Azure Data Lake Storage (ADLS) Gen2 (preview), Hadoop Distributed File System (HDFS), Google Cloud Storage (preview), and S3 (preview).

You can define multiple target filesystems to which you can migrate simultaneously. For more information about creating each target type, see below:

CommandAction
filesystem add adls2 oauthAdd an ADLS Gen2 filesystem resource using a service principal and 0Auth credentials
filesystem add adls2 sharedKeyAdd an ADLS Gen2 filesystem resource using access key credentials
filesystem clearDelete all target filesystems
filesystem deleteDelete a target filesystem
filesystem listList of target filesystems
filesystem showGet target filesystem details
filesystem typesList the types of target filesystems available

If you have ADLS Gen2 installed on a private network with a custom ADLS Gen2 endpoint, you can override the default endpoint value (dfs.core.windows.net) using the following property and example value:

Override default filesystem endpoint
        [--properties] fs.account.endpoint=<private_endpoint>

Next steps#

Once you have your source and target filesystem configured, you're ready to migrate data. If you want migrate data to a different path on your target filesystem, create path mappings first.

If you want to exclude specific file sizes or file names from your data migrations, define exclusions.