Configure target filesystems

Configure one or more target filesystems to start migrating data.

Add target filesystems with the UI

To add a target filesystem from your dashboard, select the following:

1. The relevant instance from the Products panel.

2. Add target filesystem in the Filesystem Configuration page.

3. You can create a target from one of the following supported targets:

   • Azure Data Lake Storage (ADLS) Gen2 - You can override the default value (dfs.core.windows.net) for Data Lake Storage Endpoint by replacing it with a custom or private endpoint.
     Select one of the following under Authentication Type:

     • Shared Key
     • Service Principal (OAuth2)

   • Amazon S3

   • IBM Cloud Object Storage

   • Oracle Object Storage - you must enter an Oracle Cloud endpoint to configure Oracle Cloud as an S3 target. When you enter an Oracle Cloud endpoint, all the other fields are filled automatically.

     note

     Migrations to Oracle storage use the S3 SDK and S3a. For the latest information about limitations and compatibility, see Amazon S3 Compatibility API.

     For more information about Oracle endpoints and availability domains, go to the Oracle Cloud documentation.

   • Google Cloud Storage

   • HDFS

   • Local filesystem

For information about configuring filesystem health check notifications and email alerts, see Configure email notifications with the UI.

Update target filesystem configurations

Update a target filesystem's configuration in its Filesystem Configuration page by selecting it in the Filesystems & Agents panel (or Filesystems panel if you haven't installed Hive Migrator).

note

To update existing filesystems, first stop all migrations associated with them. After saving updates to your configuration, you'll need to restart the Data Migrator service for your updates to take effect. In most supported Linux distributions, run the command service livedata-migrator restart.

Update the filesystem details and select Save.

The following details can be reconfigured:

Filesystem type	Details
Google Cloud Storage	The Key File Options can be changed. You can't change the bucket name.
Azure Data Lake Storage (ADLS) Gen2	Authentication fields can be changed, but not authentication type or container name.
HDFS target	Only configuration files can be changed.
Amazon S3	Authentication fields can be changed, but not the authentication type.
IBM Cloud Object Storage	Authentication fields and configuration files can be changed. The endpoint can't be changed.
Oracle Object Storage	Authentication fields, type, and configuration files can be changed.

Delete target filesystems

Before you delete a target filesystem, first delete any associated migrations.

Delete a target filesystem by selecting it from the Filesystems panel and then selecting Delete on the Filesystem Configuration page.

Add and manage target filesystems with the CLI

Add filesystems to provide Data Migrator with information required to read content from your source and to migrate content to your target.

A range of different filesystem types are supported as targets, including Azure Data Lake Storage (ADLS) Gen2, HDFS, Google Cloud Storage, and S3.

You can define multiple target filesystems to which you can migrate simultaneously. For more information about creating each target type, see below:

ADLS Gen2

See Configure an ADLS Gen2 target.

Google Cloud Storage

See Configure a Google Cloud Storage target filesystem.

HDFS

See Configure a HDFS target filesystem.

S3

note

You must not include dots (.) in the names of S3 buckets you want to use with Data Migrator. See the recommendation in the Amazon bucket naming rules guide.

Command	Action
filesystem add s3a	Add an S3 filesystem resource (select this when using Amazon S3, IBM Cloud Object Storage, or Oracle Object Storage)
filesystem clear	Delete all target filesystems
filesystem delete	Delete a target filesystem
filesystem list	List of target filesystems
filesystem show	Get target filesystem details
filesystem types	List the types of target filesystems available

For more information, view the S3a mandatory parameters and the S3a optional parameters.

Next steps

Once you have your source and target filesystem configured, you're ready to migrate data. If you want to migrate data to a different path on your target filesystem, create path mappings first.

If you want to exclude specific file sizes or file names from your data migrations, define exclusions.