Skip to main content
Version: 2.3 (latest)

Configure an S3 target

You can migrate data to a Simple Storage Service (S3) bucket by configuring one as a target filesystem.

Follow these steps to create an S3 target:

Prerequisites

You need the following:

  • An S3 bucket.
  • An access key and corresponding secret key for your S3 bucket.

Configure an S3 target filesystem in the UI

  1. From the Dashboard, select an instance under Instances.

  2. In the Filesystems & Agents menu, select Filesystems.

  3. Select Add target filesystem.

  4. Enter the following details:

    • Filesystem Type - The type of filesystem target. Select S3.
    • Display Name - Enter a name for your target filesystem.
    • Bucket Name - Enter the reference name of your S3 bucket.
    • Access Key - Enter the access key for your bucket. For example, RANDOMSTRINGACCESSKEY.
    • Secret Key - Enter the secret key that corresponds with your access key. For example, RANDOMSTRINGPASSWORD.
    • S3 Properties - Add the s3 endpoint of your S3-compatible object storage. For example:
      Example custom endpoint
      fs.s3a.endpoint = s3-region0.example-objectStore.com
      Add additional optional properties to your S3 target as key-value pairs.
  5. Select Save. You can now use your S3 target in data migrations.

S3a properties

For information on properties that are added by default for new S3A filesystems, see the Command reference s3a default properties.

For information on properties that you can customize for new S3A filesystems, see the Command reference s3a custom properties.

Upload buffering

Migrations using an S3 target destination will buffer all uploads. By default, the buffering will occur on the local disk of the system Data Migrator is running on, in the /tmp directory.

Data Migrator will automatically delete the temporary buffering files once they are no longer needed.

If you want to use a different type of buffering, you can change the property fs.s3a.fast.upload.buffer. The following values can be supplied:

Buffering OptionDetailsProperty Value
Array BufferBuffers the uploaded data in memory instead of on the disk, using the Java heap.array
Byte BufferBuffers the uploaded data in memory instead of on the disk, but does not use the Java heap.bytebuffer
Disk BufferingThe default option. This property buffers the upload to the disk.disk

Both the array and bytebuffer options may consume large amounts of memory. Other properties (such as fs.s3a.fast.upload.active.blocks) may be used to fine-tune the migration to avoid issues.

note

If you run out of disk space on which to buffer the migration, the migration will stall with a series of errors. To avoid this, ensure the filesystem containing the directory used for buffering (/tmp by default) has enough remaining space to facilitate the transfer.

Next steps

If you haven't already, configure a source filesystem from which to migrate data. Then, you can create a migration to migrate data to your new S3 target.