Skip to main content
Version: 2.4.3 (latest)

Configure a Google Cloud Storage source

You can migrate data from a Google Cloud Storage (GCS) filesystem by configuring it as your source filesystem for Data Migrator.

note

Data Migrator supports only one-time migrations for GCS sources.

Prerequisites​

You need the following:

Configure a GCS source filesystem with the UI​

  1. From the Dashboard, select an instance under Instances.
  2. In the Filesystems & Agents menu, select Filesystems.
  3. Select Add source filesystem
  4. Select GCS from the Filesystem Type dropdown list.
  5. Enter the following details:
    • Display Name - Enter a name for your source filesystem.
    • Bucket Name - Enter the name of your GCS bucket.
    • Key File Options - Select whether to upload a JSON or a P12 key file to authenticate to your GCS bucket. Drag and drop your key file into the box below or select Browse to browse your system for it.
      • Email Address - If you chose Upload P12 Key File, enter the email address for your Google account.
        info

        Data Migrator imports GCS credentials from your JSON key file, stores them internally as configuration properties, then removes the file.

  6. Select Save.

Configure a GCS source filesystem with the CLI​

To create a GCS source, run the filesystem add gcs command in the Data Migrator CLI using the --source parameter:

Add a GCS filesystem
    filesystem add gcs      [--source]
                            [--file-system-id] string  
                            [--service-account-json-key-file] string  
                            [--service-account-p12-key-file] string  
                            [--service-account-json-key-file-server-location] string  
                            [--service-account-p12-key-file-server-location] string  
                            [--service-account-email] string  
                            [--bucket-name] string  
                            [--properties-files] list  
                            [--properties] string

Mandatory parameters​

  • --source Enter this parameter to use the filesystem resource created as a source.
  • --file-system-id The ID to give the new filesystem resource. In the UI, this is called Display Name.
  • --bucket-name The bucket name of a GCS account. In the UI, this is called Bucket Name.

Service account key parameters​

info

Enter your service account key for the GCS bucket by choosing one of the parameters below.

You can also upload the service account key directly when using the UI (this isn't supported through the CLI).

  • --service-account-json-key-file-server-location The absolute filesystem path on the Data Migrator server of your service account key file in JSON format. You can either create a GCS service account key or use an existing one.
    In the UI, this is called Key File and becomes visible when you select Key File Options -> Provide a Path. Use this parameter if you are running the CLI on a different host than your Data Migrator server.

  • --service-account-p12-key-file-server-location The absolute filesystem path on the Data Migrator server of your service account key file in P12 format. You can either create a GCS service account key or use an existing one. In the UI, this is called Key File and becomes visible when you select Key File Options -> Provide a Path. Use this parameter if you are running the CLI on a different host than your Data Migrator server.

  • --service-account-json-key-file The absolute filesystem path on the host running the Data Migrator CLI of your service account key file in JSON format.

    info

    Data Migrator imports GCS credentials from your --service-account-json-key-file, stores them internally as configuration properties, then removes the file.

    You can delete your JSON key file after successfully creating your GCS source.

  • --service-account-p12-key-file The absolute filesystem path on the host running the CLI of your service account key file in P12 format.

Optional parameters​

  • --service-account-email The email address linked to your GCS service account. In the UI, this is called Email address and is required when selecting the Upload P12 Key File option.
  • --properties Enter properties to use in a comma-separated key/value list.
Example
filesystem add gcs --source --file-system-id gcsAgent --bucket-name myGcsBucket --service-account-p12-key-file-server-location /user/hdfs/sourceStorage/myAccountKey.p12 --service-account-email user@mydomain.com

For more CLI commands, see the command reference.

Next steps​

Configure a target filesystem to migrate data to. Then create a migration.