Skip to main content
Version: 2.1.1

Configure a HDFS source

You can migrate data from a Hadoop Distributed File System (HDFS) by configuring it as your source filesystem for Data Migrator. After you install Data Migrator to a HDFS, Data Migrator automatically configures it as a source for data migrations. However, if you have Kerberos enabled or if your Hadoop configuration isn't in a default location, you'll have to manually configure your HDFS source using the steps below.

You can also use these steps to set up a custom HDFS source instead of the default.

note

If you deleted the HDFS source that Data Migrator detected automatically, and you want to detect it again, run the command filesystem auto-discover-source hdfs in the CLI. For more information about this CLI command, see the command reference.

Prerequisites​

You need the following:

  • An HDFS cluster running Hadoop 2.6 or above.
  • If Kerberos is enabled on your filesystem, a valid keytab containing a suitable principal for the HDFS superuser must be available on the Linux host.
  • Oracle Big Data Services (BDS) - If running with Oracle's Distribution of Apache Hadoop (ODH), Data Migrator must provide fully qualified hostnames for DNS. Ensure that the following configuration property overrides are added for the source filesystem:
    Configuration property overrides are an option under Additional Configuration on the Filesystem Configuration screen.
    dfs.client.use.datanode.hostname=true
    dfs.datanode.use.datanode.hostname=true

Configure a HDFS source filesystem with the UI​

  1. Connect to the UI.

  2. Select your Data Migrator product from the Products panel.

  3. Select Add Source Filesystem.

  4. Select Hadoop Distributed File System (HDFS) from the Filesystem Type dropdown list.

  5. Enter the following details:

    • Display Name - Enter a name for your source filesystem.
    • Default FS - Enter the default filesystem from your HDFS configuration.
    • Kerberos Configuration
      • Kerberos Principal - Enter a principal that will map to the HDFS superuser using auth_to_local rules, or add the Data Migrator user principal to the superuser group on the Hadoop cluster you're using.
        • For example: Create the Kerberos principal ldmuser@realm.com. Using auth_to_local rules, ensure the principal maps to the user hdfs, or that the user ldmuser is added to the superuser group.
      • Kerberos Keytab Location - Enter the path to the Kerberos keytab file containing the Kerberos Principal. The keytab file must be accessible to the local system user running the Data Migrator service (the default is hdfs) and must be accessible from the edge node where Data Migrator is installed.
        • For example: Copy the ldmuser.keytab file (where ldmuser is your intended user) containing the Kerberos principal into the /etc/security/keytabs/ directory on the edge node running Data Migrator, make its permissions accessible to the HDFS user running Data Migrator, and enter the /etc/security/keytabs/ldmuser.keytab path during Kerberos configuration for the filesystem.
    • Additional Configuration (optional) - Enter override properties or enter additional properties by adding key/value pairs.
      • Configuration Property File Paths - Enter the directory or directories containing your HDFS configuration (such as the core-site.xml and hdfs-site.xml) on your Data Migrator host's local filesystem. This is required if you have Kerberos or a High Availability (HA) HDFS.
        note

        Data Migrator reads core-site.xml and hdfs-site.xml once, during filesystem creation, applying any configuration within paths added under Configuration Property File Paths. After creation, these paths are no longer visible in the UI. You can see all filesystem properties using the API.

      • Configuration Property Overrides (Optional) - Enter override properties or additional properties for your HDFS filesystem by adding key/value pairs.
    • Success File - Enter the file name or glob pattern that Data Migrator will use to recognize client application success files when they are created in migration directories. These files will be migrated last after all other data in the directory has been successfully migrated. You can use these files to confirm the directory they're in has finished migrating.
    • Filesystem Options
      • Live Migration - Changes to the underlying source filesystem are detected and handled in real-time. If you deselect this checkbox, this filesystem becomes a static source for one-time migrations.

  6. Select Save.

    You can now migrate data from the HDFS source.

For more information about configuring Kerberos, see the section below. If you have problems configuring Kerberos, see the troubleshooting section.

Configure Kerberos​

From the tabs below, select which filesystem(s) you want to set up Kerberos on to view the relevant instructions.

To set up Kerberos on a source filesystem, enter the Kerberos details for your source filesystem in the Kerberos parameters above.

Next steps​

Configure a target filesystem to migrate data to. Then create a migration.