Skip to main content
Version: 3.3 (latest)

Configure an Ozone source

Migrate data from your Ozone filesystem by adding as a source in Data Migrator.

Prerequisites​

  • For live migrations, ensure you have either configured an event stream from Ranger to Kafka or that you configure your source Ozone with the Cirata Ranger plug-in to add a Kafka topic.
  • If you do not have an event stream configured from Ranger to Kafka for a live migration, you will need to configure your Ozone source with the Cirata Ranger plug-in which adds events to a Kafka topic.
  • In all cases, these configurations should be managed by your system administrator.
info

When adding as a live source, because Kafka provides an unordered event stream, all migrations automatically default to use Target Match.

note

Ozone does not support ACL or extended attributes. If you are migrating from HDFS to Ozone, ACLs will not be supported.

Configure an Ozone source filesystem with the UI​

  1. From the Dashboard, select an instance under Instances.
  2. In the Filesystems & Agents menu, select Filesystems.
  3. Select Add target filesystem.
  4. Enter the following details:
    • Filesystem Type - The type of filesystem target. Select Apache Ozone.
    • Display Name - Enter a name for your source filesystem.
    • Default Filesystem - The Ozone filesystem scheme. For example, ofs://omservice or o3fs://bucket1.vol1.omservice/.
    • User - Enter the name of the filesystem user you want to migrate data with, only if your target is not Kerberized.
  5. Kerberos Configuration - The details of your Kerberos configuration. You can authenticate with Kerberos using multi-realm Kerberos, cross-realm trust or target-only Kerberos. See Configure Kerberos.
    • Kerberos Principal - Enter a principal that maps to a user with appropriate permissions for your Ozone source filesystem.
    • Kerberos Keytab Location - Enter the path to the Kerberos keytab file containing the Kerberos Principal. The keytab file must be accessible from the edge node where Data Migrator is installed.
  6. Advanced Configuration
    • Configuration Property File Paths - Enter the directory or directories containing your target filesystem's configuration (such as the core-site.xml and ozone-site.xml) on your Data Migrator host's local filesystem.
      note

      Data Migrator reads core-site.xml and ozone-site.xml once, during filesystem creation, applying any configuration within paths added under Configuration Property File Paths. After creation, these paths are no longer visible in the UI. You can see all filesystem properties using the API.

      • Configuration Property Overrides (Optional) - Enter override properties or additional properties for your Ozone filesystem by adding key/value pairs.
  7. Filesystem Options (Select either)
    • Live Migration - Select to include Live as a migration type when creating migrations.
    • One-time Migration - Select to limit migration types available to one-time. See migration types to learn more about each type.
  8. Kafka Event Source (Optional, for live migration, see prerequisites).
    • Bootstrap servers - Enter hostname and port of Kafka Bootstrap servers. Use comma-separated pairs for multiple servers. For example, hostname:9092,hostname2:9092.
    • Topic name - Enter the Kafka topic name for event delivery. For example, my-event-topic.
    • Group identifier - Enter the Kafka consumer identifier. For example, my-group-id.
    • Security Protocol - Select PLAINTEXT, SASL_PLAINTEXT, SASL_SSL If the selected security protocol is SASL_SSL, the following optional Truststore fields are available to complete:
      • Truststore Type - select None, jks, pem, pkcs12
      • Truststore Location - Enter the full local path of the Truststore file. This must be accessible to the local system user running the Data Migrator service. If the Truststore type is None this field is not available.
      • Truststore Password - Enter the Truststore password. If the Truststore type is None or pem this field is not available.
caution

The Group identifier must be unique and unused for each Data Migrator instance. Data Migrator cannot share events with other consumers to ensure no other consumer can interfere with Data Migrator retrieving events from the topic. Similarly, if you have multiple Data Migrator instances using the same Ozone source and the same Kafka topic, each must be supplied with a unique Group identifier.

info

The Truststore details are not required if:
a) you are not using self-signed certificates or
b) your self-signed certificates are provided to the JVM via CA certs

  1. Kafka Kerberos Configuration (Optional)
    • Kafka Kerberos Principal - If using Kerberos with Kafka. Enter the Kafka Kerberos principal used to authenticate with Kafka.
    • Kafka Kerberos Keytab Location - If using Kerberos with Kafka. Enter the path to the Kerberos keytab containing the Kafka Kerberos Principal supplied. The keytab file must be accessible to the local system user running the Data Migrator service.
  2. Select Save to add your Ozone filesystem.

Configure an Ozone source filesystem with the CLI​

Create an Ozone source with the filesystem add ozone command in the Data Migrator CLI. See the filesystem add ozone command reference for all options.

Example​

Add a live Ozone source
filesystem add ozone --file-system-id ozone-src 
                     --default-fs ofs://omservice 
                     --source
                     --kerberos-principal om/host@SOURCEREALM.COM
                     --kerberos-keytab /etc/security/keytabs/ozone.keytab
                     --properties-files /etc/sourceConf/core-site.xml,/etc/sourceConf/ozone-site.xml
                     --kafka-security-protocol SASL_PLAINTEXT
                     --kafka-bootstrap-servers kafkahost:9092 
                     --kafka-topic ranger_audits
                     --kafka-group-id mygroup-01
                     --kafka-kerberos-principal kafka/host@SOURCEREALM.COM
                     --kafka-kerberos-keytab /etc/security/keytabs/kafka.keytab

Update an Ozone source filesystem with the CLI​

Update an existing Ozone source filesystem with the CLI using the filesystem update ozone command.

Example​

Update an Ozone source
filesystem update ozone --file-system-id ozone-src --kafka-group-id mygroup-02

More Examples​

Several further examples of creating an Ozone source filesystem with the CLI are provided below. All of these use the filesystem add ozone command.

Add Non-Live Ozone Source, no SSL, no Kerberos
filesystem add ozone --file-system-id ozone-src 
                     --default-fs ofs://omservice 
                     --source
                     --scan-only
                     --properties-files /etc/sourceConf/core-site.xml,/etc/sourceConf/ozone-site.xml
Add Live Ozone Source (no SSL, no Kerberos)
filesystem add ozone --file-system-id ozone-src 
                     --default-fs ofs://omservice 
                     --source
                     --properties-files /etc/sourceConf/core-site.xml,/etc/sourceConf/ozone-site.xml
                     --kafka-security-protocol PLAINTEXT
                     --kafka-bootstrap-servers kafkahost:9092 
                     --kafka-topic ranger_audits
                     --kafka-group-id mygroup-01
Add Live Ozone Source (Kerberos, no SSL)
filesystem add ozone --file-system-id ozone-src 
                     --default-fs ofs://omservice 
                     --source
                     --kerberos-principal om/host@SOURCEREALM.COM
                     --kerberos-keytab /etc/security/keytabs/ozone.keytab
                     --properties-files /etc/sourceConf/core-site.xml,/etc/sourceConf/ozone-site.xml
                     --kafka-security-protocol SASL_PLAINTEXT
                     --kafka-bootstrap-servers kafkahost:9092 
                     --kafka-topic ranger_audits
                     --kafka-group-id mygroup-01
                     --kafka-kerberos-principal kafka/host@SOURCEREALM.COM
                     --kafka-kerberos-keytab /etc/security/keytabs/kafka.keytab
Add Ozone as a Live Source, PLAINTEXT auth, no Kerberos
filesystem add ozone --file-system-id ozone-src 
                     --default-fs ofs://omservice 
                     --source
                     --properties-files /etc/sourceConf/core-site.xml,/etc/sourceConf/ozone-site.xml
                     --kafka-security-protocol PLAINTEXT
                     --kafka-bootstrap-servers kafkahost:9092 
                     --kafka-topic ranger_audits
                     --kafka-group-id mygroup-01
Add Ozone as a Live Source, SASL_SSL, with Kerberos and SSL
filesystem add ozone --file-system-id ozone-src 
                     --default-fs ofs://omservice 
                     --source
                     --kerberos-principal om/host@SOURCEREALM.COM
                     --kerberos-keytab /etc/security/keytabs/ozone.keytab
                     --properties-files /etc/sourceConf/core-site.xml,/etc/sourceConf/ozone-site.xml
                     --kafka-security-protocol SASL_SSL
                     --kafka-bootstrap-servers kafkahost:9092 
                     --kafka-topic ranger_audits
                     --kafka-group-id mygroup-01
                     --kafka-kerberos-principal kafka/host@SOURCEREALM.COM
                     --kafka-kerberos-keytab /etc/security/keytabs/kafka.keytab
                     --kafka-ssl-truststore-type PKCS12 
                     --kafka-ssl-truststore-location /etc/security/kafka/ssl/client-truststore.p12 
                     --kafka-ssl-truststore-password changeit

Next steps​

Configure a target filesystem to migrate data to. Then create a migration.