Skip to main content
Version: 3.3 (latest)

Configure a Ceph source

Migrate data from your Ceph storage filesystem by adding as a source in Data Migrator.

Prerequisites​

  • For live migration, it is required that the Ceph RGW service is configured to export bucket notifications to a Kafka topic.
  • Minimum Ceph version for live migration is Ceph Reef.
info

When adding as a live source, because Kafka provides an unordered event stream, all migrations automatically default to use Target Match.

Configure a Ceph source filesystem with the UI​

  1. From the Dashboard, select an instance under Instances.
  2. In the Filesystems & Agents menu, select Filesystems.
  3. Select Add source filesystem
  4. Select Ceph Storage from the Filesystem Type dropdown list.
  5. Enter the following details:
    • Display Name - Enter a name for your source filesystem.
    • Hostname - Enter the hostname address. For example, http://cephhost:7480.
    • Bucket Name - Enter the bucket name. For example, mybucket-01.
    • Access Key - Enter the access key. For example, mykey.
    • Secret Key - Enter the secret key. For example, mypassword.
  6. Within the Filesystem Properties section, enter override properties or additional properties for your Ceph filesystem by adding key/value pairs. The following filesystem options are also available:
    • Chunked Encoding - Enabled by default and should be disabled for Ceph Reef version 18.2.4 and below, otherwise the creation of empty directories or zero-byte files will fail
    • Path Style Access - Private stores generally expect path style access to be enabled so that buckets are visible, therefore this is enabled by default
  7. Filesystem Options (Select either)
    • Live Migration - Select to include Live as a migration type when creating migrations.
    • One-time Migration - Select to limit migration types available to one-time. See migration types to learn more about each type.
  8. Kafka Event Source (Optional, for live migration, see prerequisites).
    • Bootstrap servers - Enter hostname and port of Kafka Bootstrap servers. Use comma-separated pairs for multiple servers. For example, hostname:9092,hostname2:9092.
    • Topic name - Enter the Kafka topic name for event delivery. For example, my-event-topic.
    • Group identifier - Enter the Kafka consumer identifier. For example, my-group-id.
    • Security Protocol - Select PLAINTEXT, SASL_PLAINTEXT, SASL_SSL If the selected security protocol is SASL_SSL, the following optional Truststore fields are available to complete:
      • Truststore Type - select None, jks, pem, pkcs12
      • Truststore Location - Enter the full local path of the Truststore file. This must be accessible to the local system user running the Data Migrator service. If the Truststore type is None this field is not available.
      • Truststore Password - Enter the Truststore password. If the Truststore type is None or pem this field is not available.
caution

The Group identifier must be unique and unused for each Data Migrator instance. Data Migrator cannot share events with other consumers to ensure no other consumer can interfere with Data Migrator retrieving events from the topic. Similarly, if you have multiple Data Migrator instances using the same Ceph source and the same Kafka topic, each must be supplied with a unique Group identifier.

info

The Truststore details are not required if:
a) you are not using self-signed certificates or
b) your self-signed certificates are provided to the JVM via CA certs

  1. Kafka Kerberos Configuration (Optional)
    • Kafka Kerberos Principal - If using Kerberos with Kafka. Enter the Kafka Kerberos principal used to authenticate with Kafka.
    • Kafka Kerberos Keytab Location - If using Kerberos with Kafka. Enter the path to the Kerberos keytab containing the Kafka Kerberos Principal supplied. The keytab file must be accessible to the local system user running the Data Migrator service.
  2. Select Save to add your Ceph Storage filesystem.

Configure a Ceph source filesystem with the CLI​

Create a Ceph source with the filesystem add ceph command in the Data Migrator CLI. See the filesystem add ceph command reference for all options.

Example​

Add a live Ceph source
filesystem add ceph --file-system-id ceph-src 
                    --bucket-name mybucket-01 
                    --access-key mykey 
                    --secret-key mypassword 
                    --endpoint http://cephhost:7480 
                    --source 
                    --kafka-security-protocol PLAINTEXT 
                    --kafka-bootstrap-servers http://kafkahost:9092 
                    --kafka-topic mytopic-01 
                    --kafka-group-id mygroup-01

Update a Ceph source filesystem with the CLI​

Update an existing Ceph source filesystem with the CLI using the filesystem update ceph command.

Example​

Update a Ceph source
filesystem update ceph --file-system-id ceph-src --bucket-name mybucket-02

More Examples​

Several further examples of creating a Ceph source filesystem with the CLI are provided below. All of these use the filesystem add ceph command.

Add Non-Live Ceph Source
filesystem add ceph --file-system-id ceph-src 
                    --bucket-name mybucket-01 
                    --access-key mykey 
                    --secret-key mypassword 
                    --endpoint http://cephhost:7480 
                    --source 
                    --scan-only
Add Live Ceph Source (no SSL, no Kerberos)
filesystem add ceph --file-system-id ceph-src 
                    --bucket-name mybucket-01 
                    --access-key mykey 
                    --secret-key mypassword 
                    --endpoint http://cephhost:7480 
                    --source 
                    --kafka-security-protocol PLAINTEXT 
                    --kafka-bootstrap-servers http://kafkahost:9092 
                    --kafka-topic mytopic-01 
                    --kafka-group-id mygroup-01
Add Live Ceph Source (Kerberos, no SSL)
filesystem add ceph --file-system-id ceph-src 
                    --bucket-name mybucket-01 
                    --access-key mykey 
                    --secret-key mypassword 
                    --endpoint http://cephhost:7480 
                    --source 
                    --kafka-security-protocol SASL_PLAINTEXT 
                    --kafka-bootstrap-servers http://kafkahost:9092 
                    --kafka-topic mytopic-01 
                    --kafka-group-id mygroup-01 
                    --kafka-kerberos-keytab /etc/security/keytabs/ldm.keytab 
                    --kafka-kerberos-principal ldm@EXAMPLE.COM
Add Ceph as a Live Source, PLAINTEXT auth, no Kerberos
filesystem add ceph --file-system-id ceph-src
                    --bucket-name mybucket-01
                    --access-key mykey
                    --secret-key mypassword
                    --endpoint http://cephhost:7480
                    --source
                    --kafka-bootstrap-servers kafkahost:9092
                    --kafka-topic mytopic-01
                    --kafka-group-id mygroup-01
                    --kafka-security-protocol PLAINTEXT
Add Ceph as a Live Source, SASL_PLAINTEXT - with Kerberos, but no SSL
filesystem add ceph --file-system-id ceph-src
                    --bucket-name mybucket-01
                    --access-key mykey
                    --secret-key mypassword
                    --endpoint http://cephhost:7480
                    --source
                    --kafka-bootstrap-servers kafkahost:9092
                    --kafka-topic mytopic-01
                    --kafka-group-id mygroup-01
                    --kafka-security-protocol SASL_PLAINTEXT  
                    --kafka-kerberos-keytab /etc/security/keytabs/ldm.keytab
                    --kafka-kerberos-principal ldm@EXAMPLE.COM
Add Ceph as a Live Source, SASL_SSL, with Kerberos and SSL
filesystem add ceph --file-system-id ceph-src 
                    --bucket-name mybucket-01 
                    --access-key mykey
                    --secret-key mypassword 
                    --endpoint http://cephhost:7480 
                    --source
                    --kafka-bootstrap-servers kafkahost:9092 
                    --kafka-topic mytopic-01 
                    --kafka-group-id mygroup-01
                    --kafka-security-protocol SASL_SSL  
                    --kafka-kerberos-keytab /etc/security/keytabs/ldm.keytab 
                    --kafka-kerberos-principal ldm@EXAMPLE.COM 
                    --kafka-ssl-truststore-type PKCS12 
                    --kafka-ssl-truststore-location /etc/security/kafka/ssl/client-truststore.p12 
                    --kafka-ssl-truststore-password changeit

Kafka consumer properties​

The Kafka consumer properties control the behaviour of the event stream that we consume from and have some default values set. The important properties are already controlled by the parameters we request in the process to add a live source Ceph filesystem.

It is also possible to override the values of these properties by providing them as additional properties of the source filesystem. This can be done in the UI within the Filesystem Properties section or via the CLI using the [--properties] parameter where the consumer property is prefixed by kafka.consumer.

For example, if we wanted to override the consumer property max.poll.interval.ms we would refer to this as kafka.consumer.max.poll.interval.ms. More information on these consumer properties can be found in Kafka Consumer Configs.

S3a properties​

Enter additional properties for Ceph filesystems by adding them as key-value pairs in the UI or as a comma-separated key-value pair list with the --properties parameter in the CLI. You can overwrite default property values or add new properties.

Additional properties​

Find an additional list of S3a properties in the S3a documentation.

Next steps​

Configure a target filesystem to migrate data to. Then create a migration.