Configure a secrets store

Data Migrator supports integration with secrets management systems, allowing you to securely store and manage passwords and other sensitive information, reducing the risk of unauthorized access and enhancing overall security.

With a secrets store configured with Data Migrator, you can protect specific sensitive filesystem configuration values using references to secrets store keys. You can also protect sensitive application configuration properties by adding key-value pairs to your secrets store and defining a property source in Data Migrator's secrets store configuration.

The following secrets management systems are currently supported in this release:

• HashiCorp Vault

See the secrets store configuration section to integrate Data Migrator with your secrets store and the Using secrets store references section below to learn how to protect sensitive values in application and filesystem configuration.

Limitations

• Automatically rotated secrets are not supported. Automatically or manually changing referenced values stored in secrets store locations may have undesired effects and require additional actions to take effect. For example, changes to application property key-value pairs stored in a secrets store require a restart.
• HashiCorp Dynamic Secrets are not supported.
• Different secrets stores for different Data Migrator components is not supported. A single secrets store instance must be used for all components.
• The use of binary secret data, such as the binary content of an SSL Keystore stored in the secrets store, is not supported.
• HashiCorp Vault using Unix domain socket listener mode is not supported. TCP listener mode is supported, see HashiCorp Vault listener for more information on listening modes.

tip

It's generally considered a best practice to use TLS for Vault in production environments, review HashiCorp Vault with TLS before you apply your Vault integration configuration.

Secrets store configuration

Integrate your secrets store with Data Migrator by adding configuration to Data Migrator application.properties. See the configuration steps for supported secrets management systems and their authentication types below.

HashiCorp Vault

Data Migrator supports Token and AppRole(push mode) authentication methods with HashiCorp Vault.

See the steps below applicable to each authentication type.

info

The HashiCorp AppRole pull mode authentication method is currently not supported.

HashiCorp Vault using token authentication

If your Vault server is configured to use Token authentication. Configure Data Migrator to integrate with your HashiCorp Vault using token authentication.

1. Open /etc/wandisco/livedata-migrator/application.properties.
2. Add the following properties to the end of your application.properties file:

HashiCorp Vault using token authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.token

3. Optionally, add the following property to specify secrets store locations used to store configuration key-value pairs.

    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/livedata-migrator/application.properties TOKEN

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=TOKEN
    spring.cloud.vault.token=hvs.t0k3nex4mp13
    spring.config.import=vault://secret/ldm/config, vault://secret/ldm/node1/config

4. Save the file.
5. Restart the Data Migrator service. See System service commands - Data Migrator.

tip

The spring.config.import property is optional, use this property to import additional sensitive Data Migrator application property key-value pairs from your secrets store. Use references to the paths on your secrets store containing multiple key-value pairs. See the Application property reference format for more info.

HashiCorp Vault using AppRole(push mode) authentication

If your Vault server is configured to use AppRole(push mode) authentication configure Data Migrator to integrate with your HashiCorp Vault using AppRole(push mode) authentication.

1. Open /etc/wandisco/livedata-migrator/application.properties.
2. Add the following properties to the end of your application.properties file:

HashiCorp Vault using AppRole(push mode) authentication properties

    spring.cloud.vault.enabled
    spring.cloud.vault.uri
    spring.cloud.vault.authentication
    spring.cloud.vault.app-role.role-id
    spring.cloud.vault.app-role.secret-id 

3. Optionally, add the following property to specify secrets store locations used to store configuration key-value pairs.

    spring.config.import

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/livedata-migrator/application.properties APPROLE

    spring.cloud.vault.enabled=true
    spring.cloud.vault.uri=http://127.0.0.1:8200
    spring.cloud.vault.authentication=APPROLE
    spring.cloud.vault.app-role.role-id=ex4mp13r013va1u3ex4mp13va1u3
    spring.cloud.vault.app-role.secret-id=ex4mp13s3cretva1u3ex4mp13va1u3
    spring.config.import=vault://secret/ldm/config, vault://secret/ldm/node1/config

4. Save the file.
5. Restart the Data Migrator service. See System service commands - Data Migrator.

HashiCorp Vault with TLS

If your Vault server is configured to use TLS (as is typically recommended), specify a trust store to allow Data Migrator to trust the server's certificate and use https in your spring.cloud.vault.uri. To specify a trust store in your Data Migrator Vault configuration, include the following properties in /etc/wandisco/livedata-migrator/application.properties:

HashiCorp Vault with TLS properties

    spring.cloud.vault.ssl.trust-store
    spring.cloud.vault.ssl.trust-store-password
    spring.cloud.vault.ssl.trust-store-type

See Secrets store properties for descriptions of each property.

For example:

/etc/wandisco/livedata-migrator/application.properties TLS

    spring.cloud.vault.uri=https://127.0.0.1:8200
    ...
    ...
    spring.cloud.vault.ssl.trust-store=file:/etc/cirata/tls/truststore.jks
    spring.cloud.vault.ssl.trust-store-password=truststore_password
    spring.cloud.vault.ssl.trust-store-type=JKS

Ensure the trust store file is accessible to the local system user running the Data Migrator service and restart the Data Migrator service for changes to take effect.

tip

Remember to specify https in your spring.cloud.vault.uri if your Vault server uses TLS.

Vault TLS in JVM arguments

Alternatively, if you specify a trust store in your additional Data Migrator JVM arguments in /etc/wandisco/livedata-migrator/vars.env configuration, include the Vault server's certificate in the trust store.

For example:

/etc/wandisco/livedata-migrator/vars.env

   LDM_EXTRA_JVM_ARGS="-Djavax.net.ssl.trustStore=/etc/wandisco/tls/truststore.jks -Djavax.net.ssl.trustStorePassword=truststore_password"

note

Be aware, specifying a trust store using the JVM argument -Djavax.net.ssl.trustStore applies globally to the entire JVM instance.

Using secrets store references

With a secrets store configured, you can protect both, sensitive filesystem values and Data Migrator application properties using secrets store references. Add your secrets to your store before using the referenced locations in configuration.

Reference format

For filesystem property configuration specify the path and the key, for application property references, specify the path containing the key-value pairs. See details on both formats below.

Filesystem property reference format

Use the following format to reference secret values in filesystem configuration:

<scheme>://<secret_engine_mount_path>/<path_to_secret>/<key>

For example:

Example: Reference the value with a key of 'json' on the secrets engine mounted at 'secret' and a path of 'ldm/gcs'.

vault://secret/ldm/gcs/json

• scheme - Defines the secrets store type configured. In this release only the vault scheme is supported.
• secret_engine_mount_path - Specifies the mount path of your HashiCorp Vault kv secrets engine.
• path_to_secret - Defines the location of the secret on your secrets store.
• key - Specifies the key of the secret on the path.

note

Both kv1 and kv2 HashiCorp Vault secrets engines are supported. See HashiCorp Vault secrets engines for more information.

tip

HashiCorp Vault allows the use of forward slashes in key names. If the key you need to reference contains a slash, explicitly specify the key by appending the path_to_secret with ?key=. For example: vault://secret/ldm/gcs?key=json

Application property reference format

When referencing application properties you can store multiple key-value pairs on your secrets store and use the path in the reference without the need to specify the key. Use the following format to reference multiple secret application key-value pairs from your secrets store in application configuration:

<scheme>://<secret_engine_mount_path>/<path_to_secrets>

For example:

Example: Reference the key-value pairs on the secrets engine mounted at 'secret' and a path of 'ldm/config'.

vault://secret/ldm/config

• scheme - Defines the secrets store type configured. In this release only the vault scheme is supported.
• secret_engine_mount_path - Specifies the mount path of your HashiCorp Vault kv secrets engine.
• path_to_secrets - Defines the location of the key-value secrets on your secrets store.

Use filesystem configuration references

To protect specific sensitive filesystem configuration values, add secret values to a path in your secrets store then reference this path and key when adding your filesystem.

Using secrets store references in the UI

note

In this release, only the configuration of Google Cloud Storage allows the use of secrets store references for the Key File in the UI. See the Google cloud storage steps and example below.

Google cloud storage target example

To use a HashiCorp Vault location reference as the Key File value when adding a GCS target, add the contents of your GCS Key File to your secrets store as a string then use the following steps to reference this location:

tip

Add the contents of your GCS Key File to your secrets store as a string value and not using the JSON option in your Vault.

1. From the Dashboard, select an instance under Instances.

2. In the Filesystems & Agents menu, select Filesystems.

3. Select Add target filesystem.

4. Enter the following details:

   • Filesystem Type - The type of filesystem target. Select Google Cloud Storage.
   • Display Name - Enter a name for your target filesystem.
   • Bucket Name - Enter the name of your Google Cloud Storage bucket.
   • Key File Options - Select Hashicorp Vault.
     • Hashicorp Vault - In the Key File path in secrets store field, enter the HashiCorp Vault reference to the location of the content of the GCS Key File. For example: vault://my_secret_eng/ldm/gcs/keyfile.

5. Select Save.

note

Automatically rotated secrets are not supported see limitations for more info.

Using secrets store references in the CLI

The following filesystem configuration commands currently support secrets store references with the CLI:

• Google Cloud Storage - filesystem add gcs.
• ADLS Gen2 - filesystem add adls2 sharedKey, filesystem add adls oauth.
• S3 and Amazon S3 - filesystem add s3a.

Google cloud storage CLI example

Use the filesystem add gcs command with the --service-account-json-vault-reference option to add a GCS filesystem using a secrets store reference.

Example: Add GCS target with reference to key file in secrets store

filesystem add gcs --name gcs1 --bucket-name examplebucket --service-account-json-vault-reference vault://secret/ldm/production/keyfile

note

While you can use secrets store references for sensitive values when using the filesystem add command, the GCS filesystem add gcs command uses a specific --service-account-json-vault-reference option to supply a secrets store reference.

Filesystem properties which do not accept references

The following filesystem properties don't accept references to values in your secrets store. If your filesystem configuration requires properties that don't allow references, use the explicit values in your configuration.

info

For example, it's possible to use these properties when supplied as additional properties or with the equivalent CLI option. Additional properties may be properties added to configuration in the CLI with the --properties and --properties-files options or as additional key-value pairs supplied in the UI.

Filesystem	Properties	CLI option
ADLS Gen2	fs.defaultFS, fs.container.name, fs.account.endpoint	--container-name
GPFS	fs.defaultFS, kafka.topic, kafka.bootstrap.servers	--default-fs, --kafka-topic, --kafka-bootstrap-servers
HDFS	fs.defaultFS	--default-fs
LocalFs	fs.root	--fs-root
IBMCOS	topic, bootstrap.severs, bucketName, fs.s3a.endpoint
S3A	bucketName, fs.s3a.endpoint, sqsQueue, sqsQueue.endpoint	--bucket-name, --endpoint, --sqs-queue, --sqs-endpoint

Use application property references

To protect any sensitive Data Migrator application properties (values typically stored in application.properties), add key-value pairs to a secrets store location, specify this location in Data Migrators's secrets store configuration with a reference path against the spring.config.import property. On restart Data Migrator will use these locations as a configuration source for the key-value pairs stored.

info

Ensure the key names stored on your secrets store use the same property name as used in application properties.

Example server.ssl.key-store-password

For example, to reference the property and value of server.ssl.key-store-password from your configured secrets store:

1. Configure your secrets store with Data Migrator.
2. Add a secret to your secret store with key: server.ssl.key-store-password and its value containing the key store password.
3. If you don't already have this property source defined, open /etc/wandisco/livedata-migrator/application.properties.
4. Add the location or comma-separated locations to the spring.config.import property.

/etc/wandisco/livedata-migrator/application.properties

spring.config.import=vault://secret/ldm/config, vault://secret/ldm/node1/config, vault://secret/ldm/new/example

5. Restart the Data Migrator service. See System service commands - Data Migrator. Upon restart, Data Migrator will identify keys and apply their values.