Configuring LiveData Migrator

Find details here for the configuration properties of LiveData Migrator. Properties are defined in the following file:

/etc/wandisco/livedata-migrator/application.properties

Restart the LiveData Migrator service when adding new properties or changing existing values:

service livedata-migrator restart

General configuration

These configuration properties are used to adjust general items of operation.

Name	Details
spring.jackson.serialization.INDENT_OUTPUT	Whether to apply indentation to JSON output from command results Default value: true Allowed values: true, false
springdoc.swagger-ui.path	The path at which clients can access the Swagger documentation for the LiveData Migrator REST API Default value: /ldm-api.html Allowed values: Any valid file path
pull.threads	The size of the thread pool that is used for executing activities related to notifications of changes in an HDFS environment Default value: 150 Allowed values: An integer value between 1 and 10000
engine.threads	The size of the thread pool used to perform migration of content from the source file system to targets Default value: 1000 Allowed values: An integer value between 1 and 10000
persisted.store	Reserved for future use Default value: true
server.port	The TCP port used to listen for clients interacting with the REST API Default value: 18080 Allowed values: An integer value between 1024 and 65535

Logging

Configure how LiveData Migrator logs requests made against the REST API.

Name	Details
logging.level.org.zalando.logbook	The logging level to apply to HTTP activity against the REST API of LiveData Migrator. This must be set to TRACE to record any log information. Default value: TRACE Allowed values: TRACE, NONE
logbook.format.style	The logging style applied to HTTP activity records Default value: http Allowed values: http, curl
logbook.write.max-body-size	The maximum number of bytes from an HTTP request or response body to include in a log entry Default value: 1024 Allowed values: Any integer between 1 and 2147483647
logbook.exclude	A comma-separated list of patterns that match URIs for which HTTP activity should not be logged Default value: /v3/api-docs/**,/swagger-ui/** Allowed values: Any valid comma-separated list of patterns
logbook.obfuscate.parameters	A comma-separated list of HTTP parameters that should not be recorded in log entries, for example: access_token,password Default value: (none) Allowed values: Any valid comma-separated list of HTTP parameter names
logbook.obfuscate.headers	A comma-separated list of HTTP headers that should not be recorded in log entries, for example: authorization,x-auth-password,x-auth-token,X-Secret Default value: (none) Allowed values: Any valid comma-separated list of HTTP headers
obfuscate.json.properties	A comma-separated list of JSON request properties by name that should not be recorded in log entries, for example: foo,bar Default value: ${hdfs.fs.type.masked.properties},${adls2.fs.type.masked.properties},${s3a.fs.type.masked.properties},${gcs.fs.type.masked.properties} Allowed values: Any valid comma-separated list of property names

State

LiveData Migrator uses an internally-managed database to record state during operation called the Prevayler.

Name	Details
prevayler.databaseLocation	The directory in which LiveData Migrator will write files to manage state Default value: ${install.dir}/db Allowed values: The full path to a directory in which database files will be managed. It must be writable by the user running LiveData Migrator (typically hdfs.)
prevayler.persistent	Whether LiveData Migrator will persist state to disk in files Default value: true Allowed values: true, false
prevayler.force	Whether the database performs a sync operation to ensure content is written to persistent storage on each write activity Default value: true Allowed values: true, false
prevayler.bufferedJournal	Whether buffered journal I/O is used for the database Default value: true Allowed values: true, false
prevayler.mirrored	Whether actions tracked in-memory by the database are mirrored to disk on every modification. The alternative is for operation to periodically flush to disk and flush on shutdown. Default value: true Allowed values: true, false

Security

Secure access to the LiveData Migrator REST API through configuration. Choose between no security or HTTP basic security.

Name	Details
security.type	The method of securing access to the REST API. Default value: off Allowed values: off, basic

Basic authentication

important

When basic authentication is enabled on LiveData Migrator, update the LiveData UI with the credentials to maintain functionality.

Name	Details
security.basic.user	Required when security.type=basic. The username that needs to be provided by a REST client to gain access to a secured REST API, for example: admin If basic authentication is enabled or will be enabled on the HiveMigrator REST API, use the same username for LiveData Migrator and HiveMigrator. Default value: (none) Allowed values: Any string that defines a username (no whitespace)
security.basic.password	Required when security.type=basic. A bcrypt-encrypted representation of the password that needs to be provided using HTTP basic authentication to access the REST API, for example: {bcrypt}$2a$10$mQXFoGAdLryWcZLjSP31Q.5cSgtoCPO3ernnsK4F6/gva8lyn1qgu The {bcrypt} prefix must be included before the encrypted password string as shown in the example above. Default value: (none) Allowed values: A valid bcrypt-encrypted string

Connecting to LiveData Migrator with basic authentication

When basic authentication is enabled, provide the username and password when prompted to connect to LiveData Migrator through the CLI:

Example

  connect livemigrator localhost: trying to connect...Username: adminPassword: ********connected

The username and password will also be required when accessing the LiveData Migrator REST API directly.

TLS

To enable TLS on the LiveData Migrator REST API (HTTPS), modify the following server.ssl.* properties.

note

If HTTPS is enabled on the REST API, plain HTTP requests from the CLI to the REST API will fail.

Example error

Bad RequestThis combination of host and port requires TLS.

Name	Details
server.ssl.key-store	Path or classpath to the Java keystore. Default value: (none) Allowed values: File system path or classpath (example:/path/to/keystore.p12, classpath:keystore.p12)
server.ssl.key-store-password	The Java keystore password. Default value: (none) Allowed values: Any text string
server.ssl.key-store-type	The Java keystore type. Default value: (none) Allowed values: Keystore types
server.ssl.key-alias	The alias for the server certificate entry. Default value: (none) Allowed values: Any text string.
server.ssl.ciphers	The ciphers suite enforce the security by deactivating some old and deprecated SSL ciphers, this list was tested against SSL Labs. Default value: (none but list provided below) TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 ,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA256,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA,TLS_RSA_WITH_CAMELLIA_256_CBC_SHA,TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA,TLS_RSA_WITH_CAMELLIA_128_CBC_SHA

tip

The example command below will generate a server certificate and place it inside a new Java keystore named keystore.p12:

keytool -genkey -alias livedata-migrator -storetype PKCS12 -keyalg RSA -keysize 2048 -keystore keystore.p12 -validity 365

See the keytool documentation for further information on the parameters used.

File system defaults

Each file system supported by LiveData Migrator can apply properties defined using the --properties or --properties-files parameters to the various filesystem add commands. You can set default properties that will apply to each type of file system at time of creation through these configuration items.

Name	Details
adls1.fs.type.default.properties	A comma-separated list of default properties to apply to ADLS Gen 1 file system resources on creation. Default value: fs.scheme,fs.account.name,fs.container.name,fs.auth.type,fs.oauth2.client.id,fs.insecure Allowed values: Any comma-separated list of valid ADLS Gen 1 configuration properties
adls2.fs.type.default.properties	A comma-separated list of default properties to apply to ADLS Gen 2 file system resources on creation. Default value: fs.scheme,fs.account.name,fs.container.name,fs.auth.type,fs.oauth2.client.id,fs.insecure Allowed values: Any comma-separated list of valid ADLS Gen 2 configuration properties
hdfs.fs.type.default.properties	A comma-separated list of default properties to apply to ADLS Gen 1 file system resources on creation. Default value: fs.defaultFS Allowed values: Any comma-separated list of valid HDFS configuration properties
s3a.fs.type.default.properties	A comma-separated list of default properties to apply to S3A file system resources on creation. Default value: fs.defaultFS Allowed values: Any comma-separated list of valid S3A configuration properties
gcs.fs.type.default.properties	A comma-separated list of default properties to apply to GCS resources on creation. Default value: bucket.name Allowed values: Any comma-separated list of valid GCS configuration properties
local.fs.type.default.properties	A comma-separated list of default properties to apply to local file system resources on creation. Default value: fs.root Allowed values: Any comma-separated list of valid S3A configuration properties

HDFS inotify

LiveData Migrator will poll the Hadoop cluster for NameNode events using the HDFS inotify system. These properties can be added and configured to change the default poll periods.

Name	Details
hdfs.inotify.poll.period	The length of time in milliseconds between each event listener poll. Default value: 10 Allowed values: An integer value
hdfs.inotify.sleep.period	The length of time in milliseconds for delaying the event listener poll after 10 consecutive retry failures. Default value: 10 Allowed values: An integer value

HDFS marker storage

LiveData Migrator uses marker files to manage the migration of files on paths. By default, these are stored in the HDFS user's home directory if possible. If this is not possible, they will be stored in the root directory of the migration on the source filesystem. To configure another directory to store marker files in, alter the following property:

Name	Details
hdfs.fs.marker.dir	The directory in which marker files are stored. Default value: (none) Allowed values: The full path to a directory in which database files will be managed. It must be writable by the user running LiveData Migrator (typically hdfs.)

Proxy Auto-Config (PAC) file support

LiveData Migrator allows the use of a PAC file so that traffic can be routed through HTTP proxies (examples of PAC files).

Name	Details
lm.proxy.pac	Path to the PAC file on the local filesystem. Default value: (none) Allowed values: A path that includes the file URI prefix (example: file:///tmp/proxy.pac).

important

PAC files for LiveData Migrator must contain an explicit clause that will return "DIRECT" for "localhost".

Example of a PAC file

function FindProxyForURL(url, host) {    if (dnsDomainIs(host, "localhost"))        return "DIRECT";}

Configure Notification Properties

Adjust notification properties in the application.properties file:

/etc/wandisco/livedata-migrator/application.properties

Name	Details
notifications.pending.region.warn.percent	The warning percentage threshold of events LiveData Migrator has fallen behind by. A migration exceeding this quota triggers the Data migration is falling behind system events notification. Default: 90.
notifications.pending.region.clear.percent	The warning clearing percentage threshold of events LiveData Migrator has fallen behind by. A migration that previously exceeded the notifications.pending.region.warn.percent quota that then falls below this value will automatically clear its Data migration is falling behind system events notification. Default: 80.

Directory structure

The following directories are used for the LiveData Migrator core package:

Location	Content
/var/log/wandisco/livedata-migrator	Logs
/etc/wandisco/livedata-migrator	Configuration files
/opt/wandisco/livedata-migrator	Java archive files
/opt/wandisco/livedata-migrator/db	LiveData Migrator runtime state
/var/run/livedata-migrator	Runtime files