Skip to main content
Version: 1.16.0

Command reference

Looking to learn more about LiveData Migrator commands? This reference page includes a comprehensive description of each command available from the LiveData Migrator CLI.

Each command description includes the information available from the help command. Tab-completion will also give you guidance when entering commands on the available options and help auto-complete the needed values.

tip

You can also find information about UI configuration items here. Look for them in their equivalent command line interface (CLI) mandatory and optional parameters.

Built-in commands#

The built-in commands are always available in a LiveData Migrator command line interactive session. They are unrelated to migration resources and operation (other than exit/quit), but help you to interact with LiveData Migrator and automate processing through scripts for the action prompt.

Source commands#


source clear#

Clear all information that LiveData Migrator maintains about the source filesystem by issuing the source clear command. This will allow you to define an alternative source to one previously defined or detected automatically.

Delete all sources
SYNOPSYS        source clear

source delete#

Use source delete to delete information about a specific source by ID. You can obtain the ID for a source filesystem with the output of the source fs show command.

Delete a source
SYNOPSYS        source delete [--file-system-id] string

Mandatory parameters#

  • --file-system-id The ID of the source filesystem resource you want to delete. In the UI, this is called Filesystem Name.

Example#

source delete --file-system-id auto-discovered-source-hdfs

source fs show#

Get information about the source filesystem configuration.

Show the source filesystem configuration
SYNOPSYS        source fs show [--detailed]

Optional parameters#

  • ---detailed Include all configuration properties for the source filesystem in the response.

Filesystem commands#


filesystem add adls2 oauth#

Add an Azure Data Lake Storage Gen2 (ADLS Gen2) container as a migration target using the filesystem add adls2 oauth command, which requires a service principal and OAuth 2 credentials.

note

The service principal that you want to use must have the Storage Blob Data Owner role assigned to the ADLS Gen2 storage account. For more information, see the Microsoft documentation.

Add an ADLS2 FileSystem via HCFS API FileSystem With OAuth
SYNOPSYS        filesystem add adls2 oauth [--file-system-id] string                                   [--storage-account-name] string                                   [--oauth2-client-id] string                                   [--oauth2-client-secret] string                                   [--oauth2-client-endpoint] string                                   [--container-name] string                                   [--insecure]                                   [[--properties-files] list]                                   [[--properties] string]

Mandatory parameters#

  • --file-system-id The ID to give the new filesystem resource. In the UI, this is called Filesystem Name.
  • --storage-account-name The name of the ADLS Gen2 storage account to target. In the UI, this is called Account Name.
  • --oauth2-client-id The client ID (also known as application ID) for your Azure service principal. In the UI, this is called Client ID.
  • --oauth2-client-secret The client secret (also known as application secret) for the Azure service principal. In the UI, this is called Secret.
  • --oauth2-client-endpoint The client endpoint for the Azure service principal. In the UI, this is called Endpoint.
    This will often take the form of https://login.microsoftonline.com/{tenant}/oauth2/v2.0/token where {tenant} is the directory ID for the Azure service principal. You can specify a custom URL (such as a proxy endpoint that manually interfaces with Azure Active Directory (Azure AD)).
  • --container-name The name of the container in the storage account to which content will be migrated. In the UI, this is called Container Name.

Optional parameters#

  • --insecure When provided, LiveData Migrator will not use TLS to encrypt communication with ADLS Gen2. This may improve throughput, but should only be used when you have other means of securing communication. This is referenced in the UI when Use Secure Protocol is unchecked.
  • --properties-files Reference a list of existing properties files, each containing Hadoop configuration properties in the format used by core-site.xml or hdfs-site.xml.
  • --properties Specify properties to use in a comma-separated key/value list.

Example#

filesystem add adls2 oauth --file-system-id mytarget --storage-account-name myadls2 --oauth2-client-id b67f67ex-ampl-e2eb-bd6d-client9385id --oauth2-client-secret 2IPO8*secretk-9OPs8n*TexampleHJ= --oauth2-client-endpoint https://login.microsoftonline.com/78u098ex-ampl-e498-8bce-ndpoint5f2e5/oauth2/v2.0/token --container-name lm2target

filesystem add adls2 sharedKey#

Add an ADLS Gen2 container as a migration target using the filesystem add adls2 sharedKey command, which requires credentials in the form of an account key.

Add an ADLS2 FileSystem via HCFS API FileSystem With Shared Key
SYNOPSYS        filesystem add adls2 sharedKey [--file-system-id] string                                       [--storage-account-name] string                                       [--shared-key] string                                       [--container-name] string                                       [--insecure]                                       [[--properties-files] list]                                       [[--properties] string]

Mandatory parameters#

  • --file-system-id The ID to give the new filesystem resource. In the UI, this is called Filesystem Name.
  • --storage-account-name The name of the ADLS Gen2 storage account to target. In the UI, this is called Account Name.
  • --shared-key The shared account key to use as credentials to write to the storage account. In the UI, this is called Access Key.
  • --container-name The name of the container in the storage account to which content will be migrated. In the UI, this is called Container Name.

Optional parameters#

  • --insecure When provided, LiveData Migrator will not use TLS to encrypt communication with ADLS Gen2. This may improve throughput, but should only be used when you have other means of securing communication. You can only see this in the UI when Use Secure Protocol is unchecked.
  • --properties-files Reference a list of existing properties files, each containing Hadoop configuration properties in the format used by core-site.xml or hdfs-site.xml.
  • --properties Specify properties to use in a comma-separated key/value list.

Example#

filesystem add adls2 sharedKey --file-system-id mytarget --storage-account-name myadls2 --container-name lm2target --shared-key Yi8NxHGqoQ79DBGLVn+COK/sRDwbNqAEXAMPLEDaMxRkvXt2ijUtASHAREDj/vaS/NbzR5rtjEKEY31eIopUVA==

filesystem add gcs#

Add a Google Cloud Storage as a migration target using the filesystem add myGcsBucket command, which requires credentials in the form of an account key file.

Add a Google Cloud Storage filesystem
SYNOPSYS        filesystem add gcs [--file-system-id] string                           [[--service-account-json-key-file] string]                           [[--service-account-p12-key-file] string]                           [[--service-account-json-key-file-server-location] string]                           [[--service-account-p12-key-file-server-location] string]                           [[--service-account-email] string]                           [--bucket-name] string                           [[--properties-files] list]                           [[--properties] string]

Mandatory parameters#

  • --file-system-id The ID to give the new filesystem resource. In the UI, this is called Filesystem Name.
  • --bucket-name The bucket name of a Google Cloud Storage account. In the UI, this is called Bucket Name.

Service account key parameters#

info

Provide your service account key for the Google Cloud Storage bucket by choosing one of the parameters below.

You can also upload the service account key directly when using the UI (this isn't supported through the CLI).

  • --service-account-json-key-file-server-location The absolute filesystem path on the LiveData Migrator server of your service account key file in JSON format. You can either create a Google Cloud Storage service account key or use an existing one.
    In the UI, this is called Key File and becomes visible when the Key File Options -> Provide a Path option is selected.
  • --service-account-p12-key-file-server-location The absolute filesystem path on the LiveData Migrator server of your service account key file in P12 format. You can either create a Google Cloud Storage service account key or use an existing one.
    In the UI, this is called Key File and becomes visible when the Key File Options -> Provide a Path option is selected.
  • --service-account-json-key-file The absolute filesystem path on the host running the LiveData Migrator CLI of your service account key file in JSON format. Only use this parameter if you are running the LiveData Migrator CLI on a different host to your LiveData Migrator server.
  • --service-account-p12-key-file The absolute filesystem path on the host running the LiveData Migrator CLI of your service account key file in P12 format. Only use this parameter if you are running the LiveData Migrator CLI on a different host to your LiveData Migrator server.

Optional parameters#

  • --service-account-email The email address linked to your Google Cloud Storage service account. In the UI, this is called Email address and is required when selecting the Upload P12 Key File option.
  • --properties-files Reference a list of existing properties files, each containing Hadoop configuration properties in the format used by core-site.xml or hdfs-site.xml.
  • --properties Specify properties to use in a comma-separated key/value list.

Example#

filesystem add gcs --file-system-id gcsAgent --bucket-name myGcsBucket --service-account-p12-key-file-server-location /user/hdfs/targetStorage/myAccountKey.p12 --service-account-email user@mydomain.com

filesystem add hdfs#

Add a Hadoop Distributed File System (HDFS) as either a migration source or target using the filesystem add hdfs command.

Creating a HDFS resource with this command will normally only be used when migrating to a target HDFS filesystem (rather than another storage service like ADLS Gen2 or S3a). LiveData Migrator will attempt to auto-detect the source HDFS when started from the command line unless Kerberos is enabled on your source environment.

If Kerberos is enabled on your source environment, use the filesystem auto-discover-source hdfs command to provide Kerberos credentials and auto-discover your source HDFS configuration.

Add a Hadoop Distributed File System
SYNOPSYS        filesystem add hdfs [--file-system-id] string                            [[--default-fs] string]                            [[--user] string]                            [[--kerberos-principal] string]                            [[--kerberos-keytab] string]                            [--source]                            [--scan-only]                            [[--properties-files] list]                            [[--properties] string]

Mandatory parameters#

  • --file-system-id The ID to give the new filesystem resource. In the UI, this is called Filesystem Name.
  • --default-fs A string that defines how LiveData Migrator accesses HDFS. In the UI, this is called Default FS.
    It can be specified in a number of forms:
    1. As a single HDFS URI, such as hdfs://192.168.1.10:8020 (using an IP address) or hdfs://myhost.localdomain:8020 (using a hostname).
    2. As a HDFS URI that references a nameservice if the NameNodes have high availability, for example, hdfs://mynameservice. For more information, see HDFS High Availability.

Optional parameters#

Kerberos: Cross-realm authentication required between source and target HDFS

Cross-realm authentication is required in the following scenario:

  • Migration will occur between a source and target HDFS.
  • Kerberos is enabled on both clusters.

See the links below for guidance for common Hadoop distributions:

  • --user The name of the HDFS user to be used when performing operations against the filesystem. In environments where Kerberos is disabled, this user must be the HDFS super user, such as hdfs.
  • --kerberos-principal The Kerberos principal to authenticate with and perform migrations as. This principal should map to the HDFS super user using auth_to_local rules.
  • --kerberos-keytab The Kerberos keytab containing the principal defined for the --kerberos-principal parameter. This must be accessible to the local system user running the LiveData Migrator service (default is hdfs).
  • --source Provide this parameter to use the filesystem resource created as a source. This is referenced in the UI when configuring the Unknown source.
  • --scan-only Provide this parameter to create a static source filesystem for use in one-time migrations. Requires --source.
  • --properties-files Reference a list of existing properties files that contain Hadoop configuration properties in the format used by core-site.xml or hdfs-site.xml. In the UI, this is called Provide a path to files under the Additional Configuration option.
  • --properties Specify properties to use in a comma-separated key/value list. In the UI, this is called Additional Configuration under the Additional Configuration option.
Properties files are required for NameNode HA#

If your Hadoop cluster has NameNode HA enabled, you must provide the local filesystem path to the properties files that define the configuration for the nameservice ID.

Source HDFS filesystem: These configuration files will likely be in a default location depending on the distribution of the Hadoop cluster.

Target HDFS filesystem: Ensure that the target Hadoop cluster configuration is available on your LiveData Migrator host's local filesystem.

  • For the UI, use Provide a path to files under the Additional Configuration option and define the directory containing the core-site.xml and hdfs-site.xml files.

    Example for path containing source cluster configuration
    /etc/hadoop/conf
    Example for path containing target cluster configuration
    /etc/targetClusterConfig

    Alternatively, define the absolute filesystem paths to these files:

    Example for absolute paths to source cluster configuration files
    /etc/hadoop/conf/core-site.xml/etc/hadoop/conf/hdfs-site.xml
    Example for absolute paths to target cluster configuration files
    /etc/targetClusterConfig/core-site.xml/etc/targetClusterConfig/hdfs-site.xml
  • For the CLI/API, use the --properties-files parameter and define the absolute paths to the core-site.xml and hdfs-site.xml files (see the Examples section for CLI usage of this parameter).

Examples#

HDFS as source#
Example for source NameNode HA cluster
filesystem add hdfs --file-system-id mysource --source --default-fs hdfs://sourcenameservice --properties-files /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
Example for source NameNode HA cluster with Kerberos enabled
filesystem add hdfs --file-system-id mysource --source --default-fs hdfs://sourcenameservice --properties-files /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml --kerberos-keytab /etc/security/keytabs/hdfs.headless.keytab --kerberos-principal hdfs@SOURCEREALM.COM
HDFS as target#
note

When specifying a HDFS filesystem as a target, the property files for the target cluster must exist on the local filesystem and be accessible to the LiveData Migrator system user.

Example for target NameNode HA cluster with Kerberos enabled
filesystem add hdfs --file-system-id mytarget --default-fs hdfs://targetnameservice --properties-files /etc/targetClusterConfig/core-site.xml,/etc/targetClusterConfig/hdfs-site.xml --kerberos-keytab /etc/security/keytabs/hdfs.headless.keytab --kerberos-principal hdfs@SOURCEREALM.COM
Example for target single NameNode cluster
filesystem add hdfs --file-system-id mytarget --default-fs hdfs://namenode.targetdomain:8020 --user hdfs

filesystem add local#

Add a local filesystem as either a migration target or source using the filesystem add local command.

Add a Local FileSystem via HCFS API.
SYNOPSYS        filesystem add local [--file-system-id] string                                       [[--fs-root] string]                                       [--source]                                       [--scan-only]                                       [[--properties-files] list]                                       [[--properties] string]

Mandatory parameters#

  • --file-system-id The ID to give the new filesystem resource. In the UI, this is called Filesystem Name.

Optional parameters#

  • --fs-root The directory in the local filesystem to scan for data or send data to, depending on whether the filesystem is defined as a source or a target. Should be supplied using the full directory path from the root.
  • --source Provide this parameter to use the filesystem resource created as a source. This is referenced in the UI when configuring the Unknown source.
  • --scan-only Provide this parameter to create a static source filesystem for use in one-time migrations. Requires --source.
  • --properties-files Reference a list of existing properties files.
  • --properties Specify properties to use in a comma-separated key/value list.
note

If no fs-root is specified, the file path will default to the root of your system.

Examples#

Local filesystem as source#
filesystem add local --file-system-id mytarget --fs-root ./tmp --source
Local filesystem as target#
filesystem add local --file-system-id mytarget --fs-root ./Users/username/destinationfolder/

filesystem add s3a#

Add an Amazon Simple Storage Service (Amazon S3) bucket as a target filesystem using the filesystem add s3a command. This method also supports IBM COS buckets.

SYNOPSYS        filesystem add s3a [--file-system-id] string                           [--bucket-name] string                           [[--endpoint] string]                           [[--access-key] string]                           [[--secret-key] string]                           [--credentials-provider] string                           [--source]                           [--scan-only]                           [[--properties-files] list]                           [[--properties] list]

For guidance about access, permissions, and security when adding an Amazon S3 bucket as a target filesystem, see Security best practices in IAM.

S3A mandatory parameters#

  • --file-system-id The ID for the new filesystem resource. In the UI, this is called Filesystem Name.

  • --bucket-name The name of your Amazon S3 bucket. In the UI, this is called Bucket Name.

  • --credentials-provider The Java class name of a credentials provider for authenticating with the Amazon S3 endpoint. In the UI, this is called Credentials Provider. This isn't a required parameter when adding an IBM COS bucket through the UI.
    The Provider options available include:

    • org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

      Use this provider to offer credentials as an access key and secret access key with the --access-key and --secret-key Parameters.

    • com.amazonaws.auth.InstanceProfileCredentialsProvider

      Use this provider when running LiveData Migrator on an Elastic Compute Cloud (EC2) instance that has been assigned an IAM role with policies that allow it to access the Amazon S3 bucket.

    • com.amazonaws.auth.DefaultAWSCredentialsProviderChain

      A commonly-used credentials provider chain that looks for credentials in this order:

      • Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, or AWS_ACCESS_KEY and AWS_SECRET_KEY.
      • Java System Properties - aws.accessKeyId and aws.secretKey.
      • Web Identity Token credentials from the environment or container.
      • Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI.
      • Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variable is set and security manager has permission to access the variable.
      • Instance profile credentials delivered through the Amazon EC2 metadata service.
  • Endpoint (UI & IBM COS only): This is required when adding an IBM COS bucket. IBM provide a list of available endpoints that can be found in their public documentation.

S3A optional parameters#

  • --access-key When using the org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider credentials provider, specify the access key with this parameter. In the UI, this is called Access Key. This is a required parameter when adding an IBM COS bucket.
  • --secret-key When using the org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider credentials provider, specify the secret key using this parameter. In the UI, this is called Secret Key. This is a required parameter when adding an IBM COS bucket.
  • --endpoint (S3 as a target only) Provide a specific endpoint to access the S3 bucket such as an AWS PrivateLink endpoint (for example: vpce-0e25b8cdd720f900e-argc85vg.s3.us-east-1.vpce.amazonaws.com). When using this parameter, do not use the fs.s3a.endpoint property as an additional custom property as this supersedes it. In the UI, this is called Use AWS PrivateLink -> PrivateLink VPC.
  • --source (Preview) Provide this parameter to use the filesystem resource created as a source. This is referenced in the UI when configuring the Unknown source.
  • --scan-only Provide this parameter to create a static source filesystem for use in one-time migrations. Requires --source.
  • --properties-files Reference a list of existing properties files, each containing Hadoop configuration properties in the format used by core-site.xml or hdfs-site.xml.
  • --properties Specify properties to use in a comma-separated key/value list. In the UI, this is called S3A Properties (see S3a Default Properties and S3a Custom Properties for more information).
note

Amazon S3a as a source is currently a preview feature.

S3a default properties#

These properties are defined by default when adding an S3a filesystem.

  • fs.s3a.impl (default org.apache.hadoop.fs.s3a.S3AFileSystem): The implementation class of the S3a Filesystem.
  • fs.AbstractFileSystem.s3a.impl (default org.apache.hadoop.fs.s3a.S3A): The implementation class of the S3a AbstractFileSystem.
  • fs.s3a.user.agent.prefix (default APN/1.0 WANdisco/1.0 LiveDataMigrator/1.11.6): Sets a custom value that will be pre-pended to the User-Agent header sent in HTTP requests to the S3 back-end by S3aFileSystem.
  • fs.s3a.impl.disable.cache (default true): Disables the S3 filesystem cache when set to 'true'.
  • hadoop.tmp.dir (default tmp): The parent directory for other temporary directories.
  • fs.s3a.connection.maximum (default 120) Defines the maximum number of simultaneous connections to the S3 filesystem.
  • fs.s3a.threads.max (default 150): Defines the total number of threads to make available in the filesystem for data uploads or any other queued filesystem operation.
  • fs.s3a.max.total.tasks (default 60): Defines the number of operations which can be queued for execution at a time.

S3a custom properties#

These are some of the additional properties that can be added when creating an S3a filesystem.

  • fs.s3a.fast.upload.buffer (default disk): Defines how the filesystem will buffer the upload.
  • fs.s3a.fast.upload.active.blocks (default 8): Defines how many blocks a single output stream can have uploading or queued at a given time.
  • fs.s3a.block.size (default 32M): Defines the maximum size of blocks during file transfer. Use the suffix K, M, G, T or P to scale the value in Kilobytes, Megabytes, Gigabytes, Terabytes or Petabytes respectively.
  • fs.s3a.buffer.dir (default tmp): Defines the directory used by disk buffering.

Find an additional list of S3a properties in the S3a documentation.

Upload buffering#

Migrations using an S3A target destination will buffer all uploads. By default, the buffering will occur on the local disk of system LiveData Migrator is running on, in the /tmp directory.

LiveData Migrator will automatically delete the temporary buffering files once they are no longer needed.

If you want to use a different type of buffering, you can change the property fs.s3a.fast.upload.buffer. The following values can be supplied:

Buffering OptionDetailsProperty Value
Array BufferBuffers the uploaded data in memory instead of on disk, using the Java heap.array
Byte BufferBuffers the uploaded data in memory instead of on disk, but does not use the Java heap.bytebuffer
Disk BufferingThe default option. Buffers the upload to disk.disk

Both the array and bytebuffer options may lead to the consumption of large amounts of memory. Other properties (such as fs.s3a.fast.upload.active.blocks) may be used to fine-tune the migration to avoid issues.

note

If you run out of disk space on which to buffer the migration, the migration will stall with a series of errors. To avoid this, ensure the filesystem containing the directory used for buffering (/tmp by default) has enough remaining space to facilitate the transfer.

Example#

filesystem add s3a --file-system-id mytarget --bucket-name mybucket1 --credentials-provider org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider --access-key B6ZEXAMPLEACCESSKEYA --secret-key OP87ChoExampleSecretKeyI904lT7AaDBGJpp0D

filesystem auto-discover-source hdfs#

Discover your local HDFS filesystem by specifying the Kerberos credentials for your source environment.

You can also manually configure the source HDFS filesystem using the filesystem add hdfs command.

Auto-discover-source Hadoop HDFS FileSystem FileSystem
SYNOPSYS        filesystem auto-discover-source hdfs [[--kerberos-principal] string]                                             [[--kerberos-keytab] string]

Kerberos parameters#

  • --kerberos-principal The Kerberos principal to authenticate with and perform migrations as. This principal should map to the HDFS super user using auth_to_local rules.
  • --kerberos-keytab The Kerberos keytab containing the principal defined for the --kerberos-principal parameter. This must be accessible to the local system user running the LiveData Migrator service (default is hdfs).

Example#

filesystem auto-discover-source hdfs --kerberos-keytab /etc/security/keytabs/hdfs.headless.keytab --kerberos-principal hdfs@REALM.COM

filesystem clear#

Delete all target filesystem references with the filesystem clear. This leaves any migrated content intact in those targets, but removes all resources that act as references to the target filesystems.

Delete all targets
NAME        filesystem clear - Delete all targets.
SYNOPSYS        filesystem clear

filesystem del#

Delete a specific filesystem resource by ID. This leaves all migrated content intact at that target, but removes the resource that acts as a reference to that filesystem.

Delete a target
SYNOPSYS        filesystem delete [--file-system-id] string

Mandatory parameters#

  • --file-system-id The ID of the filesystem resource to delete. In the UI, this is called Filesystem Name.

Example#

filesystem delete --file-system-id mytarget

filesystem list#

List defined filesystem resources.

List targets
SYNOPSYS        filesystem list [--detailed]

Mandatory parameters#

  • --detailed Include all properties for each filesystem in the JSON result.

filesystem show#

View details for a filesystem resource.

Get target details
SYNOPSYS        filesystem show [--file-system-id] string  [--detailed]

Mandatory parameters#

  • --file-system-id The ID of the filesystem resource to show. In the UI, this is called Filesystem Name.

Example#

filesystem show --file-system-id mytarget

filesystem types#

View information about the filesystem types available for use with LiveData Migrator. filesystems that provide an eventListenerType other than no-op can be used in migrations that will migrate ongoing changes during operation.

List the types of target filesystems available
SYNOPSYS        filesystem types

filesystem update adls2 oauth#

Update an existing Azure Data Lake Storage Gen2 container migration target with a specified filesystem ID using the filesystem update adls2 oauth command. You will be prompted to optionally update the service principal and OAuth 2 credentials.

Any optional parameters supplied will update the corresponding details of the existing filesystem. The parameters that can be changed are the same as the ones listed in the filesystem add adls2 oauth section.

All parameters are optional except --file-system-id, which specifies the filesystem you want to update.

Example#

filesystem update adls2 oauth --file-system-id mytarget --storage-account-name myadls2 --oauth2-client-id b67f67ex-ampl-e2eb-bd6d-client9385id --oauth2-client-secret 2IPO8*secretk-9OPs8n*TexampleHJ= --oauth2-client-endpoint https://login.microsoftonline.com/78u098ex-ampl-e498-8bce-ndpoint5f2e5/oauth2/v2.0/token --container-name lm2target

filesystem update adls2 sharedKey#

Update an existing Azure Data Lake Storage Gen2 container migration target using the filesystem update adls2 sharedKey command. You will be prompted to optionally update the secret key.

Any optional parameters supplied will update the corresponding details of the existing filesystem. The parameters that can be changed are the same as the ones listed in the filesystem add adls2 sharedKey section.

All parameters are optional except --file-system-id, which specifies the filesystem you want to update.

Example#

filesystem update adls2 sharedKey --file-system-id mytarget --storage-account-name myadls2 --container-name lm2target --shared-key Yi8NxHGqoQ79DBGLVn+COK/sRDwbNqAEXAMPLEDaMxRkvXt2ijUtASHAREDj/vaS/NbzR5rtjEKEY31eIopUVA==

filesystem update gcs#

Update a Google Cloud Storage migration target using the filesystem update gcs command.

Any optional parameters supplied will update the corresponding details of the existing filesystem. The parameters that can be changed are the same as the ones listed in the filesystem add gcs section.

All parameters are optional except --file-system-id, which specifies the filesystem you want to update.

Example#

filesystem update gcs --file-system-id gcsAgent --bucket-name myGcsBucket --service-account-p12-key-file-server-location /user/hdfs/targetStorage/myAccountKey.p12 --service-account-email user@mydomain.com

filesystem update hdfs#

Update either a source or target Hadoop Distributed filesystem using the filesystem update hdfs command.

Any optional parameters supplied will update the corresponding details of the existing filesystem. The parameters that can be changed are the same as the ones listed in the filesystem add hdfs section.

All parameters are optional except --file-system-id, which specifies the filesystem you want to update.

Examples#

Example for source NameNode HA cluster
filesystem update hdfs --file-system-id mysource --default-fs hdfs://sourcenameservice --properties-files /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml
Example for source NameNode HA cluster with Kerberos enabled
filesystem update hdfs --file-system-id mytarget --default-fs hdfs://sourcenameservice --properties-files /etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml --kerberos-keytab /etc/security/keytabs/hdfs.headless.keytab --kerberos-principal hdfs@SOURCEREALM.COM

filesystem update local#

Update a target or source local filesystem using the filesystem update local command.

Any optional parameters supplied will update the corresponding details of the existing filesystem. The parameters that can be changed are the same as the ones listed in the filesystem add local section.

All parameters are optional except --file-system-id, which specifies the filesystem you want to update.

Example#

filesystem update local --file-system-id mytarget --fs-root ./tmp

filesystem update s3a#

Update an S3 bucket target filesystem using the filesystem update s3a command. This method also supports IBM COS buckets.

Any optional parameters supplied will update the corresponding details of the existing filesystem. The parameters that can be changed are the same as the ones listed in the filesystem add s3a section.

All parameters are optional except --file-system-id, which specifies the filesystem you want to update.

Example#

filesystem update s3a --file-system-id mytarget --bucket-name mybucket1 --credentials-provider org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider --access-key pkExampleAccessKeyiz --secret-key eSeCreTkeYd8uEDnDHRHuV9IF3n9

Exclusion commands#


exclusion add date#

Create a date-based exclusion that checks the 'modified date' of any directory or file that the LiveData Migrator encounters during a migration to which the exclusion has been applied. If the path or file being examined by LiveData Migrator has a 'modified date' earlier than the specified date, it will be excluded from the migration.

Once associated with a migration using migration exclusion add, files that match the policy will not be migrated.

Create a new date-based rule
SYNOPSYS        exclusion add date [--exclusion-id] string                           [--description] string                           [--before-date] string

Mandatory parameters#

  • --exclusion-id The ID for the exclusion policy. In the UI, this is called Name.
  • --description A user-friendly description for the policy. In the UI, this is called Description.
  • --before-date An ISO formatted date and time, which can include an offset for a particular time zone. In the UI, this is called TBA.

Example#

exclusion add date --exclusion-id beforeDate --description "Files earlier than 2020-10-01T10:00:00PDT" --before-date 2020-10-01T10:00:00-07:00

exclusion add file-size#

Create an exclusion that can be applied to migrations to constrain the files transferred by a policy based on file size. Once associated with a migration using migration exclusion add, files that match the policy will not be migrated.

Create a new exclusion by file size policy
SYNOPSYS        exclusion add file-size [--exclusion-id] string                                [--description] string                                [--value] long                                [--unit] string

Mandatory parameters#

  • --exclusion-id The ID for the exclusion policy. In the UI, this is called Name.
  • --description A user-friendly description for the policy. In the UI, this is called Description.
  • --value The numerical value for the file size, in a unit defined by the unit parameter. In the UI, this is called Value.
  • --unit A string to define the unit used. You can use B for bytes, GB for gigabytes, KB for kilobytes, MB for megabytes, PB for petabytes, TB for terabytes, GiB for gibibytes, KiB for kibibytes, MiB for mebibytes, PiB for pebibytes, or TiB for tebibytes when creating exclusions with the CLI.

Example#

exclusion add file-size --exclusion-id 100mbfiles --description "Files greater than 100 MB" --value 100 --unit MB

exclusion add regex#

Create an exclusion using a regular expression to prevent certain files and directories being transferred based on matching file or directory names. Once associated with a migration using migration exclusion add, files and directories that match the regular expression will not be migrated.

Create a new exclusion by regular expression policy
SYNOPSYS        exclusion add regex [--exclusion-id] string                            [--description] string                            [--regex] string                            [[--type] string]

Mandatory parameters#

  • --exclusion-id The ID for the exclusion policy. In the UI, this is called Name.
  • --description A user-friendly description for the policy. In the UI, this is called Description.
  • --regex A regular expression in a syntax of either Java PCRE, Automata or GLOB type. In the UI, this is called Regex.

Optional parameters#

  • --type Choose the regular expression syntax type. There are three options available:

    1. JAVA_PCRE (default)
    2. AUTOMATA
    3. GLOB

Examples#

Example glob pattern
exclusion add regex --description "No paths or files that start with test" --exclusion-id exclusion1 --type GLOB --regex test*
Example Java PCRE pattern
exclusion add regex --description "No paths of files that start with test" --exclusion-id exclusion1 --regex ^test\.*

Using backslash characters within --regex parameter#

If you wish to use a \ character as part of your regex value, you must escape this character with an additional backslash.

Example
exclusion add regex --description "No paths that start with a backslash followed by test"  --exclusion-id exclusion2 --regex ^\\test\.*

The response displayed if running through the CLI will not hide the additional backslash. However, the internal representation will be as expected within LiveData Migrator (it will read as ^\test.*).

This workaround isn't required for API inputs, as it only affects the Spring Shell implementation used for the CLI.


exclusion del#

Delete an exclusion policy so that it is no longer available for migrations.

NAME        exclusion delete - Delete an exclusion rule.
SYNOPSYS        exclusion delete [--exclusion-id] string

Mandatory parameters#

  • --exclusion-id The ID for the exclusion policy to delete. In the UI, this is called Name.

Example#

exclusion delete --exclusion-id exclusion1

exclusion list#

List all exclusion policies defined.

List all exclusion policies
NAME        exclusion list - List all exclusion rules.
SYNOPSYS        exclusion list

exclusion show#

Get details for an individual exclusion policy by ID.

Get details for a specific exclusion rule
SYNOPSYS        exclusion show [--exclusion-id] string

Mandatory parameters#

  • --exclusion-id The ID for the exclusion policy to show. In the UI, this is called Name.

Example#

exclusion show --exclusion-id 100mbfiles

Path mapping commands#


path mapping add#

Create a path mapping that allows you to define a alternative target path for a specific target filesystem. These will be automatically applied to a new migration.

When path mapping isn't used, the source path is created on the target filesystem.

note

Path mappings can't be applied to existing migrations. Delete and recreate a migration if you want a path mapping to apply.

Create a new path mapping
SYNOPSYS        path mapping add [[--path-mapping-id] string]                         [--source-path] string                         [--target] string                         [--target-path] string                         [--description] string

Mandatory parameters#

  • --source-path The path on the source filesystem.
  • --target The target filesystem id (value defined for the --file-system-id parameter).
  • --target-path The path for the target filesystem.
  • --description Description of the path mapping enclosed in quotes ("text").

Optional parameters#

  • --path-mapping-id An ID for this path mapping. An ID will be auto-generated if one isn't provided.

Example#

Example for HDP to HDI - default Hive warehouse directory
path mapping add --path-mapping-id hdp-hdi --source-path /apps/hive/warehouse --target mytarget --target-path /hive/warehouse --description "HDP to HDI - Hive warehouse directory"

path mapping del#

Delete a path mapping.

note

Deleting a path mapping will not affect any existing migrations that have the path mapping applied. Delete and recreate a migration if you no longer want a previous path mapping to apply.

Delete a path mapping
SYNOPSYS        path mapping delete [--path-mapping-id] string

Mandatory parameters#

  • --path-mapping-id The ID of the path mapping.

Example#

path mapping delete --path-mapping-id hdp-hdi

path mapping list#

List all path mappings.

List all path mappings
SYNOPSYS        path mapping list [[--target] string]

Optional parameters#

  • --target List path mappings for the specified target filesystem id.

Examples#

Example for listing all path mappings
path mapping list --target hdp-hdi
Example for listing path mappings for a specific target
path mapping list --target hdp-hdi

path mapping show#

Show details of a specified path mapping.

Get path mapping details
SYNOPSYS        path mapping show [--path-mapping-id] string

Optional parameters#

  • --path-mapping-id The ID of the path mapping.

Example#

path mapping show --path-mapping-id hdp-hdi

Migration commands#


migration stop#

Stop a migration from transferring content to its target, placing it into the STOPPED state. Stopped migrations can be resumed.

Stop a migration
SYNOPSYS        migration stop [--name] string

Mandatory parameters#

  • --name The migration name or ID to stop.

Example#

migration stop --name 4ffa620b6ebb0cd34f2c591220d93830f91ccc7e

migration resume#

Resume a migration that you've stopped from transferring content to its target.

Resume a migration
SYNOPSYS        migration resume [--name] string

Mandatory parameters#

  • --name The migration name or ID to resume.

Example#

migration resume --name 4ffa620b6ebb0cd34f2c591220d93830f91ccc7e

migration reset#

Reset a stopped migration to the state it was in before it was started. This deletes and replaces it with a new migration that has the same settings as the old one.

Reset a migration
SYNOPSYS    migration reset [--name] string  

Mandatory parameters#

  • --name The name or ID of the migration you want to reset.

Optional parameters#

  • --action-policy Accepts two string values: com.wandisco.livemigrator2.migration.OverwriteActionPolicy causes the new migration to re-migrate all files from scratch, including those already migrated to the target filesystem, regardless of file size. com.wandisco.livemigrator2.migration.SkipIfSizeMatchActionPolicy skips migrating files that exist on both the target and source, if the file size is consistent between them. Use tab auto-completion with this parameter to view both options and a short description of each.
  • --reload-mappings Resets the migration's path mapping configuration, using the newest default path mapping configuration for LiveData Migrator.
  • --detailed or --verbose Returns additional information about the reset migration, similarly to migration show.

Example#

migration reset --name mymigration

migration delete#

Delete a stopped migration resource.

Delete a migration
SYNOPSYS        migration delete [--name] string

Mandatory parameters#

  • --name The migration name or ID to delete.

Example#

migration delete --name 4ffa620b6ebb0cd34f2c591220d93830f91ccc7e

migration exclusion add#

Associate an exclusion resource with a migration so that the exclusion policy applies to items processed for the migration. Exclusions must be associated with a migration before they take effect.

Add an exclusion to a migration
SYNOPSYS        migration exclusion add [--name] string                                [--exclusion-id] string

Mandatory parameters#

  • --name The migration name or ID with which to associate the exclusion.
  • --exclusion-id The ID of the exclusion to associate with the migration. In the UI, this is called Name.

Example#

migration exclusion add --name 4ffa620b6ebb0cd34f2c591220d93830f91ccc7e --exclusion-id myexclusion1

migration exclusion del#

Remove an exclusion from association with a migration so that its policy no longer applies to items processed for the migration.

Remove an exclusion from a migration
SYNOPSYS        migration exclusion delete [--name] string                                [--exclusion-id] string

Mandatory parameters#

  • --name The migration name or ID from which to remove the exclusion.
  • --exclusion-id The ID of the exclusion to remove from the migration. In the UI, this is called Name.

Example#

migration exclusion delete --name 4ffa620b6ebb0cd34f2c591220d93830f91ccc7e --exclusion-id myexclusion1

``

Present the list of all migrations defined.

List running and active migrations
SYNOPSYS        migration list

migration add#

Create a new migration to initiate data migration from your source filesystem.

caution

Do not write to target filesystem paths when a migration is underway. This could interfere with LiveData Migrator functionality and lead to undetermined behavior.

Use different filesystem paths when writing to the target filesystem directly (and not through LiveData Migrator).

Create a new migration
SYNOPSYS        migration add [[] string]                      [--path] string                      [--target] string                      [[--exclusions] string]                      [[--action-policy] string]                      [--auto-start]

Mandatory parameters#

  • --path Defines the source filesystem directory that is the scope of the migration. All content (other than that excluded) will be migrated to the target. In the UI, this is called Path for {source-filesystem}.

    note

    ADLS Gen2 has a filesystem restriction of 60 segments. Make sure your path has less than 60 segments when defining the path string parameter.

  • --target Specifies the name of the target filesystem resource to which migration will occur. In the UI, this is called Target.

Optional parameters#

  • --name Provide a name or ID for the new migration. An ID will be auto-generated if one isn't provided. In the UI, this is called Migration Name.
  • --exclusions A comma-separated list of exclusions by name. In the UI, this is called Add new exclusion.
  • --auto-start Provide this parameter if you want the migration to start immediately. If not provided, the migration will only take effect once run. In the UI, this is called Auto-start migration.
  • --action-policy This parameter determines what happens if the migration encounters content in the target path with the same name and size. In the UI, this is called Skip Or Overwrite Settings.
    There are two options available:
    1. com.wandisco.livemigrator2.migration.OverwriteActionPolicy (default policy)
      Every file is replaced, even if file size is identical on the target storage. In the UI, this is called Overwrite.
    2. com.wandisco.livemigrator2.migration.SkipIfSizeMatchActionPolicy
      If the file size is identical between the source and target, the file is skipped. If it’s a different size, the whole file is replaced. In the UI, this is called Skip if Size Match.

Example#

migration add --path /repl1 --target mytarget –-migration-id myNewMigration --exclusions 100mbfiles

migration start#

Start a migration that was created without the --auto-start parameter.

Start a migration
SYNOPSYS        migration start [--name] string

Mandatory parameters#

  • --name The migration name or ID to run.

Example#

migration start –-migration-id myNewMigration

migration show#

Provide a JSON description of a specific migration.

Get migration details
SYNOPSYS        migration show [--name] string

Mandatory parameters#

  • --name The migration name or ID to show.

Optional parameters#

  • --detailed Outputs additional information about the migration.

Example#

migration show --name myNewMigration

migration pending-region add#

Add a pending region to a migration.

Add pending region for rescan to migration
SYNOPSYS        migration pending-region add [--name] string [--path] string

Mandatory parameters#

  • --name The migration name or ID to add a pending region to.
  • --path The path string of the region to add for rescan.

Example#

migration pending-region add --name myMigration --path etc/files

migration verification add#

Add a migration verification for a specified migration. This will scan your source and target filesystems (in the migration path) and compare them for any discrepancies in either file sizes or in missing files.

The verification status will show the number of missing paths and files on the target filesystem and also the number of file size mismatches between the source and target. The verification status can be viewed by using migration verification show (for individual verification jobs) or migration verification list (for all verification jobs).

Once a verification job is complete, a verification report will be created in the /var/log/wandisco/livedata-migrator directory in the format of verification-report-{verificationId}-{startTime}.log. This report will contain more details including any paths that have discrepancies.

See migration verifications for more details.

Verify a migration
SYNOPSYS        migration verification add [--name] string                                   [--override]

Mandatory parameters#

  • --name The migration name or ID to start (or override) a verification on.

Optional parameters#

  • --override Stop the currently running verification and start a new one.

Examples#

Start a verification job
migration verification add --name myMigration
Stop the running verification and start a new one
migration verification add --name myMigration --override

migration verification list#

List all running migration verification jobs and their statuses (use migration verification show when just wanting the status for one verification job).

List all verification jobs
SYNOPSYS        migration verification list

migration verification show#

Show the status of a specific migration verification.

Show a verification job for a migration
SYNOPSYS        migration verification show [--name] string

Mandatory parameters#

  • --name Show the status of the current verification job running on this migration name or ID (only one verification job can be running per migration).

Example#

See verification status values for further explanation of the output.

Example status of a completed verification
WANdisco LiveData Migrator >> migration verification show --name testmig{  "migrationId" : "testmig",  "state" : "COMPLETED",  "verificationId" : "e1aedfbd-b094-4a1b-a294-69cdd5a6030a",  "verificationPath" : "/testdir",  "startTime" : "2021-04-29T13:27:44.278Z",  "completeTime" : "2021-04-29T13:27:45.392Z",  "verificationEdge" : "/testmig/testdir01/testfile01",  "scannerSummary" : {    "progressSummary" : {      "filesScanned" : 177,      "directoriesScanned" : 47,      "bytesScanned" : 1105391944,      "filesExcluded" : 51,      "dirsExcluded" : 0,      "bytesExcluded" : 0,      "baseScanCompletionTime" : "2021-04-29T13:27:45.392Z"    },    "contentSummary" : {      "byteCount" : 1105391944,      "fileCount" : 194,      "directoryCount" : 81    }  },  "verificationProgress" : {    "matchedPathCount" : 224,    "totalFailedPathCount" : 0,    "targetFilesMissing" : 0,    "targetDirectoriesMissing" : 0,    "filesizeMismatches" : 0  }}

status#

Get a text description of the overall status of migrations. Information is provided on the following:

  • Total number of migrations defined.
  • Average bandwidth being used over 10s, 60s, and 300s intervals.
  • Peak bandwidth observed over 300s interval.
  • Average file transfer rate per second over 10s, 60s, and 300s intervals.
  • Peak file transfer rate per second over a 300s interval.
  • List of migrations, including one-time migrations, with source path and migration id, and with current progress broken down by migration state: completed, live, stopped, running and ready.
Get migration status
NAME        status - Get migration status.
SYNOPSYS        status

Optional parameters#

  • --transfers Displays overall performance information about data transfers across the last 10 seconds, 1 minute and 30 minute intervals.
  • --diagnostics Returns additional information about your LiveData Migrator instance and its migrations, useful for troubleshooting.
  • --migrations Displays information about each running migration.
  • --network Displays file transfer throughput in Gib/s during the last 10 seconds, 1 minute and 30 minutes.

Examples#

Status
WANdisco LiveMigrator >> status
Network             (10s)       (1m)       (30m)---------Average Throughput: 10.4 Gib/s  9.7 Gib/s  10.1 Gib/sAverage Files/s:    425         412        403
11 Migrations                                     dd:hh:mm  dd:hh:mm-------------Complete: 1         Transferred         Excluded  Duration /static1   5a93d5        67.1 GiB       2.3 GiB  00:12:34
Live:     3         Transferred         Excluded  Duration /repl1     9088aa       143.2 GiB      17.3 GiB  00:00:34 /repl_psm1 a4a7e6       423.6 TiB       9.6 GiB  02:05:29 /repl5     ab140d       118.9 GiB       1.2 GiB  00:00:34
Running:  5         Transferred         Excluded  Duration  Remaining /repl123   e3727c  30.3/45.2 GiB 67%    9.8 GiB  00:00:34  00:00:17 /repl2     88e4e7  26.2/32.4 GiB 81%    0.2 GiB  00:01:27  00:00:12 /repl3     372056   4.1/12.5 GiB 33%    1.1 GiB  00:00:25  00:01:05 /repl4     6bc813  10.6/81.7 TiB  8%   12.4 GiB  00:04:21  01:02:43 /replxyz   dc33cb   2.5/41.1 GiB  6%    6.5 GiB  01:00:12  07:34:23
Ready:    2 /repl7     070910  543.2 GiB /repltest  d05ca0  7.3 GiB
WANdisco LiveMigrator >> status
Status with --transfers
WANdisco LiveMigrator >> status --transfers
Files (10s) (1m) (30m)
Average Migrated/s: 362 158 4781< 1 KB 14 27 3761< 1 MB 151 82 0< 1 GB 27 1 2< 1 PB 0 0 0< 1 EB 0 0 0
Peak Migrated/s: 505 161 8712< 1 KB 125 48 7761< 1 MB 251 95 4< 1 GB 29 7 3< 1 PB 0 0 0< 1 EB 0 0 0
Average Scanned/s: 550 561 467Average Rescanned/s: 24 45 56Average Excluded/s: 7 7 6
Status with --diagnostics
WANdisco LiveMigrator >> status --diagnostics
Uptime: 0 Days 1 Hours 23 Minutes 24 SecondsSystemCpuLoad: 0.1433 ProcessCpuLoad: 0.0081JVM GcCount: 192 GcPauseTime: 36 s (36328 ms)OS Connections: 1, Tx: 0 B, Rx: 0 B, Retransmit: 0Transfer Bytes (10/30/300s): 0.00 Gib/s, 0.00 Gib/s, 0.00 Gib/sTransfer Files (10/30/300s): 0.00/s 0.00/s 0.00/sActive Transfers/pull.threads: 0/100Migrations: 0 RUNNING, 4 LIVE, 0 STOPPEDActions Total: 0, Largest: "testmigration" 0, Peak: "MyMigration" 1PendingRegions Total: 0 Avg: 0, Largest: "MyMigration" 0FailedPaths Total: 0, Largest: "MyMigration" 0File Transfer Retries Total: 0, Largest: "MyMigration" 0Total Excluded Scan files/dirs/bytes: 26, 0, 8.1 MBTotal Iterated Scan files/dirs/bytes: 20082, 9876, 2.7 GBEventsBehind Current/Avg/Max: 0/0/0, RPC Time Avg/Max: 4/8EventsQueued: 0, Total Events Added: 504Transferred File Size Percentiles: 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 BTransferred File Transfer Rates Percentiles per Second: 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 B, 2 BActive File Size Percentiles: 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 BActive File Transfer Rates Percentiles per Second: 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 B, 0 B

Bandwidth limit commands#


bandwidth policy del#

Delete the current bandwidth limit and revert to unlimited bandwidth.

Allow the application to use unlimited bandwidth
SYNOPSYS        bandwidth policy del

bandwidth policy set#

Set the bandwidth limit that will determine how much bandwidth LiveData Migrator can use.

If no limit is defined, the default setting is unlimited bandwidth.

Set the application bandwidth limit, in bytes per second
SYNOPSYS        bandwidth policy set [--value] long  [--unit] string

Mandatory parameters#

  • --value Define the number of byte units.
  • --unit Define the byte unit to be used.
    Decimal units: KB, MB, GB, TB, PB, EB, ZB, YB
    Binary units: KiB, MiB, GiB, TiB, PiB, EiB, ZiB, YiB

Examples#

Set a limit of 10 Megabytes per second
bandwidth policy set --value 10 --unit MB
Set a limit of 10 Mebibytes per second
bandwidth policy set --value 10 --unit MiB

bandwidth policy show#

Display the current bandwidth limit.

Get details of the application bandwidth limit, in bytes per second
SYNOPSYS        bandwidth policy show

Hive agent commands#


hive agent add azure#

Add a local or remote Hive agent to connect to an Azure SQL database using the hive agent add azure command.

If your LiveData Migrator host can communicate directly with the Azure SQL database, then a local Hive agent will be sufficient. Otherwise, consider using a remote Hive agent.

remote deployments

For a remote Hive agent connection, specify a remote host (Azure VM, HDI cluster node) that will be used to communicate with the local LiveData Migrator server (constrained to a user-defined port).

A small service will be deployed on this remote host so that the data can transfer between the Hive agent and the remote Metastore.

Add Azure SQL agent
SYNOPSYS        hive agent add azure [[--name] string]                             [--db-server-name] string                             [--database-name] string                             [--database-user] string                             [--database-password] string                             [--storage-account] string                             [--container-name] string                             [[--root-folder] string]                             [[--hdi-version] string]                             [[--insecure] boolean]                             [[--host] string]                             [[--port] integer]                             [--no-ssl]                             [[--autodeploy] boolean]                             [[--ssh-user] string]                             [[--ssh-key] file]                             [[--ssh-port] int]                             [--use-sudo]                             [--ignore-host-checking]                             [[--file-system-id] string]                             [[--default-fs-override] string]

Mandatory parameters#

info

The Azure Hive agent requires a ADLS Gen2 storage account and container name, this is only for the purposes of generating the correct location for the metadata. The agent will not access the container and data will not be written to it.

  • --database-name The Azure SQL database name. In the UI, this is called Azure SQL Database Name.
  • --storage-account The name of the ADLS Gen2 storage account. In the UI, this is called ADLS Gen2 Storage Account Name.
  • --container-name The name of the container in the ADLS Gen2 storage account. In the UI, this is called ADLS Gen2 Container Name.
  • --hdi-version The HDI version. This is relevant if you are intending to integrate your SQL server into a HDInsights cluster. In the UI, this is called HDI Version.
  • --name The ID to give to the new Hive agent. In the UI, this is called Name.

Additionally, use only one of the following parameters:

  • --file-system-id The name of the filesystem that will be associated with this agent (for example: myadls2storage). This will ensure any path mappings are correctly linked between the filesystem and the agent. In the UI, this is called Filesystem.
  • --default-fs-override Provide an override for the default filesystem URI instead of a filesystem name (for example: abfss://mycontainer@mystorageaccount.dfs.core.windows.net). In the UI, this is called DefaultFs Override.

Optional parameters#

  • --root-folder The root directory for the Azure Metastore, this is used if you're intending to integrate the Azure SQL Database with a HDI cluster. In the UI, this is called Root Folder.
  • --insecure Define an insecure connection (TLS disabled) to the Azure SQL database server (default is false). In the UI, this is called Use Secure Protocol.

Authentication parameters#

Choose one of the authentication methods listed and include the additional parameters required for the chosen method.

  • --auth-method The authentication method to use to connect to the Azure SQL server. In the UI, this is called Authentication Method.
    The following methods can be used:
    • SQL_PASSWORD - Provide a username and password to access the database. In the UI, this is called SQL Password.
    • AD_MSI - Use a system-assigned or user-assigned managed identity. In the UI, this is called Active Directory MSI.
Required parameters for SQL_PASSWORD#
  • --database-user The user name to access the database. In the UI, this is called Database Username.
  • --database-password The user password to access the database. In the UI, this is called Database Password.
Required parameters for AD_MSI#

To use this method, the following pre-requirements must be met:

  • LiveData Migrator or the remote Azure Hive agent must be installed on an Azure resource with the managed identity assigned to it. The host must also have Azure AD authentication enabled.

  • Your Azure SQL server must be enabled for Azure AD authentication.

  • You have created a contained user in the Azure SQL database that is mapped to the Azure AD resource (where LiveData Migrator or the remote Azure Hive agent is installed).

    • The username of the contained user will depend on whether you are using a system-assigned or user-assigned identity.

      Azure SQL database command for a system-assigned managed identity
      CREATE USER "<azure_resource_name>" FROM EXTERNAL PROVIDER;ALTER ROLE db_owner ADD MEMBER "<azure_resource_name>";

      The <azure_resource_name> is the name of the Azure resource where LiveData Migrator or remote Azure Hive agent is installed (for example: myAzureVM).

      Azure SQL database command for a user-assigned managed identity
      CREATE USER <managed_identity_name> FROM EXTERNAL PROVIDER;ALTER ROLE db_owner ADD MEMBER <managed_identity_name>;

      The <managed_identity_name> is the name of the user-assigned managed identity (for example: myManagedIdentity).

Once all pre-requirements are met, see the system-assigned identity or user-assigned identity parameters.

System-assigned identity#

No other parameters are required for a system-managed identity.

User-assigned identity#

The --client-id parameter must be specified:

  • --client-id The Client ID of your Azure managed identity. In the UI, this is called MSI Client ID.

Parameters for remote Hive agents only#

  • --host The host where the remote Hive agent will be deployed.
  • --port The port for the remote Hive agent to use on the remote host. This port is used to communicate with the local LiveData Migrator server.
  • --no-ssl TLS encryption and certificate authentication is enabled by default between LiveData Migrator and the remote agent. Use this parameter to disable it.
Parameters for automated deployment#
  • --autodeploy The remote agent will be automatically deployed when this flag is used. If using this, the --ssh-key parameter must also be specified.
  • --ssh-user The SSH user to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter).
  • --ssh-key The absolute path to the SSH private key to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter).
  • --ssh-port The SSH port to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter). Default is port 22.
  • --use-sudo All commands performed by the SSH user will use sudo on the remote host when performing automatic deployment (using the --autodeploy parameter).
  • --ignore-host-checking Ignore strict host key checking when performing the automatic deployment (using the --autodeploy parameter).
Steps for manual deployment#

If you do not wish to use the --autodeploy function, follow these steps to deploy a remote Hive agent for Azure SQL manually:

  1. Transfer the remote server installer to your remote host (Azure VM, HDI cluster node):

    Example of secure transfer from local to remote host
    scp /opt/wandisco/hivemigrator/hivemigrator-remote-server-installer.sh myRemoteHost:~
  2. On your remote host, run the installer as root (or sudo) user in silent mode:

    ./hivemigrator-remote-server-installer.sh -- --silent
  3. On your remote host, start the remote server service:

    service hivemigrator-remote-server start
  4. On your local host, run the hive agent add azure command without using --autodeploy and its related parameters to configure your remote Hive agent.

    See the Example for remote Azure SQL deployment - manual example below for further guidance.

Examples#

Example for local Azure SQL deployment with SQL username/password
hive agent add azure --name azureAgent --db-server-name mysqlserver.database.windows.net --database-name mydb1 --auth-method SQL_PASSWORD --database-user azureuser --database-password mypassword --storage-account myadls2 --container-name mycontainer --root-folder /hive/warehouse --hdi-version 3.6 --file-system-id myadls2storage
Example for remote Azure SQL deployment with System-assigned managed identity - automated
hive agent add azure --name azureRemoteAgent --db-server-name mysqlserver.database.windows.net --database-name mydb1 --auth-method AD_MSI --storage-account myadls2 --container-name mycontainer --root-folder /hive/warehouse --hdi-version 3.6 --file-system-id myadls2storage --autodeploy --ssh-user root --ssh-key /root/.ssh/id_rsa --ssh-port 22 --host myRemoteHost.example.com --port 5552
Example for remote Azure SQL deployment with User-assigned managed identity - manual
hive agent add azure --name azureRemoteAgent --db-server-name mysqlserver.database.windows.net --database-name mydb1 --auth-method AD_MSI --client-id b67f67ex-ampl-e2eb-bd6d-client9385id --storage-account myadls2 --container-name mycontainer --root-folder /hive/warehouse --hdi-version 3.6 --file-system-id myadls2storage --host myRemoteHost.example.com --port 5552

hive agent add filesystem#

Add a filesystem Hive agent to connect to your host's local filesystem using the hive agent add filesystem command.

Add filesystem agent
SYNOPSYS        hive agent add filesystem [--filesystem-id] string                                  [--root-folder] string                                  [[--name] string]

Mandatory parameters#

  • --filesystem-id The filesystem ID to be used.
  • --root-folder The path to use as the root directory for the filesystem agent.

Optional parameters#

  • --name The ID to give to the new Hive agent.

Example#

hive agent add filesystem --filesystem-id myfilesystem --root-folder /var/lib/mysql --name fsAgent

hive agent add glue#

Add an AWS Glue Hive agent to connect to an AWS Glue data catalog using the hive agent add glue command.

If your LiveData Migrator host can communicate directly with the AWS Glue Data Catalog, then a local Hive agent will be sufficient. Otherwise, consider using a remote Hive agent.

remote deployments

For a remote Hive agent connection, specify a remote host (EC2 instance) that will be used to communicate with the local LiveData Migrator server (constrained to a user-defined port).

A small service will be deployed on this remote host so that the data can transfer between the Hive agent and the remote Metastore.

Add AWS Glue agent
SYNOPSYS        hive agent add glue [[--name] string]                            [[--access-key] string]                            [[--secret-key] string]                            [[--glue-endpoint] string]                            [[--aws-region] string]                            [[--glue-catalog-id] string]                            [[--credentials-provider] string]                            [[--glue-max-retries] integer]                            [[--glue-max-connections] integer]                            [[--glue-max-socket-timeout] integer]                            [[--glue-connection-timeout] integer]                            [[--file-system-id] string]                            [[--default-fs-override] string]                            [[--host] string]                            [[--port] integer]                            [--no-ssl]

Glue parameters#

  • --name The ID to give to the new Hive agent. In the UI, this is called Name.
  • --glue-endpoint The AWS Glue service endpoint for connections to the data catalog. VPC endpoint types are also supported. In the UI, this is called AWS Glue Service Endpoint.
  • --aws-region The AWS region that your data catalog is located in (default is us-east-1). If --glue-endpoint is specified, this parameter will be ignored. In the UI, this is called AWS Region.

Additionally, use only one of the following parameters:

  • --file-system-id The name of the filesystem that will be associated with this agent (for example: mys3bucket). This will ensure any path mappings are correctly linked between the filesystem and the agent. In the UI, this is called Filesystem.
  • --default-fs-override Provide an override for the default filesystem URI instead of a filesystem name (for example: s3a://mybucket/). In the UI, this is called DefaultFs Override.

Glue credential parameters#

Glue optional parameters#

  • --glue-catalog-id The AWS Account ID to access the Data Catalog. This is used if the Data Catalog is owned by a different account to the one provided by the credentials provider and cross-account access has been granted.
  • --glue-max-retries The maximum number of retries the Glue client will perform after an error.
  • --glue-max-connections The maximum number of parallel connections the Glue client will allocate.
  • --glue-max-socket-timeout The maximum time the Glue client will allow for an established connection to timeout.
  • --glue-connection-timeout The maximum time the Glue client will allow to establish a connection.

Parameters for remote Hive agents only#

  • --host The host where the remote Hive agent will be deployed.
  • --port The port for the remote Hive agent to use on the remote host. This port is used to communicate with the local LiveData Migrator server.
  • --no-ssl TLS encryption and certificate authentication is enabled by default between LiveData Migrator and the remote agent. Use this parameter to disable it.
Steps for remote agent deployment#

Follow these steps to deploy a remote Hive agent for AWS Glue:

  1. Transfer the remote server installer to your remote host (Amazon EC2 instance):

    Example of secure transfer from local to remote host
    scp /opt/wandisco/hivemigrator/hivemigrator-remote-server-installer.sh myRemoteHost:~
  2. On your remote host, run the installer as root (or sudo) user in silent mode:

    ./hivemigrator-remote-server-installer.sh -- --silent
  3. On your remote host, start the remote server service:

    service hivemigrator-remote-server start
  4. On your local host, run the hive agent add glue command to configure your remote Hive agent.

    See the Example for remote AWS Glue agent example below for further guidance.

Examples#

Example for local AWS Glue agent
hive agent add glue --name glueAgent --access-key ACCESS6HCFPAQIVZTKEY --secret-key SECRET1vTMuqKOIuhET0HAI78UIPfSRjcswTKEY --glue-endpoint glue.eu-west-1.amazonaws.com --aws-region eu-west-1 --file-system-id mys3bucket
Example for remote AWS Glue agent
hive agent add glue --name glueAgent --access-key ACCESS6HCFPAQIVZTKEY --secret-key SECRET1vTMuqKOIuhET0HAI78UIPfSRjcswTKEY --glue-endpoint glue.eu-west-1.amazonaws.com --aws-region eu-west-1 --file-system-id mys3bucket --host myRemoteHost.example.com --port 5552

hive agent add hive#

Add a Hive agent to connect to a local or remote Apache Hive Metastore using the hive agent add hive command.

remote deployments

When connecting to a remote Apache Hive Metastore, specify a host on the remote cluster that will be used to communicate with the local LiveData Migrator server (constrained to a user-defined port).

A small service will be deployed on this remote host so that the data can transfer between the Hive agent and the remote Metastore.

Add local or remote Hive agent
SYNOPSYS        hive agent add hive [[--config-path] string]                            [[--kerberos-principal] string]                            [[--kerberos-keytab] string]                            [[--name] string]                            [[--host] string]                            [[--port] integer]                            [--no-ssl]                            [--autodeploy]                            [[--ssh-user] string]                            [[--ssh-key] file]                            [[--ssh-port] int]                            [--use-sudo]                            [--ignore-host-checking]                            [[--file-system-id] string]                            [[--default-fs-override] string]

Mandatory parameters#

  • --kerberos-principal Not required if Kerberos is disabled. The Kerberos principal to use to access the Hive service (for example: hive/myhost.example.com@REALM.COM). In the UI, this is called Principal.
  • --kerberos-keytab Not required if Kerberos is disabled. The path to the Kerberos keytab containing the principal to access the Hive service (for example: /etc/security/keytabs/hive.service.keytab). In the UI, this is called Keytab.
  • --name The ID to give to the new Hive agent. In the UI, this is called Name.

Additionally, use only one of the following parameters:

  • --file-system-id The name of the filesystem that will be associated with this agent (for example: myhdfs). This will ensure any path mappings are correctly linked between the filesystem and the agent. In the UI, this is called Filesystem.
  • --default-fs-override Provide an override for the default filesystem URI instead of a filesystem name (for example: hdfs://nameservice01). In the UI, this is called DefaultFs Override.

Optional parameters#

  • --config-path The path to the directory containing the Hive configuration files. If not specified, LiveData Migrator will use the default location for the cluster distribution. In the UI, this is called Override Default Hadoop Configuration Path.

Parameters for remote Hive agents only#

  • --host The host where the remote Hive agent will be deployed.
  • --port The port for the remote Hive agent to use on the remote host. This port is used to communicate with the local LiveData Migrator server.
  • --no-ssl TLS encryption and certificate authentication is enabled by default between LiveData Migrator and the remote agent. Use this parameter to disable it.
Parameters for automated deployment#
  • --autodeploy The remote agent will be automatically deployed when this flag is used. If using this, the --ssh-key parameter must also be specified.
  • --ssh-user The SSH user to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter).
  • --ssh-key The absolute path to the SSH private key to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter).
  • --ssh-port The SSH port to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter). Default is port 22.
  • --use-sudo All commands performed by the SSH user will use sudo on the remote host when performing automatic deployment (using the --autodeploy parameter).
  • --ignore-host-checking Ignore strict host key checking when performing the automatic deployment (using the --autodeploy parameter).
Steps for manual deployment#

If you do not wish to use the --autodeploy function, follow these steps to deploy a remote Hive agent for Apache Hive manually:

  1. Transfer the remote server installer to your remote host:

    Example of secure transfer from local to remote host
    scp /opt/wandisco/hivemigrator/hivemigrator-remote-server-installer.sh myRemoteHost:~
  2. On your remote host, run the installer as root (or sudo) user in silent mode:

    ./hivemigrator-remote-server-installer.sh -- --silent
  3. On your remote host, start the remote server service:

    service hivemigrator-remote-server start
  4. On your local host, run the hive agent add hive command without using --autodeploy and its related parameters to configure your remote Hive agent.

    See the Example for remote Apache Hive deployment - manual example below for further guidance.

Examples#

Example for local Apache Hive deployment
hive agent add hive --name sourceAgent --kerberos-keytab /etc/security/keytabs/hive.service.keytab --kerberos-principal hive/_HOST@LOCALREALM.COM --file-system-id mysourcehdfs
Example for remote Apache Hive deployment - automated
hive agent add hive --name targetautoAgent --autodeploy --ssh-user root --ssh-key /root/.ssh/id_rsa --ssh-port 22 --host myRemoteHost.example.com --port 5552 --kerberos-keytab /etc/security/keytabs/hive.service.keytab --kerberos-principal hive/_HOST@REMOTEREALM.COM --config-path <example directory path> --file-system-id mytargethdfs

Replace <example directory path> with the path to a directory containing the core-site.xml, hdfs-site.xml, and hive-site.xml.

Example for remote Apache Hive deployment - manual
hive agent add hive --name targetmanualAgent --host myRemoteHost.example.com --port 5552 --kerberos-keytab /etc/security/keytabs/hive.service.keytab --kerberos-principal hive/_HOST@REMOTEREALM.COM --config-path <example directory path> --file-system-id mytargethdfs

Replace <example directory path> with the path to a directory containing the core-site.xml, hdfs-site.xml, and hive-site.xml.

If specifying Kerberos and config path information for remote agents, ensure that the directories and Kerberos principal are correct for your chosen remote host (not your local host). :::


hive agent add databricks#

note

Databricks agents are currently available as a preview feature.

info

The source table format must be Parquet to ensure a successful migration to Databricks Delta Lake.

Add a Databricks Hive agent to connect to a Databricks Delta Lake metastore (AWS, Azure or GCP) using the hive agent add databricks command.

Add Databricks agent
SYNOPSYS        hive agent add databricks [[--name] string]                                  [--jdbc-server-hostname] string                                  [--jdbc-port] int                                  [--jdbc-http-path] string                                  [--access-token] string                                  [[--fs-mount-point] string]                                  [--convert-to-delta]                                  [--delete-after-conversion]                                  [[--file-system-id] string]                                  [[--default-fs-override] string]                                  [[--host] string]                                  [[--port] integer]                                  [--no-ssl]

Enable JDBC connections to Databricks#

The following steps are required to enable Java Database Connectivity (JDBC) to Databricks Delta Lake:

  1. Download the Databricks JDBC driver.

  2. Unzip the package and upload the SparkJDBC42.jar file to the LiveData Migrator host machine.

  3. Move the SparkJDBC42.jar file to the LiveData Migrator directory below:

    /opt/wandisco/hivemigrator/agent/databricks
  4. Change ownership of the Jar file to the Hive Migrator system user and group:

    Example for hive:hadoop
    chown hive:hadoop /opt/wandisco/hivemigrator/agent/databricks/SparkJDBC42.jar

Databricks mandatory parameters#

  • --name The ID to give to the new Hive agent. In the UI, this is called Name.
  • --jdbc-server-hostname The server hostname for the Databricks cluster (AWS, Azure or GCP). In the UI, this is called JDBC Server Hostname.
  • --jdbc-port The port used for JDBC connections to the Databricks cluster (AWS, Azure or GCP). In the UI, this is called JDBC Port.
  • --jdbc-http-path The HTTP path for the Databricks cluster (AWS, Azure or GCP). In the UI, this is called JDBC Http Path.
  • --access-token The personal access token to be used for the Databricks cluster (AWS, Azure or GCP). In the UI, this is called Access Token.

Additionally, use only one of the following parameters:

important

If the --convert-to-delta option is used, the --default-fs-override parameter must also be provided with the value set to dbfs:, or a path inside the Databricks filesystem. For example, dbfs:/mount/externalStorage.

  • --file-system-id The name of the filesystem that will be associated with this agent (for example: myadls2 or mys3bucket). This will ensure any path mappings are correctly linked between the filesystem and the agent. In the UI, this is called Filesystem.
  • --default-fs-override Provide an override for the default filesystem URI instead of a filesystem name (for example: dbfs:). In the UI, this is called DefaultFs Override.

Databricks optional parameters#

  • --fs-mount-point Define the ADLS/S3/GCP location in the Databricks filesystem for containing migrations (for example: /mnt/mybucketname). In the UI, this is called FS Mount Point.

    note

    This parameter is required if --convert-to-delta is used. The Databricks agent will copy all associated table data and metadata into this location within the Databricks filesystem during conversion.

  • --convert-to-delta All underlying table data and metadata is migrated to the filesystem location defined by the --fs-mount-point parameter. Use this option to automatically copy the associated data and metadata into Delta Lake on Databricks (AWS, Azure or GCP), and convert tables into Delta Lake format. In the UI, this is called Convert to delta format.

    The following parameter can only be used if --convert-to-delta has been specified:

    • --delete-after-conversion Use this option to delete the underlying table data and metadata from the filesystem location (defined by --fs-mount-point) once it has been converted into Delta Lake on Databricks. In the UI, this is called Delete after conversion.

      important

      Only use this option if you are performing one-time migrations for the underlying table data. The Databricks agent does not support continuous (live) updates of table data when transferring to Delta Lake on Databricks.

  • If a migration to Databricks runs without the --convert-to-delta option, then some migrated data may not be visible from the Databricks side. To avoid this issue, ensure that the value of default-fs-override is set to "dbfs:" with the value of --fs-mount-point.

    Example:

    --default-fs-override dbfs:/mnt/mybucketname    

Example#

Example for local Databricks agent
hive agent add databricks --name databricksAgent --jdbc-server-hostname mydbcluster.cloud.databricks.com  --jdbc-port 443 --jdbc-http-path sql/protocolv1/o/8445611123456789/0234-125567-testy978 --access-token daexamplefg123456789t6f0b57dfdtoken4 --file-system-id mys3bucket --default-fs-override dbfs: --fs-mount-point /mnt/mybucket --convert-to-delta

hive agent add dataproc#

Add a Hive agent to connect to a local or remote Google Dataproc Metastore using the hive agent add dataproc command.

remote deployments

When connecting to a remote Dataproc Metastore, specify a host on the remote cluster that will be used to communicate with the local LiveData Migrator server (constrained to a user-defined port).

A small service will be deployed on this remote host so that the data can transfer between the Hive agent and the remote Metastore.

Add local or remote Dataproc agent
SYNOPSYS        hive agent add dataproc [[--config-path] string]                            [[--kerberos-principal] string]                            [[--kerberos-keytab] string]                            [[--name] string]                            [[--host] string]                            [[--port] integer]                            [--no-ssl]                            [--autodeploy]                            [[--ssh-user] string]                            [[--ssh-key] file]                            [[--ssh-port] int]                            [--use-sudo]                            [--ignore-host-checking]                            [[--file-system-id] string]                            [[--default-fs-override] string]

Mandatory parameters#

  • --kerberos-principal Not required if Kerberos is disabled. The Kerberos principal to use to access the Hive service (for example: hive/myhost.example.com@REALM.COM). In the UI, this is called Principal.
  • --kerberos-keytab Not required if Kerberos is disabled. The path to the Kerberos keytab containing the principal to access the Hive service (for example: /etc/security/keytabs/hive.service.keytab). In the UI, this is called Keytab.
  • --name The ID to give to the new Hive agent. In the UI, this is called Name.

Additionally, use only one of the following parameters:

  • --file-system-id The name of the filesystem that will be associated with this agent (for example: myhdfs). This will ensure any path mappings are correctly linked between the filesystem and the agent. In the UI, this is called Filesystem.
  • --default-fs-override Provide an override for the default filesystem URI instead of a filesystem name (for example: hdfs://nameservice01). In the UI, this is called DefaultFs Override.

Optional parameters#

  • --config-path The path to the directory containing the Hive configuration files. If not specified, LiveData Migrator will use the default location for the cluster distribution. In the UI, this is called Override Default Hadoop Configuration Path.

Parameters for remote Hive agents only#

  • --host The host where the remote Hive agent will be deployed.
  • --port The port for the remote Hive agent to use on the remote host. This port is used to communicate with the local LiveData Migrator server.
  • --no-ssl TLS encryption and certificate authentication is enabled by default between LiveData Migrator and the remote agent. Use this parameter to disable it.
Parameters for automated deployment#
  • --autodeploy The remote agent will be automatically deployed when this flag is used. If using this, the --ssh-key parameter must also be specified.
  • --ssh-user The SSH user to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter).
  • --ssh-key The absolute path to the SSH private key to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter).
  • --ssh-port The SSH port to use for authentication on the remote host to perform automatic deployment (when using the --autodeploy parameter). Default is port 22.
  • --use-sudo All commands performed by the SSH user will use sudo on the remote host when performing automatic deployment (using the --autodeploy parameter).
  • --ignore-host-checking Ignore strict host key checking when performing the automatic deployment (using the --autodeploy parameter).
Steps for manual deployment#

If you do not wish to use the --autodeploy function, follow these steps to deploy a remote Hive agent for Apache Hive manually:

  1. Transfer the remote server installer to your remote host:

    Example of secure transfer from local to remote host
    scp /opt/wandisco/hivemigrator/hivemigrator-remote-server-installer.sh myRemoteHost:~
  2. On your remote host, run the installer as root (or sudo) user in silent mode:

    ./hivemigrator-remote-server-installer.sh -- --silent
  3. On your remote host, start the remote server service:

    service hivemigrator-remote-server start
  4. On your local host, run the hive agent add dataproc command without using --autodeploy and its related parameters to configure your remote Hive agent.

    See the Example for remote Apache Hive deployment - manual example below for further guidance.

Examples#

Example for local Apache Hive deployment
hive agent add dataproc --name sourceAgent --file-system-id mysourcehdfs
Example for remote Apache Hive deployment - automated
hive agent add dataproc --name targetautoAgent --autodeploy --ssh-user root --ssh-key /root/.ssh/id_rsa --ssh-port 22 --host myRemoteHost.example.com --port 5552 --config-path <example directory path> --file-system-id mytargethdfs

Replace <example directory path> with the path to a directory containing the core-site.xml, hdfs-site.xml, and hive-site.xml.

Example for remote Apache Hive deployment - manual
hive agent add dataproc --name targetmanualAgent --host myRemoteHost.example.com --port 5552 --config-path <example directory path> --file-system-id mytargethdfs

Replace <example directory path> with the path to a directory containing the core-site.xml, hdfs-site.xml, and hive-site.xml.

note

If specifying Kerberos and config path information for remote agents, ensure that the directories and Kerberos principal are correct for your chosen remote host (not your local host).

hive agent add snowflake#

Add a remote agent to connect to a Snowflake warehouse using the hive agent add snowflake command.

Add a remote Snowflake agent using basic authentication
SYNOPSYS        hive agent add snowflake basic [[--account-identifier] string]                                          [[--file-system-id] string]                                      [[--name ] string]                                      [[--password] string]                                                     [[--stage] string]                                                         [[--stage-schema] string]                                                  [[--warehouse] string]                                        [[--default-fs-override] string]                                                     [[--schema] string]                                                        [[--stage-database] string]                                                [[--user] string]  
Add a remote Snowflake agent using privatekey
SYNOPSYS        hive agent add snowflake privatekey [[--account-ID] string]                                          [[--file-system-id] string]                                               [[--private-key-file  string]                                      [[--private-key-file-pwd  string]                                             [[--schema   string]                                                      [[--stage-database  string]                                               [[--warehouse  string]                                      [[--default-fs-override string]                                         [[--name  string]                                                                [[--stage  string]                                                         [[--stage-schema  string]                                         [[--user] string]

Mandatory parameters#

  • --account-identifier is the unique ID for your Snowflake account.

  • --file-system-id is the ID of the source filesystem resource. In the UI, this is called --Filesystem Name.

  • --private-key-file is the path to your private key file.

  • --private-key-file-pwd is the password that corresponds with the above private-key-file.

  • --warehouse is the Snowflake-based cluster of compute resources.

  • --default-fs-override is an override for the default filesystem URI instead of a filesystem name.

  • --name is a name that will be used to reference the remote agent.

  • --stage storage used to temporarily store data that is being moved into Snowflake. A stage can be internal (to Snowflake), or external, using a could storage service from Amazon S3, Azure or Google Cloud.

  • --user is your Snowflake username.

Optional parameters#

  • --stage-database is an optional parameter for a Snowflake stage database with the default value "WANDISO".
  • --stage-schema - is an optional parameter for a Snowflake stage schema, with the default value "PUBLIC".
  • --schema - is an optional parameter for a Snowflake schema, with the default value "PUBLIC".

Examples#

Example adding of a Snowflake remote agent with basic authentication
hive agent add snowflake basic --account-identifier test_adls2 --name snowflakeAgent --stage myAzure --user jonndaugh -- password f39£fo301!jf --warehouse DemoWH2

hive agent check#

Check the configuration of an existing Hive agent using hive agent check.

Check if agent configuration is valid & connectable
SYNOPSYS        hive agent check [--name] string

Example#

hive agent check --name azureAgent

hive agent configure azure#

Change the configuration of an existing Azure Hive agent using hive agent configure azure.

The parameters that can be changed are the same as the ones listed in the hive agent add azure section.

All parameters are optional except --name, which is required to specify the existing Hive agent that you wish to configure.

Example#

hive agent configure azure --name azureAgent --database-password CorrectPassword

hive agent configure filesystem#

Change the configuration of an existing filesystem Hive agent using hive agent configure filesystem.

The parameters that can be changed are the same as the ones listed in the hive agent add filesystem section.

All parameters are optional except --name, which is required to specify the existing Hive agent that you wish to configure.

Example#

hive agent configure filesystem --name fsAgent --root-folder /user/dbuser/databases

hive agent configure glue#

Change the configuration of an existing AWS Glue Hive agent using hive agent configure glue.

The parameters that can be changed are the same as the ones listed in the hive agent add glue section.

All parameters are optional except --name, which is required to specify the existing Hive agent that you wish to configure.

Example#

hive agent configure glue --name glueAgent --aws-region us-east-2

hive agent configure hive#

Change the configuration of an existing Apache Hive agent using hive agent configure hive.

The parameters that can be changed are the same as the ones listed in the hive agent add hive section.

All parameters are optional except --name, which is required to specify the existing Hive agent that you wish to configure.

Example#

hive agent configure hive --name sourceAgent --kerberos-keytab /opt/keytabs/hive.keytab --kerberos-principal hive/myhostname.example.com@REALM.COM

hive agent configure databricks#

Change the configuration of an existing Databricks agent using hive agent configure databricks.

The parameters that can be changed are the same as the ones listed in the hive agent add databricks section.

All parameters are optional except --name, which is required to specify the existing Hive agent that you wish to configure.

Example#

hive agent configure hive --name databricksAgent --access-token myexamplefg123456789t6fnew7dfdtoken4

hive agent configure dataproc#

Change the configuration of an existing Dataproc agent using hive agent configure dataproc.

The parameters that can be changed are the same as the ones listed in the hive agent add databroc section.

All parameters are optional except --name, which is required to specify the existing Hive agent that you wish to configure.

Example#

hive agent configure dataproc --name dataprocAgent --port 9099

hive agent configure snowflake#

Configure an existing Snowflake remote agent by using the hive agent configure snowflake command.

Add a remote Snowflake agent using basic authentication
SYNOPSYS        hive agent configure snowflake basic [[--account-identifier] string]                                               [[--file-system-id] string]                                            [[--user] string]                                                  [[--password] string]                                                           [[--stage] string]                                                              [[--stage-schema] string]                                                       [[--warehouse] string]                                            [[--default-fs-override] string]                                                [[--name] string]                                                               [[--schema] string]                                                             [[--stage-database] string]         

Example Snowflake remote agent configuration#

hive agent configure snowflake basic --user snowflakeAgent --password <password-here> --stage internal
Configure a remote Snowflake agent using privatekey authentication
SYNOPSYS        hive agent configure snowflake privatekey [[--account-identifier] string]                                                     [[--file-system-id] string]                                                         [[--private-key-file] string]                                                [[--private-key-file-pwd] string]                                                        [[--schema] string]                                                                   [[--stage-database] string]                                                           [[--warehouse] string]                                                  [[--default-fs-override] string]                                                      [[--name] string]                                                                     [[--stage] string]                                                        [[--stage-schema] string]  

Example Snowflake remote agent configuration#

hive agent configure snowflake privatekey --private-key-file-pwd <password> --private-key-file /path/to/keyfiles/ --user snowflakeAgent --schema star-schema

hive agent delete#

Delete the specified Hive agent with hive agent delete.

Delete agent
SYNOPSYS        hive agent delete [--name] string

Example#

hive agent delete --name azureAgent

hive agent list#

List configured Hive agents with hive agent list.

List already added agents
SYNOPSYS        hive agent list [--detailed]

Example#

hive agent list --detailed

hive agent show#

Show the configuration of a Hive agent with hive agent show.

Show agent configuration
SYNOPSYS        hive agent show [--name] string

Example#

hive agent show --name azureAgent

hive agent types#

Print a list of supported Hive agent types with hive agent types.

Print list of supported agent types
SYNOPSYS        hive agent types

Example#

hive agent types

Hive rule commands#


hive rule add,hive rule create#

Create a Hive migration rule that is used to define which databases and tables are migrated.

info

Specify these rules when starting a new migration to control which databases and tables are migrated.

Add new Hive migration rule
SYNOPSYS        hive rule add [--database-pattern] string                      [--table-pattern] string                      [[--name] string]
ALSO KNOWN AS        hive rule create

Mandatory parameters#

  • --database-pattern Specify a Hive DDL pattern that will match the database names you want to migrate.
  • --table-pattern Specify a Hive DDL pattern that will match the table names you want to migrate.
tip

You can use a single asterisk (*) if you want to match all databases and/or all tables within the Metastore/database.

Optional parameters#

  • --name The name for the Hive rule.

Example#

Match all database names that start with test and all tables inside of them
hive rule add --name test_databases --database-pattern test* --table-pattern *

hive rule configure#

Change the parameters of an existing Hive rule.

The parameters that can be changed are the same as the ones listed in the hive rule add,hive rule create section.

All parameters are optional except --name, which is required to specify the existing Hive rule that you wish to configure.

Example#

hive rule configure --name test_databases --database-pattern test_db*

hive rule delete#

Delete a Hive rule.

Delete selected Hive migration rule
SYNOPSYS        hive rule delete [--name] string

Example#

hive rule delete --name test_databases

hive rule list#

List all Hive rules.

Get a list of defined rules
SYNOPSYS        hive rule list

Example#

hive rule list

hive rule show#

Show details of a Hive rule.

Show rule details
SYNOPSYS        hive rule show [--name] string

Example#

hive rule show --name test_databases

Hive migration commands#


hive migration add#

Create a new Hive migration to initiate metadata migration from your source Metastore.

info

Create hive rules before initiating a Hive migration to specify which databases and tables are migrated.

Create new migration
SYNOPSYS        hive migration add [--source] string                           [--target] string                           [[--rule-names] list]                           [[--name] string]                           [--auto-start]                           [--once]

Mandatory parameters#

  • --source The name of the Hive agent for the source of migration.
  • --target The name of the Hive agent for the target of migration.

Optional parameters#

  • --rule-names The rule name or list of rule names to use with the migration. Multiple rules need to be comma-separated (for example: rule1,rule2,rule3).
  • --name The name to identify the migration with.
  • --auto-start Specify this parameter to start the migration immediately after creation.
  • --once Specify this parameter to perform a one-time migration, and not continuously scan for new or changing metadata.

Example#

hive migration add --source sourceAgent --target remoteAgent --rule-names test_dbs,user_dbs --name hive_migration --auto-start
note

Auto-completion of the --rule-names parameter will not work correctly if it is added at the end of the Hive migration parameters. See the troubleshooting guide for workarounds.


hive migration delete#

Delete a Hive migration.

note

A Hive migration must be stopped state before it can be deleted, this can be achieved by using the --force-stop parameter with this command.

Delete migration from the list, migration should be stopped
SYNOPSYS        hive migration delete [--name] string  [--force-stop]

Example#

hive migration delete --name hive_migration --force-stop

hive migration list#

List all Hive migrations.

print a list of all migrations
SYNOPSYS        hive migration list

Example#

hive migration list

hive migration show#

Display information about a Hive migration.

Show info about specific migration
SYNOPSYS        hive migration show [--name] string

Example#

hive migration show --name hive_migration
note

Specify the --detailed parameter to output additional information about the metadata migration.


hive migration start#

Start a Hive migration or a list of Hive migrations (comma-separated).

note

Specify the --once parameter to perform a one-time migration, and not continuously scan for new or changing metadata.

Start migration
SYNOPSYS        hive migration start [--name] list  [--once]

Example#

hive migration start --name hive_migration1,hive_migration2

hive migration start all#

Start all Hive migrations.

note

Specify the --once parameter to perform a one-time migration, and not continuously scan for new or changing metadata.

Start migration
SYNOPSYS        hive migration start all [--once]

Example#

hive migration start all --once

hive migration status#

Show the status of a Hive migration or a list of Hive migrations (comma-separated).

Show migration status
SYNOPSYS        hive migration status [--name] list

Example#

hive migration status --name hive_migration1,hive_migration2

hive migration status all#

Show the status of all Hive migrations.

Start migration
SYNOPSYS        hive migration status all

Example#

hive migration status all

hive migration stop#

Stop a running hive migration or a list of running hive migrations (comma-separated).

Stop running migration
SYNOPSYS        hive migration stop [--name] list

Example#

hive migration stop --name hive_migration1,hive_migration2

hive migration stop all#

Stop all running Hive migrations.

Stop all running migrations
SYNOPSYS        hive migration stop all

Example#

hive migration stop all

Hive show commands#


hive show conf#

Show the value of a configuration property from a specific agent.

Returns a description of the specified Hive configuration property.
SYNOPSYS        hive show conf [--parameter] string  [[--agent-name] string]

Hive show configuration parameters#

  • --agent-name The name of the agent.
  • --parameter The configuration parameter/property that you want to show the value of.

Example#

Example when sourceAgent is an Apache Hive agent
hive show conf --agent-name sourceAgent --parameter hive.metastore.uris

hive show database#

Show details about a database from a specified agent.

Show detailed information about a given database and agent (or sourceAgent if not set).
SYNOPSYS        hive show database [--database] string  [[--agent-name] string]

Hive show database parameters#

  • --agent-name The name of the agent.
  • --database The database name. If not specified, the default will be default.

Example#

hive show database --agent-name sourceAgent --database mydb01

hive show databases#

Show a list of databases from a specified agent.

Get databases list from a given agent or sourceAgent if agent isn't set.
SYNOPSYS        hive show databases [[--like] string]  [[--agent-name] string]

Hive show databases parameters#

  • --agent-name The name of the agent.
  • --like The Hive DDL pattern to use to match the database names (for example: testdb* will match any database name that begins with "testdb").

Example#

hive show database --agent-name sourceAgent --like testdb*

hive show indexes#

Show a list of indexes for a database/table from a specified agent.

Get indexes list for a given database/table and agent (or sourceAgent if not set).
SYNOPSYS        hive show indexes [--database] string  [--table] string  [[--agent-name] string]

Hive show indexes parameters#

  • --agent-name The name of the agent.
  • --database The database name.
  • --table The table name.

Example#

hive show indexes --agent-name sourceAgent --database mydb01 --table mytbl01

hive show partitions#

Show a list of partitions for a database/table from a specified agent.

Get partitions list for a given database/table and agent (or sourceAgent if not set).
SYNOPSYS        hive show partitions [--database] string  [--table] string  [[--agent-name] string]

Hive show partitions parameters#

  • --agent-name The name of the agent.
  • --database The database name.
  • --table The table name.

Example#

hive show partitions --agent-name sourceAgent --database mydb01 --table mytbl01

hive show table#

Show details about a table from a specified agent.

Show detailed information about a given table using the given agent (or sourceAgent if not set).
SYNOPSYS        hive show table [--database] string  [--table] string  [[--agent-name] string]

Hive show table parameters#

  • --agent-name The name of the agent.
  • --database The database name where the table is located.
  • --table The table name.

Example#

hive show table --agent-name sourceAgent --database mydb01 --table mytbl01

hive show tables#

Show a list of tables for a database from a specified agent.

Get tables list for a given database (default if not set ) and agent (sourceAgent if not set).
SYNOPSYS        hive show tables [[--like] string]  [[--database] string]  [[--agent-name] string]

Hive show tables parameters#

  • --agent-name The name of the agent.
  • --like The Hive DDL pattern to use to match the table names (for example: testtbl* will match any table name that begins with "testtbl").

Example#

hive show tables --agent-name sourceAgent --database mydb01 --like testtbl*

Notification commands#


notification email addresses add#

Add email addresses to the subscription list for email notifications.

Subscribe email address to notifications.

SYNOPSYS    notification email addresses add [--addresses] set  

Mandatory parameters#

  • --addresses A comma-separated lists of email addresses to be added.

Example#

notification email addresses add --addresses myemail@company.org,personalemail@gmail.com

notification email addresses remove#

Remove email addresses from the subscription list for email notifications.

Unsubscribe email address to notifications.

SYNOPSYS    notification email addresses remove [--addresses] set  

Mandatory parameters#

  • --addresses A comma-separated lists of email addresses to be removed. Use auto-completion to quickly select from subscribed emails.

Example#

notification email addresses remove --addresses myemail@company.org,personalemail@gmail.com

notification email smtp set#

Configure the details of an SMTP server for LiveData Migrator to connect to.

Configure the SMTP adapter.
SYNOPSYS    notification email smtp set [--host] string  [--port] integer  [--security] security-enum  [--email] string  [[--login] string]  [[--password] string]  

Mandatory parameters#

  • --host The host address of the SMTP server.
  • --port The port to connect to the SMTP server. Many SMTP servers use port 25.
  • --security The type of security the server uses. Can be either tls or none.
  • --email The email address for LiveData Migrator to use with emails sent through the SMTP server. This address will be the sender of all configured email notifications.

Optional parameters#

  • --login The username to authenticate with the SMTP server.
  • --password The password to authenticate with the SMTP server login. Required if a login is provided.

Example#

notification email smtp set --host my.internal.host --port 587 --security TLS --email livedatamigrator@wandisco.com  --login myusername --password mypassword

notification email smtp show#

Display the details of the SMTP server LiveData Migrator is configured to use.

Show the current configuration of SMTP adapter.
SYNOPSYS    notification email smtp show

notification email subscriptions show#

Show a list of currently subscribed emails and notifications.

Show email notification subscriptions.
SYNOPSYS    notification email subscriptions show

notification email types add#

Add notification types to the email notification subscription list.

See the output from the command notification email types show for a list of all currently available notification types.

Subscribe on notification types.
SYNOPSYS  notification email types add [--types] set  

Mandatory parameters#

  • --types A comma-separated list of notification types to subscribe to.

Example#

notification email types add MISSING_EVENTS,EVENTS_BEHIND,MIGRATION_AUTO_STOPPED

notification email types remove#

Remove notification types from the email notification subscription list.

Unsubscribe on notification types.
SYNOPSYS    notification email types remove [--types] set  

Mandatory parameters#

  • --types A comma-separated list of notification types to unsubscribe from.

Example#

notification email types remove MISSING_EVENTS,EVENTS_BEHIND,MIGRATION_AUTO_STOPPED

migration path status#

View all actions scheduled on a source filesystem in the specified path.

Show information on the migration status of a path on the source filesystem
SYNOPSYS        migration path status [--source-path] string [--source] string

Mandatory parameters#

  • --source-path The path on the filesystem to review actions for. Supply a full directory.
  • --source The filesystem ID of the source system the path is in.

Example#

migration path status --source-path /root/mypath/ --source mySource

notification email types show#

Return a list of all available notification types to subscribe to.

Show email notification types.
SYNOPSYS  notification email types show

notification latest#

Display the latest notification LiveData Migrator presented and additional details about the notification.

Get the latest notification.
SYNOPSYS    notification latest

notification show#

Show the details of a specific notification. Use tab autocompletion to cycle through the list of notifications received along with their type, timestamp and UUID.

Show notification details.
SYNOPSYS    notification show [--notification-id] string  

Mandatory parameters#

  • `--notification-id The UUID of the notification to be shown.

Example#

notification show --notification-id urn:uuid:6a1f2047-8445-460d-b27c-ec5c0496b727

License commands#


license show#

Show the details of the active license.

Show used license
SYNOPSYS        license show [--full]

license upload#

Upload a new license by submitting its location on the local filesystem.

Upload license file
SYNOPSYS        license upload [--path] string

Example#

license upload --path /user/hdfs/license.key

Connect commands#


connect livemigrator#

Connect to the LiveData Migrator service on your LiveData Migrator host with this command.

note

This is a manual method of connecting to the LiveData Migrator service as the livedata-migrator command (shown in CLI - Log in) will attempt to establish this connection automatically.

Connect Livemigrator
SYNOPSYS        connect livemigrator [--host] string                             [--ssl]                             [[--port] int]                             [[--timeout] integer]                             [[--user] string]

Mandatory parameters#

  • --host The hostname or IP address for the LiveData Migrator host.

Optional parameters#

  • --ssl Specify this parameter if you want to establish a TLS connection to LiveData Migrator. Enable Server TLS on the LiveData Migrator service before using this parameter.
  • --port The LiveData Migrator port to connect on (default is 18080).
  • --timeout Define the connection timeout in milliseconds. Set this parameter to override the default connection timeout of 5 minutes (300000ms).
  • --user The username to use for authenticating to the LiveData Migrator service. Used only when the LiveData Migrator instance has basic authentication enabled. You will still be prompted to provide the user password.

Example#

connect livemigrator --host localhost --port 18080

connect hivemigrator#

Connect to the Hive Migrator service on your LiveData Migrator host with this command.

note

This is a manual method of connecting to the Hive Migrator service as the livedata-migrator command (shown in CLI - Log in section) will attempt to establish this connection automatically.

Connect Hivemigrator
SYNOPSYS        connect hivemigrator [--host] string                             [--ssl]                             [[--port] int]                             [[--timeout] long]                             [[--user] string]

Mandatory parameters#

  • --host The hostname or IP address for the LiveData Migrator host that contains the Hive Migrator service.

Optional parameters#

  • --ssl Specify this parameter if you want to establish a TLS connection to Hive Migrator.
  • --port The Hive Migrator service port to connect on (default is 6780).
  • --timeout Define the connection timeout in milliseconds. Set this parameter to override the default connection timeout of 5 minutes (300000ms).
  • --user The username to use for authenticating to the Hive Migrator service. Used only when Hive Migrator has basic authentication enabled. You will still be prompted to provide the user password.

Example#

connect hivemigrator --host localhost --port 6780

Built-in commands#


clear#

Clear the interactive action prompt screen output with the clear command. You can also type <Ctrl-L> to achieve the same, even while typing another command.

Clear the shell screen
SYNOPSYS        clear

echo#

Prints whatever text that you write to the console. This can be used to sanity check a command before running it (for example: echo migration add --path /repl1 --target mytarget –-migration-id myNewMigration --exclusions 100mbfiles).

Print message
SYNOPSYS        echo [--message] string

exit or quit#

Entering either exit or quit will stop operation of LiveData Migrator when it is run from the command line. All processing will cease, and you will be returned to your system shell.

If your LiveData Migrator command line is connected to a LiveData Migrator system service, this command will end your interactive session with that service, which will remain in operation to continue processing Live migrations.

If this command is encountered during non-interactive processing of input (such as when you pipe input to an instance as part of another shell script) no further commands contained in that input will be processed.

Exit the shell
SYNOPSYS        exit
ALSO KNOWN AS        quit

help#

Use the help command to get details of all commands available from the action prompt.

Display help about available commands
SYNOPSYS        help [[-C] string]

For longer commands, you can use backslashes (\) to indicate continuation, or use quotation marks (") to enclose the full command. When using quotation marks, you can press Tab on your keyboard to make LiveData Migrator automatically suggest the remainder of your typed command.

See the examples below for reference.

Example#

help connect
NAME        connect - Connect to LiveData Migrator and Hive Migrator.
SYNOPSYS        connect [[--host] string]  [--ssl]  [[--lm2port] int]  [[--hvm-port] int]  [[--timeout] integer]  [[--user] string]  
Use of backslashes
help hive\ migration\ add
NAME        hive migration add - Create new migration.
SYNOPSYS        hive migration add [--source] string  [--target] string  [[--name] string]  [--auto-start]  [--once]  [--rule-names] list  
Use of quotation marks
help "filesystem add local"
NAME        filesystem add local - Add a Local filesystem via HCFS API.
SYNOPSYS        filesystem add local [--file-system-id] string  [[--fs-root] string]  [--source]  [--scan-only]  [[--properties-files] list]  [[--properties] string]  

history#

Enter history at the action prompt to list all previously entered commands.

Entering history --file <filename> will save up to 500 most recently entered commands in text form to the file specified. Use this to record commands that you have executed.

Display or save the history of previously run commands
SYNOPSYS        history [[--file] file]

Optional parameters#

  • --file The name of the file in which to save the history of commands.

script#

Load and execute commands from a text file using the script --file <filename> command. This file should have one command per line, and each will be executed as though they were entered directly at the action prompt in that sequence.

Use scripts outside of the LiveData Migrator CLI by referencing the script when running the livedata-migrator command (see examples).

Read and execute commands from a file
SYNOPSYS        script [--file] file

Mandatory parameters#

  • --file The name of the file containing script commands.
Example contents of a script file
hive agent check --name sourceAgenthive agent check --name azureAgent

Examples#

info

These examples assume that myScript is inside the working directory.

Example inside CLI
script --file myScript
Example outside of CLI (non-interactive)
livedata-migrator --script=./myScript

stacktrace#

Use the stacktrace command to get full technical information about the source of an error during LiveData Migrator operation.

Display the full stacktrace of the last error
SYNOPSYS        stacktrace

Action prompt features#

The action prompt provides many features to guide you during operation.

FeatureHow to use it
Review available commandsCommands that can't be used without creating other resources first are tagged with * in the output of the help command.
Command completionHit the <tab> key at any time to get assistance or to complete partially-entered commands.
Cancel inputType <Ctrl-C> before entering a command to return to an empty action prompt.
Syntax indicationInvalid commands are highlighted as you type.
Clear the displayType <Ctrl-L> at any time.
Previous commandsNavigate previous commands using the up and down arrows, and use standard emacs shortcuts.
Interactive or scripted operationYou can interact with the command line interface directly, or send it commands on standard input to incorporate it into shell scripts. See script for more information and examples.

System service commands#

The service scripts can be used to control operation of each individual service at any time.

LiveData Migrator#

service livedata-migrator start|stop|force-reload|restart|status

Hive Migrator#

service hivemigrator start|stop|force-reload|restart|status

LiveData UI#

service livedata-ui start|stop|force-reload|restart|status

Log commands#

The following commands will only affect logging of the CLI terminal, and will not affect other components of LiveData Migrator:

log off
log info
log debug
log trace

External commands#

Use these commands outside of the LiveData Migrator CLI.

livedata-migrator#

Launch LiveData Migrator and its connected services.

Optional parameters#

  • --version List the versions of all LiveData Migrator components without starting LiveData Migrator. Includes LiveData Migrator, LiveData UI, LiveData Migrator CLI, Hive Migrator and the Hive Migrator Azure Libraries.

Example output#

# livedata-migrator --versionlivedata-migrator 1.12.0-1462livedata-ui 6.6.1-1914livedata-migrator-cli 1.3.0-209hivemigrator 1.3.0-514hivemigrator-azure-hdi 1.3.0-514