Download and install LiveData Migrator

Follow these steps to install and deploy the LiveData Migrator service.

Get your install key#

The install key is a string that is unique to your migrator resource and will need to be provided when running the installation script.

  1. In the Azure Portal, navigate to the LiveData Migrator resource page.

  2. From LiveData Migrator -> Get Started, click Get Install Key.

  3. The Install Key is provided as a Base64 encoded string, for example:

    Qc-GfyyhsofHJUosKKghsoHUNOPPSAPLhyTRDCju-ID

    This string will be used in the install steps.

Install the LiveData Migrator service#

  1. Use SSH to sign in to your HDFS cluster edge node or HDFS Sandbox, depending on your setup.

    1. HDFS edge node

      Log in to HDFS edge node
      ssh <user>@<hdfs_edge_node_hostname>
    2. HDFS Sandbox

      Log in to HDFS Sandbox host
      ssh <user>@<hdfs_sandbox_hostname>
      Log in to HDFS Sandbox container
      docker-compose exec -w /tmp sandbox-hdp-vanilla bash
  2. Download the installation script to your own environment (HDFS edge node) or the HDFS Sandbox by running the following command:

    wget https://cdn.wandisco.com/ldma/install_ldma.sh
  3. Make the install script executable and run it using the following command:

    Red Hat/CentOS
    chmod +x install_ldma.sh && ./install_ldma.sh
  4. When prompted, provide your install key:

    Example
    Install key: Qc-GfyyhsofHJUosKKghsoHUNOPPSAPLhyTRDCju-ID

    The installation will now continue, wait for it to complete before moving on to the next step.

  5. Enable the LiveData Migrator service:

    CentOS 6 only
    ln -s /opt/wandisco/livedata-migrator/livedata-migrator.jar /etc/init.d/livedata-migrator
    Debian/Ubuntu/Red Hat/CentOS 7
    systemctl enable livedata-migrator
  6. Check the LiveData Migrator service is running:

    CentOS 6 only
    service livedata-migrator status
    Red Hat/Debian/CentOS 7
    systemctl status livedata-migrator
  7. Check the Hive Migrator service is running:

    CentOS 6 only
    service hivemigrator.service status
    Red Hat/Debian/CentOS 7
    systemctl status hivemigrator.service

If you need help, run az livedata migrator create --help for more options.

The LiveData Migrator and Hive Migrator services are run using the following user/group settings:

ServiceDefault User/GroupConfiguration fileConfiguration properties
LiveData Migratorhdfs:hdfs/etc/wandisco/livedata-migrator/vars.envUSERNAME=hdfs and GROUPNAME=hdfs
Hive Migratorhive:hadoop/etc/wandisco/hivemigrator/vars.shHIVE_MIGRATOR_SERVER_USER=hive and HIVE_MIGRATOR_SERVER_GROUP=hadoop

To run services with different users or groups, edit the noted configuration properties in their corresponding configuration file.

If you're running a systemd-based operating system, create an override service file:

systemctl edit livedata-migrator

Next steps#

Once you've installed LiveData Migrator, configure Kerberos if the HDFS cluster you are using is running Kerberos.

If the HDFS cluster is not running Kerberos, you may skip this step and create a target for your data migrations and/or create a metadata target for metadata migration.

Troubleshooting#

If you are having problems downloading and installing LiveData Migrator, you can find solutions in the troubleshooting guide.

If you are running Kerberos on the HDFS cluster you have installed LiveData Migrator to and the dashboard displays either an Install State or Meta Install State of "InstalledNoSourceDetected", configure Kerberos for LiveData Migrator.