Skip to main content
Version: 1.15.1

Prerequisites

Ready to start? Check the prerequisites and then follow these steps to get ready for LiveData Migrator. This latest version of LiveData Migrator also includes a metadata migration feature. Prerequisites to migrate metadata are included in the list below, and you'll find instructions explaining how to use the feature throughout this guide.

Read the release notes to see the latest information about the current version of LiveData Migrator, including supported filesystem versions.

Recommended Technical Knowledge#

System administration#

  • Linux operating system installation
  • Disk management
  • Memory monitoring and management
  • Command line administration and manually editing configuration files
  • Service configuration and management

Networking#

  • IP Address assignation
  • TCP/IP ports and Firewall setup or server certificates (for TLS)

Cloud Storage Technologies#

  • Proficiency with intended target storage technologies, such as ADLS Gen 2, GCS, HDFS, and Amazon Simple Storage Service (S3).
    • For Amazon Web Services, this includes:
      • Knowledge of AWS Marketplace, Amazon Simple Storage Service (Amazon S3), AWS Glue Data Catalog, and AWS Command Line Tool.
      • Understanding any storage persistence and related costs.
      • Ability to monitor and troubleshoot AWS services.

LiveData#

  • An understanding of the installation procedures for your platform, as provided in this guide.

If you’re not confident about meeting the requirements, you can discuss a supported installation by contacting WANdisco.

Prerequisites#

  • Linux host
  • Java 1.8 64-bit.
  • Network connectivity from your LiveData Migrator host to your target filesystem (for example: ADLS Gen2 endpoint or S3 bucket).
  • Port 8081 accessible on your Linux host (to access the UI through a web browser).
  • If migrating from HDFS:
    • Hadoop client libraries must be installed on the Linux host.
    • Ability to authenticate as the HDFS superuser (for example: hdfs).
    • If Kerberos is enabled on your Hadoop cluster, a valid keytab containing a suitable principal for the HDFS superuser must be available on the Linux host.
  • If you want to migrate metadata to or from Apache Hive:
    • The Hive service must be present on the cluster.
    • SSH/CLI access to the cluster.
    • If Kerberos is enabled on your Hadoop cluster, a valid keytab containing a suitable principal for the Hive service user must be available. The host for the keytab will depend on whether you deploy locally, remotely, or both (see the hive agent add hive section for more information).
      • The keytab must be owned by the same user running LiveData Migrator's metadata migration component.

Machine Specification#

  • 16 CPUs, 48 GB RAM (minimum 4 CPUs, 32 GB RAM)
    • If deploying LiveData Migrator on a Hadoop cluster, the host machine should be an edge node that matches at least 75-80% of the NameNode host specification.
  • 200 GB (minimum 100 GB)
    • SSD-based storage is recommended.
  • 2 Gbps minimum network capacity
    • Your network bandwidth must be able to cope with transferring data and ongoing changes from your source filesystem.

Production Use Configuration#

We recommended you configure Data Migration properties on your Hadoop Distributed File System to ensure smooth operation.

Next Steps#

Once you have all the prerequisites, set up your network and then download and install LiveData Migrator.