Test drive LiveData Migrator for Azure

What is LiveData Migrator for Azure?#

LiveData Migrator for Azure moves data from Hadoop to Azure Data Lake Storage Gen2 (ADLS Gen2).
The Test Drive service contains a sample Hadoop environment and all the components necessary to experience LiveData Migrator. Test Drive allows you to sign in to an Azure Test Drive subscription so you don't need your own, and it has the following preconfigured components:

  • Resource group
  • Storage account
  • Container

This Test Drive experience allows you to do the following:

  • Deploy and use LiveData Migrator for Azure to demonstrate how you can migrate actively-changing data from a Hadoop cluster to an ADLS Gen2 account.
  • Familiarize yourself with LiveData Migrator’s features and native integration into Azure Portal.
  • Interact with data from the Hadoop environment and show how easy it is to use the system to effectively migrate your data.

Get started with your Test Drive#

  1. Go to Azure Marketplace and select Test Drive.

    You'll receive an email in approximately 10 minutes with sign-in credentials.

    Test Drive email

    note

    Test Drive is available for two hours from the time you receive the email with your sign-in credentials. A timer is displayed at the top of your screen.

  2. Open a private browser session using Chrome incognito mode, for example, or log out of your own Azure Portal account.

  3. Go to Azure Portal.

  4. Sign in with the Test Drive user name and password provided in the email.

  5. Select Yes to stay signed in.

  6. Skip the tour of Microsoft Azure by selecting Maybe Later.

Connect to the virtual machine to view data in the Hadoop test cluster#

  1. In Azure Portal, go to All resources.

    All resources

    note

    If you weren't prompted to use the sign-in credentials, and you see a Recent resources section, you're in your own subscription. Sign out or use private browsing.

  2. To form a Secure Shell (SSH) connection, select the Virtual Machine (VM).

    Virtual machine

  3. Locate and take note of the public IP address

    IP address

  4. Connect to the VM by selecting Connect > SSH.

    Connect

  5. Open your terminal, and type and run the following command:

    ssh -L 8092:localhost:8888 wandisco@[Public IP address]
  6. Enter the password Wandisco123!.

    note

    If you are using Windows, paste text by right-clicking the icon in the top left. Then select Edit, Paste.

    The wandisco@wandisco-ldma prompt appears.

    Terminal command 2

    Leave this window open.

Check LiveData Migrator is installed#

  1. Go to Azure Portal.

  2. Select the resource group name, and then select your LiveData Migrator.

    Install 1

  3. Check the install state is InstalledAndOperational.

    Install 2

    note

    If the install state is not InstalledAndOperational, check back in a few minutes. Starting up the cluster and installing LiveData Migrator can take a little over 10 minutes depending on Azure resources.

Connect to Hue to view files in Hadoop#

  1. Open a new browser tab and go to http://127.0.0.1:8092.

    Hue 1

  2. Enter the following credentials:

    • User name: hdfs
    • Password: hdfs

    Hue 2 The connection is made.

    info

    In Hue, if you get an error that you're no longer connected to the VM through SSH, please SSH in again using your CLI.

  3. Select Files from the left pane, and then go up two levels to the root level.

    Hue 3

  4. Drill down into the Demo/1GB folder.

    Hue 4

  5. Take a note of the two data files in Demo/1GB.

Specify the migration target#

  1. To create the target for your data migration, select Targets.

  2. Select Create.

    Target 1

  3. Enter the following details:

    • Target name
    • Storage account
    • Storage container
  4. Select Review + create, and then on the next page, select Create.

    Target 2

  5. Once completed, select Go to resource to check your target for data migration has been created.

    Target 3

  6. Check the provisioning state is Succeeded. Note the entry for the Storage account/container specifies where your target data will be migrated to.

    Target 4

Create the migration#

  1. Select the resource group.

  2. Select the LiveData Migrator.

    Migration 1

  3. Select Migrations > Create.

    Migration 2

  4. Enter the following migration details:

    • Enter a migration name.
    • Select the preconfigured target storage.
    • Select the preconfigured path.

    Migration 3

  5. Select Next: Migration Settings.

  6. For Automatically Start Migration, select Yes.

  7. Select Review + create > Create.

    Migration 4

    Now your migration is running.

  8. Check the migration has been created and is running by selecting Go to resource.

    Migration 5

  9. Check the following details:

    • Provisioning state is Succeeded.
    • Migration status is Live whcih means migrations have started.
    • The path is the /demo source folder.

    Migration 6

Check the initial data migration to ADLS Gen2 storage#

  1. In the top search bar, search for storage accounts.

    ADLS 1

  2. Select your storage account.

  3. Select Containers in the left menu.

    ADLS 2

  4. Select your container.

    ADLS 3

  5. Drill down to /Demo/1GB to check the data migrated.

    ADLS 4

  6. Compare with the files in Hue.

    ADLS 5

Check updates to source data are reflected in the target#

To check live changes made to source files in Hue are added to the container, complete the following steps:

  1. Add a file to the source location in Hue by going to Hue and selecting Upload.

    ADLS 6

  2. Follow prompts to add a file from your computer.

  3. Check the file is added.

    ADLS 7

  4. Go to your container in Azure Portal and select Refresh.

    ADLS 8 The file has been added.

  5. Delete a file from the source location in Hue by going to Hue and selecting a file to delete.

  6. Select Delete forever and confirm.

    ADLS 9

  7. Go to your container in Azure Portal and select Refresh.

    ADLS 10 The file has been deleted.

  8. Rename a file in the source location going to Hue and selecting a file to rename.

  9. Select Actions > Rename.

    ADLS 11

  10. Enter a new name and confirm.

  11. Check the file is renamed in Azure Portal.

    ADLS 12

You completed the Test Drive exercises. Your environment will be deleted at the end of the two hour Test Drive.