Skip to main content

Install data transfer agents

Data transfer agents let you scale beyond the limitations of a single host by sharing the workload of transferring data across additional hosts with access to your source. Data transfer agents accelerate data transfer by removing network, memory, and CPU bottlenecks.

This means LiveData Migrator for Azure can be scaled to the capacity of your wide area network or another limitation, such as the data transfer capability of your storage environment.

You can add an unlimited number of data transfer agents, subject to Azure limits.

To add a data transfer agent:

  1. Install an agent to a host using the installer script.
  2. Copy the authentication token provided by the installer.
  3. Add the agent using the copied authentication token.
note

Data transfer agents don't scale metadata migrations
Data transfer agents assist with moving data but aren't involved in scaling metadata migrations, which are typically under a much lower load and don't experience bottlenecks from being deployed on a single host.

info

If you use data transfer agents, agents become responsible for transferring the data to your target filesystem(s). LiveData Migrator for Azure doesn't transfer the data. If you stop the agents, your data isn't migrated. If you want to stop using agents, and use LiveData Migrator for Azure to move data, you must first remove all agents.

Data transfer agents don't apply bandwidth limits. If you need to manage traffic, use LiveData Migrator for Azure instead.

Prerequisites​

  • You are the system administrator able to run installer scripts and install packages with root or sudo.

  • Additional hosts are deployed on your network that can access the source storage environment.

  • Port 1433 is open between the host running LiveData Migrator for Azure and all hosts running data transfer agents. See Network requirements.

For recommended machine specifications, see the Installation Prerequisites.

Install an agent​

Install each agent on a separate host with client access to the source filesystem.

  1. Download the data transfer agent installer:

    Download the installer
       wget https://cdn.wandisco.com/ldma/install_dta.sh

  2. Make the installer executable: 

    chmod +x install_dta.sh

  3. Run the installation command as the root (or sudo) user: 

    ./install_dta.sh
info

Install with the default system user and default user(hdfs) group(hdfs), with the command

./install_dta.sh.

Install using a custom user or user group, with

./install_dta.sh -- --user <custom user> --group <custom group>.

Authentication token​

A successful agent installation prints the authentication token in your terminal.
Copy the authentication token then paste it in when Adding an agent.

Token format​

The format of the generated token is: 

----- BEGIN AGENT TOKEN

----<example_token_text>----

END AGENT TOKEN -----
note

The begin and end tags must be included when you paste in the token.

The authentication token is generated with the system's hostname in the /etc/hostname file. The hostname may differ from the hostname of the node on the network. To ensure the node is valid when added to the Azure Portal, the generator includes a script - gen-certs-and-token.sh - that allows you to create custom hostnames and ports manually.

Retrieve authentication token from CLI​

A successful agent installation, described in steps 1 to 3 above, prints the authentication token in your terminal. The directories in which agent files are stored are also shown.

See the output example below.

Example output from a successful installation
..
 
Connection Token will be stored in /opt/wandisco/livedata-migrator-data-agent/connection_token,
Security config file is /etc/wandisco/livedata-migrator-data-agent/security.properties,
Registration request file is /opt/wandisco/livedata-migrator-data-agent/reg_data_agent.json

(Re)start the livedata-migrator-data-agent service and call the livedata-migrator API to (re)register an agent:
curl -XPOST -H "Content-Type: application/json" -d @/opt/wandisco/livedata-migrator-data-agent/reg_data_agent.json http://host:18080/scaling/dataagents/
Agent connection token:
----- BEGIN AGENT TOKEN -----
H4sIAAAAAAAAAKVU246jOBT8lVFes2kIhA7ZN2NubtokBEJCxAu3kHCxczcw2n9fMtvq6d3eWbW0SJY4JUy5TtXx90ESwex8Hfw+GD0eRTOQ/Q1qSw/pCAJP+4GGBCOkrDsIgQNzwJAC8n55wFLy/LQvi/nCcVRQgAt2KbNAoPp9rUL9FJk+iVTQhOR1s73iVjpsDX2cGvItFfTb1qi6QJBvgTC7OrzGzH1i4wIzXGg89lA79/B43WO2ill/hu4D+h9cv2IKyUeutQc8JU/eTo+Un0qwojAbAuAm0qYqd8PVcHlEbPecJCF5jizhwux8bNdc5zISFMCdK8OgKdrphnbEjVM6FVbxfkHq4jky7Ik/WyxdpvOFRd14ExJ9foK0wUzzwELJbd9cYgXsZE3xgAock8OAGhCeDBdPZgrAEPAlwFoAkaPK91mmhaQ15TMThI0/Pwrnl5ktO1q8tRCvG+Jq9lLulKWBZjnaA9xxXJtPuaPv3x0q3txaaHeRFZLaB/j4Kpyz6lCvpRMoynYRkh9Wa7b6L/YPfhsk1SEj169mxe+gwp16RbkG+6X+UxWGrLEKQB8d6GuM1kearlOKVe0akrTmu9f6pQw2Ph+3khispTLa2EX/XsTCWHU8kOmMb3EHWuw5DS6QhDs/6rFuroLe6Y8ohnxjdeD4zlX/5PoVU0jeuLDnoa9kZWnsxJRVkqhf6HAOV2ZI5Iln22fTZyC249eiSlUd+Qt4n7GJu94F4nAWeWdnu/IzSeN3t7FQ0+gK+f0aVCsOllJIjI0V0QYwtQD4Y1b6SQOfc5L3OVlBpJ/lcabrHNW4kDQ8OSjSaeMV4yZfqFCqptVWm2d3uV7S67wDBw3I5zIvls8pnqeK7XFmvqu76vVe3l9qPyTEHU5l/uZeWc3xeRDeeF5Mvx4WK2v/npfFEvn9t98sLXjPi+H1GeklYkNp/1LU1+/qANNMxSxZ+rh3HCdvkgbRObjZnXQyPDsIiRRIxRA11YWqDNzchi0mDc/F1+5g5DAPkEW3qCv4fr6y/RJ89iskD8f+j18heTj28OtDaz5LfW+NmyXn7DFJs3j3PJ5Ok5E4nkqjiTzJRtE0k0dS1F84cRRnkRz3u6K837Sgj+EbT0TxDTDp5fGPYn/L9/x4dK/FpziNblf6xCKSHi4JfUpoPfjjT0r88TbmBQAA
----- END AGENT TOKEN -----

WANdisco LiveData Migrator Data Agent installed successfully.

Add the agent using the token from the output in your terminal.

Retrieve authentication token from host​

Alternatively, you can find the generated token on the agent host in the following location. 

/opt/wandisco/livedata-migrator-data-agent/connection_token

Add the agent using the token output from the file.