Troubleshooting

This article provides details on some of the issues you may face when installing and using LiveData Platform for Azure, and provides steps to help you resolve these problem. Please ensure you have read the Prerequisites as if any of these items are missed, issues may be encountered.

Registering the LiveData resource provider#

The following section provides details on errors you may encounter when registering the LiveData resource provider.

Insufficient authorization#

To register resource providers, you must have the Contributor role (or greater).

If you do not have the relevant Azure Active Directory (AAD) permissions then you will get an error similar to:

The client 'user.name@example.com' with object id 'ba878430-1a00-4056-acf3-5678b8k0e3de' does not have authorization to perform action 'Wandisco.Fusion/register/action' over scope '/subscriptions/9087fget-7697-4e7d-g908-a5a3ae908654' or the scope is invalid. If access was recently granted, please refresh your credentials.

You will need to get in contact with the team that manages your subscription and either get the appropriate permissions or ask them to register the LiveData resource provider for you.

Install the LiveData extension with the Azure CLI#

The following section provides details on errors you may encounter when installing Azure CLI with LiveData Plane Extension.

AZ: command not found#

If you do not have the Azure CLI installed when trying to add the LiveData extension you may get the output:

az extension add --source https://extension.whl
az: command not found

Please check the Azure CLI is installed and try again. You can check if the Azure client is running/installed by entering az and pressing enter.

See Install the Azure CLI on Linux in the Microsoft documentation for installation instructions.

Update the LiveData extension#

The LiveData extension must be removed and readded to update to the latest version.

az extension remove --name livedata
az extension add --source https://cdn.wandisco.com/az-cli/extensions/frp/0/livedata-0-py2.py3-none-any.whl

The az extension update --name livedata command is currently unsupported.

LiveData extension is not compatible#

The following errors can occur if the --source URL is incorrect.

Example 1
az extension add --source https://cdn.wandisco.com/az-cli/extensions/frp/0/fusion-0-py2.py2-none-any.whl
Are you sure you want to install this extension? (y/n): y
The 'None' extension is not compatible with this version of the CLI.
You have CLI core version 2.3.1, and this extension requires a min of 2.0.67 and max of 2.1.0.
Please install a compatible extension version or remove it.
Example 2
az extension add --source https://wandiscopublicfusion.blob.core.windows.net/wandiscopublicfusion/az-cli/extensions/frp/0.1.0/fusion-0.1.0-py2.py3-none-an.whl
Are you sure you want to install this extension? (y/n): y
Request to https://wandiscopublicfusion.blob.core.windows.net/wandiscopublicfusion/az-cli/extensions/frp/0.1.0/fusion-0.1.0-py2.py3-none-an.whl failed with 404

See the Install Azure CLI with LiveData Plane Extension section for the correct URL.

Installation prerequisites not met for the LiveData extension#

In certain cases and on certain operating systems, the LiveData extension may fail to install due to missing prerequisites, often with an error about the missing dependency.

Examples of the error:

Example 1
knack.util.CLIError: An error occurred. Pip failed with status code 1. Use --debug for more information.
Example 2
cli.azure.cli.core.azclierror: An error occurred. Pip failed with status code 1. Use --debug for more information.

If you encounter this problem, please ensure you have the following prerequisites installed for the operating systems below:

Ubuntu#

  • Python3-pip: Obtainable via apt-get install python3-pip
  • Python3-dev: Obtainable via apt-get install python3-dev --fix-missing
  • Gcc: Obtainable via apt-get install gcc --fix-missing

Redhat/CentOS#

  • Python3-pip: Obtainable via yum install python3-pip
  • Python3-dev: Obtainable via yum install python3-devel
  • Gcc: Obtainable via yum install gcc

Miscellaneous prerequisites#

These are dependencies that you may need in some cases regardless of the operating system, all installed via Pip.

  • Azure CLI: Obtainable via pip install azure-zli
  • Pip tools: Obtainable via pip install pip-tools
  • Cryptography: Obtainable via pip install cryptography

LiveData Migrator#

Download command is not found#

Wget may not be pre-installed on your distribution. If you get a wget: command not found error, install wget by running yum install wget on an RPM-based operating system, or apt-get install wget on a Debian-based operating system. Try the download command again after.

Access is denied when installing LiveData Migrator#

Ensure that you make the LiveData Migrator install script install_ldma.sh executable and run it as the root user. For example:

chmod +x install_ldma.sh && sudo ./install_ldma.sh

Resource group not found when creating a migrator instance#

When you run the command to create a LiveData Migrator resource in the Azure CLI, you'll get a "resource group not found" error if your resource group doesn't exist in the subscription you're using.

To fix it, create a new resource group in the subscription you're using and use its name in the migrator resource creation command.

You'll also receive this error if you're using the wrong subscription. To fix this, switch your subscription to the one with the correct resource group.

DeploymentFailed error when creating a migrator#

When creating a Migrator in Microsoft Azure, you need to have the Wandisco.Fusion resource provider registered in your subscription.

If you do not, you may get the following error:

"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceCreationFailed\",\r\n \"message\": \"S2S Pipeline Failed\"\r\n }\r\n ]\r\n }\r\n}"
}

To fix this issue:

  1. Register the WANdisco Fusion provider again.
  2. Create a Migrator again.

Source filesystem can't be found#

If your Hadoop client isn't available within the systemd environment, LiveData Migator won't be able to detect your source filesystem and you'll get an error when you try to create a migration. To resolve this, export the Path to system variables to make the Hadoop tool available to the service with the following command:

sudo systemctl set-environment PATH=$PATH

ResourceCreationFailed: Failed to Authenticate with ADLS#

When attempting to create a MigratorTarget, the above response can indicate an error authenticating the credentials provided.

Possible reasons for this error include:

  1. Your entered credentials are incorrect.
  2. Your shared secret has expired. In Azure, this may happen automatically if an exp (expiry time) is configured for the secret.
  3. Your shared secret has been revoked.
  4. Clock drift on the host has caused the client's time to be out of sync with the server, rendering calls invalid due to illogical time values specified in the call. For example, the call may claim it is from the future or claim it is from the past (queries from the past are considered replays by the server).

'Migrator is down' error when adding a target or creating an exclusion#

If you see a Migrator is down error in Azure Portal when adding a target, it means LiveData Platform couldn't communicate with with the LiveData Migrator instance on your filesystem. This can happen in several cases:

  1. Your filesystem isn't running.
  2. The LiveData Migrator service isn't running on the filesystem.
  3. Connection to your filesystem is lost.

Check your filesystem is active and run the following command to start the LiveData Migrator service: systemctl start livedata-migrator. Configure your network to allow communication between your LiveData Migrator instance and the Azure Portal.

The service has not been started.#

Ensure the service is running. Run the following command: systemctl start livedata-migrator.

ResourcePurchaseValidationFailed#

If your Microsoft Azure purchase state fails to validate with Microsoft Azure, you'll get the following error:

"code": "ResourcePurchaseValidationFailed",
"message": "User failed validation to purchase resources. Error message: 'Offer with PublisherId: 'wandisco', OfferId: 'ldm' cannot be purchased due to validation errors. For more information see details. Correlation Id: '9ab979d4-2bb2-4bac-b45c-fdb1a2552978' The 'unknown' payment instrument(s) is not supported for offer with OfferId: 'ldm', PlanId 'metered-v1'. Correlation Id

You need a paid Microsoft Azure subscription to use LiveData Migrator. Make sure you're using a Microsoft payment plan that's tied to a credit card.

LiveData Migrator's install state/meta install state show as "InstalledNoSourceDetected" in the overview panel#

During installation, LiveData Migrator automatically sets its HDFS host as the source filesystem. However, the installation will fail if the filesystem runs Kerberos and you have not specified.

If you are running Kerberos on your HDFS cluster, you must configure Kerberos for data migration and metadata migration to allow LiveData Migrator to access and transfer data.

LiveData Migrator's install state shows as "NotInstalled" in the overview panel#

The clock on your HDFS node may be running several minutes slow. This causes messages sent to the Azure Portal to be wrongly considered to have expired already, since the send time appears to have been several minutes before the information arrived at Microsoft Azure.

To fix this issue, use the NTP Pool to synchronize your node's clock.

Validation Failure: Your CLI LiveData Extension version is no longer supported#

You'll get the following error if you're running an Azure CLI LiveData extension version lower than 0.14.1:

ERROR: ValidationFailure: Your CLI LiveData Extension version is no longer supported

To fix this, install the Azure CLI LiveData extension again to upgrade it to the latest version. You can use the az extension show command to check you're running a higher version than 0.14.1:

az extension show --name livedata

How do I get more help?#

If you have a technical query about LiveData Migrator for Azure and can't find help on this page, you can contact the WANdisco Customer Success team for support.