Skip to main content
Version: 2.4.3 (latest)

Monitoring with Prometheus and Grafana

Data Migrator provides detailed metrics about recent and ongoing events, activity, and performance through the UI or the CLI. Use Prometheus and Grafana to capture and visualize this data:

  • Prometheus is an open-source monitoring system that records real-time metrics in a time series database.

  • Grafana is an open-source interactive visualization web application that provides charts, graphs, and alerting functionality when connected to supported data sources.

Prometheus endpoints​

Collect Prometheus metrics for Data Migrator or the UI using each component's specific endpoint.

Data Migrator Prometheus endpoint​

Data Migrator Prometheus endpoint located at: 

 http(s)://<hostname>:18080/actuator/prometheus
info

If you are not doing so already, you may need to enable access to the REST API on your network. Learn more here.

UI Prometheus endpoint​

To access the UI endpoint, enable it in the UI configuration, and restart the service.

  1. Edit the UI configuration file.

    vi /etc/wandisco/ui/application-prod.properties

  2. Add the following configuration line to enable the endpoint.

    management.metrics.export.prometheus.enabled=true

  3. Restart the UI service.

    systemctl restart livedata-ui

The UI Prometheus endpoint when enabled is located at: 

http://<hostname>:8081/management/prometheus
tip

If you have https enabled on the UI, the endpoint will redirect to https on port 8443 automatically

Set up monitoring with Prometheus and Grafana​

Set up a host machine​

You will need a machine with network access to the Data Migrator node on port 18080.

note

We do not recommend installing additional applications on the Data Migrator node.

For more information on the minimum requirements to run Grafana, see the official documentation.

Install Prometheus​

See First steps with Prometheus for information on installing Prometheus on your new node.

Example​

.tar install on CentOS 7.5 host:
mkdir -p /var/tmp/builds/prometheus
cd /var/tmp/builds/prometheus/
wget https://github.com/prometheus/prometheus/releases/download/v2.33.5/prometheus-2.33.5.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz

Install Grafana​

See Install Grafana for information on installing Grafana on your new node.

Example​

The official documentation lists several installation options. In these examples, we use the .tar file installation steps.

Example​

.tar install on CentOS 7.5 host:
mkdir -p /var/tmp/builds/prometheus
cd /var/tmp/builds/prometheus/
wget https://github.com/prometheus/prometheus/releases/download/v2.33.5/prometheus-2.33.5.linux-amd64.tar.gz
tar xvfz prometheus-*.tar.gz

Configure Prometheus​

See the official documentation for information on configuring Prometheus.

To let Prometheus know which source to collect data from, create or update the Prometheus configuration file to point to the correct endpoint on the Data Migrator host. Use the default configuration file (prometheus.yml) or create your own and point to it when running Prometheus.

note

The Data Migrator Prometheus endpoint used here is http(s)://hostname:18080/actuator/prometheus so we can collect Data Migrator specific metrics.

The following example configuration shows Prometheus pointing to a Data Migrator instance with IP 172.31.44.14 and a metrics_path value of /actuator/prometheus:

scrape_interval: 1s
evaluation_interval: 1s

alerting:
alertmanagers:
- static_configs:
    - targets:

scrape_configs:
- job_name: "Data Migrator Prometheus"
metrics_path: '/actuator/prometheus'
static_configs:
  - targets: ["172.31.44.14:18080"]

Collect data​

Start Prometheus​

See the official documentation for your version for information on how to run Prometheus.

Example​
Starting Prometheus pointing to a specified config file from the.tar file installation method:
/var/tmp/builds/prometheus/prometheus-2.33.5.linux-amd64/prometheus --config.file="prometheus.yml

Prometheus will start collecting data from the endpoint specified in its configuration and store it in its own time-series database, ready to be analyzed. When you stop the process, it will stop collecting data.

Visualize data​

Start Grafana​

See the official documentation for your version for information on how to run Grafana.

Starting Grafana - .tar file installation method:
/var/tmp/grafana/grafana-8.4.3/bin/grafana-server

While Grafana is running, go to the hostname or IP on port 3000 to sign in. Enter the following default credentials and update them when prompted:

**Username**: admin
**Password**: admin

Add the data source​

When signed in to Grafana for the first time, you won't see any data until you've set up a data source. Create a Prometheus data source to get started.

Visualize the data​

See the Grafana documentation for information on using panels, visualizations, and dashboards to visualize your data.

View Prometheus metrics​

See the endpoint below for the latest list of available metrics for Prometheus:

URL for Data Migrator Prometheus metrics documentation on your server:
http://localhost:18080/actuator/prometheus
URL for Hive Migrator Prometheus metrics documentation on your server:
http://localhost:6780/prometheus
info

For more information on accessing the Data Migrator REST API documentation, see API Reference.