Installation and configuration of the Prometheus Metrics stack

Introduction

Prometheus is a cloud-native metrics and monitoring solution. It can get metrics of any type of software like web servers, system usage or networking equipment.

This tutorial covers the basic steps to query the performance data of a server and show it in a dashboard using Grafana.

Prerequisites

2 Debian 10 servers
Allow port 9100 in the iptables firewall or the cloud firewall for incoming traffic (INPUT chain) on the monitored node
Allow port 3000 in the iptables firewall or the cloud firewall for incoming traffic (INPUT chain) on the monitoring server

Test Setup

2001:db8::1 node1.example.com
2001:db8::2 prometheus.example.com

Step 1 - Install Node Exporter on target node

The Prometheus stack consists of multiple components. The metrics storage is prometheus itself, which scrapes metrics in periodic intervals from http endpoints. These endpoints can be provided by any software. You can find a list of exporters and integration here.

For this tutorial, we will use node_exporter to get basic system metrics such as cpu usage, network throughput or disk IO metrics. The following commands install and configure the target (monitored node).

apt update
apt install prometheus-node-exporter

The following commands are used to start and enable the node_exporter daemon for starting automatically after a reboot.

systemctl enable prometheus-node-exporter.service
systemctl start prometheus-node-exporter.service

Now, the http endpoint of the exporter should listen at tcp port 9100. Any http client like curl is able to collect the metrics:

root@node1.example.com:~# curl [2001:db8::1]:9100/metrics

# HELP apt_upgrades_pending Apt package pending updates by origin.
# TYPE apt_upgrades_pending gauge
apt_upgrades_pending{arch="amd64",origin="Debian-Security:10/stable"} 8
# HELP node_filesystem_size_bytes Filesystem size in bytes.
# TYPE node_filesystem_size_bytes gauge
node_filesystem_size_bytes{device="/dev/sda1",fstype="ext4",mountpoint="/"} 1.9970912256e+10
node_filesystem_size_bytes{device="/dev/sda15",fstype="vfat",mountpoint="/boot/efi"} 1.2594176e+08
# HELP node_forks_total Total number of forks.
# TYPE node_forks_total counter
node_forks_total 90780
# HELP node_intr_total Total number of interrupts serviced.
# TYPE node_intr_total counter
node_intr_total 1.6144664e+07
# HELP node_load1 1m load average.
# TYPE node_load1 gauge
node_load1 0.09
# HELP node_arp_entries ARP entries by device
# TYPE node_arp_entries gauge
node_arp_entries{device="eth0"} 3

Step 2 - Install Prometheus server on collector node

The prometheus server is used to collect the exposed metrics of node_exporter into a time series database called OpenTSDB. It can be installed as a single file deployment as follows.

apt update
apt install prometheus jq

The Prometheus server must be configured to know the node_exporter metrics endpoint address and the scrape interval. # vim /etc/prometheus/prometheus.yml

---
global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'node_exporter'
    metrics_path: "/metrics"
    static_configs:
      - targets:
        - 'node1.example.com:9100'

Finally, start Prometheus and enable it to start automatically on reboot.

systemctl start prometheus.service
systemctl enable prometheus.service

systemctl status prometheus confirms the successful launch of the service.

To view the status of monitoring targets, you can execute the following command. The output should look as follows:

root@prometheus.example.com:~# curl [::1]:9090/api/v1/targets 2>/dev/null| jq -r '.data.activeTargets[]|.scrapeUrl,.health,.lastError'
http://node1.example.com:9100/metrics
up

Step 3 - Setup Grafana dashboard server

Now that the prometheus server is collecting frequently metrics from our node, the data is stored for 15 days. There are multiple ways to display the metrics. The widely used way is to setup Grafana which is a dashboard server for multiple data sources.

Enable the Grafana repository and install Grafana with the following commands:

apt-get install -y apt-transport-https
apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/deb stable main" | tee -a /etc/apt/sources.list.d/grafana.list

apt update
apt install grafana

Finally, enable and start the Grafana server daemon:

systemctl start grafana-server.service
systemctl enable grafana-server.service

Step 4 - Configure Grafana data source and dashboard

The Grafana server provides a web interface. Please be careful to use only credentials for testing while the connection between your browser and the dashboard server is not secure.

You can access the interface by browsing to

http://prometheus.example.com:3000/

The default credentials are as follows:

Username: admin
Password: admin

After login, you are asked to change this password. In this state, there is no data source or dashboard configured. First, add a data source.

Select type Prometheus and type http://localhost:9090 into the URL field. After that, click Save & test to create the data source.

There are two ways to create dashboards. You can create every panel individually or import predefined dashboards from the community. For displaying node_exporter metrics, a good dashboard can be found here.

Copy the ID to import the dashboard into your Grafana server.

Press the import button, insert the copied ID and click load. You need to select the Prometheus data source for the dashboard:

Finally, the Dashboard is imported and displays data of the monitored node.

Conclusion

You now have configured a Prometheus stack and can monitor a single node. This setup has the capability of monitoring more nodes/services. Please consider to apply a reverse proxy in front of the Grafana frontend as described in this tutorial.

Tutoriais para uso no Raspberry Pi 3 B

Pages