If you've been using Kubernetes for a while, you've almost certainly thought about choosing a tool to monitor the state of your cluster nodes. This article is just about that: we will tell you about the available tools and how to work with them. We will look at how to monitor Kubernetes with VictoriaMetrics and Prometheus + Grafana and show you how to avoid common problems.
Any Kubernetes cluster consists of worker nodes, which are individual servers running applications. The nodes run under the control of the Kubernetes Control Plane master server. Monitoring includes tracking the state of the master server itself and various system components. However, there are no built-in solutions for monitoring the state of individual nodes that make up the cluster.
It's true that Kubernetes Control Plane has a Kubelet module that provides information about the state of the API server, the main Kubernetes Control Plane node, and CPU and RAM consumption by containers. However, collecting metrics from user applications has certain difficulties. Applications are constantly updated, which entails the need to update configuration files. It is time-consuming and hardly possible for large applications. That is why we have to use third-party tools, which we'll discuss further in this article.
VictoriaMetrics is a time series database (TSDB) compatible with the Kubernetes API. It works on a subscription basis, responding to changes in pods (a pod is a set of containers sharing a common namespace).
To install the latest version of VictoriaMetrics, open this page on GitHub, scroll down, and select the version for your operating system. Next, unzip the .gz
archive, and you can start running it. There are quite a lot of startup parameters, so we recommend reading the documentation. Now, let's check the main VictoriaMetrics tools that will help us with Kube monitoring.
The relabel_config
, a set of sequential rules, is used to monitor the state of the pod. It consists of the following 6 lines:
source_labels
, where you can specify the list of elements (labels) to perform certain operations on;
action
is the action to be performed on labels (e.g. replace
, keep
, drop
);
modulus
is a rarely used hashmod
module value, required to work with targets collected as a result of monitoring (for example, to scale them);
regex
is a regular expression for matching
, and the values specified in this string can be used to replace the label
value;
separator
is a separator for source_labels
;
target_label
is where you specify the label to which the result of the action will be written.
The above rules are used for actions with labels (e.g., they can be added, deleted, and their names and values can be changed) and filtering of detected targets.
As for actions, VictoriaMetrics supports all actions available for Prometheus (we will talk about it below). Besides the already mentioned actions replace
, keep
, drop
, there are also:
labelmap
labelkeep
labeldrop
hashmod
In addition, VictoriaMetrics has its own set of unique actions:
replace_all
keep_metrics
drop_metrics
labelmap_all
keep_if_equal
drop_if_equal
That's how it works:
scrape_configs:
- job_name: k8s-pods
kubernetes_sd_configs:
- role: pod
relabel_config:
- source_labels: [_meta_kubernetes_pod_namespace]
- action: keep
- regex: ^default$
This entry means that we only store pods that are in the default namespace, as indicated by the expression in the regex
line.
VictoriaMetrics also supports selecting objects from the Kubernetes API according to their role, which, again, can be assigned by TSDB tools. The roles can be:
Pod. Used to return a list with labels
for each container port with an IP pod.
Service. Objects with this role represent subscriptions to changes in specific services. For each port of a specific service, a label
is created with metadata like server_name.namespace.svc
and port number.
Endpoints. Used to link services and pods, displaying the latter's state. In addition, Endpoints direct traffic only to currently up-and-running pods.
Endpointslice. Same as Endpoints, but with modified label
prefix names. Endpointslice is used when the number of endpoints is large (more than a thousand) so that Kubernetes does not cut them off.
Node. That's your node with the relevant information.
Ingress. This role is used only for monitoring. It writes the host name
into the address, and the path
into a separate variable.
These roles are assigned using the role line (shown in the code above).
Also, you will probably need to filter pods to collect metrics only from selected pods. This can be implemented with another useful tool, Prometheus.
Strictly speaking, Prometheus is a whole set of utilities that often comes with Grafana, a web interface that visualizes what Prometheus collects. You can install this suite in several ways. For example, with helm
. After cloning the repository, enter the following commands:
cd charts/stable/prometheus-operator
helm dependency update
And then:
chelm install --name prometheus --namespace monitoring prometheus-operator
The Prometheus pod contains two useful plugins: config-reloader
for tracking changes and reloading the prometheus.yaml
configuration file, and rules-configmap-reloader
for tracking changes in Prometheus rules.
The Alertmanager pod contains the manager itself, designed to automatically create notifications for specified rules, and config-reloader
, a plugin for reloading the manager in case of changes in the configuration file.
In the Grafana pod, you will find the web interface that takes data for mapping from the Prometheus database and the Grafana-sc-dashboard
for describing ConfigMap
resources and generating JSONs.
After launching the pods, enter the following commands:
kubectl port-forward prometheus-prometheus-prometheus-oper-prometheus-0 9090:9090
Open http://localhost:9090
in a browser. To view the list of collected metrics, enter:
kubectl get servicemonitors.monitoring.coreos.com
In the previous chapter, we wrote that Prometheus helps filter pods. To do this, we must write the following directives in the scrape_config.yaml
:
relabel_configs:
- source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
action: keep
regex: true
- source_labels: [__meta_kubernetes_namespace]
action: replace
target_label: kubernetes_namespace
- source_labels: [__meta_kubernetes_pod_name]
action: replace
target_label: kubernetes_pod_name
In addition, you will need to specify the port number by creating a string like this:
prometheus.io/port: <port_number>
And the path where the metrics will be fetched:
prometheus.io/path: <path>
With Prometheus, monitoring almost any object, including servers, is easy, and the OS installed (Windows or Linux) does not matter. Among the parameters you can monitor are, for example, RAM, CPU and disk space usage. As to applications, you can monitor the number of errors, the level of requests, and the time of their execution.
You don't need additional settings for receiving metrics from some services, while special scripts called exporters
are used to export the rest. They are already configured, so you only need to find and install the right one from here. All metrics are stored in TSDB, and visualized through Grafana using an API that sends requests to TSDB. Of course, Grafana is not the only way to visualize Prometheus metrics, but it is probably the most convenient.
When you need to collect metrics from multiple ports with the Prometheus configuration, a problem may arise because prometheus.io/port can only return a single port. You can use different workarounds here—for example, those offered by VictoriaMetrics.
It has a tool called VictoriaMetrics Operator that uses Custom Resource Definitions to extend the Kubernetes API. This tool has extensive documentation describing all Custom Resources in detail. We are interested in VMPodScrape, which is launched as follows:
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMPodScrape
metadata:
name: vm
namespace: monitoring
spec:
podMetricsEndpoints:
- port: http
- port: backup-metrics
selector:
matchLabels:
app.kubernetes.io/name: vm
Note that we have specified two ports. But you can enable multi-port collection in a different way, using the familiar rule sets:
scrape configs
- job name: apiserver/0
kubernetes_sd_configs:
- role: endpoints
scheme: http
relabel_configs:
…
- action: keep
source_labels:
- __meta_kubernetes_service_label_component
regex: apiserver
- action: keep
source_labels:
- __meta_kubernetes_endpoint_port_name
regex: https
- job name: coredns/0
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
…
- action: keep
source_labels:
- __meta_kubernetes_service_label_app
regex: coredns
- action: keep
source_labels:
- __meta_kubernetes_endpoint_port_name
regex: http-metrics
This approach of creating separate blocks for applications in the config file is also quite workable. So choose the solution you prefer.
Both VictoriaMetrics (VM) and Prometheus are convenient tools for working with metrics. But which one is more cost-effective? Such tests were conducted on Google Compute Engine at the end of 2020, and it turned out that:
VM is 7 times more economical in terms of stored data space;
disk read bursts are 6.3 times higher on Prometheus;
VM is 5.3 times more economical in terms of RAM usage (4.3 vs. 23 Gb for Prometheus, which it needs for stable operation).
As we can see, VictoriaMetrics allows you to significantly save on hardware.