Filtering Prometheus Metrics
As of Sysdig agent 9.8.0, a lightweight Prometheus server is embedded in agents named promscrape
and a prometheus.yaml
file is included as part of configuration files. Using the open source Prometheus capabilities, Sysdig leverages a Prometheus feature to allow you to filter Prometheus metrics at the source before ingestion. To do so, you will:
Ensure that the Prometheus scraping is enabled in the
dragent.yaml
file.prometheus: enabled: true
On agent v9.8.0 and above, enable the feature by setting the
use_promscrape
parameter to true in thedragent.yaml
. See Enable Filtering at Ingestion. On agent v10.0 promscrape is used by default for scraping metrics.Edit the configuration in the
prometheus.yaml
file. See Edit Prometheus Configuration File.Sysdig-specific configuration is found in the
prometheus.yaml
file.
Enable Filtering at Ingestion
On agent v9.8.0, in order for target filtering to work, the use_promscrape
parameter in the dragent.yaml
must be set to true. For more information on configuration, see Configuring Sysdig Agent.
use_promscrape: true
Note
On agent v10.0, use_promscrape
is enabled by default. Implies, promscrape is used for scraping Prometheus metrics.
Filtering configuration is optional. The absence of prometheus.yaml
will not change the existing behavior of the agent.
Edit Prometheus Configuration File
About the Prometheus Configuration File
The prometheus.yaml
file contains mostly the filtering/relabeling configuration in a list of key-value pairs, representing target process attributes.
You replace keys and values with the desired tags corresponding to your environment.
In this file, you will configure the following:
Default scrape interval (optional).
For example:
scrape_interval: 10s
Of the labeling parameters that Prometheus offers, Sysdig supports only metric_relabel_configs. The relabel_config parameter is not supported.
Zero or more process-specific filtering configurations (optional).
See Kubernetes Environments and Docker Environments
The filtering configuration includes:
Filtering rules
For example:
- source_labels: [container_label_io_kubernetes_pod_name]
Limit on number of scraped samples (optional)
For example:
sample_limit: 2000
Default filtering configuration (optional). See Default Configuration.
The filtering configuration includes:
Filtering rules
For example:
- source_labels: [car]
Limit on number of scraped samples (optional)
For example:
sample_limit: 2000
The prometheus.yaml
file is installed alongside dragent.yaml
. For the most part, the syntax of prometheus.yaml
complies with the standard Prometheus configuration.
Default Configuration
A configuration with empty key-value pairs is considered a default configuration. The default configuration will be applied to all the processes to be scraped that don't have a matching filtering configuration. In Sample Prometheus Configuration File, the job_name: 'default'
section represents the default configuration.
Kubernetes Environments
If the agent runs in Kubernetes environments (Open Source/OpenShift/GKE), include the following Kubernetes objects as key-value pairs. See Agent Install: Kubernetes for details on agent installation.
For example:
sysdig_sd_configs: - tags: namespace: backend deployment: my-api
In addition to the aforementioned tags, any of these object types can be matched against:
daemonset: my_daemon deployment: my_deployment hpa: my_hpa namespace: my_namespace node: my_node pod: my_pode replicaset: my_replica replicationcontroller: my_controller resourcequota: my_quota service: my_service stateful: my_statefulset
For Kubernetes/OpenShift/GKE deployments, prometheus.yaml
shares the same ConfigMap with dragent.yaml
.
Docker Environments
In Docker environments, include attributes such as container, host, port, and more. For example:
sysdig_sd_configs: - tags: host: my-host port: 8080
For Docker-based deployments, prometheus.yaml
can be mounted from the host.
Sample Prometheus Configuration File
global: scrape_interval: 20s scrape_configs: - job_name: 'default' sysdig_sd_configs: # default config relabel_configs: - job_name: 'my-app-job' sample_limit: 2000 sysdig_sd_configs: # apply this filtering config only to my-app - tags: namespace: backend deployment: my-app metric_relabel_configs: # Drop all metrics starting with http_ - source_labels: [__name__] regex: "http_(.+)" action: drop metric_relabel_configs: # Drop all metrics for which the city label equals atlantis - source_labels: [city] regex: "atlantis" action: drop