Filtering Prometheus Metrics

As of Sysdig agent 9.8.0, a lightweight Prometheus server is embedded in agents named promscrape and a prometheus.yaml file is included as part of configuration files. Using the open source Prometheus capabilities, Sysdig leverages a Prometheus feature to allow you to filter Prometheus metrics at the source before ingestion. To do so, you will:

  • Ensure that the Prometheus scraping is enabled in the  dragent.yaml file.

    prometheus:
  enabled: true

  • On agent v9.8.0 and above, enable the feature by setting the use_promscrape parameter to true in the dragent.yaml. See Enable Filtering at Ingestion. On agent v10.0 promscrape is used by default for scraping metrics.

  • Edit the configuration in the prometheus.yaml file. See Edit Prometheus Configuration File.

    Sysdig-specific configuration is found in the prometheus.yaml file.

Enable Filtering at Ingestion

On agent v9.8.0, in order for target filtering to work, the use_promscrape parameter in the dragent.yaml must be set to true. For more information on configuration, see Configuring Sysdig Agent.Configuring Sysdig Agent

use_promscrape: true

Note

On agent v10.0, use_promscrape is enabled by default. Implies, promscrape is used for scraping Prometheus metrics.

Filtering configuration is optional. The absence of prometheus.yaml  will not change the existing behavior of the agent.

Edit Prometheus Configuration File

About the Prometheus Configuration File

The prometheus.yaml file contains mostly the filtering/relabeling configuration in a list of key-value pairs, representing target process attributes.

You replace keys and values with the desired tags corresponding to your environment.

In this file, you will configure the following:

  • Default scrape interval (optional).

    For example: scrape_interval: 10s

  • Of the labeling parameters that Prometheus offers, Sysdig supports only metric_relabel_configs. The relabel_config parameter is not supported.

  • Zero or more process-specific filtering configurations (optional).

    See Kubernetes Environments and Docker Environments

    The filtering configuration includes:

    • Filtering rules

      For example: - source_labels: [container_label_io_kubernetes_pod_name]

    • Limit on number of scraped samples (optional)

      For example: sample_limit: 2000

  • Default filtering configuration (optional). See Default Configuration.

    The filtering configuration includes:

    • Filtering rules

      For example: - source_labels: [car]

    • Limit on number of scraped samples (optional)

      For example: sample_limit: 2000

The prometheus.yaml file is installed alongside dragent.yaml. For the most part, the syntax of prometheus.yaml complies with the standard Prometheus configuration. 

Default Configuration

A configuration with empty key-value pairs is considered a default configuration. The default configuration will be applied to all the processes to be scraped that don't have a matching filtering configuration. In Sample Prometheus Configuration File, the job_name: 'default' section represents the default configuration.

Kubernetes Environments

If the agent runs in Kubernetes environments (Open Source/OpenShift/GKE), include the following Kubernetes objects as key-value pairs. See Agent Install: Kubernetes for details on agent installation.

For example:

sysdig_sd_configs:
- tags:
    namespace: backend
    deployment: my-api

In addition to the aforementioned tags, any of these object types can be matched against:

daemonset: my_daemon
deployment: my_deployment
hpa: my_hpa
namespace: my_namespace
node: my_node
pod: my_pode
replicaset: my_replica
replicationcontroller: my_controller
resourcequota: my_quota
service: my_service
stateful: my_statefulset

For Kubernetes/OpenShift/GKE deployments, prometheus.yaml shares the same ConfigMap with dragent.yaml.

Docker Environments

In Docker environments, include attributes such as container, host, port, and more. For example:

sysdig_sd_configs:
- tags:
    host: my-host
    port: 8080

For Docker-based deployments, prometheus.yaml can be mounted from the host.

Sample Prometheus Configuration File

global:
  scrape_interval: 20s
scrape_configs:
- job_name: 'default'
  sysdig_sd_configs: # default config
  relabel_configs:
- job_name: 'my-app-job'
  sample_limit: 2000
  sysdig_sd_configs:  # apply this filtering config only to my-app
  - tags:
      namespace: backend
      deployment: my-app
  metric_relabel_configs:
  # Drop all metrics starting with http_
  - source_labels: [__name__]
    regex: "http_(.+)"
    action: drop
  metric_relabel_configs:
  # Drop all metrics for which the city label equals atlantis
  - source_labels: [city]
    regex: "atlantis"
    action: drop