This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

    Filtering Prometheus Metrics

    As of Sysdig agent 9.8.0, a lightweight Prometheus server is embedded in agents named promscrape and a prometheus.yaml file is included as part of configuration files. Using the open source Prometheus capabilities, Sysdig leverages a Prometheus feature to allow you to filter Prometheus metrics at the source before ingestion. To do so, you will:

    • Ensure that the Prometheus scraping is enabled in the  dragent.yaml file.

      prometheus:
        enabled: true
      
    • On agent v9.8.0 and above, enable the feature by setting the use_promscrape parameter to true in the dragent.yaml. See Enable Filtering at Ingestion. On agent v10.0 promscrape is used by default for scraping metrics.

    • Edit the configuration in the prometheus.yaml file. See Edit Prometheus Configuration File.

      Sysdig-specific configuration is found in the prometheus.yaml file.

    Enable Filtering at Ingestion

    On agent v9.8.0, in order for target filtering to work, the use_promscrape parameter in the dragent.yaml must be set to true. For more information on configuration, see Configuring Sysdig Agent.

    use_promscrape: true
    

    On agent v10.0, use_promscrape is enabled by default. Implies, promscrape is used for scraping Prometheus metrics.

    Filtering configuration is optional. The absence of prometheus.yaml  will not change the existing behavior of the agent.

    Edit Prometheus Configuration File

    About the Prometheus Configuration File

    The prometheus.yaml file contains mostly the filtering/relabeling configuration in a list of key-value pairs, representing target process attributes.

    You replace keys and values with the desired tags corresponding to your environment.

    In this file, you will configure the following:

    • Default scrape interval (optional).

      For example: scrape_interval: 10s

    • Of the labeling parameters that Prometheus offers, Sysdig supports only metric_relabel_configs. The relabel_config parameter is not supported.

    • Zero or more process-specific filtering configurations (optional).

      See Kubernetes Environments and Docker Environments

      The filtering configuration includes:

      • Filtering rules

        For example: - source_labels: [container_label_io_kubernetes_pod_name]

      • Limit on number of scraped samples (optional)

        For example: sample_limit: 2000

    • Default filtering configuration (optional). See Default Configuration.

      The filtering configuration includes:

      • Filtering rules

        For example: - source_labels: [car]

      • Limit on number of scraped samples (optional)

        For example: sample_limit: 2000

    The prometheus.yaml file is installed alongside dragent.yaml. For the most part, the syntax of prometheus.yaml complies with the standard Prometheus configuration

    Default Configuration

    A configuration with empty key-value pairs is considered a default configuration. The default configuration will be applied to all the processes to be scraped that don’t have a matching filtering configuration. In Sample Prometheus Configuration File, the job_name: 'default' section represents the default configuration.

    Kubernetes Environments

    If the agent runs in Kubernetes environments (Open Source/OpenShift/GKE), include the following Kubernetes objects as key-value pairs. See Agent Install: Kubernetes for details on agent installation.

    For example:

    sysdig_sd_configs:
    - tags:
        namespace: backend
        deployment: my-api
    

    In addition to the aforementioned tags, any of these object types can be matched against:

    daemonset: my_daemon
    deployment: my_deployment
    hpa: my_hpa
    namespace: my_namespace
    node: my_node
    pod: my_pode
    replicaset: my_replica
    replicationcontroller: my_controller
    resourcequota: my_quota
    service: my_service
    stateful: my_statefulset
    

    For Kubernetes/OpenShift/GKE deployments, prometheus.yaml shares the same ConfigMap with dragent.yaml.

    Docker Environments

    In Docker environments, include attributes such as container, host, port, and more. For example:

    sysdig_sd_configs:
    - tags:
        host: my-host
        port: 8080
    

    For Docker-based deployments, prometheus.yaml can be mounted from the host.

    Sample Prometheus Configuration File

    global:
      scrape_interval: 20s
    scrape_configs:
    - job_name: 'default'
      sysdig_sd_configs: # default config
      relabel_configs:
    - job_name: 'my-app-job'
      sample_limit: 2000
      sysdig_sd_configs:  # apply this filtering config only to my-app
      - tags:
          namespace: backend
          deployment: my-app
      metric_relabel_configs:
      # Drop all metrics starting with http_
      - source_labels: [__name__]
        regex: "http_(.+)"
        action: drop
      metric_relabel_configs:
      # Drop all metrics for which the city label equals atlantis
      - source_labels: [city]
        regex: "atlantis"
        action: drop