This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

    Migrating from Promscrape V1 to V2

    Promscrape is the lightweight Prometheus server in the Sysdig agent. An updated version of promscrape, named Promscrape V2 is available. This configuration is controlled by the prom_discovery_service parameter in the dragent.yaml file. To use the latest features, such as Service Discovery and Monitoring Integrations, you need to have this option enabled in your environment.

    Compare Promscrape V1 and V2

    The main difference between V1 and V2 is how scrape targets are determined.

    In v1 targets are found through process-filtering rules configured in dragent.yaml or dragent.default.yaml (if no rules are given in dragent.yaml).The process-filtering rules are applied to all the running processes on the host. Matches are made based on process attributes, such as process name or TCP ports being listened to, as well as associated contexts from docker or Kubernetes, such as container labels or Kubernetes annotations.

    With Promscrape V2, scrape targets are determined by scrape_configs fields in a prometheus.yaml file (or the prometheus-v2.default.yaml file if no prometheus.yaml exists). Because promscrape is adapted from the open-source Prometheus server, the scrape_config settings are compatible with the normal Prometheus configuration. For more information, see Configuration.

    Migrate Using Default Configuration

    The default configuration for Promscrape v1 triggers the scraping based on standard Kubernetes pod annotations and container labels. The default configuration for v2 currently triggers scraping only based on the standard Kubernetes pod annotations leveraging the Prometheus native Kubernetes service discovery. Use the following configuration:

    Annotation

    Value

    Description

    prometheus.io/scrape

    true

    Required field.

    prometheus.io/port

    The port number to scrape

    Optional. It will scrape all pod-registered ports if omitted.

    prometheus.io/scheme

    <http|https>

    The default is http.

    prometheus.io/path

    (required field)

    The URL

    The default is /metrics.

    • Users running Kubernetes with Promscrape v1 default rules and triggering scraping based on pod annotations, need not take any action to migrate to v2. The migration happens automatically.

    • Users operating non-Kubernetes environments might need to continue using v1 for now, depending on how scraping is triggered. As of today promscrape.v2 doesn’t support leveraging container and Docker labels to discover Prometheus metrics endpoints. If your environment is one of these, define static jobs with the IP:port to be scrapped.

    Migrate Using Custom Rules

    If you relying on custom process_filter rules to collect metrics, use any method using standard Prometheus configuration syntax to scrape the endpoints. We recommend one of the following:

    • Adopt the standard approach of adding the standard Prometheus annotations to their pods. For more information, see Migrate Using Default Configuration.
    • Write a Prometheus scrape_config by using Kubernetes pods service discovery and use the appropriate pod metadata to trigger their scrapes.

    See the below example for converting your process_filter rules to Prometheus terminology.

    process_filter

    Prometheus

    - include:
        kubernetes.pod.annotation.sysdig.com/test: true
    - action: keep
      source_labels: [__meta_kubernetes_pod_annotation_sysdig_com_test]
      regex: true
    - include:
        kubernetes.pod.label.app: sysdig
    - action: keep
      source_labels: [__meta_kubernetes_pod_label_app]
      regex: 'sysdig'
    -include:
       container.label.com.sysdig.test: true

    Not supported.

    - include:
        process.name: test

    Not supported.

    - include:
        process.cmdline: sysdig-agent

    Not supported.

    - include:
        port: 8080
    - action: keep
      source_labels: [__meta_kubernetes_pod_container_port_number]
      regex: '8080'
    - include:
        container.image: sysdig-agent

    Not supported.

    - include:
        container.name: sysdig-agent
    - action: keep
      source_labels: [__meta_kubernetes_pod_container_name]
      regex: 'sysdig-agent'
    - include:
        appcheck.match: sysdig

    Appchecks are not compatble with Promscrape v2. See Configure Monitoring Integrations for supported integrations.