Promscrape is the lightweight Prometheus server in the Sysdig agent. An
updated version of
promscrape, named Promscrape V2 is available. This configuration is controlled by the
prom_discovery_service parameter in the
dragent.yaml file. To use the latest features, such as Service Discovery and Monitoring Integrations, you need to have this option enabled in your environment.
Compare Promscrape V1 and V2
The main difference between V1 and V2 is how scrape targets are determined.
In v1 targets are found through
process-filtering rules configured in
dragent.default.yaml (if no rules are given in
process-filtering rules are applied to all the
running processes on the host. Matches are made based on process
attributes, such as process name or TCP ports being listened to, as well
as associated contexts from docker or Kubernetes, such as container
labels or Kubernetes annotations.
With Promscrape V2, scrape targets are determined by
fields in a
prometheus.yaml file (or the
file if no
prometheus.yaml exists). Because
promscrape is adapted
from the open-source Prometheus server, the
scrape_config settings are
compatible with the normal Prometheus configuration. Here is an example:
global: scrape_interval: 10s scrape_configs: - job_name: 'my_pod_job' sample_limit: 40000 tls_config: insecure_skip_verify: true kubernetes_sd_configs: - role: pod relabel_configs: # Look for pod name starting with "my_pod_prefix" in namespace "my_namespace" - action: source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_name,__meta_kubernetes_pod_label] separator: / regex: my_namespace/my_pod_prefix.+ - action: keep source_labels: [__meta_kubernetes_pod_label_app] regex: my_app_metrics # In those pods try to scrape from port 9876 - source_labels: [__address__] action: replace target_label: __address__ regex: (.+?)(\\:\\d)? replacement: $1:9876 # Trying to ensure we only scrape local targets # __HOSTIPS__ is replaced by promscrape with a regex list of the IP addresses # of all the active network interfaces on the host - action: keep source_labels: [__meta_kubernetes_pod_host_ip] regex: __HOSTIPS__ # Holding on to pod-id and container name so we can associate the metrics # with the container (and cluster hierarchy) - action: replace source_labels: [__meta_kubernetes_pod_uid] target_label: sysdig_k8s_pod_uid - action: replace source_labels: [__meta_kubernetes_pod_container_name] target_label: sysdig_k8s_pod_container_name
Migrate Using Default Configuration
The default configuration for Promscrape v1 triggers the scraping based on standard Kubernetes pod annotations and container labels. The default configuration for v2 currently triggers scraping only based on the standard Kubernetes pod annotations leveraging the Prometheus native service discovery.
Example Pod Annotations
The port number to scrape
Optional. It will scrape all pod-registered ports if omitted.
The default is
The default is
Example Static Job
- job_name: 'static10' static_configs: - targets: ['localhost:5010']
Users running Kubernetes with Promscrape v1 default rules and triggering scraping based on pod annotations need not take any action to migrate to v2. The migration happens automatically.
Users operating non-Kubernetes environments might need to continue using v1 for now, depending on how scraping is triggered. As of today
promscrape.v2doesn’t support leveraging container and Docker labels to discover Prometheus metrics endpoints. If your environment is one of these, define static jobs with the
IP:portto be scrapped.
Migrate Using Custom Rules
If you relying on custom
process_filter rules to collect metrics, use
any method using standard Prometheus configuration syntax to scrape the
endpoints. We recommend one of the following:
- Adopt the standard approach of adding the standard Prometheus annotations to their pods. For more information, see Migrate Using Default Configuration.
- Write a Prometheus
scrape_configby using Kubernetes pods service discovery and use the appropriate pod metadata to trigger their scrapes.
See the below example for converting your
process_filter rules to
Appchecks are not compatble with Promscrape v2. See Configure Monitoring Integrations for supported integrations.
If you have any queries related to promscrape migration, contact Sysdig Support.