Migrating from Promscrape V1 to V2
To use the latest features, such as Service Discovery and Monitoring Integrations, enable the prom_service_discovery
option in your environment. This configuration is controlled by the prom_discovery_service
parameter in the dragent.yaml
file.
You can either modify directly the dragent.yaml
file or add the parameter in the values.yaml file of the Helm chart.
prometheus:
prom_service_discovery: true # Enables Promscrape V2 (enabled by default)
For the sysdig-deploy chart, set the configuration under the agent.sysdig.settings
key.
agent:
sysdig:
settings:
prometheus:
prom_service_discovery: true # Enables Promscrape V2 (enabled by default)
For the shield chart, write the configuration under the host.additional_settings
key.
host:
additional_settings:
prometheus:
prom_service_discovery: true # Enables Promscrape V2 (enabled by default)
How to check which version of Promscrape is running
To check which version of Promscrape is running, you can use the following command in the Agent Console:
$ prometheus ?
$ prometheus config ?
$ prometheus config show ?
Similarly, you can look for prometheus.prom_discovery_service:
in the draios.log
file to check if Promscrape V2 is enabled.
Compare Promscrape V1 and V2
The main difference between V1 and V2 is how scrape targets are determined.
In V1 targets are found through process-filtering
rules configured in dragent.yaml
or dragent.default.yaml
(if no rules are given in dragent.yaml
). The process-filtering
rules are applied to all the
running processes on the host. Matches are made based on process
attributes, such as process name or TCP ports being listened to, as well
as associated contexts from docker or Kubernetes, such as container
labels or Kubernetes annotations.
With Promscrape V2, scrape targets are determined by scrape_configs
fields in a prometheus.yaml
file (or the prometheus-v2.default.yaml
file if no prometheus.yaml
exists). The scrape_config
settings are
compatible with the normal Prometheus configuration. Here is an example:
global:
scrape_interval: 10s
scrape_configs:
- job_name: 'my_pod_job'
sample_limit: 40000
tls_config:
insecure_skip_verify: true
kubernetes_sd_configs:
- role: pod
relabel_configs:
# Look for pod name starting with "my_pod_prefix" in namespace "my_namespace"
- action:
source_labels: [__meta_kubernetes_namespace,__meta_kubernetes_pod_name,__meta_kubernetes_pod_label]
separator: /
regex: my_namespace/my_pod_prefix.+
- action: keep
source_labels: [__meta_kubernetes_pod_label_app]
regex: my_app_metrics
# In those pods try to scrape from port 9876
- source_labels: [__address__]
action: replace
target_label: __address__
regex: (.+?)(\\:\\d)?
replacement: $1:9876
# Trying to ensure we only scrape local targets
# __HOSTIPS__ is replaced by promscrape with a regex list of the IP addresses
# of all the active network interfaces on the host
- action: keep
source_labels: [__meta_kubernetes_pod_host_ip]
regex: __HOSTIPS__
# Holding on to pod-id and container name so we can associate the metrics
# with the container (and cluster hierarchy)
- action: replace
source_labels: [__meta_kubernetes_pod_uid]
target_label: sysdig_k8s_pod_uid
- action: replace
source_labels: [__meta_kubernetes_pod_container_name]
target_label: sysdig_k8s_pod_container_name
Migrate Using Default Configuration
The default configuration for Promscrape V1
triggers the scraping based on standard Kubernetes pod annotations
and container labels. Users running Kubernetes with Promscrape V1
default rules and
triggering scraping based on pod annotations do not need to take any action to migrate to V2.
The migration happens automatically.
The default configuration for Promscrape V2
currently triggers scraping only based on the standard Kubernetes
pod annotations leveraging the Prometheus native service discovery.
Example Pod Annotations
Annotation | Value | Description |
---|---|---|
|
| Required field. |
| The port number to scrape | Optional. It will scrape all pod-registered ports if omitted. |
| < | The default is |
| The URL | The default is |
Non-Kubernetes Environments
Users operating non-Kubernetes environments might need to continue
using V1 for now, depending on how scraping is triggered. As of
today Promscrape V2
doesn’t support leveraging container and
Docker labels to discover Prometheus metrics endpoints. If your
environment is one of these, define static jobs with the IP:port
to be scrapped.
Example Static Job
- job_name: 'static10'
static_configs:
- targets: ['localhost:5010']
Migrate Using Custom Rules
If you relying on custom process_filter
rules to collect metrics, use
any method using standard Prometheus configuration syntax to scrape the
endpoints. We recommend one of the following:
- Adopt the standard approach of adding the standard Prometheus annotations to their pods. For more information, see Migrate Using Default Configuration.
- Write a Prometheus
scrape_config
by using Kubernetes pods service discovery and use the appropriate pod metadata to trigger their scrapes.
See the below example for converting your process_filter
rules to Prometheus terminology.
process_filter (Promscrape V1) | Prometheus (Promscrape V2) |
---|---|
|
|
|
|
| Not supported. |
| Not supported. |
| Not supported. |
|
|
| Not supported. |
|
|
| Appchecks are not compatible with Promscrape v2. See Configure Monitoring Integrations for supported integrations. |
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.