Working with Prometheus Metrics

The Sysdig Monitor Agent uses its visibility to all running processes (at both the host and container levels) to find eligible targets for scraping Prometheus metrics. By default, no scraping is attempted. Once the feature is enabled, the agent assembles a list of eligible targets, apply filtering rules, and sends back to Sysdig collector:

  1. A process is determined to be eligible for possible scraping if it positively matches against a series of Process Filter include/exclude rules. See Process Filter for more information.

  2. The Agent will then attempt to scrape an eligible process at a /metrics endpoint on all of its listening TCP ports unless the additional configuration is present to restrict scraping to a subset of ports and/or another endpoint name.

  3. As of agent v9.8.0, filtering metrics at ingestion can be enabled. If enabled, filtering rules are applied at ingestion as it receives the metrics. See Filtering Prometheus Metrics for more information.

  4. Upon receiving the metrics, the agent applies the following before sending them to the Sysdig collector.

    • Applies the global metrics_filter rules.

    • Imposes the configured max_metrics to limit the number of metrics.

The metrics ultimately appear in the Sysdig Monitor Explore interface in the Prometheus section.


Within the set of eligible processes/ports/endpoints, the Agent uses an auto-discover approach to scrape only ports that are exporting Prometheus metrics and will stop attempting to scrape or retry on ports based on how they respond to attempts to connect and scrape them. It is therefore strongly recommended that you create a configuration that restricts the process and ports for attempted scraping to the minimum expected range for your exporters. This minimizes the potential for unintended side-effects in both the Agent and your applications due to repeated failed connection attempts.