Sysdig Documentation

Working with Prometheus Metrics

The Sysdig Monitor Agent uses its visibility to all running processes (at both the host and container levels) to find eligible targets for scraping Prometheus metrics. By default, no scraping is attempted. Once the feature is enabled, the Agent assembles a list of eligible targets in two steps:

  1. A process is determined to be eligible for possible scraping if it positively matches against a series of Process Filter include/exclude rules.

  2. The Agent will then attempt to scrape an eligible process at a /metrics endpoint on all of its listening TCP ports, unless the additional configuration is present to restrict scraping to a subset of ports and/or another endpoint name.

The metrics ultimately appear in the Sysdig Monitor Explore interface in the Prometheus section.

373654583.png

Within the set of eligible processes/ports/endpoints, the Agent uses an auto-discover approach to scrape only ports that are exporting Prometheus metrics, and will stop attempting to scrape or retry on ports based on how they respond to attempts to connect and scrape them. It is therefore strongly recommended that you create a configuration that restricts the process and ports for attempted scraping to the minimum expected range for your exporters. This minimizes the potential for unintended side-effects in both the Agent and your applications due to repeated failed connection attempts.