Dynamic Sampling

Dynamic sampling supports scraping a rotating set of Prometheus endpoints based on the total amount of time series scraped from each endpoint. With dynamic sampling turned on, you get consistent and up-to-date data from every Prometheus endpoints on a given node at dynamic intervals, while maintaining the data collection frequency and fidelity of Prometheus metrics via the Sysdig agent.

Sysdig’s ability to collect and process the volumes of data scraped from different Prometheus endpoints is controlled by a mix of the amount of time serires scraped from each endpoint, the total amount of Prometheus time series collected by the agent in each time window, and the frequency at which Sysdig agent collects and sends the data to the Sysdig backend. On nodes with multiple Prometheus endpoints sending high volumes of time series, the Sysdig agent may need to skip some endpoints depending on the overall volume and scrape frequency. Dynamic sampling addresses this by cycling through individual Prometheus endpoints and scraping the latest time series from each endpoint on a rotational basis such that all time series from all Prometheus endpoints are processed at dynamic intervals. The result is more timeseries being scraped and processed overall at a lower frequency. For example, instead of receiving 50,000 timeseries every 10 seconds, you might receive 100,000 timeseries every 20 seconds.

Dynamic Sampling Considerations

  • Any alerts that depend on dynamically-sampled metrics will have the same interval as the metric. Using the example here, the alerts related to either endpoint will be raised, at most, every 20 seconds.

  • The time series from all the endpoints are sent to the backend in manner to prevent data integrity issues. Therefore, if total timeseries from a particular endpoint is greater than the maximum allowed limit by any agent, the time series from that endpoint may be dropped irrespective of whether dynamic sampling is turned on or not.

  • Sysdig agent always maximizes the allowed limit for every interval as long as all the time series of an endpoint can be fit into that allowed limit.

Enable Dynamic Sampling

To configure dynamic sampling:

  1. Open the dragent.yaml file.

  2. Add the following line:

    promscrape_emit_all: true

  3. Restart the agent.