Dynamic Sampling

Sysdig’s ability to collect and process the volumes of data scraped from different Prometheus endpoints depends on the number of time series scraped from each endpoint, the total Prometheus time series collected by the agent in each time window, and the frequency at which Sysdig agent collects and sends the data to the backend.

When nodes have multiple Prometheus endpoints sending high volumes of time series, the Sysdig agent may skip some endpoints depending on the overall volume and scrape frequency.

Dynamic sampling addresses this by cycling through individual Prometheus endpoints and scraping the latest time series from each endpoint on a rotational basis, ensuring all time series from all Prometheus endpoints are processed at dynamic intervals. This results in more timeseries being scraped and processed overall at a lower frequency. For example, instead of receiving 50,000 timeseries every 10 seconds, you might receive 100,000 timeseries every 20 seconds.

Dynamic Sampling Considerations

• Any alerts that depend on dynamically-sampled metrics will have the same interval as the metric. Using the example here, the alerts related to either endpoint will be raised, at most, every 20 seconds.

• The time series from all the endpoints are sent to the backend in order to prevent data integrity issues. Therefore, if total number of timeseries from a particular endpoint is greater than the maximum allowed limit by any agent, the time series from that endpoint will be dropped irrespective of whether dynamic sampling is turned on or not.

• Sysdig agent always maximizes the allowed limit for every interval as long as all the time series of an endpoint fits in the allowed limit.

Enable Dynamic Sampling

To configure dynamic sampling:

1. Open the dragent.yaml file.

2. Add the following line:

   promscrape_emit_all: true

3. Restart the agent.