Reduce Metrics Consumption

Limitations in Excluding Default Metrics and Prometheus Remote Write Metrics

Once the Sysdig Agent is installed, thousands of metrics, such as sysdig_container_cpu_used_percent, are collected by the agent at no additional cost. These are known as default metrics and their collection cannot be disabled. Metrics that are not collected by default are called custom metrics. The collection of custom metrics counts towards your time series entitlement while default metrics do not.

To stop custom metrics from being forwarded to Sysdig via Prometheus Remote Write, configuration changes must be made directly in the prometheus.yaml file. You cannot drop a metric once it has already been ingested.

Excluding metrics from collection only applies to custom metrics that are collected by the Sysdig Agent.

Exclude Clusters from Collection

Admins can exclude specific Kubernetes clusters from metrics collection for all supported monitoring integrations, except for the Prometheus Default Job (k8s-pods). To disable a cluster integration:

1. Log in to Sysdig Monitor as an Admin, and navigate to Integrations > Monitoring Integrations from the left navigation bar.

2. On the Monitoring Integrations page, select an integration that is Reporting Metrics.

3. Click Manage this Integration.

4. Under Exceptions, select the cluster(s) you wish to exclude.

   

5. Click Confirm, then Close the window.

This will exclude all metrics associated with this integration for the specified cluster from being collected by the agent. The change may require a few minutes to take effect.

Exclude Workloads from Collection

The Sysdig agent collects metrics from any workload labelled with the prometheus.io/scrape='true' annotation. To prevent these metrics from being collected, either remove the annotation prometheus.io/scrape='true' or apply the following annotation:

spec:
  template:
    metadata:
      annotations:
        promcat.sysdig.com/omit: 'true'

You can apply this annotation across an entire namespace with the following kubectl command:

kubectl get pods -n <namespace> -o name | xargs -I {} kubectl annotate {} promcat.sysdig.com/omit='true'

Exclude Metrics Collection Jobs with the Sysdig API

You can use the Sysdig API to exclude specific metrics without turning off the integration or omitting all metrics at the workload level, This can be particularly useful for the default Prometheus Default Job, where disabling the entire k8s-pods job may have unintended consequences for workloads that rely on the collection of metrics via kubernetes annotation.

Disable Collection of a Single Metric for a Specified Prometheus Job

Some metrics with high cardinality may not offer substantial value. Disabling the collection of such metrics can be an effective strategy to reduce time series consumption.

In the following example, http_roundtrip_duration_seconds_bucket is a metric that is collected by the Prometheus Default k8s-pods job. To exclude the http_roundtrip_duration_seconds_bucket metric from being collected, use the following bash command:

curl --location --request POST 'https://<regional-endpoint>/monitor/prometheus-jobs/v1/disabled-metrics' \
--header 'Authorization: Bearer <token>' \
--header 'content-type: application/json' \
--header 'X-sysdig-public-notation: true' \
--data-raw '{
   "data": [
      {
         "jobName": "k8s-pods",
         "metrics": [
            {
               "metricName": "http_roundtrip_duration_seconds_bucket",
               "isDisabled": true
            }
         ]
      }
   ]
}'

Re-enable Metrics Collection for a Specified Prometheus Job

To re-enable a metric whose collection was previously disabled, you can use the same endpoint:

curl --location --request POST 'https://<regional-endpoint>/monitor/prometheus-jobs/v1/disabled-metrics' \
--header 'Authorization: Bearer <token>' \
--header 'content-type: application/json' \
--header 'X-sysdig-public-notation: true' \
--data-raw '{
   "data": [
      {
         "jobName": "k8s-pods",
         "metrics": [
            {
               "metricName": "http_roundtrip_duration_seconds_bucket",
               "isDisabled": false
            }
         ]
      }
   ]
}'

List All Disabled Metrics for a Specific Prometheus Job

To list all disabled metrics for a specific Prometheus job, use the following bash command:

curl --location --request GET 'https://<regional-endpoint>/monitor/prometheus-jobs/v1/disabled-metrics' \
--header "Authorization: Bearer <token>"

Access Next Gen API Docs Using Regional Endpoints

For further detail regarding API usage, Access Next Gen API docs.