Configure Prometheus Remote Write

In Sysdig terminology, the remote endpoints that can read Prometheus metrics are known as Prometheus Remote Write. Prometheus Remote Write does not require the Sysdig agent to be installed in the Prometheus environment. This expands your monitoring capabilities beyond Kubernetes and regular Linux kernels to environments where the Sysdig agent cannot be installed.

Prometheus Remote Write can collect metrics from:

• An existing Prometheus server

• Additional environments:

  • Windows

  • Managed Cloud Environments, such as AWS and IBM

  • Fargate

  • IoT

Use the Sysdig agent in environments where an agent can be installed. However, use the Prometheus Remote Write to collect metrics from ephemeral or batch jobs that may not exist long enough to be scraped by the agent.

With the Prometheus Remote Write, you can either monitor metrics through the Monitor UI or you can use PromQL to query the data by using the standard Prometheus query language.

Prerequisites

• Collect your Sysdig API key
• Confirm your endpoints and region

Endpoints and Regions

Prometheus Remote Write resides in the ingest endpoints for each region under /prometheus/remote/write. The public Prometheus Remote Write endpoints for each region are listed in SaaS Regions and IP Ranges.

Configure Remote Write in Prometheus Server

You need to configure remote_write in your Prometheus server to send metrics to Sysdig Prometheus Remote Write.

The configuration of your Prometheus server depends on your installation. In general, you configure the remote_write section in the prometheus.yml configuration file:

global:
  external_labels:
    [ <labelname>: <labelvalue> ... ]
remote_write:
    - url: "https://<region-url>/prometheus/remote/write"
      bearer_token: "<your API Token>"

The communication between your Prometheus server and Prometheus Remote Write should use the authorization header with the Sysdig API key (not the agent access key) as the bearer token.

Alternatively, you can also use the bearer_token_file entry to refer to a file instead of directly including the API token.

Customize Metrics

To enable customization, Sysdig provides additional options to control which metrics you want to send to Prometheus Remote Write.

Manage Metrics

Prometheus Remote Write by default sends all the metrics to Sysdig Prometheus Remote Write. These metrics are sent with a remote_write: true label appended to it so you can easily identify them.

Label Metrics

You can specify custom label-value pairs and send them with each time series to the Prometheus Remote Write. Use the external_labels block in the global section in the Prometheus configuration file. This is similar to setting an agent tag and allowing you to filter or scope the metrics in Sysdig Monitor.

For example, if you have two Prometheus servers configured to remote write to Prometheus Remote Write, you can include an external label to identify them easily:

Prometheus 1

global:
  external_labels:
    provider: prometheus1
remote_write:
- url: ...

Prometheus 2

global:
  external_labels:
    provider: prometheus2
remote_write:
- url: ...

Filter Metrics

With the general configuration, all the metrics are by default remotely written to Prometheus Remote Write. You can control the metrics that you collect and send to Sysdig. To select which series and labels to collect, drop, or replace, and reduce the number of active series that are sent to Sysdig, you can set up relabel configurations by using the write_relabel_configs block within your remote_write section.

For example, you can send metrics from one specific namespace called myapp-ns as follows:

remote_write:
- url: https://<region-url>/prometheus/remote/write
  bearer_token_file: /etc/secrets/sysdig-api-token
  write_relabel_configs:
  - source_labels: [__meta_kubernetes_namespace]
    regex: 'myapp-ns'
    action: keep

Verify Installation

To verify that the Prometheus Remote Write has been installed correctly:

1. Log in to Sysdig Monitor.

2. Do one of the following:

   • Open Explore > PromQL Query and runt the following query:

     sum(sysdig_ts_usage{metric_category="PROMETHEUS_REMOTE_WRITE"})

     You will see the time series usage graph on the screen.

     

   • Open Dashboards > Sysdig Monitor > Time Series Usage Dashboard.

   ​ Under the Number of time series ingested per category panel, check the time series category.

   ​

   

   ​ You will see PROMETHEUS_REMOTE_WRITE as one of the categories.

Rate Limit

The default limits are configured set for each user and can be raised as required. The defaults are good for most users, and the limits help protect against any misconfigurations.

Team Scoping

It is possible to scope a Sysdig Team to only access metrics matching certain labels sent via Prometheus Remote Write. See Manage Teams and Roles

Limitations

• Alerts based on metrics ingested through Prometheus Remote Write will be delayed by approximately 5 minutes to ensure all the data has been correctly ingested.

• Metrics sent to Prometheus Remote Write can be accessed in Explore, but they are not compatible with the scope tree.

• Label enrichment is unavailable at this point. Only labels collected at the source can be used. You should add additional labels to perform further scoping or pivoting in Sysdig.

• Currently, Sysdig Dashboards do not support mixing metrics with different sampling. For example, 10 seconds and 1-minute samples. For optimal experience, configure the scrape interval to be 10s to combine remote write metrics with agent metrics.

• Remote write functionality does not support sending metric metadata. Upstream Prometheus recently added support for propagation of metadata (metric type, unit, description, info) and this functionality will be supported in a future update to Prometheus Remote Write.

  • Suffix the metric name with _total, _sum , or _count to store them as a counter. Otherwise, the metrics will be handled as a gauge.

  • Units can be set in Dashboards manually.

Learn More

• Using PromQL

• Run PromQL Queries Faster with Extended Label Set

• PromQL Query Explorer

• PromQL Library

• PromQL Alerts