Prometheus Remote Write
You can collect Prometheus metrics from environments where the Sysdig
agent is not available. Sysdig uses the
remote_write capabilities to
help you do so.
In Sysdig terminology, the remote endpoints that can read Prometheus metrics are known as Prometheus Remote Write. Prometheus Remote Write does not require the Sysdig agent to be installed in the Prometheus environment. This facility expands your monitoring capabilities beyond Kubernetes and regular Linux kernels to environments where the Sysdig agent cannot be installed.
Prometheus Remote Write can collect metrics from:
An existing Prometheus server
Managed Cloud Environments, such as AWS and IBM
Use Sysdig agent in environments where an agent can be installed. However, use the Prometheus Remote Write to collect metrics from ephemeral or batch jobs that may not exist long enough to be scraped by the agent.
With the Prometheus Remote Write, you can either monitor metrics through the Monitor UI or you can use PromQL to query the data by using the standard Prometheus query language.
Enable Prometheus Remote Write
Contact your Sysdig representative to enable Prometheus Remote Write in your environment.
Endpoints and Regions
Prometheus Remote Write resides in the ingest endpoints for each region
/prometheus/remote/write. The public Prometheus Remote Write
endpoints for each region are listed below:
|Asia Pacific (Sydney)||https://app.au1.sysdig.com/prometheus/remote/write|
Configure Remote Write in Prometheus Server
You need to configure
remote_write in your Prometheus server in order
to send metrics to Sysdig Prometheus Remote Write.
The configuration of your Prometheus server depends on your
installation. In general, you configure the
remote_write section in
prometheus.yml configuration file:
global: external_labels: [ <labelname>: <labelvalue> ... ] remote_write: - url: "https://<region-url>/prometheus/remote/write" bearer_token: "<your API Token>" tls_config: insecure_skip_verify: true
The communication between your Prometheus server and Prometheus Remote Write should use the authorization header with the Sysdig API key (not the agent access key) as the bearer token.
Alternatively, you can also use the
bearer_token_file entry to refer
to a file instead of directly including the API token.
Prometheus does not reveal the
bearer_token value on the UI.
To enable customization, Sysdig provides additional options to control which metrics you want to send to Prometheus Remote Write.
Prometheus Remote Write by default sends all the metrics to Sysdig
Prometheus Remote Write. These metrics are sent with a
remote_write: true label appended to it so you can easily identify
You can specify custom label-value pairs and send them with each time
series to the Prometheus Remote Write. Use the
global section in the Prometheus configuration file. This is
similar to setting an agent tag and allowing you to filter or scope the
metrics in Sysdig Monitor.
For example, if you have two Prometheus servers configured to remote write to Prometheus Remote Write, you can include an external label to identify them easily:
Prometheus 1 global: external_labels: provider: prometheus1 remote_write: - url: ... Prometheus 2 global: external_labels: provider: prometheus2 remote_write: - url: ...
With the general configuration, all the metrics are by default remotely
written to Prometheus Remote Write. You can control the metrics that you
collect and send to Sysdig. To select which series and labels to
collect, drop, or replace, and reduce the number of active series that
are sent to Sysdig, you can set up relabel configurations by using
write_relabel_configs block within your
For example, you can send metrics from one specific namespace called
myapp-ns as give below:
remote_write: - url: https://<region-url>/prometheus/remote/write bearer_token_file: /etc/secrets/sysdig-api-token write_relabel_configs: - source_labels: [__meta_kubernetes_namespace] regex: ‘myapp-ns’ action: keep
The default limits are configured set for each user and can be raised as required. The defaults are good for most users, and the limits help protect against any misconfigurations.
100 concurrent requests.
This doesn’t necessarily mean 100 Prometheus servers because the time at which the data is written is distributed.
Data points per minute
The number of data points sent depends on how often metrics are submitted to Sysdig. A scrape interval of 10s will submit more DPM than an interval of 60s.
Number of writes per minute
Metrics sent to Prometheus Remote Write can be accessed in Explore, but they are not compatible with the scope tree.
Prometheus Remote Write metrics won’t work with Team Scope.
Label enrichment is unavailable at this point. Only labels collected at the source can be used. You should add additional labels to perform further scoping or pivoting in Sysdig.
Currently, Sysdig Dashboards do not support mixing metrics with different sampling. For example, 10 seconds and 1-minute samples. For optimal experience, configure the scrape interval to be 10s to combine remote write metrics with agent metrics.
Remote write functionality does not support sending metric metadata. Upstream Prometheus recently added support for propagation of metadata (metric type, unit, description, info) and this functionality will be supported in a future update to Prometheus Remote Write.
Suffix the metric name with
_countto store them as a counter. Otherwise, the metrics will be handled as a gauge.
Units can be set in Dashboards manually.