Using PromQL
Note
PromQL is available only in Sysdig SaaS editions. The feature is not yet supported by Sysdig on-premises installations.
The Prometheus Query Language (PromQL) is the defacto standard for querying Prometheus metric data. PromQL is designed to allow the user to select and aggregate time-series data.
Sysdig Monitor's PromQL support includes all of the features, functions, and aggregations in standard open-source PromQL. The PromQL language is documented at Prometheus Query Basics.
For new functionalities released as part of agent v10.0.0, see Collect Prometheus Metrics.
Construct a PromQL Query
In the Dashboard Panel, select the PromQL type to query data using PromQL.

Display: Specify a meaningful display name for the legend. The text you enter replaces the metric name displayed in the legend. The default legend title is the metric name.
Query: Enter a PromQL query directly. For example:
sum(rate(promraw_go_memstats_heap_released_bytes_total[5m])) by(kubernetes_cluster_name)
Specify the following:
Metrics: Search the desired metric. The field supports auto-complete. Enter the text and the rest of the text you type is predicted so you can filter the metric easily. In the example:
promraw_go_memstats_heap_released_bytes_total
.Segmentation: This is the process of categorizing aggregated data with labels to provide precise control over the data. Choose an appropriate value for segmenting the aggregated PromQL data. In this example,
kubernetes_cluster_name
.
The PromQL query field supports the following variables. Both the variables are replaced in the UI in real-time.
$__range
: A built-in variable. It represents the time range currently selected in the time navigation. In the Live mode, the value is constantly updated to reflect the new time range.$__interval
: A built-in variable. It represents a time interval and is automatically configured based on the time range.
Unit and Y-Axes: Specify the unit of scale and display format.
Compare To: (Form-Based) Allows you to compare the current value to historical values. Not supported if Segmentation is specified.
Axes: Determine scale, unit, display format, and gauge for the Y-axes.
Legend: Determine the position of the legend in the Dashboard.
Panel: Specify a name and add details about the panel.
See Create a New Panel for details.
Applying a Dashboard Scope to a PromQL Query
The dashboard scope is automatically applied only to form-based panels. To scope a panel built from a PromQL query, you must use a scope variable within the query. The variable will take the value of the referenced scope parameter, and the PromQL panel will change accordingly.
There are two predefined variables available:
$__interval
represents the time interval defined based on the time range. This will help to adapt the time range for different operations, such asrate
andavg_over_time
, and prevent displaying empty graphs due to the change in the granularity of the data.$__range
represents the time interval defined for the dashboard. This is used to adapt operations like calculating average for a time frame selected.
The following examples show how to use scope variables within PromQL queries.
CPU Used Percent
The following query returns the CPU used percent for all the hosts, regardless of the scope configured at the dashboard level, with a mobile average depending on the time span defined.
avg_over_time(sysdig_host_cpu_used_percent[$__interval])
To scope this query, you must set up an appropriate scope variable. A key step is to provide a variable name that is referenced as part of the query.

In this example, hostname
is used as the variable name. The host can then be referenced using $hostname
as follows:
avg_over_time(sysdig_host_cpu_used_percent{host_name=$hostname}[$__interval])
Depending on the operator specified while configuring scope values, you might need to use a different operator within the query. If you are not using the correct operator for the scope type, the system will perform the query but will show a warning as the results may not be the expected ones.
Scope Operator | PromQL Filter Operator | Example |
---|---|---|
|
| sysdig_host_cpu_used_percent{host_name=$hostname} |
|
| sysdig_host_cpu_used_percent{host_name=~$hostname} |
Enrich Metrics with Labels
Running PromQL queries in Sysdig Monitor by default returns only a minimum set of labels. To enrich the return results of PromQL queries with additional labels, such as Kubernetes cluster name, you need to use a vector matching operation. The vector matching operation in Prometheus is similar to the SQL-like join operation.
Info Metrics
Prometheus returns different information metrics that have a value of 1 with several labels. The information that the info metrics return might not be useful as it is. However, joining the labels of an info metric with a non-info metric can provide useful information, such as the value of metric X across an infrastructure/application/deployment.
Vector Matching Operation
The vector matching operation is similar to an SQL join. You use a vector matching operation to build a PromQL query that can return metrics with information from your infrastructure. Vector matching helps filter and enrich labels, usually adding information labels to the metrics you are trying to visualize.
See Mapping Between Classic Metrics and PromQL Metrics for a list of info metrics.
Example 1: Return a Metric Filtered by Cluster
This example shows a metric returned by an application, say myapp_guage, running on Kubernetes. The query attempts at getting an aggregated value of a cluster, by having one cluster selected in the scope. We assume that previously you have set a $cluster
variable in your scope.
To do so, run the following query to return the myapp_guage metrics:
sum (myapp_gauge * on (container_id) kube_pod_container_info{cluster=$cluster})
The query performs the following operations, not necessarily in this order:
The
kube_pod_container_info
info metrics is filtered, selecting only those timeseries and the associated cluster values you want to see. The selection is based on thecluster
label.The
myapp_gauge
metric is matched with thekube_pod_container_info
metric where thecontainer_id
label has the same value, multiplying both the values. Because the info metric has the value 1, multiplying the values doesn't change the result. As the info metric has already been filtered by a cluster, only those values associated with the cluster will be kept.The resultant timeseries with the value of
myapp_gauge
are then aggregated with the sum function and the result is returned.
Example 2: Calculate the GC Latency
This example shows calculating the GC latency in a go application deployed on a specific Kubernetes namespace.
To calculate the GC latency, run the following query:
go_gc_duration_seconds * on (container_id,host_mac) group_left(pod,namespace) kube_pod_container_info{namespace=~$namespace}
The query is performing the following operations:
The
kube_pod_container_info
info metrics are filtered based on the namespace variable.The metrics associated with
go_gc_duration_seconds
is matched in a many-to-one way with the filteredkube_pod_container_info
.The pod and namespace labels are added from the
kube_pod_container_info
metric to the result. The query keeps only those metrics that have the matchingcontainer_id
andhost_mac
labels on both sides.The values are multiplied and the resulting metrics are returned. The new metrics will only have the values associated with
go_gc_duration_seconds
because the info metric value is always 1.
You can use any Prometheus metric in the query. For example, the query above can be rewritten for a sample Apache metric as follows:
appinfo_apache_net_bytes * on (container_id) group_left(pod, namespace) kube_pod_container_info{namespace=~$namespace}
Example 3: Calculate Average CPU Used Percent in AWS Hosts
This example shows calculating the average CPU used percent per AWS account and region, having the hosts filtered by account and region.
avg by(region,account_id) (sysdig_host_cpu_used_percent * on (host_mac) group_left(region,account_id) sysdig_cloud_provider_info{account_id=~$AWS_account, region=~$AWS_region})
The query performs the following operations:
Filters the
sysdig_cloud_provider_info
metric based on theaccount_id
andregion
labels that come from the dashboard scope as variables.Matches the
sysdig_host_cpu_used_percent
metrics withsysdig_cloud_provider_info
. Only those metrics with the samehost_mac
label on both sides are extracted, addingregion
andaccount_id
labels to the resulting metrics.Calculates the average of the new metrics by
account_id
andregion
.
Example 4: Calculate Total CPU Usage in Deployments
This example shows calculating the total CPU usage per deployment. The value can also be filtered by cluster, namespace, and deployment by using the dashboard scope.
sum by(cluster,namespace,owner_name) ((sysdig_container_cpu_cores_used * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~$deployment,cluster=~$cluster,namespace=~$namespace})
sysdig_container_cpu_cores_used
can be replaced by any metric that has thecontainer_id
label.To connect the
sysdig_container_cpu_cores_used
metric with the pod, usekube
_pod_container_info
and then, usekube
_pod_owner
to connect the pod to other kubernetes objects.
The query performs the following:
sysdig_container_cpu_cores_used * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info
:The
sysdig_container_cpu_cores_used
metric value is multiplied withkube_pod_container_info
(which has the value of 1), by matchingcontainer_id
and by keeping the pod, namespace and cluster labels as it is._name_='sysdig_container_cpu_cores_used',container='<label>', container_id='<label>',container_type='DOCKER`,host_mac='<label>'
The new metrics will be
cluster='<label>',container='<label>', container_id='<label>',container_type='DOCKER`,host_mac='<label>',namespace='<label>, pod='<label>'
The value extracted from the previous result is multiplied with
kube
_pod_owner
(which has the value of 1) by matching on the pod, namespace, and cluster labels and keeping the owner name from the value ofkube
_pod_owner
. The owner can be deployment, replicaset, service, daemonset, or statefulset object.The name of the deployment to filter upon is extracted from the
kube
_pod_owner
metrics.The pod, namespace, and cluster names are extracted from the
kube_pod_container_info
metrics.
The new metrics will be:
cluster='<matched_label>',container='<matched_container_label>', container_id='<label>',container_type='DOCKER`,host_mac='<label>',namespace='<label>, owner_name ='<label>', pod='<label>'
The
kube_pod_owner
will have a labelowner_name
that is the name of the object that owns the pod. This value is extracted by filtering:kube_pod_owner{owner_kind="Deployment",owner_name=~$deployment,cluster=~$cluster,namespace=~$namespace}
The
owner_kind
provides the deployment name and the origin ofowner_name
, that is the dashboard scope.The sum aggregation is applied and the time series are aggregated by cluster, namespace, and deployment.
The following table helps understand the labels applied in each step of the query:
__name__ | container_id | container | container_type | host_mac | pod | namespace | cluster | owner_name | |
---|---|---|---|---|---|---|---|---|---|
| Yes | Yes | Yes | Yes | Yes | No | No | No | No |
sysdig_container_cpu_cores_used * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No |
(sysdig_container_cpu_cores_used * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~$deployment,cluster=~$cluster,namespace=~$namespace} | No | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
sum by(cluster,namespace,owner_name) ((sysdig_container_cpu_cores_used * on(container_id) group_left(pod,namespace,cluster) kube_pod_container_info) * on(pod,namespace,cluster) group_left(owner_name) kube_pod_owner{owner_kind="Deployment",owner_name=~$deployment,cluster=~$cluster,namespace=~$namespace}) | No | No | No | No | No | No | Yes | Yes | Yes |
Formatting
Sysdig Monitor supports percentages only as 0-100 values. In calculated ratios, you can skip multiplying the whole query times 100 by selecting percentage as a 0-1 value.