Configure Default Integrations

This section explains how to configure the default integrations in Monitor.

It’s only for PromScrape V2.

Each Monitoring Integration holds a specific job that scrapes its metrics and sends them to Sysdig Monitor. To optimize metrics scraping for building dashboards and alerts in Sysdig Monitor, Sysdig offers default jobs for these integrations. Periodically, the Sysdig agent connects with Sysdig Monitor retrieves the default jobs, and makes the Monitoring Integrations available for use. See the list of the available integrations and corresponding jobs.

You can find all the jobs in the /opt/draios/etc/promscrape.yaml file in the sysdig-agent container in your cluster.

Two types of integrations are found:

  • Enabled by default: Upon installing the Sysdig Agent, the integration starts sending metrics to the Sysdig backend.
  • Out of the box: An integration that requires no exporter or extra configuration to retrieve metrics from the application. The application is already orchestrated.

Supported Monitoring Integrations

IntegrationOut of the BoxEnabled by defaultJob name in config file
Apacheapache-exporter-default, apache-grok-default
Calicocalico-node-default, calico-controller-default
Consulconsul-server-default, consul-envoy-default
Fluentdfluentd-default, openshift-fluentd-default
HAProxy Ingresshaproxy-default
HAProxy Ingress OpenShifthaproxy-router
Harborharbor-exporter-default, harbor-core-default, harbor-registry-default, harbor-jobservice-default
IBM Kubernetes API Serveriks-apiservers-default
Kubernetes API serverkubernetes-apiservers-default
K8s cAdvisork8s-cadvisor-default
Kubernetes controller managerkube-controller-manager-default
Kubernetes CoreDNSkube-dns-default
Kubernetes etcdetcd-default, etcd-legacy-default
Kubernetes kubeletk8s-kubelet-default
Kubernetes kube-proxykubernetes-kube-proxy-default
Kubernetes PVCk8s-pvc-default
Kubernetes Schedulerkube-scheduler-default
Kubernetes storagek8s-storage-default
Kafkakafka-exporter-default, kafka-jmx-default
Knativeknative-operator-default, knative-serving-controller-default, knative-serving-autoscaler-default, knative-serving-activator-default, knative-serving-webhook-default, knative-eventing-broker-filter-default, knative-eventing-broker-ingress-default, knative-eventing-controller-default, knative-eventing-imc-controller-default, knative-eventing-imc-dispatcher-default, knative-eventing-apiserver-source-default
NGINX Ingressnginx-ingress-default
OpenShift API-Serveropenshift-apiserver-default
OpenShift Controller Manageropenshift-controller-manager-default
OpenShift CoreDNSopenshift-dns-default
OpenShift Etcdopenshift-etcd-default
OpenShift Scheduleropenshift-scheduler-default
OpenShift State Metricsopenshift-state-metrics
Portworxportworx-default, portworx-openshift-default
Prometheus Default Jobk8s-pods
Rancher RKE API Serverrancher-rke-api-server-default
Rancher RKE Controller Managerrancher-rke-controller-manager-default
Rancher RKE CoreDNSrancher-rke-coredns-default
Rancher RKE Kube Proxyrancher-rke-kube-proxy-default
Rancher RKE Schedulerrancher-rke-scheduler-default
Rancher RKE2 API Serverrancher-rke2-api-server-default
Rancher RKE2 Controller Managerrancher-rke2-controller-manager-default
Rancher RKE2 CoreDNSrancher-rke2-coredns-default
Rancher RKE2 Etcdrancher-rke2-etcd-default
Rancher RKE2 Kube Proxyrancher-rke2-kube-proxy-default
Rancher RKE2 Schedulerrancher-rke2-scheduler-default
Redis Enterpriseredis-enterprise-default
Sysdig Admission Controllersysdig-admission-controller-default

Enable and Disable Integrations

Some integrations are disabled by default due to the potential high cardinality of their metrics. To enable or disable integrations, follow the below steps:

Navigate to the Monitoring Integrations page (Integrations > Monitoring Integrations).

Select any integration (eg. Kubernetes API server), and click on Manage this integration.

Select whether this integration is enabled or disabled, and click on Confirm.

If the Manage This integration button is not available, please contact Support.

Customize a Default Job

The default jobs offered by Sysdig for integrations are optimized to scrape the metrics for building dashboards and alerts in Sysdig Monitor. Instead of processing all the metrics available, you can determine which metrics to include or exclude for your requirements. To do so, you can overwrite the default configuration in the prometheus.yaml file. The prometheus.yaml file is located in the sysdig-agent ConfigMap in the sysdig-agent namespace.

You can overwrite the default job for a specific integration by adding a new job to the prometheus.yaml file with the same name as the default job that you want to replace. For example, if you want to create a new job for the Apache integration, create a new job with the name apache-default. The jobs defined by the user has precedence over the default ones.

See Supported Monitoring Integrations for the complete list of integrations and corresponding job names.

Add Sysdig Annotations in Exporters

Sysdig provides a set of Helm charts that helps you configure the exporters for the integrations. For more information on installing Monitor Integrations, see the Monitoring Integrations option in Sysdig Monitor. Additionally, the Helm charts are publicly available in the Sysdig Helm repository.

If exporters are already installed in your cluster, you can use the standard Prometheus annotations and the Sysdig agent will automatically scrape them.

For example, if you use the annotation given below, the incoming metrics will have the information about the pod that generates the metrics.

      annotations: /metrics '9100' 'true'

If you use an exporter, the incoming metrics will be associated with the exporter pod, not the application pod. To change this behavior, you can use the Sysdig-provided annotations and configure the exporter on the agent.

Annotate the Exporter

Use the following annotations to configure the exporter:

      annotations: '9187' my-namespace deployment my-workload my-integration
  • port: The port to scrape for metrics on the exporter.
  • target_ns: The namespace of the workload corresponding to the application (not the exporter).
  • target_workload_type: The type of the workload of the application (not the exporter). The possible values are deployment, statefulset, and daemonset.
  • target_workload_name: The name of the workload corresponding to the application (not the exporter).
  • integration_type: The type of the integration. The job created in the Sysdig agent use this value to find the exporter.

Configure a New Job

Edit the prometheus.yaml file to configure a new job in Sysdig agent. The file is located in the sysdig-agent ConfigMap in the sysdig-agent namespace.

You can use the following example template:

- job_name: my-integration
    insecure_skip_verify: true
    - role: pod
    - action: keep
      source_labels: [__meta_kubernetes_pod_host_ip]
      regex: __HOSTIPS__
    - action: drop
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_omit]
      regex: true
    - source_labels: [__meta_kubernetes_pod_phase]
      action: keep
      regex: Running
    - action: keep
        - __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
      regex: 'my-integration' # Use here the integration type that you defined in your annotations
    - action: replace
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_ns]
      target_label: kube_namespace_name
    - action: replace
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_workload_type]
      target_label: kube_workload_type
    - action: replace
      source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_workload_name]
      target_label: kube_workload_name
    - action: replace
      replacement: true
      target_label: sysdig_omit_source
    - action: replace
      source_labels: [__address__, __meta_kubernetes_pod_annotation_promcat_sysdig_com_port]
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: replace
      source_labels: [__meta_kubernetes_pod_uid]
      target_label: sysdig_k8s_pod_uid
    - action: replace
      source_labels: [__meta_kubernetes_pod_container_name]
      target_label: sysdig_k8s_pod_container_name

Exclude a Deployment from Being Scraped

If you want the agent to exclude a deployment from being scraped, use the following annotation:

      annotations: 'true'

Learn More