Configure Default Integrations

Each Monitoring Integration holds a specific job that scrapes its metrics and sends them to Sysdig Monitor. To optimize metrics scraping for building dashboards and alerts in Sysdig Monitor, Sysdig offers default jobs for these integrations. Periodically, the Sysdig agent connects with Sysdig Monitor and retrieves the default jobs and make the Monitoring Integrations available for use. See the list of the available integrations and corresponding jobs.

You can find all the jobs in the /opt/draios/etc/promscrape.yaml file in the sysdig-agent container in your cluster.

Customize a Default Job

The default jobs offered by Sysdig for integrations are optimized to scrape the metrics for building dashboards and alerts in Sysdig Monitor. Instead of processing all the metrics available, you can determine which metrics to include or exclude for your requirements. To do so, you can overwrite the default configuration in the prometheus.yaml file. The prometheus.yaml file is located in the sysdig-agent ConfigMap in the sysdig-agent namespace.

You can overwrite the default job for a specific integration by adding a new job to the prometheus.yaml file with the same name as the default job that you want to replace. For example, if you want to create a new job for the Apache integration, create a new job with the name apache-default. The jobs defined by the user has precedence over the default ones.

See Supported Monitoring Integrations for thee complete list of integrations and corresponding job names.

Use Sysdig Annotations in Exporters

Sysdig provides a set of Helm charts that helps you configure the exporters for the integrations. For more information on installing Monitor Integrations, see the Monitoring Integrations option in the Sysdig Monitor. Additionally, the Helm charts are publicly available in the Sysdig Helm repository.

If exporters are already installed in your cluster, you can use the standard Prometheus annotations and the Sysdig agent will automatically scrape them.

For example, if you use the annotation given below, the incoming metrics will have the information about the pod that generates the metrics.

      annotations: /metrics "9100" "true"

If you use an exporter, the incoming metrics will be associated with the exporter pod, not the application pod. To change this behavior, you can use the Sysdig-provided annotations and configure the exporter with special settings on the agent.

Annotate the Exporter

Use the following annotations to configure the exporter:

      annotations: "9187" my-namespace deployment my-workload my-integration
  • port: The port to scrape for metrics on the exporter.
  • target_ns: The namespace of the workload corresponding to the application (not the exporter).
  • target_workload_type: The type of the workload of the application (not the exporter). The possible values are deployment, statefulset, and daemonset.
  • target_workload_name: The name of the workload corresponding to the application (not the exporter).
  • integration_type: The type of the integration. The job created in the Sysdig agent use this value to find the exporter.

Configure a New Job

Edit the prometheus.yaml file to configure a new job in Sysdig agent. The file is located in the sysdig-agent ConfigMap in the sysdig-agentnamespace.

You can use this example template:

- job_name: my-integration
    insecure_skip_verify: true
  - role: pod
  - action: keep
    source_labels: [__meta_kubernetes_pod_host_ip]
    regex: __HOSTIPS__
  - action: drop
    source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_omit]
    regex: true
  - action: keep
    - __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
    regex: "my-integration" # Use here the integration type that you defined in your annotations
  - action: replace
    source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_ns]
    target_label: kube_namespace_name
  - action: replace
    source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_workload_type]
    target_label: kube_workload_type
  - action: replace
    source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_target_workload_name]
    target_label: kube_workload_name
  - action: replace
    replacement: true
    target_label: sysdig_omit_source
  - action: replace
    source_labels: [__address__, __meta_kubernetes_pod_annotation_promcat_sysdig_com_port]
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: replace
    source_labels: [__meta_kubernetes_pod_uid]
    target_label: sysdig_k8s_pod_uid
  - action: replace
    source_labels: [__meta_kubernetes_pod_container_name]
    target_label: sysdig_k8s_pod_container_name

Exclude a Deployment from Being Scraped

If you want the Agent to exclude a deployment from being scraped, you can use the following annotation:

      annotations: "true"

Supported Monitoring Integrations

IntegrationOut of the BoxJob name in config file
Harbor✔️harbor-exporter-default, harbor-core-default, harbor-registry-default, harbor-jobservice-default
K8s Controller Manager✔️kube-controller-manager
K8s CoreDNS✔️kube-dns
K8s Scheduler✔️kube-scheduler
NGingx Kubernetes ingress controller✔️nginx-ingress-default
OPA - Gatekeeper✔️opa-default
Sysdig Admission Controller✔️sysdig-admission-controller-default

Last modified October 14, 2021