(Legacy) Integrations for Sysdig Monitor

• 1: (Legacy)Collect Prometheus Metrics
• 1.1: (Legacy) Working with Prometheus Metrics
• 1.2: (Legacy) Set up the Environment
• 1.3: (Legacy) Configuring Sysdig Agent
• 1.4: (Legacy) Filtering Prometheus Metrics
• 1.5: (Legacy) Example Configuration
• 1.6: (Legacy) Logging and Troubleshooting
• 1.7: (Legacy) Collecting Prometheus Metrics from Remote Hosts
• 2: (Legacy) Integrate Applications (Default App Checks)
• 2.1: Apache
• 2.2: Apache Kafka
• 2.3: Consul
• 2.4: Couchbase
• 2.5: Elasticsearch
• 2.6: etcd
• 2.7: fluentd
• 2.8: Go
• 2.9: HAProxy
• 2.10: HTTP
• 2.11: Jenkins
• 2.12: Lighttpd
• 2.13: Memcached
• 2.14: Mesos/Marathon
• 2.15: MongoDB
• 2.16: MySQL
• 2.17: NGINX and NGINX Plus
• 2.18: NTP
• 2.19: PGBouncer
• 2.20: PHP-FPM
• 2.21: PostgreSQL
• 2.22: RabbitMQ
• 2.23: RedisDB
• 2.24: SNMP
• 2.25: Supervisord
• 2.26: TCP
• 2.27: Varnish
• 3: (Legacy) Create a Custom App Check
• 4: (Legacy) Create Per-Container Custom App Checks

1 - (Legacy)Collect Prometheus Metrics

Sysdig supports collecting, storing, and querying Prometheus native metrics and labels. You can use Sysdig in the same way that you use Prometheus and leverage Prometheus Query Language (PromQL) to create dashboards and alerts. Sysdig is compatible with Prometheus HTTP API to query your monitoring data programmatically using PromQL and extend Sysdig to other platforms like Grafana.

From a metric collection standpoint, a lightweight Prometheus server is directly embedded into the Sysdig agent to facilitate metric collection. This also supports targets, instances, and jobs with filtering and relabeling using Prometheus syntax. You can configure the agent to identify these processes that expose Prometheus metric endpoints on its own host and send it to the Sysdig collector for storing and further processing.

The Prometheus product itself does not necessarily have to be installed for Prometheus metrics collection.

See the Sysdig agent versions and compatibility with Prometheus features:

• Latest versions of agent (v12.0.0 and above): The following features are enabled by default:

  • Automatically scraping any Kubernetes pods with the following annotation set: prometheus.io/scrape=true
  • Automatically scrape applications supported by Monitoring Integrations.

• Sysdig agent prior to v12.0.0: Manually enable Prometheus in dragent.yaml file:

    prometheus:
         enabled: true
  

Learn More

The following topics describe in detail how to configure the Sysdig agent for service discovery, metrics collection, and further processing.

• Working with Prometheus Metrics

• Setting up the Environment

• Configuring Sysdig Agent

• Example Configuration

• Enable Prometheus Native Service Discovery

• Configure Sysdig with Grafana

• Logging and Troubleshooting

• Collecting Prometheus Metrics from Remote Hosts

See the following blog posts for additional context on the Prometheus metric and how such metrics are typically used.

• Instrumenting Prometheus Metrics

• Sysdig’s Prometheus Monitoring

• Sysdig Introduces the First Cloud-Scale Prometheus Monitoring Offering

• Challenges Using Prometheus at scale

1.1 - (Legacy) Working with Prometheus Metrics

The Sysdig agent uses its visibility to all running processes (at both the host and container levels) to find eligible targets for scraping Prometheus metrics. By default, no scraping is attempted. Once the feature is enabled, the agent assembles a list of eligible targets, apply filtering rules, and sends back to the Sysdig collector.

Latest Prometheus Features

Sysdig agents v12.0 or above is required for the following capabilities:

• Monitoring Integrations

Sysdig agents v10.0 or above is required for the following capabilities:

• New capabilities of using Prometheus data:

  • Ability to visualize data using PromQL queries. See Using PromQL.

  • Create alerts from PromQL-based Dashboards. See Create Panel Alerts.

  • Backward compatibility for dashboards v2 and alerts.

• New metrics limit per agent

• 10-second data granularity

• Higher retention rate on the new metric store.

Prerequisites and Guidelines

• Sysdig agent v 10.0.0 and above is required for the latest Prometheus features.

• Prometheus feature is enabled in the dragent.yaml file.

  prometheus:
    enabled: true
  

  See Setting up the Environment for more information.

• The endpoints of the target should be available on a TCP connection to the agent. The agent scrapes a target, remote or local, specified by the IP: Port or the URL in dragent.yaml.

Service Discovery

To use native Prometheus service discovery, enable Promscrape V2 as described in Enable Prometheus Native Service Discovery. This section covers the Sysdig way of service discovery that involves configuring process filters in the Sysdig agent.

The way service discovery works in the Sysdig agent differs from that of the Prometheus server. While the Prometheus server has built-in integration with several service discovery mechanisms and the prometheus.yml file to read the configuration settings from, the Sysdig agent auto-discovers any process (exporter or instrumented) that matches the specifications in the dragent.yaml, file and instructs the embedded lightweight Prometheus server to retrieve the metrics from it.

The lightweight Prometheus server in the agent is named promscrape and is controlled by the flag of the same name in the dragent.yaml file. See Configuring Sysdig Agent for more information.

Unlike the Prometheus server that can scrape processes running on all the machines in a cluster, the agent can scrape only those processes that are running on the host that it is installed on.

Within the set of eligible processes/ports/endpoints, the agent scrapes only the ports that are exporting Prometheus metrics and will stop attempting to scrape or retry on ports based on how they respond to attempts to connect and scrape them. It is therefore strongly recommended that you create a configuration that restricts the process and ports for attempted scraping to the minimum expected range for your exporters. This minimizes the potential for unintended side-effects in both the Agent and your applications due to repeated failed connection attempts.

The end to end metric collection can be summarized as follows:

1. A process is determined to be eligible for possible scraping if it positively matches against a series of Process Filter include/exclude rules. See Process Filter for more information.

2. The Agent will then attempt to scrape an eligible process at a /metrics endpoint on all of its listening TCP ports unless the additional configuration is present to restrict scraping to a subset of ports and/or another endpoint name.

3. Upon receiving the metrics, the agent applies the following rules before sending them to the Sysdig collector.

   • Applies the global metrics_filter rules.

     See Include/Exclude Custom Metrics.

   • Imposes the configured max_metrics to limit the number of metrics.

The metrics ultimately appear in the Sysdig Monitor Explore interface in the Prometheus section.

1.2 - (Legacy) Set up the Environment

Quick Start For Kubernetes Environments

Prometheus users who are already leveraging Kubernetes Service Discovery (specifically the approach in this sample prometheus-kubernetes.yml) may already have Annotations attached to the Pods that mark them as eligible for scraping. Such environments can quickly begin scraping the same metrics using the Sysdig Agent in a couple of easy steps.

1. Enable the Prometheus metrics feature in the Sysdig Agent. Assuming you are deploying using DaemonSets, the needed config can be added to the Agent’s dragent.yaml by including the following in your DaemonSet YAML (placing it in the env section for the sysdig-agent container):

   - name: ADDITIONAL_CONF
     value: "prometheus:\n  enabled: true"
   

2. Ensure the Kubernetes Pods that contain your Prometheus exporters have been deployed with the following Annotations to enable scraping (substituting the listening exporter-TCP-port) :

   spec:
     template:
       metadata:
         annotations:
           prometheus.io/scrape: "true"
           prometheus.io/port: "exporter-TCP-port"
   

   The configuration above assumes your exporters use the typical endpoint called /metrics. If an exporter is using a different endpoint, this can also be specified by adding the following additional optional Annotation, substituting the exporter-endpoint-name:

   prometheus.io/path: "/exporter-endpoint-name"
   

If you try this Kubernetes Deployment of a simple exporter, you will quickly see auto-discovered Prometheus metrics being displayed in Sysdig Monitor. You can use this working example as a basis to similarly Annotate your own exporters.

If you have Prometheus exporters not deployed in annotated Kubernetes Pods that you would like to scrape, the following sections describe the full set of options to configure the Agent to find and scrape your metrics.

Quick Start for Container Environments

In order for Prometheus scraping to work in a Docker-based container environment, set the following labels to the application containers, substituting <exporter-port> and <exporter-path> with the correct port and path where metrics are exported by your application:

• io.prometheus.scrape=true

• io.prometheus.port=<exporter-port>

• io.prometheus.path=<exporter-path>

For example, if mysqld-exporter is to be scraped, spin up the container as follows:

docker -d -l io.prometheus.scrape=true -l io.prometheus.port=9104 -l io.prometheus.path=/metrics mysqld-exporter

1.3 - (Legacy) Configuring Sysdig Agent

As is typical for the agent, the default configuration for the feature is specified in dragent.default.yaml, and you can override the defaults by configuring parameters in the dragent.yaml. For each parameter, you do not set in dragent.yaml, the defaults in dragent.default.yaml will remain in effect.

Main Configuration Parameters

promscrape

Promscrape is a lightweight Prometheus server that is embedded with the Sysdig agent. The use_promscrape parameter controls whether to use it to scrape Prometheus endpoints.

Promscrape has two versions: Promscrape V1 and Promscrape V2. With V1, Sysdig agent discovers scrape targets through the process_filter rules. With V2, promscrape itself discovers targets by using the standard Prometheus configuration, allowing the use of relabel_configs to find or modify targets.

prometheus

The prometheus section defines the behavior related to Prometheus metrics collection and analysis. It allows for turning the feature on, set a limit from the agent side on the number of metrics to be scraped, and determines whether to report histogram metrics and log failed scrape attempts.

Process Filter

The process_filter section specifies which of the processes known by an agent may be eligible for scraping.

Note that once you specify a process_filter in your dragent.yaml, this replaces the entire Prometheus process_filter section (i.e. all the rules) shown in the dragent.default.yaml.

The Process Filter is specified in a series of include and exclude rules that are evaluated top-to-bottom for each process known by an Agent. If a process matches an include rule, scraping will be attempted via a /metrics endpoint on each listening TCP port for the process, unless a conf section also appears within the rule to further restrict how the process will be scraped. See conf for more information.

Multiple patterns can be specified in a single rule, in which case all patterns must match for the rule to be a match (AND logic).

Within a pattern value, simple “glob” wildcarding may be used, where * matches any number of characters (including none) and ? matches any single character. Note that due to YAML syntax, when using wildcards, be sure to enclose the value in quotes ("*").

The table below describes the supported patterns in Process Filter rules. To provide realistic examples, we’ll use a simple sample Prometheus exporter (source code here) which can be deployed as a container using the Docker command line below. To help illustrate some of the configuration options, this sample exporter presents Prometheus metrics on /prometheus instead of the more common /metrics endpoint, which will be shown in the example configurations further below.

# docker run -d -p 8080:8080 \
    --label class="exporter" \
    --name my-java-app \
    luca3m/prometheus-java-app

# ps auxww | grep app.jar
root     11502 95.9  9.2 3745724 753632 ?      Ssl  15:52   1:42 java -jar /app.jar --management.security.enabled=false

# curl http://localhost:8080/prometheus
...
random_bucket{le="0.005",} 6.0
random_bucket{le="0.01",} 17.0
random_bucket{le="0.025",} 51.0
...

- include:
    container.image: luca3m/prometheus-java-app
    container.name: my-java-app
    container.label.class: exporter
    process.name: java
    process.cmdline: "*app.jar*"
    port: 8080

conf

Each include rule in the port_filter may include a conf portion that further describes how scraping will be attempted on the eligible process. If a conf portion is not included, scraping will be attempted at a /metrics endpoint on all listening ports of the matching process. The possible settings:

host: 192.168.1.101
- or -
host: subdomain.example.com
- or -
host: localhost

Authentication Integration

As of agent version 0.89, Sysdig can collect Prometheus metrics from endpoints requiring authentication. Use the parameters below to enable this function.

• For username/password authentication:

  • username

  • password

• For authentication using a token:

  • auth_token_path

• For certificate authentication with a certificate key:

  • auth_cert_path

  • auth_key_path

conf Authentication Example

Below is an example of the dragent.yaml section showing all the Prometheus authentication configuration options, on OpenShift, Kubernetes, and etcd.

In this example:

• The username/password are taken from a default annotation used by OpenShift.

• The auth token path is commonly available in Kubernetes deployments.

• The certificate and key used here for etcd may normally not be as easily accessible to the agent. In this case they were extracted from the host namespace, constructed into Kubernetes secrets, and then mounted into the agent container.

prometheus:
  enabled: true
  process_filter:
    - include:
        port: 1936
        conf:
            username: "{kubernetes.service.annotation.prometheus.openshift.io/username}"
            password: "{kubernetes.service.annotation.prometheus.openshift.io/password}"
    - include:
        process.name: kubelet
        conf:
            port: 10250
            use_https: true
            auth_token_path: "/run/secrets/kubernetes.io/serviceaccount/token"
    - include:
        process.name: etcd
        conf:
            port: 2379
            use_https: true
            auth_cert_path: "/run/secrets/etcd/client-cert"
            auth_key_path: "/run/secrets/etcd/client-key"

Kubernetes Objects

As described above, there are multiple configuration options that can be set based on auto-discovered values for Kubernetes Labels and/or Annotations. The format in each case begins with "kubernetes.OBJECT.annotation." or "kubernetes.OBJECT.label." where OBJECT can be any of the following supported Kubernetes object types:

• daemonSet

• deployment

• namespace

• node

• pod

• replicaSet

• replicationController

• service

• statefulset

The configuration text you add after the final dot becomes the name of the Kubernetes Label/Annotation that the Agent will look for. If the Label/Annotation is discovered attached to the process, the value of that Label/Annotation will be used for the configuration option.

Note that there are multiple ways for a Kubernetes Label/Annotation to be attached to a particular process. One of the simplest examples of this is the Pod-based approach shown in Quick Start For Kubernetes Environments. However, as an example alternative to marking at the Pod level, you could attach Labels/Annotations at the Namespace level, in which case auto-discovered configuration options would apply to all processes running in that Namespace regardless of whether they’re in a Deployment, DaemonSet, ReplicaSet, etc.

1.4 - (Legacy) Filtering Prometheus Metrics

As of Sysdig agent 9.8.0, a lightweight Prometheus server is embedded in agents named promscrape and a prometheus.yaml file is included as part of configuration files. Using the open source Prometheus capabilities, Sysdig leverages a Prometheus feature to allow you to filter Prometheus metrics at the source before ingestion. To do so, you will:

• Ensure that the Prometheus scraping is enabled in the  dragent.yaml file.

  prometheus:
    enabled: true
  

• On agent v9.8.0 and above, enable the feature by setting the

  use_promscrape parameter to true in the dragent.yaml. See Enable Filtering at Ingestion.

• Edit the configuration in the prometheus.yaml file. See Edit Prometheus Configuration File.

  Sysdig-specific configuration is found in the prometheus.yaml file.

Enable Filtering at Ingestion

On agent v9.8.0, in order for target filtering to work, the use_promscrape parameter in the dragent.yaml must be set to true. For more information on configuration, see Configuring Sysdig Agent.

use_promscrape: true

Filtering configuration is optional. The absence of prometheus.yaml  will not change the existing behavior of the agent.

Edit Prometheus Configuration File

About the Prometheus Configuration File

The prometheus.yaml file contains mostly the filtering/relabeling configuration in a list of key-value pairs, representing target process attributes.

You replace keys and values with the desired tags corresponding to your environment.

In this file, you will configure the following:

• Default scrape interval (optional).

  For example:

  scrape_interval: 10s

• Of the labeling parameters that Prometheus offers, Sysdig supports only metric_relabel_configs. The relabel_config parameter is not supported.

• Zero or more process-specific filtering configurations (optional).

  See Kubernetes Environments and Docker Environments

  The filtering configuration includes:

  • Filtering rules

    For example:

    - source_labels: [container_label_io_kubernetes_pod_name]

  • Limit on number of scraped samples (optional)

    For example:

    sample_limit: 2000

• Default filtering configuration (optional). The filtering configuration includes:

  • Filtering rules

    For example:

    - source_labels: [car]

  • Limit on number of scraped samples (optional)

    For example:

    sample_limit: 2000

The prometheus.yaml file is installed alongside dragent.yaml. For the most part, the syntax of prometheus.yaml complies with the standard Prometheus configuration. 

Default Configuration

A configuration with empty key-value pairs is considered a default configuration. The default configuration will be applied to all the processes to be scraped that don’t have a matching filtering configuration. In Sample Prometheus Configuration File, the job_name: 'default' section represents the default configuration.

Kubernetes Environments

If the agent runs in Kubernetes environments (Open Source/OpenShift/GKE), include the following Kubernetes objects as key-value pairs. See Agent Install: Kubernetes for details on agent installation.

For example:

sysdig_sd_configs:
- tags:
    namespace: backend
    deployment: my-api

In addition to the aforementioned tags, any of these object types can be matched against:

daemonset: my_daemon
deployment: my_deployment
hpa: my_hpa
namespace: my_namespace
node: my_node
pod: my_pode
replicaset: my_replica
replicationcontroller: my_controller
resourcequota: my_quota
service: my_service
stateful: my_statefulset

For Kubernetes/OpenShift/GKE deployments, prometheus.yaml shares the same ConfigMap with dragent.yaml.

Docker Environments

In Docker environments, include attributes such as container, host, port, and more. For example:

sysdig_sd_configs:
- tags:
    host: my-host
    port: 8080

For Docker-based deployments, prometheus.yaml can be mounted from the host.

Sample Prometheus Configuration File

global:
  scrape_interval: 20s
scrape_configs:
- job_name: 'default'
  sysdig_sd_configs: # default config
  relabel_configs:
- job_name: 'my-app-job'
  sample_limit: 2000
  sysdig_sd_configs:  # apply this filtering config only to my-app
  - tags:
      namespace: backend
      deployment: my-app
  metric_relabel_configs:
  # Drop all metrics starting with http_
  - source_labels: [__name__]
    regex: "http_(.+)"
    action: drop
  metric_relabel_configs:
  # Drop all metrics for which the city label equals atlantis
  - source_labels: [city]
    regex: "atlantis"
    action: drop

1.5 - (Legacy) Example Configuration

This topic introduces you to default and specific Prometheus configurations.

Default Configuration

As an example that pulls together many of the configuration elements shown above, consider the default Agent configuration that’s inherited from the dragent.default.yaml.

prometheus:
  enabled: true
  interval: 10
  log_errors: true
  max_metrics: 1000
  max_metrics_per_process: 100
  max_tags_per_metric: 20

  # Filtering processes to scan. Processes not matching a rule will not
  # be scanned
  # If an include rule doesn't contain a port or port_filter in the conf
  # section, we will scan all the ports that a matching process is listening to.
  process_filter:
    - exclude:
        process.name: docker-proxy
    - exclude:
        container.image: sysdig/agent
    # special rule to exclude processes matching configured prometheus appcheck
    - exclude:
        appcheck.match: prometheus
    - include:
        container.label.io.prometheus.scrape: "true"
        conf:
            # Custom path definition
            # If the Label doesn't exist we'll still use "/metrics"
            path: "{container.label.io.prometheus.path}"

            # Port definition
            # - If the Label exists, only scan the given port.
            # - If it doesn't, use port_filter instead.
            # - If there is no port_filter defined, skip this process
            port: "{container.label.io.prometheus.port}"
            port_filter:
                - exclude: [9092,9200,9300]
                - include: 9090-9500
                - include: [9913,9984,24231,42004]
    - exclude:
        container.label.io.prometheus.scrape: "false"
    - include:
        kubernetes.pod.annotation.prometheus.io/scrape: true
        conf:
            path: "{kubernetes.pod.annotation.prometheus.io/path}"
            port: "{kubernetes.pod.annotation.prometheus.io/port}"
    - exclude:
        kubernetes.pod.annotation.prometheus.io/scrape: false

Consider the following about this default configuration:

• All Prometheus scraping is disabled by default. To enable the entire configuration shown here, you would only need to add the following to your dragent.yaml:

  prometheus:
    enabled: true
  

  Enabling this option and any pods (in case of Kubernetes) that have the right annotation set or containers (if not) that have the labels set will automatically be scrapped.

• Once enabled, this default configuration is ideal for the use case described in the Quick Start For Kubernetes Environments.

• A Process Filter rule excludes processes that are likely to exist in most environments but are known to never export Prometheus metrics, such as the Docker Proxy and the Agent itself.

• Another Process Filter rule ensures that any processes configured to be scraped by the legacy Prometheus application check will not be scraped.

• Another Process Filter rule is tailored to use container Labels. Processes marked with the container Label io.prometheus.scrape will become eligible for scraping, and if further marked with container Labels io.prometheus.port and/or io.prometheus.path, scraping will be attempted only on this port and/or endpoint. If the container is not marked with the specified path Label, scraping the /metrics endpoint will be attempted. If the container is not marked with the specified port Label, any listening ports in the port_filter will be attempted for scraping (this port_filter in the default is set for the range of ports for common Prometheus exporters, with exclusions for ports in the range that are known to be used by other applications that are not exporters).

• The final Process Filter Include rule is tailored to the use case described in the Quick Start For Kubernetes Environments.

Scrape a Single Custom Process

If you need to scrape a single custom process, for instance, a java process listening on port 9000 with path /prometheus, add the following to the dragent.yaml:

prometheus:
  enabled: true
  process_filter:
    - include:
        process.name: java
        port: 9000
        conf:
          # ensure we only scrape port 9000 as opposed to all ports this process may be listening to
          port: 9000
          path: "/prometheus"

This configuration overrides the default process_filter section shown in Default Configuration. You can add relevant rules from the default configuration to this to further filter down the metrics.

Scrape a Single Custom Process Based on Container Labels

If you still want to scrape based on container labels, you could just append the relevant rules from the defaults to the process_filter. For example:

prometheus:
  enabled: true
  process_filter:
    - include:
        process.name: java
        port: 9000
        conf:
          # ensure we only scrape port 9000 as opposed to all ports this process may be listening to
          port: 9000
          path: "/prometheus"
    - exclude:
        process.name: docker-proxy
    - include:
        container.label.io.prometheus.scrape: "true"
        conf:
            path: "{container.label.io.prometheus.path}"
            port: "{container.label.io.prometheus.port}"

Container Environment

With this default configuration enabled, a containerized install of our example exporter shown below would be automatically scraped via the Agent.

# docker run -d -p 8080:8080 \
    --label io.prometheus.scrape="true" \
    --label io.prometheus.port="8080" \
    --label io.prometheus.path="/prometheus" \
    luca3m/prometheus-java-app

Kubernetes Environment

In a Kubernetes-based environment, a Deployment with the Annotations as shown in this example YAML would be scraped by enabling the default configuration.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-java-app
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-java-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/prometheus"
        prometheus.io/port: "8080"
    spec:
      containers:
        - name: prometheus-java-app
          image: luca3m/prometheus-java-app
          imagePullPolicy: Always

Non-Containerized Environment

This is an example of a non-containerized environment or a containerized environment that doesn’t use Labels or Annotations. The following dragent.yaml would override the default and do per-second scrapes of our sample exporter and also a second exporter on port 5005, each at their respective non-standard endpoints. This can be thought of as a conservative “whitelist” type of configuration since it restricts scraping to only exporters that are known to exist in the environment and the ports on which they’re known to export Prometheus metrics.

prometheus:
  enabled: true
  interval: 1
  process_filter:
    - include:
        process.cmdline: "*app.jar*"
        conf:
          port: 8080
          path: "/prometheus"
    - include:
        port: 5005
        conf:
          port: 5005
          path: "/wacko"

1.6 - (Legacy) Logging and Troubleshooting

Logging

After the Agent begins scraping Prometheus metrics, there may be a delay of up to a few minutes before the metrics become visible in Sysdig Monitor. To help quickly confirm your configuration is correct, starting with Agent version 0.80.0, the following log line will appear in the Agent log the first time since starting that it has found and is successfully scraping at least one Prometheus exporter:

2018-05-04 21:42:10.048, 8820, Information, 05-04 21:42:10.048324 Starting export of Prometheus metrics

As this is an INFO level log message, it will appear in Agents using the default logging settings. To reveal even more detail,increase the Agent log level to DEBUG , which produces a message like the following that reveals the name of a specific metric first detected. You can then look for this metric to be visible in Sysdig Monitor shortly after.

2018-05-04 21:50:46.068, 11212, Debug, 05-04 21:50:46.068141 First prometheus metrics since agent start: pid 9583: 5 metrics including: randomSummary.95percentile

Troubleshooting

See the previous section for information on expected log messages during successful scraping. If you have enabled Prometheus and are not seeing the Starting export message shown there, revisit your configuration.

It is also suggested to leave the configuration option in its default setting of log_errors: true , which will reveal any issues scraping eligible processes in the Agent log.

For example, here is an error message for a failed scrape of a TCP port that was listening but not accepting HTTP requests:

2017-10-13 22:00:12.076, 4984, Error, sdchecks[4987] Exception on running check prometheus.5000: Exception('Timeout when hitting http://localhost:5000/metrics',)
2017-10-13 22:00:12.076, 4984, Error, sdchecks, Traceback (most recent call last):
2017-10-13 22:00:12.076, 4984, Error, sdchecks, File "/opt/draios/lib/python/sdchecks.py", line 246, in run
2017-10-13 22:00:12.076, 4984, Error, sdchecks, self.check_instance.check(self.instance_conf)
2017-10-13 22:00:12.076, 4984, Error, sdchecks, File "/opt/draios/lib/python/checks.d/prometheus.py", line 44, in check
2017-10-13 22:00:12.076, 4984, Error, sdchecks, metrics = self.get_prometheus_metrics(query_url, timeout, "prometheus")
2017-10-13 22:00:12.076, 4984, Error, sdchecks, File "/opt/draios/lib/python/checks.d/prometheus.py", line 105, in get_prometheus_metrics
2017-10-13 22:00:12.077, 4984, Error, sdchecks, raise Exception("Timeout when hitting %s" % url)
2017-10-13 22:00:12.077, 4984, Error, sdchecks, Exception: Timeout when hitting http://localhost:5000/metrics

Here is an example error message for a failed scrape of a port that was responding to HTTP requests on the /metrics endpoint but not responding with valid Prometheus-format data. The invalid endpoint is responding as follows:

# curl http://localhost:5002/metrics
This ain't no Prometheus metrics!

And the corresponding error message in the Agent log, indicating no further scraping will be attempted after the initial failure:

2017-10-13 22:03:05.081, 5216, Information, sdchecks[5219] Skip retries for Prometheus error: could not convert string to float: ain't
2017-10-13 22:03:05.082, 5216, Error, sdchecks[5219] Exception on running check prometheus.5002: could not convert string to float: ain't

1.7 - (Legacy) Collecting Prometheus Metrics from Remote Hosts

(Legacy) Collecting Prometheus Metrics from Remote Hosts

Sysdig Monitor can collect Prometheus metrics from remote endpoints with minimum configuration. Remote endpoints (remote hosts) refer to hosts where Sysdig Agent cannot be deployed. For example, a Kubernetes master node on managed Kubernetes services such as GKE and EKS where user workload cannot be deployed, which in turn implies no Agents involved. Enabling remote scraping on such hosts is as simple as identifying an Agent to perform the scraping and declaring the endpoint configurations with a remote services section in the Agent configuration file.

The collected Prometheus metrics are reported under and associated with the Agent that performed the scraping as opposed to associating them with a process.

Preparing the Configuration File

Multiple Agents can share the same configuration. Therefore, determine which one of those Agents scrape the remote endpoints with the dragent.yaml file. This is applicable to both

• Create a separate configuration section for remote services in the Agent configuration file under the prometheus configuration.

• Include a configuration section for each remote endpoint, and add either a URL or host/port (and an optional path) parameter to each section to identify the endpoint to scrape. The optional path identifies the resource at the endpoint. An empty path parameter defaults to the "/metrics" endpoint for scraping.

• Optionally, add custom tags for each endpoint configuration for remote services. In the absence of tags, metric reporting might not work as expected when multiple endpoints are involved. Agents cannot distinguish similar metrics scraped from multiple endpoints unless those metrics are uniquely identified by tags.

To help you get started, an example configuration for Kubernetes is given below:

prometheus:
  remote_services:
        - prom_1:
            kubernetes.node.annotation.sysdig.com/region: europe
            kubernetes.node.annotation.sysdig.com/scraper: true
            conf:
                url: "https://xx.xxx.xxx.xy:5005/metrics"
                tags:
                    host: xx.xxx.xxx.xy
                    service: prom_1
                    scraping_node: "{kubernetes.node.name}"
        - prom_2:
            kubernetes.node.annotation.sysdig.com/region: india
            kubernetes.node.annotation.sysdig.com/scraper: true
            conf:
                host: xx.xxx.xxx.yx
                port: 5005
                use_https: true
                tags:
                    host: xx.xxx.xxx.yx
                    service: prom_2
                    scraping_node: "{kubernetes.node.name}"
        - prom_3:
            kubernetes.pod.annotation.sysdig.com/prom_3_scraper: true
            conf:
                url: "{kubernetes.pod.annotation.sysdig.com/prom_3_url}"
                tags:
                    service: prom_3
                    scraping_node: "{kubernetes.node.name}"
        - haproxy:
            kubernetes.node.annotation.yourhost.com/haproxy_scraper: true
            conf:
                host: "mymasternode"
                port: 1936
                path: "/metrics"
                username: "{kubernetes.node.annotation.yourhost.com/haproxy_username}"
                password: "{kubernetes.node.annotation.yourhost.com/haproxy_password}"
                tags:
                    service: router

In the above example, scraping is triggered by node and pod annotations. You can add annotations to nodes and pods by using the kubectl annotate command as follows:

kubectl annotate node mynode --overwrite sysdig.com/region=india sysdig.com/scraper=true haproxy_scraper=true yourhost.com/haproxy_username=admin yourhost.com/haproxy_password=admin

In this example, you set annotation on a node to trigger scraping of the prom2 and haproxy services as defined in the above configuration.

Preparing Container Environments

An example configuration for Docker environment is given below:

prometheus:
  remote_services:
        - prom_container:
            container.label.com.sysdig.scrape_xyz: true
            conf:
                url: "https://xyz:5005/metrics"
                tags:
                    host: xyz
                    service: xyz

In order for remote scraping to work in a Docker-based container environment, set the com.sysdig.scrape_xyz=true label to the Agent container. For example:

docker run -d --name sysdig-agent --restart always --privileged --net host --pid host -e ACCESS_KEY=<KEY> -e COLLECTOR=<COLLECTOR> -e SECURE=true -e TAGS=example_tag:example_value -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro --shm-size=512m sysdig/agent

Substitute <KEY>, <COLLECTOR>, TAGS with your account key, collector, and tags respectively.

Syntax of the Rules

The syntax of the rules for the remote_services is almost identical to those of process_filter with an exception to the include/exclude rule. The remote_services section does not use include/exclude rules. The process_filter uses include and exclude rules of which only the first match against a process is applied, whereas, in the remote_services section, each rule has a corresponding service name and all the matching rules are applied.

Rule Conditions

The rule conditions work the same way as those for the process_filter. The only caveat is that the rules will be matched against the Agent process and container because the remote process/context is unknown. Therefore, matches for container labels and annotations work as before but they must be applicable to the Agent container as well. For instance, node annotations will apply because the Agent container runs on a node.

For annotations, multiple patterns can be specified in a single rule, in which case all patterns must match for the rule to be a match (AND operator). In the following example, the endpoint will not be considered unless both the annotations match:

kubernetes.node.annotation.sysdig.com/region_scraper: europe
kubernetes.node.annotation.sysdig.com/scraper: true

That is, Kubernetes nodes belonging to only the Europe region are considered for scraping.

Authenticating Sysdig Agent

Sysdig Agent requires necessary permissions on the remote host to scrape for metrics. The authentication methods for local scraping works for authenticating agents on remote hosts as well, but the authorization parameters work only in the agent context.

• Authentication based on certificate-key pair requires it to be constructed into Kubernetes secret and mounted to the agent.

• In token-based authentication, make sure the agent token has access rights on the remote endpoint to do the scraping.

• Use annotation to retrieve username/password instead of passing them in plaintext. Any annotation enclosed in curly braces will be replaced by the value of the annotation. If the annotation doesn’t exist the value will be an empty string. Token substitution is supported for all the authorization parameters. Because authorization works only in the Agent context, credentials cannot be automatically retrieved from the target pod. Therefore, use an annotation in the Agent pod to pass them. To do so, set the password into an annotation for the selected Kubernetes object.

In the following example, an HAProxy account is authenticated with the password supplied in the yourhost.com/haproxy_password annotation on the agent node.

- haproxy:
            kubernetes.node.annotation.yourhost.com/haproxy_scraper: true
            conf:
                host: "mymasternode"
                port: 1936
                path: "/metrics"
                username: "{kubernetes.node.annotation.yourhost.com/haproxy_username}"
                password: "{kubernetes.node.annotation.yourhost.com/haproxy_password}"
                tags:
                    service: router

2 - (Legacy) Integrate Applications (Default App Checks)

The Sysdig agent supports additional application monitoring capabilities with application check scripts or ‘app checks’. These are a set of plugins that poll for custom metrics from the specific applications which export them via status or management pages: e.g. NGINX, Redis, MongoDB, Memcached and more.

Many app checks are enabled by default in the agent and when a supported application is found, the correct app check script will be called and metrics polled automatically.

However, if default connection parameters are changed in your application, you will need to modify the app check connection parameters in the Sysdig Agent configuration file (dragent.yaml) to match your application.

In some cases, you may also need to enable the metrics reporting functionality in the application before the agent can poll them.

This page details how to make configuration changes in the agent’s configuration file, and provides an application integration example. Click the Supported Applications links for application-specific details.

Edit dragent.yaml to Integrate or Modify Application Checks

Out of the box, the Sysdig agent will gather and report on a wide variety of pre-defined metrics. It can also accommodate any number of custom parameters for additional metrics collection.

The agent relies on a pair of configuration files to define metrics collection parameters:

The “dragent.yaml” file can be accessed and edited in several ways, depending on how the agent was installed.

Review Understanding the Agent Config Files for details.

The examples in this section presume you are entering YAML code directly intodragent.yaml, under the app_checks section.

Find the default settings

To find the default app-checks for already supported applications, check the dragent.default.yaml file.

(Location: /opt/draios/etc/dragent.default.yaml.)

Sample format

app_checks:
  - name: APP_NAME
    check_module: APP_CHECK_SCRIPT
    pattern:
      comm: PROCESS_NAME
    conf:
      host: IP_ADDR
      port: PORT

Change the default settings

To override the defaults:

1. Copy relevant code blocks from dragent.default.yaml into dragent.yaml . (Or copy the code from the appropriate app check integration page in this documentation section.)

   Any entries copied into dragent.yaml file will override similar entries in dragent.default.yaml.

   Never modify dragent.default.yaml, as it will be overwritten whenever the agent is updated.

2. Modify the parameters as needed.

   Be sure to use proper YAML. Pay attention to consistent spacing for indents (as shown) and list all check entries under an app_checks: section title.

3. Save the changes and restart the agent.

   Use service restart agent or docker restart sysdig-agent.

Metrics for the relevant application should appear in the Sysdig Monitor interface under the appropriate name.

Example 1: Change Name and Add Password

Here is a sample app-check entry for Redis. The app_checks section was copied from the dragent.default.yaml file and modified for a specific instance.

customerid: 831f3-Your-Access-Key-9401
tags: local:sf,acct:dev,svc:db
app_checks:
  - name: redis-6380
    check_module: redisdb
    pattern:
      comm: redis-server
    conf:
      host: 127.0.0.1
      port: PORT
      password: PASSWORD

Edits made:

• The name to be displayed in the interface

• A required password.

Example 2: Increase Polling Interval

The default interval for an application check to be run by the agent is set to every second. You can increase the interval per application check by adding the interval: parameter (under the -name section) and the number of seconds to wait before each run of the script.

Example: Run the NTP check once per minute:

app_checks:
  - name: ntp
    interval: 60
    pattern:
      comm: systemd
    conf:
      host: us.pool.ntp.org

Disabling

Disable a Single Application Check

Sometimes the default configuration shipped with the Sysdig agent does not work for you or you may not be interested in checks for a single application. To turn a single check off, add an entry like this to disable it:

app_checks:
 - name: nginx
   enabled: false

This entry overrides the default configuration of the nginx check, disabling it.

If you are using the ADDITIONAL_CONF parameter to modify your container agent’s configuration, you would add an entry like this to your Docker run command (or Kubernetes manifest):

-e ADDITIONAL_CONF="app_checks:\n  - name: nginx\n    enabled: false\n"

Disable ALL Application Checks

If you do not need it or otherwise want to disable the application check functionality, you can add the following entry to the agent’s user settings configuration file /opt/draios/etc/dragent.yaml:

app_checks_enabled: false

Restart the agent as shown immediately above for either the native Linux agent installation or the container agent installation.

Optional: Configure a Custom App-Check

Sysdig allows custom application check-script configurations to be created for each individual container in the infrastructure, via the environment variable SYSDIG_AGENT_CONF. This avoids the need for multiple edits and entries to achieve the container-specific customization, by enabling application teams to configure their own checks.

The SYSDIG_AGENT_CONF variable stores a YAML-formatted configuration for the app check, and is used to match app-check configurations. It can be stored directly within the Docker file.

The example below defines a per container app-check for Redis in the Dockerfile, using the SYSDIG_AGENT_CONF environment variable:

FROM redis
# This config file adds a password for accessing redis instance
ADD redis.conf /

ENV SYSDIG_AGENT_CONF { "app_checks": [{ "name": "redis", "check_module": "redisdb", "pattern": {"comm": "redis-server"}, "conf": { "host": "127.0.0.1", "port": "6379", "password": "protected"} }] }
ENTRYPOINT ["redis-server"]
CMD [ "/redis.conf" ]

The example below shows how parameters can be added to a container started with docker run, by either using the -e/–envflag variable, or injecting the parameters using an orchestration system (for example, Kubernetes):

PER_CONTAINER_CONF='{ "app_checks": [{ "name": "redis", "check_module": "redisdb", "pattern": {"comm": "redis-server"}, "conf": { "host": "127.0.0.1", "port": "6379", "password": "protected"} }] }'

docker run --name redis -v /tmp/redis.conf:/etc/redis.conf -e SYSDIG_AGENT_CONF="${PER_CONTAINER_CONF}" -d redis /etc/redis.conf

Metrics Limit

Metric limits are defined by your payment plan. If more metrics are needed please contact your sales representative with your use case.

Note that a metric with the same name but different tag name will count as a unique metric by the agent. Example: a metric 'user.clicks' with the tag 'country=us' and another 'user.clicks' with the 'tag country=it'are considered two metrics which count towards the limit.

Supported Applications

Below is the supported list of applications the agent will automatically poll.

Some app-check scripts will need to be configured since no defaults exist, while some applications may need to be configured to output their metrics. Click a highlighted link to see application-specific notes.

• Active MQ
• Apache
• Apache CouchDB
• Apache HBase
• Apache Kafka
• Apache Zookeeper
• Consul
• CEPH
• Couchbase
• Elasticsearch
• etcd
• fluentd
• Gearman
• Go
• Gunicorn
• HAProxy
• HDFS
• HTTP
• Jenkins
• JVM
• Lighttpd
• Memcached
• Mesos/Marathon
• MongoDB
• MySQL
• NGINX and NGINX Plus
• NTP
• PGBouncer
• PHP-FPM
• Postfix
• PostgreSQL
• Prometheus
• RabbitMQ
• RedisDB
• Supervisord
• SNMP
• TCP

You can also

• Create a Custom App Check from scratch

• Create Per-Container Custom App Checks

2.1 - Apache

Apache web server is an open-source, web server creation, deployment, and management software. If Apache is installed on your environment, the Sysdig agent will connect using the mod_status module on Apache. You may need to edit the default entries in the agent configuration file to connect. See the Default Configuration, below.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Apache Setup

Install mod_status on your Apache servers and enable ExtendedStatus.

The following configuration is required. If it is already present, then un-comment the lines, otherwise add the configuration.

LoadModule status_module modules/mod_status.so
...

<Location /server-status>
    SetHandler server-status
    Order Deny,Allow
    Deny from all
    Allow from localhost
</Location>
...

ExtendedStatus On

Sysdig Agent Configuration

Review how to edit dragent.yaml to Integrate or Modify Application Checks.

Apache has a common default for exposing metrics. The process command name can be either apache2 or httpd. By default, the Sysdig agent will look for the process apache2. If named differently in your environment (e.g. httpd), edit the configuration file to match the process name as shown in Example 1.

Default Configuration

By default, Sysdig’s dragent.default.yaml uses the following code to connect with Apache and collect all metrics.

app_checks:
  - name: apache
    check_module: apache
    pattern:
      comm: apache2
    conf:
      apache_status_url: "http://localhost:{port}/server-status?auto"
    log_errors: false

Example

If it is necessary to edit dragent.yaml to change the process name, use the following example and update the comm with the value httpd.

app_checks:
  - name: apache
    check_module: apache
    pattern:
      comm: httpd
    conf:
      apache_status_url: "http://localhost/server-status?auto"
    log_errors: false

Metrics Available

The Apache metrics are listed in the metrics dictionary here: Apache Metrics.

UI Examples

2.2 - Apache Kafka

Apache Kafka is a distributed streaming platform. Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. If Kafka is installed on your environment, the Sysdig agent will automatically connect. See the Default Configuration, below.

The Sysdig agent automatically collects metrics from Kafka via JMX polling. You need to provide consumer names and topics in the agent config file to collect consumer-based Kafka metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Kafka Setup

Kafka will automatically expose all metrics. You do not need to add anything on the Kafka instance.

Sysdig Agent Configuration

Review how to edit dragent.yaml to Integrate or Modify Application Checks.

Metrics from Kafka via JMX polling are already configured in the agent’s default-settings configuration file. Metrics for consumers, however, need to use app-checks to poll the Kafka and Zookeeper API. You need to provide consumer names and topics in dragent.yaml file.

Default Configuration

Since consumer names and topics are environment-specific, a default configuration is not present in dragent.default.yaml.

Refer to the following examples for adding Kafka checks to dragent.yaml.

Example 1: Basic Configuration

A basic example with sample consumer and topic names:

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern:
      comm: java
      arg: kafka.Kafka
    conf:
      kafka_connect_str: "127.0.0.1:9092" # kafka address, usually localhost as we run the check on the same instance
      zk_connect_str: "localhost:2181" # zookeeper address, may be different than localhost
      zk_prefix: /
      consumer_groups:
        sample-consumer-1: # sample consumer name
          sample-topic-1: [0, ] # sample topic name and partitions
        sample-consumer-2: # sample consumer name
          sample-topic-2: [0, 1, 2, 3] # sample topic name and partitions

Example 2: Store Consumer Group Info (Kafka 9+)

From Kafka 9 onwards, you can store consumer group config info inside Kafka itself for better performance.

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern:
      comm: java
      arg: kafka.Kafka
    conf:
      kafka_connect_str: "localhost:9092"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      kafka_consumer_offsets: true
      consumer_groups:
        sample-consumer-1: # sample consumer name
          sample-topic-1: [0, ] # sample topic name and partitions

Example 3: Aggregate Partitions at the Topic Level

To enable aggregation of partitions at the topic level, use kafka_consumer_topics with aggregate_partitions = true.

In this case the app check will aggregate the lag & offset values at the partition level, reducing the number of metrics collected.

Set aggregate_partitions = false to disable metrics aggregation at the partition level. In this case, the appcheck will show lag and offset values for each partition.

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern:
      comm: java
      arg: kafka.Kafka
    conf:
      kafka_connect_str: "localhost:9092"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      kafka_consumer_offsets: true
      kafka_consumer_topics:
        aggregate_partitions: true
      consumer_groups:
        sample-consumer-1: # sample consumer name
          sample-topic-1: [0, ] # sample topic name and partitions
        sample-consumer-2: # sample consumer name
          sample-topic-2: [0, 1, 2, 3] # sample topic name and partitions

Example 4: Custom Tags

Optional tags can be applied to every emitted metric, service check, and/or event.

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern:
      comm: java
      arg: kafka.Kafka
    conf:
      kafka_connect_str: "localhost:9092"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      consumer_groups:
        sample-consumer-1: # sample consumer name
          sample-topic-1: [0, ] # sample topic name and partitions
    tags:  ["key_first_tag:value_1", "key_second_tag:value_2", "key_third_tag:value_3"]

Example 5: SSL and Authentication

If SSL and authentication are enabled on Kafka, use the following configuration.

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern:
      comm: java
      arg: kafka.Kafka
    conf:
      kafka_consumer_offsets: true
      kafka_connect_str: "127.0.0.1:9093"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      consumer_groups:
        test-group:
          test: [0, ]
          test-4: [0, 1, 2, 3]
      security_protocol: SASL_SSL
      sasl_mechanism: PLAIN
      sasl_plain_username: <USERNAME>
      sasl_plain_password: <PASSWORD>
      ssl_check_hostname: true
      ssl_cafile:  <SSL_CA_FILE_PATH>
      #ssl_context: <SSL_CONTEXT>
      #ssl_certfile: <CERT_FILE_PATH>
      #ssl_keyfile: <KEY_FILE_PATH>
      #ssl_password: <PASSWORD>
      #ssl_crlfile: <SSL_FILE_PATH>

Configuration Keywords and Descriptions

security_protocol (str)

sasl_mechanism (str)

sasl_plain_username (str) 

sasl_plain_password (str) 

ssl_context (ssl.SSLContext) 

ssl_check_hostname (bool)

ssl_cafile (str)

ssl_certfile (str)

ssl_keyfile (str)

ssl_password (str) 

ssl_crlfile (str)

Example 6: Regex for Consumer Groups and Topics

As of Sysdig agent version 0.94, the Kafka app check has added optional regex (regular expression) support for Kafka consumer groups and topics.

Regex Configuration:

• No new metrics are added with this feature

• The new parameter consumer_groups_regex is added, which includes regex for consumers and topics from Kafka. Consumer offsets stored in Zookeeper are not collected.

• Regex for topics is optional. When not provided, all topics under the consumer will be reported.

• The regex Python syntax is documented here: https://docs.python.org/3.7/library/re.html#regular-expression-syntax

• If both consumer_groups and consumer_groups_regex are provided at the same time, matched consumer groups from both parameters will be merged

Sample configuration:

app_checks:
  - name: kafka
    check_module: kafka_consumer
    pattern:
      comm: java
      arg: kafka.Kafka
    conf:
      kafka_connect_str: "localhost:9092"
      zk_connect_str: "localhost:2181"
      zk_prefix: /
      kafka_consumer_offsets: true
      # Regex can be provided in following format
      # consumer_groups_regex:
      #   'REGEX_1_FOR_CONSUMER_GROUPS':
      #      - 'REGEX_1_FOR_TOPIC'
      #      - 'REGEX_2_FOR_TOPIC'
      consumer_groups_regex:
        'consumer*':
          - 'topic'
          - '^topic.*'
          - '.*topic$'
          - '^topic.*'
          - 'topic\d+'
          - '^topic_\w+'

Example

Metrics Available

Kafka Consumer Metrics (App Checks)

See Apache Kafka Consumer Metrics.

JMX Metrics

See Apache Kafka JMX Metrics.

Result in the Monitor UI

2.3 - Consul

Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud. If Consul is installed on your environment, the Sysdig agent will automatically connect and collect basic metrics. If the Consul Access Control List (ACL) is configured, you may need to edit the default entries to connect. Also, additional latency metrics can be collected by modifying default entries. See the Default Configuration, below.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Consul Configuration

Consul is ready to expose metrics without any special configuration.

Sysdig Agent Configuration

Review how to edit dragent.yaml to Integrate or Modify Application Checks.

Default Configuration

By default, Sysdig’s dragent.default.yaml ``uses the following code to connect with Consul and collect basic metrics.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      catalog_checks: yes

With the dragent.default.yaml file, the below set of metrics are available in the Sysdig Monitor UI:

Additional metrics and event can be collected by adding configuration in dragent.yaml file. The ACL token must be provided if enabled. See the following examples.

Example 1: Enable Leader Change Event

self_leader_check An enabled node will watch for itself to become the leader and will emit an event when that happens. It can be enabled on all nodes.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      catalog_checks: yes
      self_leader_check: yes
    logs_enabled: true

Example 2: Enable Latency Metrics

If the network_latency_checks flag is enabled, then the Consul network coordinates will be retrieved and the latency calculated for each node and between data centers.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      catalog_checks: yes
      network_latency_checks: yes
    logs_enabled: true

With the above changes, you can see the following additional metrics:

Example 3: Enable ACL Token

When the ACL Systemis enabled in Consul, the ACL Agent Token must be added in dragent.yaml in order to collect metrics.

Follow Consul’s official documentation to Configure ACL, Bootstrap ACL and Create Agent Token.

app_checks:
  - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      acl_token: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" #Add agent token
      catalog_checks: yes
      logs_enabled: true

Example 4: Collect Metrics from Non-Leader Node

Required: Agent 9.6.0+

With agent 9.6.0, you can use the configuration option single_node_install (Optional. Default: false). Set this option to true and the app check will be performed on non-leader nodes of Consul.

app_checks:
   - name: consul
    pattern:
      comm: consul
    conf:
      url: "http://localhost:8500"
      catalog_checks: yes
      single_node_install: true

StatsD Metrics

In addition to the metrics from the Sysdig app-check, there are many other metrics that Consul can send using StatsD. Those metrics will be automatically collected by the Sysdig agent’s StatsD integration if Consul is configured to send them.

Add statsd_address under telemetry to the Consul config file. The default config file location is /consul/config/local.json

{
...
  "telemetry": {
     "statsd_address": "127.0.0.1:8125"
  }
...
}

See Telemetry Metrics for more details.

Metrics Available

See Consul Metrics.

Result in the Monitor UI

2.4 - Couchbase

Couchbase Server is a distributed, open-source, NoSQL database engine. The core architecture is designed to simplify building modern applications with a flexible data model and simpler high availability, high scalability, high performance, and advanced security. If Couchbase is installed on your environment, the Sysdig agent will automatically connect. If authentication is configured, you may need to edit the default entries to connect. See the Default Configuration, below.

The Sysdig agent automatically collects all bucket and node metrics. You can also edit the configuration to collect query metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Couchbase Setup

Couchbase will automatically expose all metrics. You do not need to configure anything on the Couchbase instance.

Sysdig Agent Configuration

Review how to edit dragent.yaml to Integrate or Modify Application Checks.

Default Configuration

By default, Sysdig’s dragent.default.yaml uses the following code to connect with Couchbase and collect all bucket and node metrics.

app_checks:
  - name: couchbase
    pattern:
      comm: beam.smp
      arg: couchbase
      port: 8091
    conf:
      server: http://localhost:8091

If authentication is enabled, you need to edit dragent.yaml file to connect with Couchbase. See Example 1.

Example 1: Authentication

Replace <username> and <password> with appropriate values and update the dragent.yaml file.

app_checks:
  - name: couchbase
    pattern:
      comm: beam.smp
      arg: couchbase
      port: 8091
    conf:
      server: http://localhost:8091
      user: <username>
      password: <password>
      # The following block is optional and required only if the 'path' and
      # 'port' need to be set to non-default values specified here
      cbstats:
        port: 11210
        path: /opt/couchbase/bin/cbstats

Example 2: Query Stats

Additionally, you can configure query_monitoring_url to get query monitoring stats. This is available from Couchbase version 4.5. See Query Monitoring for more detail.

app_checks:
  - name: couchbase
    pattern:
      comm: beam.smp
      arg: couchbase
      port: 8091
    conf:
      server: http://localhost:8091
      query_monitoring_url: http://localhost:8093

Metrics Available

See Couchbase Metrics.

Result in the Monitor UI

2.5 - Elasticsearch

Elasticsearch is an open-source, distributed, document storage and search engine that stores and retrieves data structures in near real-time. Elasticsearch represents data in the form of structured JSON documents and makes full-text search accessible via RESTful API and web clients for languages like PHP, Python, and Ruby. It’s also elastic in the sense that it’s easy to scale horizontally—simply add more nodes to distribute the load. If Elasticsearch is installed on your environment, the Sysdig agent will automatically connect in most of the cases. See the Default Configuration, below.

The Sysdig Agent automatically collects default metrics. You can also edit the configuration to collect Primary Shard stats.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Elasticsearch Setup

Elasticsearch is ready to expose metrics without any special configuration.

Sysdig Agent Configuration

Review how to edit dragent.yaml to Integrate or Modify Application Checks.

Default Configuration

By default, Sysdig’s dragent.default.yaml uses the following code to connect with Elasticsearch and collect basic metrics.

app_checks:
  - name: elasticsearch
    check_module: elastic
    pattern:
      port: 9200
      comm: java
    conf:
      url: http://localhost:9200

For more metrics, you may need to change the elasticsearch default setting in dragent.yaml:

Example 1: Agent authentication to Elasticsearch Cluster with Authentication

Password Authentication

app_checks:
  - name: elasticsearch
    check_module: elastic
    pattern:
      port: 9200
      comm: java
    conf:
      url: https://sysdigcloud-elasticsearch:9200
      username: readonly
      password: some_password
      ssl_verify: false

Certificate Authentication

app_checks:
   - name: elasticsearch
     check_module: elastic
     pattern:
       port: 9200
       comm: java
     conf:
       url: https://localhost:9200
       ssl_cert: /tmp/certs/ssl.crt
       ssl_key: /tmp/certs/ssl.key
       ssl_verify: true

ssl_cert: Path to the certificate chain used for validating the authenticity of the Elasticsearch server.

ssl_key: Path to the certificate key used for authenticating to the Elasticsearch server.

Example 2: Enable Primary shard Statistics

app_checks:
  - name: elasticsearch
    check_module: elastic
    pattern:
      port: 9200
      comm: java
    conf:
      url: http://localhost:9200
      pshard_stats : true

pshard-specific Metrics

Enable pshard_stats to monitor the following additional metrics:

Example 3: Enable Primary shard Statistics for Master Node only

app_checks:
  - name: elasticsearch
    check_module: elastic
    pattern:
      port: 9200
      comm: java
    conf:
      url: http://localhost:9200
      pshard_stats_master_node_only: true

Note that this option takes precedence over the pshard_stats option (above). This means that if the following configuration were put into place, only the pshard_stats_master_node_only option would be respected:

app_checks:
  - name: elasticsearch
    check_module: elastic
    pattern:
      port: 9200
      comm: java
    conf:
      url: http://localhost:9200
      pshard_stats: true
      pshard_stats_master_node_only: true

All Available Metrics

With the default settings and the pshard setting, the total available metrics are listed here: Elasticsearch Metrics.

Result in the Monitor UI

2.6 - etcd

etcdis a distributed key-value store that provides a reliable way to store data across a cluster of machines. If etcd is installed on your environment, the Sysdig agent will automatically connect. If you are using ectd older than version 2, you may need to edit the default entries to connect. See the Default Configuration section, below.

The Sysdig Agent automatically collects all metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

etcd Versions

etcd v2

The app check functionality described on this page supports etcd metrics from APIs that are specific to v2 of etcd.

These APIs are present in etcd v3 as well, but export metrics only for the v2 datastores. For example, after upgrading from etcd v2 to v3, if the v2 datastores are not migrated to v3, the v2 APIs will continue exporting metrics for these datastores. If the v2 datastores are migrated to v3, the v2 APIs will no longer export metrics for these datastores.

etcd v3

etcd v3 uses a native Prometheus exporter. The exporter only exports metrics for v3 datastores. For example, after upgrading from etcd v2 to v3, if v2 datastores are not migrated to v3, the Prometheus endpoint will not export metrics for these datastores. The Prometheus endpoint will only export metrics for datastores migrated to v3 or datastores created after the upgrade to v3.

If your etcd version is v3 or higher, use the information on this page to enable an integration: Integrate Prometheus Metrics.

etcd Setup

etcd will automatically expose all metrics. You do not need to add anything to the etcd instance.

Sysdig Agent Configuration

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

The default agent configuration for etcd will look for the application on localhost, port 2379. No customization is required.

Default Configuration

By default, Sysdig’s dragent.default.yaml uses the following code to connect with etcd and collect all metrics.

app_checks:
  - name: etcd
    pattern:
      comm: etcd
    conf:
      url: "http://localhost:2379"

etcd (before version 2) does not listen on localhost, so the Sysdig agent will not connect to it automatically. In such case, you may need edit the dragent.yaml file with the hostname and port. See Example 1.

Alternatively, you can add the option -bind-addr 0.0.0.0:4001 to the etcd command line to allow the agent to connect.

Example 1

You can use {hostname} and {port} as a tokens in the conf: section. This is the recommended setting for Kubernetes customers.

app_checks:
  - name: etcd
    pattern:
      comm: etcd
    conf:
      url: "http://{hostname}:{port}"

Alternatively you can specify the real hostname and port.

app_checks:
  - name: etcd
    pattern:
      comm: etcd
    conf:
      url: "http://my_hostname:4000"  #etcd service listening on port 4000

Example 2: SSL/TLS Certificate

If encryption is used, add the appropriate SSL/TLS entries. Provide correct path of SSL/TLS key and certificates used in etcd configuration in fields ssl_keyfile, ssl_certfile, ssl_ca_certs.

app_checks:
  - name: etcd
    pattern:
      comm: etcd
    conf:
      url: "https://localhost:PORT"
      ssl_keyfile:  /etc/etcd/peer.key  # Path to key file
      ssl_certfile: /etc/etcd/peer.crt  # Path to SSL certificate
      ssl_ca_certs: /etc/etcd/ca.crt    # Path to CA certificate
      ssl_cert_validation: True

Metrics Available

See etcd Metrics.

Result in the Monitor UI

2.7 - fluentd

Fluentd is an open source data collector, which allows unifying data collection and consumption to better use and understand data. Fluentd structures data as JSON as much as possible, to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations. If Fluentd is installed on your environment, the Sysdig agent will automatically connect. See See the Default Configuration section, below. The Sysdig agent automatically collects default metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Fluentd Setup

Fluentd can be installed as a package (.deb, .rpm, etc) depending on the OS flavor, or it can be deployed in a Docker container. Fluentd installation is documented here. For the examples on this page, a .deb package installation is used.

After installing Fluentd, add following lines in fluentd.conf :

<source>
  @type monitor_agent
  bind 0.0.0.0
  port 24220
</source>

Sysdig Agent Configuration

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

Default Configuration

By default, Sysdig’sdragent.default.yaml uses the following code to connect with Fluentd and collect default metrics.

(If you use a non-standard port for monitor_agent , you can configure it as usual in the agent config file dragent.yaml.)

  - name: fluentd
    pattern:
      comm: fluentd
    conf:
      monitor_agent_url: http://localhost:24220/api/plugins.json

Example

To generate the metric data, it is necessary to generate some logs through an application. In the following example, HTTP is used. (For more information, see Life of a Fluentd event.)

Execute the following command on in the Fluentd environment:

$ curl -i -X POST -d 'json={"action":"login","user":2}' http://localhost:8888/test.cycle

Expected output: (Note: Here the status code is 200 OK, as HTTP traffic is successfully generated; it will vary per application.)

HTTP/1.1 200 OK
Content-type: text/plain
Connection: Keep-Alive
Content-length: 0

Metrics Available

See fluentd Metrics.

Result in the Monitor UI

2.8 - Go

Golang expvaris the standard interface designed to instrument and expose custom metrics from a Go program via HTTP. In addition to custom metrics, it also exports some metrics out-of-the-box, such as command line arguments, allocation stats, heap stats, and garbage collection metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

Go_expvar Setup

You will need to create a custom entry in the user settings config file for your Go application, due to the difficulty in determining if an application is written in Go by looking at process names or arguments. Be sure your app has expvars enabled, which means importing the expvar module and having an HTTP server started from inside your app, as follows:

import (
    ...
    "net/http"
    "expvar"
    ...
)

// If your application has no http server running for the DefaultServeMux,
// you'll have to have a http server running for expvar to use, for example
// by adding the following to your init function
func init() {
    go http.ServeAndListen(":8080", nil)
}

// You can also expose variables that are specific to your application
// See http://golang.org/pkg/expvar/ for more information

var (
    exp_points_processed = expvar.NewInt("points_processed")
)

func processPoints(p RawPoints) {
    points_processed, err := parsePoints(p)
    exp_points_processed.Add(points_processed)
    ...
}

See also the following blog entry: How to instrument Go code with custom expvar metrics.

Sysdig Agent Configuration

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

Default Configuration

No default configuration for Go is provided in the Sysdig agent dragent.default.yaml file. You must edit the agent config file as described in Example 1.

Example

Add the following code sample to dragent.yaml to collect Go metrics.

app_checks:
  - name: go-expvar
    check_module: go_expvar
    pattern:
          comm: go-expvar
    conf:
      expvar_url: "http://localhost:8080/debug/vars" # automatically match url using the listening port
      # Add custom metrics if you want
      metrics:
        - path: system.numberOfSeconds
          type: gauge # gauge or rate
          alias: go_expvar.system.numberOfSeconds
        - path: system.lastLoad
          type: gauge
          alias: go_expvar.system.lastLoad
        - path: system.numberOfLoginsPerUser/.* # You can use / to get inside the map and use .* to match any record inside
          type: gauge
        - path: system.allLoad/.*
          type: gauge

Metrics Available

See Go Metrics.

Result in the Monitor UI

2.9 - HAProxy

HAProxy provides a high-availability load balancer and proxy server for TCP- and HTTP-based applications which spreads requests across multiple servers.

The Sysdig agent automatically collects haproxy metrics. You can also edit the agent configuration file to collect additional metrics.

This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

HAProxy Setup

The stats feature must be enabled on your HAProxy instance. This can be done by adding the following entry to the HAProxy configuration file /etc/haproxy/haproxy.cfg

listen stats
  bind :1936
  mode http
  stats enable
  stats hide-version
  stats realm Haproxy\ Statistics
  stats uri /haproxy_stats
  stats auth stats:stats

Sysdig Agent Configuration

Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

Default Configuration

By default, Sysdig’s dragent.default.yaml uses the following code to connect with HAProxy and collect haproxy metrics:

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/
      collect_aggregates_only: True
    log_errors: false

You can get a few additional status metrics by editing the configuration in dragent.yaml,as in the following examples.

Example: Collect Status Metrics Per Service

Enable the collect_status_metrics flag to collect the metrics haproxy.count_per_status, and haproxy.backend_hosts.

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/haproxy_stats
      collect_aggregates_only: True
      collect_status_metrics: True
    log_errors: false

Example: Collect Status Metrics Per Host

Enable:

• collect_status_metrics_by_host: Instructs the check to collect status metrics per host, instead of per service. This only applies if `collect_status_metrics` is true.

• tag_service_check_by_host: When this flag is set, the hostname is also passed with the service check ‘haproxy.backend_up’.

  By default, only the backend name and service name are associated with it.

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/haproxy_stats
      collect_aggregates_only: True
      collect_status_metrics: True
      collect_status_metrics_by_host: True
      tag_service_check_by_host: True
    log_errors: false

Example: Collect HAProxy Stats by UNIX Socket

If you’ve configured HAProxy to report statistics to a UNIX socket, you can set the url in dragent.yaml to the socket’s path (e.g., unix:///var/run/haproxy.sock).

Set up HAProxy Config File

Edit your HAProxy configuration file ( /etc/haproxy/haproxy.cfg ) to add the following lines to the global section:

global
    [snip]
       stats socket /run/haproxy/admin.sock mode 660 level admin
       stats timeout 30s
    [snip]

Edit dragent.yaml url

Add the socket URL from the HAProxy config to the dragent.yaml file:

app_checks:
      - name: haproxy
        pattern:
          comm: haproxy
        conf:
          url: unix:///run/haproxy/admin.sock
        log_errors: True

Metrics Available

See HAProxy Metrics.

Example: Enable Service Check

Required: Agent 9.6.0+

enable_service_check: Enable/Disable service check haproxy.backend.up.

When set to false , all service checks will be disabled.

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/haproxy_stats
      collect_aggregates_only: true
      enable_service_check: false

Example: Filter Metrics Per Service

Required: Agent 9.6.0+

services_exclude (Optional): Name or regex of services to be excluded.

services_include (Optional): Name or regex of services to be included

If a service is excluded with services_exclude, it can still be be included explicitly by services_include. The following example excludes all services except service_1 and service_2.

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/haproxy_stats
      collect_aggregates_only: true
      services_exclude:
        - ".*"
      services_include:
        - "service_1"
        - "service_2"

Additional Options: active_tag, headers

Required: Agent 9.6.0+

There are two additional configuration options introduced with agent 9.6.0:

• active_tag (Optional. Default: false):

  Adds tag active to backend metrics that belong to the active pool of connections.

• headers (Optional):

  Extra headers such as auth-token can be passed along with requests.

app_checks:
  - name: haproxy
    pattern:
      comm: haproxy
      port: 1936
    conf:
      username: stats
      password: stats
      url: http://localhost:1936/haproxy_stats
      collect_aggregates_only: true
      active_tag: true
      headers:
        <HEADER_NAME>: <HEADER_VALUE>
        <HEADER_NAME>: <HEADER_VALUE>