Example Configuration

This topic introduces you to default and specific Prometheus configurations.

Default Configuration

As an example that pulls together many of the configuration elements shown above, consider the default Agent configuration that's inherited from the dragent.default.yaml.

prometheus:
  enabled: true
  interval: 10
  log_errors: true
  max_metrics: 1000
  max_metrics_per_process: 100
  max_tags_per_metric: 20
 
  # Filtering processes to scan. Processes not matching a rule will not
  # be scanned
  # If an include rule doesn't contain a port or port_filter in the conf
  # section, we will scan all the ports that a matching process is listening to.
  process_filter:
    - exclude:
        process.name: docker-proxy
    - exclude:
        container.image: sysdig/agent
    # special rule to exclude processes matching configured prometheus appcheck
    - exclude:
        appcheck.match: prometheus
    - include:
        container.label.io.prometheus.scrape: "true"
        conf:
            # Custom path definition
            # If the Label doesn't exist we'll still use "/metrics"
            path: "{container.label.io.prometheus.path}"
 
            # Port definition
            # - If the Label exists, only scan the given port.
            # - If it doesn't, use port_filter instead.
            # - If there is no port_filter defined, skip this process
            port: "{container.label.io.prometheus.port}"
            port_filter:
                - exclude: [9092,9200,9300]
                - include: 9090-9500
                - include: [9913,9984,24231,42004]
    - exclude:
        container.label.io.prometheus.scrape: "false"
    - include:
        kubernetes.pod.annotation.prometheus.io/scrape: true
        conf:
            path: "{kubernetes.pod.annotation.prometheus.io/path}"
            port: "{kubernetes.pod.annotation.prometheus.io/port}"
    - exclude:
        kubernetes.pod.annotation.prometheus.io/scrape: false

Consider the following about this default configuration:

  • All Prometheus scraping is disabled by default. To enable the entire configuration shown here, you would only need to add the following to your dragent.yaml:

    prometheus:
  enabled: true

    Enabling this option and any pods (in case of Kubernetes) that have the right annotation set or containers (if not) that have the labels set will automatically be scrapped.

  • Once enabled, this default configuration is ideal for the use case described in the Quick Start For Kubernetes Environments.

  • A Process Filter rule excludes processes that are likely to exist in most environments but are known to never export Prometheus metrics, such as the Docker Proxy and the Agent itself.

  • Another Process Filter rule ensures that any processes configured to be scraped by the legacy Prometheus application check will not be scraped.

  • Another Process Filter rule is tailored to use container Labels. Processes marked with the container Label io.prometheus.scrape will become eligible for scraping, and if further marked with container Labels io.prometheus.port and/or io.prometheus.path, scraping will be attempted only on this port and/or endpoint. If the container is not marked with the specified path Label, scraping the /metrics endpoint will be attempted. If the container is not marked with the specified port Label, any listening ports in the port_filter will be attempted for scraping (this port_filter in the default is set for the range of ports for common Prometheus exporters, with exclusions for ports in the range that are known to be used by other applications that are not exporters).

  • The final Process Filter Include rule is tailored to the use case described in the Quick Start For Kubernetes Environments.

Scrape a Single Custom Process

If you need to scrape a single custom process, for instance, a java process listening on port 9000 with path /prometheus, add the following to the dragent.yaml:

prometheus:
  enabled: true
  process_filter:
    - include:
        process.name: java
        port: 9000
        conf:
          # ensure we only scrape port 9000 as opposed to all ports this process may be listening to
          port: 9000
          path: "/prometheus"

This configuration overrides the default process_filter section shown in Default Configuration. You can add relevant rules from the default configuration to this to further filter down the metrics.

Note

port has different purposes depending on where it's placed in the configuration. When placed under the include section, it is a condition for matching the include rule.

Placing a port under conf indicates that only that particular port is scraped when the rule is matched as opposed to all the ports that the process could be listening on.

In this example, the first rule will be matched for the Java process listening on port 9000. The java process listening only on port 9000 will be scrapped.

Scrape a Single Custom Process Based on Container Labels

If you still want to scrape based on container labels, you could just append the relevant rules from the defaults to the process_filter. For example:

prometheus:
  enabled: true
  process_filter:
    - include:
        process.name: java
        port: 9000
        conf:
          # ensure we only scrape port 9000 as opposed to all ports this process may be listening to
          port: 9000
          path: "/prometheus"
    - exclude:
        process.name: docker-proxy
    - include:
        container.label.io.prometheus.scrape: "true"
        conf:
            path: "{container.label.io.prometheus.path}"
            port: "{container.label.io.prometheus.port}"

Note

port has a different meaning depending on where it's placed in the configuration. When placed under the include section, it's a condition for matching the include rule.

Placing port under conf indicates that only that port is scraped when the rule is matched as opposed to all the ports that the process could be listening on.

In this example, the first rule will be matched for the process listening on port 9000. The java process listening only on port 9000 will be scrapped.

Container Environment

With this default configuration enabled, a containerized install of our example exporter shown below would be automatically scraped via the Agent.

# docker run -d -p 8080:8080 \
    --label io.prometheus.scrape="true" \
    --label io.prometheus.port="8080" \
    --label io.prometheus.path="/prometheus" \
    luca3m/prometheus-java-app

Kubernetes Environment

In a Kubernetes-based environment, a Deployment with the Annotations as shown in this example YAML would be scraped by enabling the default configuration.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus-java-app
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus-java-app
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/path: "/prometheus"
        prometheus.io/port: "8080"
    spec:
      containers:
        - name: prometheus-java-app
          image: luca3m/prometheus-java-app
          imagePullPolicy: Always

Non-Containerized Environment

This is an example of a non-containerized environment or a containerized environment that doesn't use Labels or Annotations. The following dragent.yaml would override the default and do per-second scrapes of our sample exporter and also a second exporter on port 5005, each at their respective non-standard endpoints. This can be thought of as a conservative "whitelist" type of configuration since it restricts scraping to only exporters that are known to exist in the environment and the ports on which they're known to export Prometheus metrics.

prometheus:
  enabled: true
  interval: 1
  process_filter:
    - include:
        process.cmdline: "*app.jar*"
        conf:
          port: 8080
          path: "/prometheus"
    - include:
        port: 5005
        conf:
          port: 5005
          path: "/wacko"

Note

port has a different meaning depending on where it's placed in the configuration. When placed under the include section, it's a condition for matching the include rule. Placing port under conf indicates that only that port is scraped when the rule is matched as opposed to all the ports that the process could be listening on.

In this example, the first rule will be matched for the process *app.jar*. The java process listening only on port 8080 will be scrapped as opposed to all the ports that *app.jar* could be listening on. The second rule will be matched for port 5005 and the process listening only on 5005 will be scraped.