Example Configuration
This topic introduces you to default and specific Prometheus configurations.
Default Configuration
As an example that pulls together many of the configuration elements shown above, consider the default Agent configuration that's inherited from the dragent.default.yaml
.
prometheus: enabled: true interval: 10 log_errors: true max_metrics: 1000 max_metrics_per_process: 100 max_tags_per_metric: 20 # Filtering processes to scan. Processes not matching a rule will not # be scanned # If an include rule doesn't contain a port or port_filter in the conf # section, we will scan all the ports that a matching process is listening to. process_filter: - exclude: process.name: docker-proxy - exclude: container.image: sysdig/agent # special rule to exclude processes matching configured prometheus appcheck - exclude: appcheck.match: prometheus - include: container.label.io.prometheus.scrape: "true" conf: # Custom path definition # If the Label doesn't exist we'll still use "/metrics" path: "{container.label.io.prometheus.path}" # Port definition # - If the Label exists, only scan the given port. # - If it doesn't, use port_filter instead. # - If there is no port_filter defined, skip this process port: "{container.label.io.prometheus.port}" port_filter: - exclude: [9092,9200,9300] - include: 9090-9500 - include: [9913,9984,24231,42004] - exclude: container.label.io.prometheus.scrape: "false" - include: kubernetes.pod.annotation.prometheus.io/scrape: true conf: path: "{kubernetes.pod.annotation.prometheus.io/path}" port: "{kubernetes.pod.annotation.prometheus.io/port}" - exclude: kubernetes.pod.annotation.prometheus.io/scrape: false
Consider the following about this default configuration:
All Prometheus scraping is disabled by default. To enable the entire configuration shown here, you would only need to add the following to your
dragent.yaml
:prometheus: enabled: true
Enabling this option and any pods (in case of Kubernetes) that have the right annotation set or containers (if not) that have the labels set will automatically be scrapped.
Once enabled, this default configuration is ideal for the use case described in the Quick Start For Kubernetes Environments.
A Process Filter rule excludes processes that are likely to exist in most environments but are known to never export Prometheus metrics, such as the Docker Proxy and the Agent itself.
Another Process Filter rule ensures that any processes configured to be scraped by the legacy Prometheus application check will not be scraped.
Another Process Filter rule is tailored to use container Labels. Processes marked with the container Label
io.prometheus.scrape
will become eligible for scraping, and if further marked with container Labelsio.prometheus.port
and/orio.prometheus.path
, scraping will be attempted only on this port and/or endpoint. If the container is not marked with the specified path Label, scraping the/metrics
endpoint will be attempted. If the container is not marked with the specified port Label, any listening ports in theport_filter
will be attempted for scraping (thisport_filter
in the default is set for the range of ports for common Prometheus exporters, with exclusions for ports in the range that are known to be used by other applications that are not exporters).The final Process Filter Include rule is tailored to the use case described in the Quick Start For Kubernetes Environments.
Scrape a Single Custom Process
If you need to scrape a single custom process, for instance, a java process listening on port 9000 with path /prometheus
, add the following to the dragent.yaml
:
prometheus: enabled: true process_filter: - include: process.name: java port: 9000 conf: # ensure we only scrape port 9000 as opposed to all ports this process may be listening to port: 9000 path: "/prometheus"
This configuration overrides the default process_filter
section shown in Default Configuration. You can add relevant rules from the default configuration to this to further filter down the metrics.
Note
port
has different purposes depending on where it's placed in the configuration. When placed under the include
section, it is a condition for matching the include rule.
Placing a port
under conf
indicates that only that particular port is scraped when the rule is matched as opposed to all the ports that the process could be listening on.
In this example, the first rule will be matched for the Java process listening on port 9000. The java process listening only on port 9000 will be scrapped.
Scrape a Single Custom Process Based on Container Labels
If you still want to scrape based on container labels, you could just append the relevant rules from the defaults to the process_filter
. For example:
prometheus: enabled: true process_filter: - include: process.name: java port: 9000 conf: # ensure we only scrape port 9000 as opposed to all ports this process may be listening to port: 9000 path: "/prometheus" - exclude: process.name: docker-proxy - include: container.label.io.prometheus.scrape: "true" conf: path: "{container.label.io.prometheus.path}" port: "{container.label.io.prometheus.port}"
Note
port
has a different meaning depending on where it's placed in the configuration. When placed under the include
section, it's a condition for matching the include rule.
Placing port
under conf
indicates that only that port is scraped when the rule is matched as opposed to all the ports that the process could be listening on.
In this example, the first rule will be matched for the process listening on port 9000. The java process listening only on port 9000 will be scrapped.
Container Environment
With this default configuration enabled, a containerized install of our example exporter shown below would be automatically scraped via the Agent.
# docker run -d -p 8080:8080 \ --label io.prometheus.scrape="true" \ --label io.prometheus.port="8080" \ --label io.prometheus.path="/prometheus" \ luca3m/prometheus-java-app
Kubernetes Environment
In a Kubernetes-based environment, a Deployment with the Annotations as shown in this example YAML would be scraped by enabling the default configuration.
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: prometheus-java-app spec: replicas: 1 template: metadata: labels: app: prometheus-java-app annotations: prometheus.io/scrape: "true" prometheus.io/path: "/prometheus" prometheus.io/port: "8080" spec: containers: - name: prometheus-java-app image: luca3m/prometheus-java-app imagePullPolicy: Always
Non-Containerized Environment
This is an example of a non-containerized environment or a containerized environment that doesn't use Labels or Annotations. The following dragent.yaml
would override the default and do per-second scrapes of our sample exporter and also a second exporter on port 5005, each at their respective non-standard endpoints. This can be thought of as a conservative "whitelist" type of configuration since it restricts scraping to only exporters that are known to exist in the environment and the ports on which they're known to export Prometheus metrics.
prometheus: enabled: true interval: 1 process_filter: - include: process.cmdline: "*app.jar*" conf: port: 8080 path: "/prometheus" - include: port: 5005 conf: port: 5005 path: "/wacko"
Note
port
has a different meaning depending on where it's placed in the configuration. When placed under the include
section, it's a condition for matching the include rule. Placing port
under conf
indicates that only that port is scraped when the rule is matched as opposed to all the ports that the process could be listening on.
In this example, the first rule will be matched for the process *app.jar*. The java process listening only on port 8080 will be scrapped as opposed to all the ports that *app.jar* could be listening on. The second rule will be matched for port 5005 and the process listening only on 5005 will be scraped.