This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

    Apache Kafka

    Apache Kafka is a distributed streaming platform. Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies. If Kafka is installed on your environment, the Sysdig agent will automatically connect. See the Default Configuration, below.

    The Sysdig agent automatically collects metrics from Kafka via JMX polling. You need to provide consumer names and topics in the agent config file to collect consumer-based Kafka metrics.

    This page describes the default configuration settings, how to edit the configuration to collect additional information, the metrics available for integration, and a sample result in the Sysdig Monitor UI.

    Kafka Setup

    Kafka will automatically expose all metrics. You do not need to add anything on the Kafka instance.

    Zstandard, one of the compressions available in the Kafka integration, is only included in Kafka versions 2.1.0 or newer. See also Apache documentation.

    Sysdig Agent Configuration

    Review how to Edit dragent.yaml to Integrate or Modify Application Checks.

    Metrics from Kafka via JMX polling are already configured in the agent’s default-settings configuration file. Metrics for consumers, however, need to use app-checks to poll the Kafka and Zookeeper API. You need to provide consumer names and topics in dragent.yaml file.

    Default Configuration

    Since consumer names and topics are environment-specific, a default configuration is not present in dragent.default.yaml.

    Refer to the following examples for adding Kafka checks to dragent.yaml.

    Remember! Never edit dragent.default.yaml directly; always edit only dragent.yaml.

    Example 1: Basic Configuration

    A basic example with sample consumer and topic names:

    app_checks:
      - name: kafka
        check_module: kafka_consumer
        pattern:
          comm: java
          arg: kafka.Kafka
        conf:
          kafka_connect_str: "127.0.0.1:9092" # kafka address, usually localhost as we run the check on the same instance
          zk_connect_str: "localhost:2181" # zookeeper address, may be different than localhost
          zk_prefix: /
          consumer_groups:
            sample-consumer-1: # sample consumer name
              sample-topic-1: [0, ] # sample topic name and partitions
            sample-consumer-2: # sample consumer name
              sample-topic-2: [0, 1, 2, 3] # sample topic name and partitions
    

    Example 2: Store Consumer Group Info (Kafka 9+)

    From Kafka 9 onwards, you can store consumer group config info inside Kafka itself for better performance.

    app_checks:
      - name: kafka
        check_module: kafka_consumer
        pattern:
          comm: java
          arg: kafka.Kafka
        conf:
          kafka_connect_str: "localhost:9092"
          zk_connect_str: "localhost:2181"
          zk_prefix: /
          kafka_consumer_offsets: true
          consumer_groups:
            sample-consumer-1: # sample consumer name
              sample-topic-1: [0, ] # sample topic name and partitions
    

    If kafka_consumer_offsets entry is set to true the app check will look for consumer offsets in Kafka. The appcheck will also look in Kafka if zk_connect_str is not set.

    Example 3: Aggregate Partitions at the Topic Level

    To enable aggregation of partitions at the topic level, use kafka_consumer_topics with aggregate_partitions = true.

    In this case the app check will aggregate the lag & offset values at the partition level, reducing the number of metrics collected.

    Set aggregate_partitions = false to disable metrics aggregation at the partition level. In this case, the appcheck will show lag and offset values for each partition.

    app_checks:
      - name: kafka
        check_module: kafka_consumer
        pattern:
          comm: java
          arg: kafka.Kafka
        conf:
          kafka_connect_str: "localhost:9092"
          zk_connect_str: "localhost:2181"
          zk_prefix: /
          kafka_consumer_offsets: true
          kafka_consumer_topics:
            aggregate_partitions: true
          consumer_groups:
            sample-consumer-1: # sample consumer name
              sample-topic-1: [0, ] # sample topic name and partitions
            sample-consumer-2: # sample consumer name
              sample-topic-2: [0, 1, 2, 3] # sample topic name and partitions
    

    Example 4: Custom Tags

    Optional tags can be applied to every emitted metric, service check, and/or event.

    app_checks:
      - name: kafka
        check_module: kafka_consumer
        pattern:
          comm: java
          arg: kafka.Kafka
        conf:
          kafka_connect_str: "localhost:9092"
          zk_connect_str: "localhost:2181"
          zk_prefix: /
          consumer_groups:
            sample-consumer-1: # sample consumer name
              sample-topic-1: [0, ] # sample topic name and partitions
        tags:  ["key_first_tag:value_1", "key_second_tag:value_2", "key_third_tag:value_3"]
    

    Example 5: SSL and Authentication

    If SSL and authentication are enabled on Kafka, use the following configuration.

    app_checks:
      - name: kafka
        check_module: kafka_consumer
        pattern:
          comm: java
          arg: kafka.Kafka
        conf:
          kafka_consumer_offsets: true
          kafka_connect_str: "127.0.0.1:9093"
          zk_connect_str: "localhost:2181"
          zk_prefix: /
          consumer_groups:
            test-group:
              test: [0, ]
              test-4: [0, 1, 2, 3]
          security_protocol: SASL_SSL
          sasl_mechanism: PLAIN
          sasl_plain_username: <USERNAME>
          sasl_plain_password: <PASSWORD>
          ssl_check_hostname: true
          ssl_cafile:  <SSL_CA_FILE_PATH>
          #ssl_context: <SSL_CONTEXT>
          #ssl_certfile: <CERT_FILE_PATH>
          #ssl_keyfile: <KEY_FILE_PATH>
          #ssl_password: <PASSWORD>
          #ssl_crlfile: <SSL_FILE_PATH>
    

    Configuration Keywords and Descriptions

    Keyword

    Description

    Default Value

    security_protocol (str)

    Protocol used to communicate with brokers.

    PLAINTEXT

    sasl_mechanism (str)

    String picking SASL mechanism when security_protocol is SASL_PLAINTEXT or SASL_SSL

    Currently only PLAIN is supported

    sasl_plain_username (str) 

    Username for SASL PLAIN authentication.

    sasl_plain_password (str) 

    Password for SASL PLAIN authentication.

    ssl_context (ssl.SSLContext) 

    Pre-configured SSLContext for wrapping socket connections. If provided, all other ssl_* configurations will be ignored.

    none

    ssl_check_hostname (bool)

    Flag to configure whether SSL handshake should verify that the certificate matches the broker's hostname.

    true

    ssl_cafile (str)

    Optional filename of ca file to use in certificate veriication.

    none

    ssl_certfile (str)

    Optional filename of file in pem format containing the client certificate, as well as any CA certificates needed to establish the certificate's authenticity.

    none

    ssl_keyfile (str)

    Optional filename containing the client private key.

    none

    ssl_password (str) 

    Optional password to be used when loading the certificate chain.

    none

    ssl_crlfile (str)

    Optional filename containing the CRL to check for certificate expiration. By default, no CRL check is done.

    When providing a file, only the leaf certificate will be checked against this CRL. The CRL can only be checked with 2.7.9+.

    none

    Example 6: Regex for Consumer Groups and Topics

    As of Sysdig agent version 0.94, the Kafka app check has added optional regex (regular expression) support for Kafka consumer groups and topics.

    Regex Configuration:

    • No new metrics are added with this feature

    • The new parameter consumer_groups_regex is added, which includes regex for consumers and topics from Kafka. Consumer offsets stored in Zookeeper are not collected.

    • Regex for topics is optional. When not provided, all topics under the consumer will be reported.

    • The regex Python syntax is documented here: https://docs.python.org/3.7/library/re.html#regular-expression-syntax

    • If both consumer_groups and consumer_groups_regex are provided at the same time, matched consumer groups from both parameters will be merged

    Sample configuration:

    app_checks:
      - name: kafka
        check_module: kafka_consumer
        pattern:
          comm: java
          arg: kafka.Kafka
        conf:
          kafka_connect_str: "localhost:9092"
          zk_connect_str: "localhost:2181"
          zk_prefix: /
          kafka_consumer_offsets: true
          # Regex can be provided in following format
          # consumer_groups_regex:
          #   'REGEX_1_FOR_CONSUMER_GROUPS':
          #      - 'REGEX_1_FOR_TOPIC'
          #      - 'REGEX_2_FOR_TOPIC'
          consumer_groups_regex:
            'consumer*':
              - 'topic'
              - '^topic.*'
              - '.*topic$'
              - '^topic.*'
              - 'topic\d+'
              - '^topic_\w+'
    

    Example

    Regex

    Description

    Examples Matched

    Examples NOT Matched

    topic_\d+

    All strings having keyword topic followed by _ and one or more digit characters (equal to [0-9])

    my-topic_1

    topic_23

    topic_5-dev

    topic_x

    my-topic-1

    topic-123

    topic

    All strings having topic keyword

    topic_x

    x_topic123

    xyz

    consumer*

    All strings have consumer keyword

    consumer-1

    sample-consumer

    sample-consumer-2

    xyz

    ^topic_\w+

    All strings starting with topic followed by _ and any one or more word characters (equal to [a-zA-Z0-9_])

    topic_12

    topic_x

    topic_xyz_123

    topic-12

    x_topic

    topic__xyz

    ^topic.*

    All strings starting with topic

    topic-x

    topic123

    x-topic

    x_topic123

    .*topic$

    All strings ending with topic

    x_topic

    sampletopic

    topic-1

    x_topic123

    Metrics Available

    Kafka Consumer Metrics (App Checks)

    See Apache Kafka Consumer Metrics.

    JMX Metrics

    See Apache Kafka JMX Metrics.

    Result in the Monitor UI