This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

  • 1:
    • 2:
      • 2.1:
        • 2.2:
          • 2.3:
          • 3:
            • 4:
              • 4.1:
                • 4.2:
                  • 4.3:
                    • 4.4:
                      • 4.4.1:
                      • 4.5:
                        • 4.6:
                          • 4.7:
                          • 5:
                            • 6:
                              • 7:
                                • 8:
                                  • 8.1:
                                    • 8.2:
                                      • 8.3:
                                      • 9:
                                        • 10:

                                          Agent Configuration

                                          Out of the box, the Sysdig agent will gather and report on a wide variety of pre-defined metrics. It can also accommodate any number of custom parameters for additional metrics collection.

                                          Use this section when you need to change the default or pre-defined settings by editing the agent configuration files, or for other special circumstances.

                                          Integrations for Sysdig Monitor also require editing the agent config files.

                                          By default, the Sysdig agent is configured to collect metric data from a range of platforms and applications. You can edit the agent config files to extend the default behavior, including additional metrics for JMX, StatsD, Prometheus, or a wide range of other applications. You can also monitor log files for targeted text strings.

                                          1 -

                                          Understand the Agent Configuration

                                          Out of the box, the Sysdig agent will gather and report on a wide variety of pre-defined metrics. It can also accommodate any number of custom parameters for additional metrics collection.

                                          The agent relies on a pair of configuration files to define metrics collection parameters:

                                          dragent.default.yaml

                                          The core configuration file. You can look at it to understand more about the default configurations provided.

                                          Location: /opt/draios/etc/dragent.default.yaml.

                                          CAUTION. This file should never be edited.

                                          dragent.yaml or configmap.yaml (Kubernetes)

                                          The configuration file where parameters can be added, either directly in YAML as name/value pairs, or using environment variables such as ADDITIONAL_CONFLocation: /opt/draios/etc/dragent.yaml.

                                          The dragent.yaml file can be accessed and edited in several ways, depending on how the agent was installed. This document describes how to modify dragent.yaml.

                                          One additional file, dragent.auto.yaml is also created and used in special circumstances. See Optional: Agent Auto-Config for more detail.

                                          Access and Edit the Configuration File

                                          There are various ways to add or edit parameters indragent.yaml.

                                          Option 1: With dragent.yaml (for testing)

                                          It is possible to edit the container’s file directly on the host.

                                          Add parameters directly in YAML.

                                          1. Access dragent.yamldirectly at"/opt/draios/etc/dragent.yaml."

                                          2. Edit the file. Use proper YAML syntax.

                                            See the examples at the bottom of the page.

                                          3. Restart the agent for changes to take effect

                                          • Native agent: service dragent restart

                                          • Container agent: docker restart sysdig-agent

                                          Option 2: With configmap.yaml(Kubernetes)

                                          Configmap.yaml is the configuration file where parameters can be added, either directly in YAML as name/value pairs, or using environment variables such as ‘ADDTIONAL_CONF."

                                          If you install agents as DaemonSets on a system running Kubernetes, you use configmap.yaml to connect with and manipulate the underlyingdragent.yamlfile.

                                          See also: Agent Install: Kubernetes | GKE | OpenShift | IBM

                                          Add parameters directly in YAML.

                                          Edit the files locally and apply with the changes withkubectl -f.

                                          1. Access theconfigmap.yaml.

                                          2. Edit the file as needed.

                                          3. Apply the changes:

                                            kubectl apply -f sysdig-agent-configmap.yaml

                                          Running agents will automatically pick the new configuration after Kubernetes pushes the changes across all the nodes in the cluster.

                                          Option 3: With Docker Run (Docker)

                                          Add -e ADDITIONAL_CONF="<VARIABLES>" to a Docker run command, where <VARIABLES> contains all the customized parameters you want to include, in a single-line format.

                                          Convert YAML Parameters to Single-Line Format

                                          To insert ADDITIONAL_CONF parameters in a Docker run command or a daemonset file, you must convert the YAML code into a single-line format.

                                          You can do the conversion manually for short snippets. To convert longer portions of YAML, use echo|sed commands.

                                          In earlier versions, the Sysdig Agent connected to port 6666. This behavior has been deprecated, as the Sysdig agent now connects to port 6443.

                                          The basic procedure:

                                          1. Write your configuration in YAML, as it would be entered directly in dragent.yaml.

                                          2. In a bash shell, use echo and sed to convert to a single line.

                                            sed script: echo "" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'

                                          3. Insert the resulting line into a Docker run command or add it to the daemonset file as an ADDITIONAL_CONF.

                                          Example: Simple

                                          Insert parameters to turn off StatsD collection and blacklist port 6443.

                                          Sysdig agent uses port 6443 for both inbound and outbound communication with the Sysdig backend. The agent initiates a request and keeps a connection open with the Sysdig backend for the backend to push configurations, Falco rules, policies, and so on. Ensure that you allow the agents’ inbound and outbound communication on TCP 6443 from the respective IPs associated with your SaaS Regions. Note that you are allowing the agent to send communication outbound on TCP 6443 to the inbound IP ranges listed in the SaaS Regions.

                                          YAML format

                                          statsd:
                                              enabled: false
                                              blacklisted_ports:
                                              - 6443
                                          

                                          Single-line format (manual)

                                          Use spaces, hyphens, and \n correctly when manually converting to a single line:

                                          ADDITIONAL_CONF="statsd:\n enabled: false\n blacklisted_ports:\n - 6443"

                                          Here the single line is incorporated into a full agent startup Docker command.

                                          docker run
                                            --name sysdig-agent \
                                            --privileged \
                                            --net host \
                                            --pid host \
                                            -e ACCESS_KEY=1234-your-key-here-1234 \
                                            -e TAGS=dept:sales,local:NYC \
                                            -e ADDITIONAL_CONF="statsd:\n    enabled: false\n    blacklisted_ports:\n    - 6443" \
                                            -v /var/run/docker.sock:/host/var/run/docker.sock \
                                            -v /dev:/host/dev \
                                            -v /proc:/host/proc:ro \
                                            -v /boot:/host/boot:ro \
                                            -v /lib/modules:/host/lib/modules:ro \
                                            -v /usr:/host/usr:ro \
                                          quay.io/sysdig/agent
                                          
                                          Example: Complex

                                          Insert parameters to override the default configuration for a RabbitMQ app check.

                                          YAML format

                                          app_checks:
                                            - name: rabbitmq
                                              pattern:
                                                port: 15672
                                              conf:
                                                rabbitmq_api_url: "http://localhost:15672/api/"
                                                rabbitmq_user: myuser
                                                rabbitmq_pass: mypassword
                                                queues:
                                                  - MyQueue1
                                                  - MyQueue2
                                          

                                          Single-line format (echo |sed)

                                          From a bash shell, issue the echo command and sed script.

                                          echo "app_checks:
                                            - name: rabbitmq
                                              pattern:
                                                port: 15672
                                              conf:
                                                rabbitmq_api_url: "http://localhost:15672/api/"
                                                rabbitmq_user: myuser
                                                rabbitmq_pass: mypassword
                                                queues:
                                                  - MyQueue1
                                                  - MyQueue2
                                          " | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'
                                          

                                          This results in the single-line format to be used with ADDITIONAL_CONF in a Docker command or daemonset file.

                                          "app_checks:\n - name: rabbitmq\n  pattern:\n    port: 15672\n  conf:\n    rabbitmq_api_url: http://localhost:15672/api/\n    rabbitmq_user: myuser\n    rabbitmq_pass: mypassword\n    queues:\n      - MyQueue1\n      - MyQueue2\n"
                                          

                                          Option 4: With HELM Format

                                          If you installed the Sysdig agent in Kubernetes using a Helm chart, then no configmap.yaml file was downloaded. You edit dragent.yaml using Helm syntax:

                                          Example

                                          $ helm install
                                              --name sysdig-agent
                                              --set sysdig.settings.tags='linux:ubuntu\,dept:dev\,local:nyc'
                                              --set clusterName='my_cluster'
                                            sysdig/sysdig
                                          

                                          Will be transformed into

                                          data:
                                           dragent.yaml: |
                                            tags: linux:ubuntu,dept:dev,local:nyc
                                            k8s_cluster_name: my_cluster
                                          

                                          Table 1: Environment Variables for Agent Config File

                                          Name

                                          Value

                                          Description

                                          ACCESS_KEY

                                          <your Sysdig access key>

                                          Required

                                          TAGS

                                          <meaningful tags you want applied to your instances>

                                          Optional. These are displayed in Sysdig Monitor for ease of use.

                                          For example:

                                          tags: linux:ubuntu,dept:dev,local:nyc

                                          See sysdig-agent-configmap.yaml.

                                          COLLECTOR

                                          <collector-hostname.com> or 111.222.333.400

                                          Enter the host name or IP address of the Sysdig collector service. Note that when used within dragent.yaml, must be lowercase collector.

                                          For SaaS regions, see: SaaS Regions and IP Ranges.

                                          COLLECTOR_PORT

                                          6443

                                          On-prem only. The port used by the Sysdig collector service; default 6443.

                                          SECURE

                                          "true"

                                          On-prem only. If using SSL/TLS to connect to collector service value = "true" otherwise "false."

                                          CHECK_CERTIFICATE

                                          "false"

                                          On-prem only. Set to "true" when using SSL/TLS to connect to the collector service and should check for valid SSL/TLS certificate.

                                          ADDITIONAL_CONF

                                          Optional. A place to provide custom configuration values to the agent as environment variables .

                                          SYSDIG_PROBE_URL

                                          Optional. An alternative URL to download precompiled kernel module.

                                          Sample Docker Command Using Variables

                                          docker run \
                                            --name sysdig-agent \
                                            --privileged \
                                            --net host \
                                            --pid host \
                                            -e ACCESS_KEY=3e762f9a-3936-4c60-9cf4-c67e7ce5793b \
                                            -e COLLECTOR=mycollector.elb.us-west-1.amazonaws.com \
                                            -e COLLECTOR_PORT=6443 \
                                            -e CHECK_CERTIFICATE=false \
                                            -e TAGS=my_tag:some_value \
                                            -e ADDITIONAL_CONF="log:\n file_priority: debug\n console_priority: error" \
                                            -v /var/run/docker.sock:/host/var/run/docker.sock \
                                            -v /dev:/host/dev \
                                            -v /proc:/host/proc:ro \
                                            -v /boot:/host/boot:ro \
                                            -v /lib/modules:/host/lib/modules:ro \
                                            -v /usr:/host/usr:ro \
                                            --shm-size=350m \
                                          quay.io/sysdig/agent
                                          

                                          2 -

                                          Configure Agent Modes

                                          Agent modes provide the ability to control metric collection to fit your scale and specific requirement. You can choose one of the following modes to do so:

                                          • Monitor

                                          • Monitor Light

                                          • Troubleshooting

                                          • Secure

                                          • Custom Metrics Only

                                          Using a stripped-down mode limits collection of unneeded metrics, which in turn prevents the consumption of excess resources and helps reduce expenses.

                                          Monitor

                                          The Monitor mode offers an extensive collection of metrics. We recommend this mode to monitor enterprise environments.

                                          monitor is the default mode if you are running the Enterprise tier. To switch back to the Monitor mode from a different mode, do one of the following:

                                          • Add the following to the dragent.yaml file and restart the agent:

                                            feature:
                                              mode: monitor
                                            
                                          • Remove the parameter related to the existing mode from the dragent.yaml file and restart the agent. For example, to switch from troubleshooting mode to monitor, delete the following lines:

                                            feature:
                                              mode: troubleshooting
                                            

                                          Monitor Light

                                          Monitor Light caters to the users that run agents in a resource-restrictive environment, or to those who are interested only in a limited set of metrics.

                                          Monitor Light provides CPU, Memory, File, File system, and Network metrics. For more information, see Metrics Available in Monitor Light.

                                          Enable Monitor Light Mode

                                          To switch to the Monitor Light mode, edit the dragent.yaml file:

                                          1. Open the dragent.yaml file.

                                          2. Add the following configuration parameter:

                                            feature:
                                              mode: monitor_light
                                            
                                          3. Restart the agent.

                                          Troubleshooting

                                          Troubleshooting mode offers sophisticated metrics with detailed diagnostic capabilities. Some of these metrics are heuristic in nature.

                                          In addition to the extensive metrics available in the Monitor mode, Troubleshooting mode provides additional metrics such as net.sql and additional segmentation for file and network metrics. For more information, see Additional Metrics Values Available in Troubleshooting.

                                          Enable Troubleshooting Mode

                                          To switch to the Troubleshooting mode, edit the dragent.yaml file:

                                          1. Open the dragent.yaml file.

                                          2. Add the following configuration parameter:

                                            feature:
                                              mode: troubleshooting
                                            
                                          3. Restart the agent.

                                          Secure Mode

                                          The secure mode supports only Sysdig Secure features.

                                          Sysdig agent collects no metrics in the secure mode, which, in turn, minimizes network consumption and storage requirement in the Sysdig backend. Lower resource usage can help reduce costs and improve performance.

                                          In the Secure mode, the Monitor UI shows no data because no metrics are sent to the collector.

                                          This feature requires agent v10.5.0 or above.

                                          Enabling Secure Mode

                                          1. Open the dragent.yaml file.

                                          2. Add the following:

                                            feature:
                                              mode: secure
                                            
                                          3. Restart the agent.

                                          Custom Metrics Only Mode

                                          Custom Metrics Only mode collects the same metrics as the Monitor Light mode, but also adds the ability to collect the following:

                                          • Custom Metrics: StatsD, JMX, App Checks, and Prometheus
                                          • Kubernetes State Metrics

                                          As such, Custom Metrics Only mode is suitable if would like to use most of the features of Monitor mode but are limited in resources.

                                          This mode is not compatible with Secure. If your account is configured for Secure, you must explicitly disable Secure in the agent configuration if you wish to use this mode.

                                          This mode requires agent v12.4.0 or above.

                                          Enabling Custom Metrics Only Mode

                                          1. Open the dragent.yaml file.

                                          2. Add the following configuration parameter:

                                            feature:
                                              mode: custom-metrics-only
                                            
                                          3. If your account is enabled for Secure, add the following:

                                            security:
                                              enabled: false
                                            secure_audit_streams:
                                              enabled: false
                                            falcobaseline:
                                              enabled: false
                                            

                                            This configuration explicitly disables the Secure features in the agent. If you do not disable Secure, the agent will not start due to incompatiblity issues.

                                          4. Restart the agent.

                                          2.1 -

                                          Metrics Available in Monitor Light

                                          Monitor Light provides cpu, memory, file, file system, and network metrics.

                                          MetricsDescription
                                          cpu.cores.usedSee System.System
                                          cpu.cores.used.percent
                                          cpu.idle.percent
                                          cpu.iowait.percent
                                          cpu.nice.percent
                                          cpu.stolen.percent
                                          cpu.system.percent
                                          cpu.used.percent
                                          cpu.user.percent
                                          load.average.percpu.1m
                                          load.average.percpu.5m
                                          load.average.percpu.15m
                                          memory.bytes.available
                                          memory.bytes.total
                                          memory.bytes.used
                                          memory.bytes.used
                                          memory.bytes.virtual
                                          memory.pageFault.major
                                          memory.pageFault.minor
                                          memory.swap.bytes.available
                                          memory.swap.bytes.total
                                          memory.swap.bytes.used
                                          memory.swap.used.percent
                                          memory.used.percent
                                          file.bytes.in
                                          file.bytes.out
                                          file.bytes.total
                                          file.iops.in
                                          file.iops.out
                                          file.iops.total
                                          file.open.count
                                          file.time.in
                                          file.time.out
                                          file.time.total
                                          fs.bytes.free
                                          fs.bytes.total
                                          fs.bytes.used
                                          fs.free.percent
                                          fs.inodes.total.count
                                          fs.inodes.used.count
                                          fs.inodes.used.percent
                                          fs.largest.used.percent
                                          fs.root.used.percent
                                          fs.used.percent
                                          net.bytes.in
                                          net.bytes.out
                                          net.bytes.total
                                          proc.count
                                          thread.count
                                          container.count
                                          system.uptime
                                          uptime

                                          2.2 -

                                          Additional Metrics Values Available in Troubleshooting

                                          In addition to the extensive set of metrics available in the monitor mode, additional metrics, such as net.sql and net.mongodb, as well as additional segmentations for file and network metrics are available.

                                          MetricsAdditional Metrics Values Available When Segmented bySupported Agent Versions
                                          file.error.total.countfile.name and file.mount labelsVersion 10.1.0 or above
                                          file.bytes.total
                                          file.bytes.in
                                          file.bytes.out
                                          file.open.count
                                          file.time.total
                                          host.count
                                          host.error.count
                                          proc.count
                                          proc.start.count
                                          net.mongodb.collectionallVersion 10.2.0 or above
                                          net.mongodb.error.count
                                          net.mongodb.operation
                                          net.mongodb.request.count
                                          net.mongodb.request.time
                                          net.sql.queryall
                                          net.sql.error.count
                                          net.sql.query.type
                                          net.sql.request.count
                                          net.sql.request.time
                                          net.sql.table
                                          net.http.error.countnet.http.urlVersion 10.3.0 or above
                                          net.http.method
                                          net.http.request.count
                                          net.http.request.time
                                          net.bytes.in
                                          net.bytes.out
                                          net.request.time.worst.out
                                          net.request.count
                                          net.request.time
                                          net.bytes.total
                                          net.http.request.time.worstall

                                          2.3 -

                                          Metrics Not Available in Essentials Mode

                                          The following metrics will not be reported in the essentials mode when compared with monitor mode:

                                          MetricsSegmented By
                                          net.bytes.innet.connection.server, net.connection.direction, net.connection.l4proto , and net.connection.client labels
                                          net.bytes.out
                                          net.connection.count.total
                                          net.connection.count.in
                                          net.connection.count.out
                                          net.request.count
                                          net.request.count.in
                                          net.request.count.out
                                          net.request.time
                                          net.request.time.in
                                          net.request.time.out
                                          net.bytes.total
                                          net.mongodb.collectionall
                                          net.mongodb.error.count
                                          net.mongodb.operation
                                          net.mongodb.request.count
                                          net.mongodb.request.time
                                          net.sql.queryall
                                          net.sql.error.count
                                          net.sql.query.type
                                          net.sql.request.count
                                          net.sql.request.time
                                          net.sql.table
                                          net.sql.queryall
                                          net.sql.error.count
                                          net.sql.query.type
                                          net.sql.request.count
                                          net.sql.request.time
                                          net.sql.table
                                          net.http.method
                                          net.http.request.count
                                          net.http.request.time
                                          net.http.statusCode
                                          net.http.url

                                          3 -

                                          Enable HTTP Proxy for Agents

                                          You can configure the agent to allow it to communicate with the Sysdig collector through an HTTP proxy. HTTP proxy is usually configured to offer greater visibility and better management of the network.

                                          Agent Behaviour

                                          The agent can connect to the collector through an HTTP proxy by sending an HTTP CONNECT message and receiving a response. The proxy then initiates a TCP connection to the collector. These two connections form a tunnel that acts like one logical connection.

                                          By default, the agent will encrypt all messages sent through this tunnel. This means that after the initial CONNECT message and response, all the communication on that tunnel is encrypted by SSL end-to-end. This encryption is controlled by the top-level ssl parameter in the agent configuration.

                                          Optionally, the agent can add a second layer of encryption, securing the CONNECT message and response. This second layer of encryption may be desired in the case of HTTP authentication if there is a concern that network packet sniffing could be used to determine the user’s credentials. This second layer of encryption is enabled by setting the ssl parameter to true in the http_proxy section of the agent configuration. See Examples for details.

                                          Configuration

                                          You specify the following parameters at the same level as http_proxy in the dragent.yaml file. These existing configuration options affect the communication between the agent and collector (both with and without a proxy).

                                          • ssl: If set to false, the metrics sent from the agent to the collector are unencrypted (default is true).

                                          • ssl_verify_certificate: Determines whether the agent verifies the SSL certificate sent from the collector (default is true).

                                          The following configuration options affect the behavior of the HTTP Proxy setting. You specify them under the http_proxy heading in the dragent.yaml file.

                                          • proxy_host: Indicates the hostname of the proxy server. The default is an empty string, which implies communication through an HTTP proxy is disabled.

                                          • proxy_port: Specifies the port on the proxy server the agent should connect to. The default is 0, which indicates that the HTTP proxy is disabled.

                                          • proxy_user : Required if HTTP authentication is configured. This option specifies the username for the HTTP authentication. The default is an empty string, which indicates that authentication is not configured.

                                          • proxy_password : Required if HTTP authentication is configured. This option specifies the password for the HTTP authentication. The default is an empty string. Specifying proxy_user with no proxy_password is allowed.

                                          • ssl: If set to true, the connection between the agent and the proxy server is encrypted.

                                            Note that this parameter requires the top-level ssl parameter to be enabled, as the agent does not support SSL to the proxy but unencrypted traffic to the collector. This additional security prevents you from misconfiguring the agent assuming the metrics are as well encrypted end-to-end when they are not.

                                          • ssl_verify_certificate: Determines whether the agent will verify the certificate presented by the proxy.

                                            This option is configured independently of the top-level ssl_verify_certificate parameter. This option is enabled by default. If the provided certificate is not correct, this option can cause the connection to the proxy server to fail.

                                          • ca_certificate: The path to the CA certificate for the proxy server. If ssl_verify_certificate is enabled, the CA certificate must be signed appropriately.

                                          Examples

                                          No SSL

                                          The following example shows no SSL connection between the agent and the proxy server as well as between the proxy server and the collector.

                                          collector_port: 6667
                                          ssl: false
                                          http_proxy:
                                                  proxy_host: squid.yourdomain.com
                                                  proxy_port: 3128
                                                  ssl: false
                                          

                                          SSL Between Proxy and Collector

                                          In this example, SSL is enabled only between the proxy server and the collector.

                                          collector_port: 6443
                                          ssl: true
                                          ssl_verify_certificate: true
                                          http_proxy:
                                                  proxy_host: squid.yourdomain.com
                                                  proxy_port: 3128
                                          

                                          SSL

                                          The following example shows SSL is enabled between the agent and the proxy server as well as between the proxy server and the collector.

                                          collector_port: 6443
                                          ssl: true
                                          http_proxy:
                                                  proxy_host: squid.yourdomain.com
                                                  proxy_port: 3129
                                                  ssl: true
                                                  ssl_verify_certificate: true
                                                  ca_certificate: /usr/proxy/proxy.crt
                                          

                                          SSL with Username and Password

                                          The following configuration instructs the agent to connect to a proxy server located at squid.yourdomain.com on port 3128. The agent will request the proxy server to establish an HTTP tunnel to the Sysdig collector at collector-your.sysdigcloud.com on port 6443. The agent will authenticate with the proxy server using the given user and password combination.

                                          collector: collector-your.sysdigcloud.com
                                          collector_port: 6443
                                          http_proxy:
                                              proxy_host: squid.yourdomain.com
                                              proxy_port: 3128
                                              proxy_user: sysdig_customer
                                              proxy_password: 12345
                                              ssl: true
                                              ssl_verify_certificate: true
                                              ca_certificate: /usr/proxy/proxy_cert.crt
                                          
                                          

                                          4 -

                                          Filter Data

                                          The dragent.yaml file elements are wide-reaching. This section describes the parameters to edit in dragent.yaml to perform a range of activities:

                                          4.1 -

                                          Blacklist Ports

                                          Use the blacklisted_ports parameter in the agent configuration file to block network traffic and metrics from unnecessary network ports.

                                          Note: Port 53 (DNS) is always blacklisted.

                                          1. Access the agent configuration file, using one of the options listed.

                                          2. Add blacklisted_ports with desired port numbers.

                                            Example (YAML):

                                            blacklisted_ports:

                                            - 6443

                                            - 6379

                                          3. Restart the agent (if editing dragent.yaml file directly), using either the service dragent restart or docker restart sysdig-agent command as appropriate.

                                          4.2 -

                                          Enable/Disable Event Data

                                          Sysdig Monitor supports event integrations with certain applications by default. The Sysdig agent will automatically discover these services and begin collecting event data from them.

                                          The following applications are currently supported:

                                          • Docker

                                          • Kubernetes

                                          Other methods of ingesting custom events into Sysdig Monitor are touched upon in Custom Events.

                                          By default, only a limited set of events is collected for a supported application, and are listed in the agent’s default settings configuration file (/opt/draios/etc/dragent.default.yaml).

                                          To enable collecting other supported events, add an events entry to dragent.yaml.

                                          You can also change log entry in dragent.yaml to filter events by severity.

                                          Learn more about it in the following sections.

                                          Supported Application Events

                                          Events marked with * are enabled by default; see the dragent.default.yaml file.

                                          Docker Events

                                          The following Docker events are supported.

                                            docker:
                                              container:
                                                - attach       # Container Attached      (information)
                                                - commit       # Container Committed     (information)
                                                - copy         # Container Copied        (information)
                                                - create       # Container Created       (information)
                                                - destroy      # Container Destroyed     (warning)
                                                - die          # Container Died          (warning)
                                                - exec_create  # Container Exec Created  (information)
                                                - exec_start   # Container Exec Started  (information)
                                                - export       # Container Exported      (information)
                                                - kill         # Container Killed        (warning)*
                                                - oom          # Container Out of Memory (warning)*
                                                - pause        # Container Paused        (information)
                                                - rename       # Container Renamed       (information)
                                                - resize       # Container Resized       (information)
                                                - restart      # Container Restarted     (warning)
                                                - start        # Container Started       (information)
                                                - stop         # Container Stopped       (information)
                                                - top          # Container Top           (information)
                                                - unpause      # Container Unpaused      (information)
                                                - update       # Container Updated       (information)
                                              image:
                                                - delete # Image Deleted  (information)
                                                - import # Image Imported (information)
                                                - pull   # Image Pulled   (information)
                                                - push   # Image Pushed   (information)
                                                - tag    # Image Tagged   (information)
                                                - untag  # Image Untaged  (information)
                                              volume:
                                                - create  # Volume Created    (information)
                                                - mount   # Volume Mounted    (information)
                                                - unmount # Volume Unmounted  (information)
                                                - destroy # Volume Destroyed  (information)
                                              network:
                                                - create     # Network Created       (information)
                                                - connect    # Network Connected     (information)
                                                - disconnect # Network Disconnected  (information)
                                                - destroy    # Network Destroyed     (information)
                                          

                                          Kubernetes Events

                                          The following Kubernetes events are supported.

                                            kubernetes:
                                              node:
                                                - TerminatedAllPods       # Terminated All Pods      (information)
                                                - RegisteredNode          # Node Registered          (information)*
                                                - RemovingNode            # Removing Node            (information)*
                                                - DeletingNode            # Deleting Node            (information)*
                                                - DeletingAllPods         # Deleting All Pods        (information)
                                                - TerminatingEvictedPod   # Terminating Evicted Pod  (information)*
                                                - NodeReady               # Node Ready               (information)*
                                                - NodeNotReady            # Node not Ready           (information)*
                                                - NodeSchedulable         # Node is Schedulable      (information)*
                                                - NodeNotSchedulable      # Node is not Schedulable  (information)*
                                                - CIDRNotAvailable        # CIDR not Available       (information)*
                                                - CIDRAssignmentFailed    # CIDR Assignment Failed   (information)*
                                                - Starting                # Starting Kubelet         (information)*
                                                - KubeletSetupFailed      # Kubelet Setup Failed     (warning)*
                                                - FailedMount             # Volume Mount Failed      (warning)*
                                                - NodeSelectorMismatching # Node Selector Mismatch   (warning)*
                                                - InsufficientFreeCPU     # Insufficient Free CPU    (warning)*
                                                - InsufficientFreeMemory  # Insufficient Free Mem    (warning)*
                                                - OutOfDisk               # Out of Disk              (information)*
                                                - HostNetworkNotSupported # Host Ntw not Supported   (warning)*
                                                - NilShaper               # Undefined Shaper         (warning)*
                                                - Rebooted                # Node Rebooted            (warning)*
                                                - NodeHasSufficientDisk   # Node Has Sufficient Disk (information)*
                                                - NodeOutOfDisk           # Node Out of Disk Space   (information)*
                                                - InvalidDiskCapacity     # Invalid Disk Capacity    (warning)*
                                                - FreeDiskSpaceFailed     # Free Disk Space Failed   (warning)*
                                              pod:
                                                - Pulling           # Pulling Container Image          (information)
                                                - Pulled            # Ctr Img Pulled                   (information)
                                                - Failed            # Ctr Img Pull/Create/Start Fail   (warning)*
                                                - InspectFailed     # Ctr Img Inspect Failed           (warning)*
                                                - ErrImageNeverPull # Ctr Img NeverPull Policy Violate (warning)*
                                                - BackOff           # Back Off Ctr Start, Image Pull   (warning)
                                                - Created           # Container Created                (information)
                                                - Started           # Container Started                (information)
                                                - Killing           # Killing Container                (information)*
                                                - Unhealthy         # Container Unhealthy              (warning)
                                                - FailedSync        # Pod Sync Failed                  (warning)
                                                - FailedValidation  # Failed Pod Config Validation     (warning)
                                                - OutOfDisk         # Out of Disk                      (information)*
                                                - HostPortConflict  # Host/Port Conflict               (warning)*
                                              replicationController:
                                                - SuccessfulCreate    # Pod Created        (information)*
                                                - FailedCreate        # Pod Create Failed  (warning)*
                                                - SuccessfulDelete    # Pod Deleted        (information)*
                                                - FailedDelete        # Pod Delete Failed  (warning)*
                                          

                                          Enable/Disable Events Collection with events Parameter

                                          To customize the default events collected for a specific application (by either enabling or disabling events), add an events entry to dragent.yaml as described in the examples below.

                                          An entry in a section in dragent.yaml overrides the entire section in the default configuration.

                                          For example, the Pulling entry below will permit only kubernetes pod Pulling events to be collected and all other kubernetes pod events settings in dragent.default.yaml will be ignored.

                                          However, other kubernetes sections - node and replicationController - remain intact and will be used as specified in dragent.default.yaml.

                                          Example 1: Collect Only Certain Events

                                          Collect only ‘Pulling’ events from Kubernetes for pods:

                                          events:
                                            kubernetes:
                                              pod:
                                                 - Pulling
                                          

                                          Example 2: Disable All Events in a Section

                                          To disable all events in a section, set the event section to none:

                                          events:
                                            kubernetes: none
                                            docker: none
                                          

                                          Example 3: Combine Methods

                                          These methods can be combined. For example, disable all kubernetes node and docker image events and limit docker container events to [attach, commit, copy] (components events in other sections will be collected as specified by default):

                                          events:
                                            kubernetes:
                                              node: none
                                            docker:
                                              image: none
                                              container:
                                                - attach
                                                - commit
                                                - copy
                                          

                                          Note: Format Sequences as List or Single Line

                                          In addition to bulleted lists, sequences can also be specified in a bracketed single line, eg.:

                                          events:
                                            kubernetes:
                                              pod: [Pulling, Pulled, Failed]
                                          

                                          So, the following two settings are equivalent, permitting only Pulling, Pulled, Failed events for pods to be emitted:

                                          events:
                                            kubernetes:
                                              pod: [Pulling, Pulled, Failed]
                                          
                                          events:
                                            kubernetes:
                                              pod:
                                                - Pulling
                                                - Pulled
                                                - Failed
                                          

                                          Change Event Collection by Severity with log Parameter

                                          Events are limited globally at the agent level based on severity, using the log settings in dragent.yaml.

                                          The default setting for the events severity filter is information (only warning and higher severity events are transmitted).

                                          Valid severity levels are: none, emergency, alert, critical, error, warning, notice, information, debug.

                                          Example 1: Block Low-Severity Messages

                                          Block all low-severity messages (notice, information, debug):

                                          log:
                                            event_priority: warning
                                          

                                          Example 2: Block All Event Collection

                                          Block all event collection:

                                          log:
                                            event_priority: none
                                          

                                          For other uses of the log settings see Optional: Change the Agent Log Level.

                                          4.3 -

                                          Include/Exclude Custom Metrics

                                          For more information, see Integrate Applications (Default App Checks).

                                          It is possible to filter custom metrics in the following ways:

                                          • Ability to include/exclude custom metrics using configurable patterns,

                                          • Ability to log which custom metrics are exceeding limits

                                          After you identify those key custom metrics that must be received, use the new ‘include’ and ’exclude’ filtering parameters to make sure you receive them before the metrics limit is hit.

                                          Filter Metrics Example

                                          Here is an example configuration entry that would be put into the agent config file: (/opt/draios/etc/dragent.yaml)

                                          metrics_filter:
                                            - include: test.*
                                            - exclude: test.*
                                            - include: haproxy.backend.*
                                            - exclude: haproxy.*
                                            - exclude: redis.*
                                          

                                          Given the config entry above, this is the action for these metrics:

                                          test.* → send

                                          haproxy.backend.request → send

                                          haproxy.frontend.bytes → drop

                                          redis.keys → drop

                                          The semantic is: whenever the agent is reading metrics, they are filtered according to configured filters and the filtering rule order - the first rule that matches will be applied. Thus since the inclusion item for test.* was listed first it will be followed and that second ’exclude’ rule for the same exact metric entry will be ignored.

                                          Logging Accepted/Dropped Metrics

                                          Logging is disabled by default. You can enable logging to see which metrics are accepted or dropped by adding the following configuration entry into the dragent.yaml config file:

                                          metrics_excess_log: true
                                          

                                          When logging of excess metrics is enabled, logging occurs at INFO-level, every 30 seconds and lasts for 10 seconds. The entries that can be seen in /opt/draios/logs/draios.log will be formatted like this:

                                          +/-[type] [metric included/excluded]: metric.name (filter: +/-[metric.filter])
                                          

                                          The first ‘+’ or ‘-’, followed by ’type’ provides an easy way to quickly scan the list of metrics and spot which are included or excluded (’+’ means “included”, ‘-’ means “excluded”).

                                          The second entry specifies metric type (“statsd”, “app_check”, “service_check”, or “jmx”).

                                          A third entry spells out whether “included” or “excluded”, followed by the metric name. Finally, inside the last entry (in parentheses), there is information about filter applied and its effect (’+’ or ‘-’, meaning “include” or “exclude”).

                                          With this example filter rule set:

                                          metrics_filter:
                                            - include: mongo.statsd.net*
                                            - exclude: mongo.statsd.*
                                          

                                          We might see the following INFO-level log entries (timestamps stripped):

                                          -[statsd] metric excluded: mongo.statsd.vsize (filter: -[mongo.statsd.*])
                                          +[statsd] metric included: mongo.statsd.netIn (filter: +[mongo.statsd.net*])
                                          
                                          

                                          4.4 -

                                          Prioritize Designated Containers

                                          To get the most out of Sysdig Monitor, you may want to customize the way in which container data is prioritized and reported. Use this page to understand the default behavior and sorting rules, and to implement custom behavior when and where you need it. This can help reduce agent and backend load by not monitoring unnecessary containers, or– if encountering backend limits for containers– you can filter to ensure that the important containers are always reported.

                                          Overview

                                          By default, a Sysdig agent will collect metrics from all containers it detects in an environment. When reporting to the Monitor interface, it uses default sorting behavior to prioritize what container information to display first.

                                          Understand Default Behavior

                                          Out of the box, it chooses the containers with the highest

                                          • CPU

                                          • Memory

                                          • File IO

                                          • Net IO

                                          and allocates approximately 1/4 of the total limit to each stat type.

                                          Understand Simple Container Filtering

                                          As of agent version 0.86, it is possible set a use_container_filter parameter in the agent config file, tag/label specific containers, and set include/exclude rules to push those containers to the top of the reporting hierarchy.

                                          This is an effective sorting tool when:

                                          • You can manually mark each container with an include or exclude tag, AND

                                          • The number of includes is small (say, less than 100)

                                          In this case, the containers that explicitly match the include rules will take top priority.

                                          Understand Smart Container Reporting

                                          In some enterprises, the number of containers is too high to tag with simple filtering rules, and/or the include_all group is too large to ensure that the most-desired containers are consistently reported. As of Sysdig agent version 0.91, you can append another parameter to the agent config file, smart_container_reporting.

                                          This is an effective sorting tool when:

                                          • The number of containers is large and you can’t or won’t mark each one with include/exclude tags, AND

                                          • There are certain containers you would like to always prioritize

                                          This helps ensure that even when there are thousands of containers in an environment, the most-desired containers are consistently reported.

                                          Container filtering and smart container reporting affect the monitoring of all the processes/metrics within a container, including StatsD, JMX, app-checks, and built-in metrics.

                                          Prometheus metrics are attached to processes, rather than containers, and are therefore handled differently.

                                          The container limit is set in dragent.yaml under containers:limit:

                                          Understand Sysdig Aggregated Container

                                          The sydig_aggregated parameter is automatically activated when smart container reporting is enabled, to capture the most-desired metrics from the containers that were excluded by smart filtering and report them under a single entity. It appears like any other container in the Sysdig Monitor UI, with the name “sysdig_aggregated.

                                          Sysdig_aggregated can report on a wide array of metrics; see Sysdig_aggregated Container Metrics. However, because this is not a regular container, certain limitations apply:

                                          • container_id and container_image do not exist.

                                          • The aggregated container cannot be segmented by certain metrics that are excluded, such as process.

                                          • Some default dashboards associated with the aggregated container may have some empty graphs.

                                          Use Simple Container Filtering

                                          By default, the filtering feature is turned off. It can be enabled by adding the following line to the agent configuration:

                                          • use_container_filter: true

                                          When enabled, the agent will follow include/exclude filtering rules based on:

                                          • container image

                                          • container name

                                          • container label

                                          • Kubernetes annotation or label

                                          The default behavior in default.dragent.yaml excludes based on a container label (com.sysdig.report) and/or a Kubernetes pod annotation (.sysdig.com/report ).

                                          Container Condition Parameters and Rules

                                          Parameters

                                          The condition parameters are described in the following table:

                                          Pattern name

                                          Description

                                          Example

                                          container.image

                                          Matches if the process is running inside a container running the specified image

                                          - include:

                                          container.image: luca3m/prometheus-java-app

                                          container.name

                                          Matches if the process is running inside a container with the specified name

                                          - include:

                                          container.name: my-java-app

                                          container.label.*

                                          Matches if the process is running in a container that has a Label matching the given value

                                          - include:

                                          container.label.class: exporter

                                          kubernetes.<object>.annotation.* kubernetes.<object>.label.*

                                          Matches if the process is attached to a Kubernetes object (Pod, Namespace, etc.) that is marked with the Annotation/Label matching the given value.

                                          - include:

                                          kubernetes.pod.annotation.prometheus.io/scrape: true

                                          all

                                          Matches all. Use as last rule to determine default behavior.

                                          - include:

                                          all

                                          Rules

                                          Once enabled (when use_container_filter: true is set), the agent will follow filtering rules from the container_filter section.

                                          • Each rule is an include or exclude rule which can contain one or more conditions.

                                          • The first matching rule in the list will determine if the container is included or excluded.

                                          • The conditions consist of a key name and a value. If the given key for a container matches the value, the rule will be matched.

                                          • If a rule contains multiple conditions they all need to match for the rule to be considered a match.

                                          Default Configuraton

                                          The dragent.default.yaml contains the following default configuration for container filters:

                                          use_container_filter: false
                                          
                                          container_filter:
                                            - include:
                                                container.label.com.sysdig.report: true
                                            - exclude:
                                                container.label.com.sysdig.report: false
                                            - include:
                                                kubernetes.pod.annotation.sysdig.com/report: true
                                            - exclude:
                                                kubernetes.pod.annotation.sysdig.com/report: false
                                            - include:
                                                  all
                                          

                                          Note that it excludes via a container.label and by a kubernetes.pod.annotation.

                                          The examples on this page show how to edit in the dragent.yaml file directly. Convert the examples to Docker or Helm commands, if applicable for your situation.

                                          Enable Container Filtering in the Agent Config File

                                          Option 1: Use the Default Configuration

                                          To enable container filtering using the default configuration in default.dragent.yaml (above), follow the steps below.

                                          1. Apply Labels and/or Annotations to Designated Containers

                                          To set up, decide which containers should be excluded from automatic monitoring.

                                          Apply the container label .com.sysdig.report and/or the Kubernetes pod annotation sysdig.com/report to the designated containers.

                                          2. Edit the Agent Configuration

                                          Add the following line to dragent.yaml to turn on the default functionality:

                                          use_container_filter: true
                                          

                                          Option 2: Define Your Own Rules

                                          You can also edit dragent.yaml to apply your own container filtering rules.

                                          1. Designate Containers

                                          To set up, decide which containers should be excluded from automatic monitoring.

                                          Note the image, name, label, or Kubernetes pod information as appropriate, and build your rule set accordingly.

                                          2. Edit the Agent Configuration

                                          For example:

                                          use_container_filter: true
                                          
                                          container_filter:
                                            - include:
                                                container.name: my-app
                                            - include:
                                                container.label.com.sysdig.report: true
                                            - exclude:
                                                kubernetes.namespace.name: kube-system
                                                container.image: "gcr.io*"
                                            - include:
                                                all
                                          

                                          The above example shows a container_filter with 3 include rules and 1 exclude rule.

                                          • If the container name is “my-app” it will be included.

                                          • Likewise, if the container has a label with the key “com.sysdig.report” and with the value “true”.

                                          • If neither of those rules is true, and the container is part of a Kubernetes hierarchy within the “kube-system” namespace and the container image starts with “gcr.io”, it will be excluded.

                                          • The last rule includes all, so any containers not matching an earlier rule will be monitored and metrics for them will be sent to the backend.

                                          Use Smart Container Reporting

                                          As of Sysdig agent version 0.91, you can add another parameter to the config file: smart_container_reporting = true

                                          This enables several new prioritization checks:

                                          • container_filter (you would enable and set include/exclude rules, as described above)

                                          • container age

                                          • high stats

                                          • legacy patterns

                                          The sort is modified with the following rules in priority order:

                                          1. User-specified containers come before others

                                          2. Containers reported previously should be reported before those which have never been reported

                                          3. Containers with higher usage by each of the 4 default stats should come before those with lower usage

                                          Enable Smart Container Reporting and sysdig_aggregated

                                          1. Set up any simple container filtering rules you need, following either Option 1 or Option 2, above.

                                          2. Edit the agent configuration:

                                            smart_container_reporting: true
                                            
                                          3. This turns on both smart_container_reporting and sysdig_aggregated. The changes will be visible in the Sysdig Monitor UI.

                                            See also Sysdig_aggregated Container Metrics..

                                          Logging

                                          When the log level is set to DEBUG, the following messages may be found in the logs:

                                          messagemeaning
                                          container <id>, no filter configuredcontainer filtering is not enabled
                                          container <id>, include in reportcontainer is included
                                          container <id>, exclude in reportcontainer is excluded
                                          Not reporting thread <thread-id> in container <id>Process thread is excluded

                                          See also: Optional: Change the Agent Log Level.

                                          4.4.1 -

                                          Sysdig Aggregated Container Metrics

                                          Sysdig_aggregated containers can report on the following metrics:

                                          • tcounters

                                            • other

                                              • time_ns

                                              • time_percentage

                                              • count

                                            • io_file

                                              • time_ns_in

                                              • time_ns_out

                                              • time_ns_other

                                              • time_percentage_in

                                              • time_percentage_out

                                              • time_percentage_other

                                              • count_in

                                              • count_out

                                              • count_other

                                              • bytes_in

                                              • bytes_out

                                              • bytes_other

                                            • io_net

                                              • time_ns_in

                                              • time_ns_out

                                              • time_ns_other

                                              • time_percentage_in

                                              • time_percentage_out

                                              • time_percentage_other

                                              • count_in

                                              • count_out

                                              • count_other

                                              • bytes_in

                                              • bytes_out

                                              • bytes_other

                                            • processing

                                              • time_ns

                                              • time_percentage

                                              • count

                                          • reqcounters

                                            • other

                                              • time_ns

                                              • time_percentage

                                              • count

                                            • io_file

                                              • time_ns_in

                                              • time_ns_out

                                              • time_ns_other

                                              • time_percentage_in

                                              • time_percentage_out

                                              • time_percentage_other

                                              • count_in

                                              • count_out

                                              • count_other

                                              • bytes_in

                                              • bytes_out

                                              • bytes_other

                                            • io_net

                                              • time_ns_in

                                              • time_ns_out

                                              • time_ns_other

                                              • time_percentage_in

                                              • time_percentage_out

                                              • time_percentage_other

                                              • count_in

                                              • count_out

                                              • count_other

                                              • bytes_in

                                              • bytes_out

                                              • bytes_other

                                            • processing

                                              • time_ns

                                              • time_percentage

                                              • count

                                          • max_transaction_counters

                                            • time_ns_in

                                            • time_ns_out

                                            • count_in

                                            • count_out

                                          • resource_counters

                                            • connection_queue_usage_pct

                                            • fd_usage_pct

                                            • cpu_pct

                                            • resident_memory_usage_kb

                                            • swap_memory_usage_kb

                                            • major_pagefaults

                                            • minor_pagefaults

                                            • fd_count

                                            • cpu_shares

                                            • memory_limit_kb

                                            • swap_limit_kb

                                            • count_processes

                                            • proc_start_count

                                            • threads_count

                                          • syscall_errors

                                            • count

                                            • count_file

                                            • count_file_opened

                                            • count_net

                                          • protos

                                            • http

                                              • server_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                              • client_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                            • mysql

                                              • server_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                              • client_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                            • postgres

                                              • server_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                              • client_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                            • mongodb

                                              • server_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                              • client_totals

                                                • ncalls

                                                • time_tot

                                                • time_max

                                                • bytes_in

                                                • bytes_out

                                                • nerrors

                                          • names

                                          • transaction_counters

                                            • time_ns_in

                                            • time_ns_out

                                            • count_in

                                            • count_out

                                          4.5 -

                                          Include/Exclude Processes

                                          In addition to filtering data by container, it is also possible to filter independently by process. Broadly speaking, this refinement helps ensure that relevant data is reported while noise is reduced. More specifically, use cases for process filtering may include: 

                                          • Wanting to alert reliably whenever a given process goes down.  The total number of processes can exceed the reporting limit; when that happens, some processes are not reported. In this case, an unreported process could be misinterpreted as being “down.” Specify a filter for 30-40 processes to guarantee that they will always be reported.

                                          • Wanting to limit the number of noisy but inessential processes being reported, for example: sed, awk, grep, and similar tools that may be used infrequently.

                                          • Wanting to prioritize workload-specific processes, perhaps from integrated applications such as NGINX, Supervisord or PHP-FPM.

                                          Note that you can report on processes and containers independently; the including/excluding of one does not affect the including/excluding of the other.

                                          Prerequisites_Processes

                                          This feature requires the following Sysdig  component versions: 

                                          • Sysdig agent version 0.91 or higher

                                          • For on-premises installations: version 3.2.0.2540 or higher

                                          Understand Process Filtering Behavior

                                          By default, processes are reported according to internal criteria such as resource usage (CPU/memory/file and net IO) and container count.

                                          If you choose to enable process filtering, processes in the include list will be given preference over other internal criteria.

                                          Processes are filtered based on a standard priority filter description already used in Sysdig yaml files. It is comprised of -include and -exclude statements which are matched in order, with evaluation ceasing with the first matched statement. Statements are considered matched if EACH of the conditions in the statement is met.

                                          Use Process Filtering

                                          Edit dragent.yaml per the following patterns to implement the filtering you need.

                                          Process Condition Parameters and Rules

                                          The process: condition parameters and rules are described below.

                                          NameValueDescription
                                          app_checks_always_send:true/falseLegacy config that causes the agent to emit any process with app check. With process filtering, this translates to an extra “include” clause at the head of the process filter which matches a process with any app check, thereby overriding any exclusions. Still subject to limit.
                                          flush_filter:Definition of process filter to be used if flush_filter_enabled == true. Defaults to -include all
                                          flush_filter_enabled:true/falseDefaults to false (default process reporting behavior). Set to true to use the rest of the process filtering options.
                                          limit:N (chosen number)Defines the approximate limit of processes to emit to the backend, within 10 processes or so. Default is 250 processes.
                                          top_n_per_container:N (chosen number)Defines how many of the top processes per resource category per emitted container to report after included processes. Still subject to limit. Defaults to 1.
                                          top_n_per_host:N (chosen number)Defines how many of the top processes per resource category per host are reported before included processes. Still subject to limit. Defaults to 1.

                                          The process: Condition Parameters

                                          Rules

                                          • container.image: my_container_image  Validates whether the container image associated with the process is a wild card match of the provided image name

                                          • container.name: my_container_name  Validates whether the container name associated with the process is a wild card match of the provided image name

                                          • container.label.XYZ: value  Validates whether the label XYZ of the container associated with the process is a wildcard match of the provided value

                                          • process.name: my_process_name  Validates whether the name of the process is a wild card match of the provided value

                                          • process.cmdline: value  Checks whether the executable name of a process contains the specified value, or any argument to the process is a wildcard match of the provided value

                                          • appcheck.match: value  Checks whether the process has any appcheck which is a wildcard match of the given value

                                          • all  Matches all processes, but does not whitelist them, nor does it blacklist them. If no filter is provided, the default is -include all. However, if a filter is provided and no match is made otherwise, then all unmatched processes will be blacklisted. In most cases, the definition of a process filter should end with -include: all.

                                          Examples

                                          Block All Processes from a Container

                                          Block all processes from a given container. No processes from some_container_name will be reported.

                                          process:
                                            flush_filter_enabled: true
                                            flush_filter:
                                            - exclude:
                                                container.name: some_container_name
                                            - include:
                                                allprocess:   flush_filter: - exclude: container.name: some_container_name - include: all
                                          

                                          Prioritize Processes from a Container

                                          Send all processes from a given container at high priority.

                                          process:
                                            flush_filter_enabled: true
                                            flush_filter:
                                              - include:
                                                  container.name: some_container_name
                                              - include:
                                                  all
                                          

                                          Prioritize “java” Processes

                                          Send all processes that contain ‘java" in the name at high priority.

                                          process:
                                            flush_filter_enabled: true
                                            flush_filter:
                                              - include:
                                                  process.name: java
                                              - include:
                                                  all
                                          

                                          Prioritize “java” Processes from a Particular Container

                                          Send processes containing “java” from a given container at high priority.

                                          process:
                                            flush_filter_enabled: true
                                            flush_filter:
                                              - include:
                                                  container.name: some_container_name
                                                  process.name: java
                                              - include:
                                                  all
                                          

                                          Prioritize “java” Processes not in a Particular Container

                                          Send all processes that contain “java” in the name that are not in container some_container_nane.

                                          process:
                                            flush_filter_enabled: true
                                            flush_filter:
                                              - exclude:
                                                  container.name: some_container_name
                                              - include:
                                                  process.name: java
                                              - include:
                                                  all
                                          

                                          Prioritize “java” Processes even from an Excluded Container

                                          Send all processes containing “java” in the name. If a process does not contain “java” in the name and if the container within which the process runs is named  some_container_name,  then exclude it.

                                          Note that each include/exclude rule is handled sequentially and hierarchically so that even if the container is excluded, it can still report “java” processes.

                                          flush_filter:
                                             - flush_filter_enabled: true
                                             - include:
                                                 process.name: java
                                             - exclude:
                                                container.name: some_container_name
                                             - include:
                                                all
                                          

                                          Prioritize “java” Processes and “sql” Processes from Different Containers

                                          Send Java processes from one container and SQL processes from another at high priority.

                                          process:
                                            flush_filter:
                                              - flush_filter_enabled: true
                                              - include:
                                                 container.name: java_container_name
                                                 process.name: java
                                              - include
                                                 container.name: sql_container_name
                                                 process.name: sql
                                              - include
                                                  all
                                          

                                          Report ONLY Processes in a Particular Container

                                          Only send processes running in a container with a given label.

                                          process:
                                            flush_filter:
                                               - flush_filter_enabled: true
                                               - include:
                                           =container.label.report_processes_from_this_container_example_label: true
                                               - exclude:
                                                   all
                                          
                                          

                                          4.6 -

                                          Collect Metrics from Remote File Systems

                                          Sysdig agent does not automatically discover and collect metrics from external file systems, such as NFS, by default. To enable collecting these metrics, add the following entry to the dragent.yaml file:

                                          remotefs = true
                                          

                                          In addition to the remote file systems, the following mount types are also excluded because they cause high load.

                                          mounts_filter:
                                            - exclude: "*|autofs|*"
                                            - exclude: "*|proc|*"
                                            - exclude: "*|cgroup|*"
                                            - exclude: "*|subfs|*"
                                            - exclude: "*|debugfs|*"
                                            - exclude: "*|devpts|*"
                                            - exclude: "*|fusectl|*"
                                            - exclude: "*|mqueue|*"
                                            - exclude: "*|rpc_pipefs|*"
                                            - exclude: "*|sysfs|*"
                                            - exclude: "*|devfs|*"
                                            - exclude: "*|devtmpfs|*"
                                            - exclude: "*|kernfs|*"
                                            - exclude: "*|ignore|*"
                                            - exclude: "*|rootfs|*"
                                            - exclude: "*|none|*"
                                            - exclude: "*|tmpfs|*"
                                            - exclude: "*|pstore|*"
                                            - exclude: "*|hugetlbfs|*"
                                            - exclude: "*|*|/etc/resolv.conf"
                                            - exclude: "*|*|/etc/hostname"
                                            - exclude: "*|*|/etc/hosts"
                                            - exclude: "*|*|/var/lib/rkt/pods/*"
                                            - exclude: "overlay|*|/opt/stage2/*"
                                            - exclude: "/dev/mapper/cl-root*|*|/opt/stage2/*"
                                            - exclude: "*|*|/dev/termination-log*"
                                            - include: "*|*|/var/lib/docker"
                                            - exclude: "*|*|/var/lib/docker/*"
                                            - exclude: "*|*|/var/lib/kubelet/pods/*"
                                            - exclude: "*|*|/run/secrets"
                                            - exclude: "*|*|/run/containerd/*"
                                            - include: "*|*|*"
                                          

                                          To include a mount type:

                                          1. Open the dragent.yaml file.

                                          2. Remove the corresponding line from the exclude list in the mount_filter.

                                          3. Add the file mount to the include list under mount_filter .

                                            The format is:

                                            # format of a mount filter is:
                                            # ```
                                            # mounts_filter:
                                            #   - exclude: "device|filesystem|mount_directory"
                                            #   - include: "pattern1|pattern2|pattern3"
                                            

                                            For example:

                                            mounts_filter:
                                              - include: "*|autofs|*"mounts_filter:
                                              - include: "overlay|*|/opt/stage2/*"
                                              - include: "/dev/mapper/cl-root*|*|/opt/stage2/*"
                                            
                                          4. Save the configuration changes and restart the agent.

                                          4.7 -

                                          Disable Captures

                                          Sometimes, security requirements dictate that capture functionality should NOT be triggered at all (for example, PCI compliance for payment information).

                                          To disable Captures altogether:

                                          1. Access using one of the options listed.

                                            This example accesses dragent.yaml directly. ``

                                          2. Set the parameter:

                                            sysdig_capture_enabled: false
                                            
                                          3. Restart the agent, using the command: ``

                                            service dragent restart
                                            

                                          See Captures for more information on the feature

                                          5 -

                                          Reduce Memory Consumption in Agent

                                          Sysdig provides a configuration option called thin cointerface to reduce the memory footprint in the agent. When the agent is installed as a Kubernetes daemonset, you can optionally enable the thin cointerface in the sysdig-agent configmap.

                                          Pros

                                          • Reduces memory consumption
                                          • Particularly useful on very large Kubernetes clusters (>10,000 pods)

                                          Cons

                                          • Less frequently used option which is therefore less battle-tested
                                          • If a watch is dropped and re-list is required (e.g. in case of a network issue, and apiserver update, etc.), there is no cache to maintain the resources. In this case, the agent must process many additional events.

                                          How It Works

                                          In a typical Kubernetes cluster, two instances of agent daemonset are installed to retrieve the data. They are automatically connected to the Kubernetes API server to retrieve the metadata associated with the entities running on the cluster and sends the global Kubernetes state to the Sysdig backend. Sysdig uses this data to generate kube state metrics.

                                          A delegated agent will not have a higher CPU or memory footprint than a non-delegated agent.

                                          On very large Kubernetes clusters (in the range of 10,000 pods) or clusters with several replication controllers, the agent’s data ingestion can have a significant memory footprint on itself and on the Kubernetes API server. Thin cointerface is provided to reduce this impact.

                                          Enabling this option changes the way the agent communicates with the API server and reduces the need to cache data, which in turn reduces the overall memory usage. Thin cointerface does this by moving some processing from the agent’s cointerface process to the dragent process. This change does not alter the data which is ultimately sent to the backend nor will it impact any Sysdig feature.

                                          The thin cointerface feature is disabled by default.

                                          To Enable:

                                          Add the following in either the sysdig-agent’s configmap or via the dragent.yaml file:

                                          thin_cointerface_enabled: true
                                          

                                          6 -

                                          Enable Kube State Metrics

                                          Agent Versions 12.5.0 and Onward

                                          HPA kube state metrics are no longer collected by default. To enable the agent to collect HPA kube state metrics, you must edit the agent configuration file, dragent.yaml, and include it along with the other resources you would like to collect.

                                          For example, to collect all supported resources including HPAs, add the following to dragent.yaml:

                                          k8s_extra_resources:
                                              include:
                                                - services
                                                - resourcequotas
                                                - persistentvolumes
                                                - persistentvolumeclaims
                                                - horizontalpodautoscalers
                                          

                                          Agent Versions 12.3.x and 12.4.x

                                          The Sysdig agent collects HPA, PVS, PV, Resourcequota, and Services kube state metrics by default.

                                          To disable some of them, you must edit the agent config file, dragent.yaml, as follows:

                                          k8s_extra_resources:
                                              include:
                                                - services
                                                - resourcequotas
                                                - persistentvolumes
                                                - persistentvolumeclaims
                                                - horizontalpodautoscalers
                                          

                                          The above list includes all the supported resources so you must remove the resources you are not interested in. For example, if you wanted to disable Services, it should look like the following:

                                          k8s_extra_resources:
                                              include:
                                                - resourcequotas
                                                - persistentvolumes
                                                - persistentvolumeclaims
                                                - horizontalpodautoscalers
                                          

                                          For more information, see Understanding the Agent Configuration Files.

                                          7 -

                                          Process Kubernetes Events

                                          Use Go to Process Kubernetes Events

                                          Required: Sysdig agent version 92.1 or higher.

                                          As of agent version 9.5.0, go_k8s_user_events:true is the default setting. Set to false to use the older, C++-based version.

                                          To streamline Sysdig agent processing times and reduce CPU load, you can use an updated processing engine written in Go.

                                          To do so, edit the following code in dragent.yaml:

                                          go_k8s_user_events: true
                                          

                                          Kubernetes Audit Events

                                          The agent listens on /k8s-audit for Kubernetes audit events. Configure the path using the following configuration option:

                                          security:{k8s_audit_server_path_uris: [path1, path2]}
                                          

                                          For more information, see Kubernetes Audit Logging.

                                          Working with containerd in K3S

                                          If you have containerd using a custom socket, you can specify this parameter in the agent configuration to correctly capture the containers’ metadata:

                                          cri:
                                            socket_path: /run/k3s/containerd/containerd.sock
                                          

                                          8 -

                                          Manage Agent Log Levels

                                          Sysdig allows you to configure file log levels for agents globally and granularly.

                                          8.1 -

                                          Change Agent Log Level Globally

                                          The Sysdig agent generates log entries in /opt/draios/logs/draios.log. The agent will rotate the log file when it reaches 10MB in size, keeping the 10 most recent log files archived with a date-stamp appended to the filename.

                                          In order of increasing detail, the log levels available are: [ none | critical| error | warning |notice | info | debug | trace ].

                                          The default level (info) creates an entry for each aggregated metrics transmission to the backend servers, once per second, in addition to entries for any warnings and errors.

                                          Setting the value lower than info may prohibit troubleshooting agent-related issues.

                                          The type and amount of logging can be changed by adding parameters and log level arguments shown below to the agent’s user settings configuration file here:

                                          /opt/draios/etc/dragent.yaml

                                          After editing the dragent.yaml file, restart the agent at the shell with: service dragent restart to affect changes.

                                          Note that dragent.yaml code can be written in both YAML and JSON. The examples below use YAML.

                                          File Log Level

                                          When troubleshooting agent behavior, increase the logging to debug for full detail:

                                          log:
                                            file_priority: debug
                                          

                                          If you wish to reduce log messages going to the /opt/draios/logs/draios.log file, add the log: parameter with one of the following arguments under it and indented two spaces: [ none | error | warning | info | debug | trace ]

                                          log:
                                            file_priority: error
                                          

                                          Container Console Logging

                                          If you are running the containerized agent, you can also reduce container console output by adding the additional parameter console_priority: with the same arguments [ none | error | warning | info | debug | trace ]

                                          log:
                                            console_priority: warning
                                          

                                          Note that troubleshooting a host with less than the default ‘info’ level will be more difficult or not possible. You should revert to ‘info’ when you are done troubleshooting the agent.

                                          A level of ’error’ will generate the fewest log entries, a level of ’trace’ will give the most, ‘info’ is the default if no entry exists.

                                          Example in dragent.yaml

                                          customerid: 831f3-Your-Access-Key-9401
                                          tags: local:sf,acct:eng,svc:websvr
                                          log:
                                           file_priority: warning
                                           console_priority: info
                                          

                                          OR

                                          customerid: 831f3-Your-Access-Key-9401
                                          tags: local:sf,acct:eng,svc:websvr
                                          log: { file_priority: debug, console_priority: debug }
                                          

                                          Docker run command

                                          If you are using the “ADDITIONAL_CONF” parameter to start a Docker containerized agent, you would specify this entry in the Docker run command:

                                          -e ADDITIONAL_CONF="log:  { file_priority: error, console_priority: none }"
                                          -e ADDITIONAL_CONF="log:\n  file_priority: error\n  console_priority: none"
                                          

                                          Kubernetes Infrastructure

                                          When running in a Kubernetes infrastructure (installed using the v1 method, comment in the “ADDITIONAL_CONF” line in the agent sysdig-daemonset.yaml manifest file, and modify as needed:

                                          - name: ADDITIONAL_CONF #OPTIONAL pass additional parameters to the agent
                                            value: "log:\n file_priority: debug\n console_priority: error"
                                          
                                          

                                          8.2 -

                                          Manage File Logging for Agent Components

                                          Sysdig Agent provides the ability to set component-wise log levels that override the global file logging level controlled by the file_priority configuration option. The components represent internal software modules and can be found in /opt/draios/logs/draios.log.

                                          By controlling logging at the fine-grained component level, you can avoid excessive logging from certain components in draios.log or enable extra logging from specific components for troubleshooting.

                                          Components can also have an optional feature level logging that can provide a way to control the logging for a particular feature in Sysdig Agent.

                                          To set feature-level or component-level logging:

                                          1. Determine the agent feature or component you want to set the log level:

                                            To do so,

                                            1. Open the /opt/draios/logs/draios.log file.

                                            2. Copy the component name.

                                              The format of the log entry is:

                                              <timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
                                              

                                              For example, the given snippet from a sample log file shows log messages from promscrape featture, sdjagent, mountedfs_reader, watchdog_runnable, protobuf_file_emitter, connection_manager, and dragent.

                                              2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
                                              2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
                                              2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
                                              2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
                                              2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
                                              2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
                                              2020-09-07 18:52:40.065, 27979.27980, Debug,       promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
                                              2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
                                              
                                          2. To set feature-level logging:

                                            1. Open /opt/draios/etc/dragent.yaml.

                                            2. Edit the dragent.yaml file and add the desired feature:

                                              In this example, you are setting the global level to notice and promscrape feature level to info.

                                              log:
                                                file_priority: notice
                                                file_priority_by_component:
                                                  - "promscrape: info"
                                              

                                              The log levels specified for feature override global settings.

                                          3. To set component-level logging:

                                            1. Open /opt/draios/etc/dragent.yaml.

                                            2. Edit the dragent.yaml file and add the desired feature:

                                              In this example, you are setting the global level to notice and promscrape feature level to info, sdjagent, mountedfs_reader component log level to debug, watchdog_runnable component log level to warning and promscrape:prom_emitter component log level to debug.

                                              log:
                                                file_priority: notice
                                                file_priority_by_component:
                                                  - "promscrape: info"
                                                  - "promscrape:prom_emitter: debug"
                                                  - "watchdog_runnable: warning"
                                                  - "sdjagent: debug"
                                                  - "mountedfs_reader: debug" 
                                              

                                              The log levels specified for feature override global settings. The log levels specified for component overide feature and global settings.

                                          4. Restart the agent.

                                            For example, if you have installed the agent as a service, then run:

                                            $ service dragent restart
                                            
                                            

                                          8.3 -

                                          Manage Console Logging for Agent Components

                                          Sysdig Agent provides the ability to set component-wise log levels that override the global console logging level controlled by the console_priority configuration option. The components represent internal software modules and can be found in /opt/draios/logs/draios.log.

                                          By controlling logging at the fine-grained component level, you can avoid excessive logging from certain components in draios.log or enable extra logging from specific components for troubleshooting.

                                          Components can also have an optional feature level logging that can provide a way to control the logging for a particular feature in Sysdig Agent.

                                          To set feature-level or component-level logging:

                                          1. Determine the agent component you want to set the log level:

                                            To do so,

                                            1. Look at the console output.

                                              If you’re using an orchestrator like Kubernetes, the log viewer facility, such as the kubectl log command, shows the console log output.

                                            2. Copy the component name.

                                              The format of the log entry is:

                                              <timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
                                              

                                              For example, the given snippet from a sample log file shows log messages from promscrape featture, sdjagent, mountedfs_reader, watchdog_runnable, protobuf_file_emitter, connection_manager, and dragent.

                                              2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
                                              2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
                                              2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
                                              2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
                                              2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
                                              2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
                                              2020-09-07 18:52:40.065, 27979.27980, Debug,       promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
                                              2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
                                              
                                          2. To set feature-level logging:

                                            1. Open /opt/draios/etc/dragent.yaml.

                                            2. Edit the dragent.yaml file and add the desired feature:

                                              In this example, you are setting the global level to notice and promscrape feature level to info.

                                              log:
                                                console_priority: notice
                                                console_priority_by_component:
                                                  - "promscrape: info"
                                              

                                              The log levels specified for feature override global settings.

                                          3. To set component-level logging:

                                            1. Open /opt/draios/etc/dragent.yaml.

                                            2. Edit the dragent.yaml file and add the desired feature:

                                              In this example, you are setting the global level to notice and promscrape feature level to info, sdjagent, mountedfs_reader component log level to debug, watchdog_runnable component log level to warning and promscrape:prom_emitter component log level to debug.

                                              log:
                                                console_priority: notice
                                                console_priority_by_component:
                                                  - "promscrape: info"
                                                  - "promscrape:prom_emitter: debug"
                                                  - "watchdog_runnable: warning"
                                                  - "sdjagent: debug"
                                                  - "mountedfs_reader: debug" 
                                              

                                              The log levels specified for feature override global settings. The log levels specified for component overide feature and global settings.

                                          4. Restart the agent.

                                            For example, if you have installed the agent as a service, then run:

                                            $ service dragent restart
                                            
                                            

                                          9 -

                                          Agent Auto-Config

                                          Introduction

                                          If you want to maintain centralized control over the configuration of your Sysdig agents, one of the following approaches is typically ideal:

                                          1. Via an orchestration system, such as using Kubernetes or Mesos/Marathon.

                                          2. Using a configuration management system, such as Chef or Ansible.

                                          However, if these approaches are not viable for your environment, or to further augment your Agent configurations via central control, Sysdig Monitor provides an Auto-Config option for agents. The feature allows you to upload fragments of YAML configuration to Sysdig Monitor that will be automatically pushed and applied to some/all of your Agents based on your requirements.

                                          Enable Agent Auto-Config

                                          Independent of the Auto-Config feature, typical Agent configuration lives in /opt/draios/etc and is derived from a combination of base config in the dragent.default.yaml file and any overrides that may be present in dragent.yaml. See also Understanding the Agent Config Files.

                                          Agent Auto-Config adds a middle layer of possible overrides in an additional file dragent.auto.yaml.When present, the the order of config application from highest precedence to lowest now becomes:

                                          1. dragent.yaml

                                          2. dragent.auto.yaml

                                          3. dragent.default.yaml

                                          While all Agents are by default prepared to receive and make use of Auto-Config data, the file dragent.auto.yaml will not be present on an Agent until you’ve pushed central Auto-Config data to be applied to that Agent.

                                          Auto-Config settings are performed via Sysdig Monitor’s REST API. Simplified examples are available that use the Python client library to get or set current Auto-Config settings. Detailed examples using the REST API are shown below.

                                          The REST endpoint for Auto-Config is /api/agents/config. Use the GET method to review the current configuration. The following example shows the initial empty settings that result in no dragent.auto.yaml files being present on your Agents.

                                          curl -X GET \
                                                 --header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
                                                 https://app.sysdigcloud.com/api/agents/config
                                          
                                          
                                          Output:
                                          {
                                              "files": []
                                          }
                                          

                                          Use the PUT method to centrally push YAML that will be distributed and applied to your Agents as dragent.auto.yaml files. The content parameter must contain syntactically-correct YAML. The filter option is used to specify if the config should be sent to one agent or all of them, such as in this example to globally enable Debug logging on all Agents:

                                          curl -X PUT \
                                                 --header "Content-Type: application/json" \
                                                 --header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
                                                 https://app.sysdigcloud.com/api/agents/config -d '
                                          {
                                            "files": [
                                              {
                                                "filter": "*",
                                                "content": "log:\n  console_priority: debug"
                                              }
                                            ]
                                          }'
                                          

                                          Alternatively, the filter can specify a hardware MAC address for a single Agent that should receive a certain YAML config. All MAC-specific configs should appear at the top of the JSON object and are not additive to any global Auto-Config specified with “filter”: “*” at the bottom. For example, when the following config is applied, the one Agent that has the MySQL app check configured would not have Debug logging enabled, but all others would.

                                          curl -X PUT \
                                                 --header "Content-Type: application/json" \
                                                 --header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
                                                 https://app.sysdigcloud.com/api/agents/config -d '
                                          {
                                            "files": [
                                              {
                                                "filter": "host.mac = \"08:00:27:de:5b:b9\"",
                                                "content": "app_checks:\n  - name: mysql\n    pattern:\n      comm: mysqld\n    conf:\n      server: 127.0.0.1\n      user: sysdig-cloud\n      pass: sysdig-cloud-password"
                                              },
                                              {
                                                "filter": "*",
                                                "content": "log:\n  console_priority: debug"
                                              }
                                            ]
                                          }'
                                          

                                          To update the active central Auto-Config settings, simply PUT a complete replacement JSON object.

                                          All connected Agents will receive centrally-pushed Auto-Config updates that apply to them based on the filter settings. Any Agent whose Auto-Config is enabled/disabled/changed based on the centrally-pushed settings will immediately restart, putting the new configuration into effect. Any central Auto-Config settings that would result in a particular Agent’s Auto-Config remaining the same will not trigger a restart.

                                          Disable Agent Auto-Config

                                          To clear all Agent Auto-Configs, use the PUT method to upload the original blank config setting of ’{ “files”: [] }’.

                                          It is also possible to override active Auto-Config on an individual Agent. To do so, follow these steps for your Agent:

                                          1. Add the following config directly to the dragent.yaml file: auto_config: false.

                                          2. Delete the file /opt/draios/etc/dragent.auto.yaml.

                                          3. Restart the Agent.

                                          For such an Agent to opt-in to Auto-Config again, remove auto_config: false from the dragent.yaml and restart the Agent.

                                          Restrictions

                                          To prevent the possibility of pushing Auto-Config that would damage an Agent’s ability to connect, the following keys will not be accepted in the centrally-pushed YAML.

                                          • auto_config

                                          • customerid

                                          • collector

                                          • collector_port

                                          • ssl

                                          • ssl_verify_certificate

                                          • ca_certificate

                                          • compression

                                          10 -

                                          Using the Agent Console

                                          Sysdig provides an Agent Console to interact with the Sysdig agent. This is a troubleshooting tool to help you view configuration files and investigate agent configuration problems quickly.

                                          Access Agent Console

                                          1. From Explore click the Groupings drop-down.

                                          2. Select Hosts & Container or Nodes.

                                          3. Click the desired host to investigate the corresponding agent configuration.

                                          4. Click Options (three dots) on the right upper corner of the Explore tab.

                                          5. Click Agent Console.

                                          Agent Console Commands

                                          View Help

                                          The ? command displays the commands to manage Prometheus configuration and targets monitored by the Sysdig agent.

                                          $ prometheus ?
                                          $ prometheus config ?
                                          $ prometheus config show ?
                                          

                                          Command Syntax

                                          The syntax of the Agent Console commands is as follows:

                                          directory command
                                          directory sub-directory command
                                          directory sub-directory sub-sub-directory command
                                          

                                          View Version

                                          Run the following to find the version of the agent running in your environment:

                                          $ version
                                          

                                          An example output:

                                          12.0.0
                                          

                                          Troubleshoot Prometheus Metrics Collection

                                          These commands help troubleshoot Prometheus targets configured in your environment.

                                          For example, the following commands display and scrape the Prometheus endpoints respectively.

                                          $ prometheus target show
                                          $ prometheus target scrape
                                          

                                          Sub-Directory Commands

                                          The Promscrape CLI consists of the following sections.

                                          • config: Manages Sysdig agent-specific Prometheus configuration.

                                          • metadata: Manages metadata associated with the Prometheus targets monitored by the Sysdig agent.

                                          • stats: Helps view the global- and job-specific Prometheus statistics.

                                          • target: Manages Prometheus endpoints monitored by Sysdig agent.

                                          Prometheus Commands

                                          Show

                                          The show command displays the information about the subsection. For example, the following example displays the configuration of the Prometheus server.

                                          $ prometheus config show
                                          
                                          5  Configuration      Value
                                          6  Enabled            True
                                          7  Target discovery   Prometheus service discovery
                                          8  Scraper            Promscrape v2
                                          9  Ingest raw         True
                                          10  Ingest calculated  True
                                          11  Metric limit       2000
                                          
                                          Scrape

                                          The scrape command scrapes a Prometheus target and displays the information. The syntax is:

                                          $ prometheus target scrape -url <URL>
                                          

                                          For example:

                                          $ prometheus target scrape -url http://99.99.99.3:10055/metrics
                                          
                                          # HELP go_gc_duration_seconds A summary of the GC invocation durations.
                                          7  # TYPE go_gc_duration_seconds summary
                                          8  go_gc_duration_seconds{quantile="0"} 7.5018e-05
                                          9  go_gc_duration_seconds{quantile="0.25"} 0.000118155
                                          10  go_gc_duration_seconds{quantile="0.5"} 0.000141586
                                          11  go_gc_duration_seconds{quantile="0.75"} 0.000171626
                                          12  go_gc_duration_seconds{quantile="1"} 0.00945638
                                          13  go_gc_duration_seconds_sum 0.114420898
                                          14  go_gc_duration_seconds_count 607
                                          

                                          View Agent Configuration

                                          The Agent configuration commands have a different syntax.

                                          Run the following to view the configuration of the agent running in your environment:

                                          $ configuration show-dragent-yaml
                                          $ configuration show-configmap-yaml
                                          $ configuration show-default-yaml
                                          $ configuration show-backend-yaml 
                                           
                                          

                                          The output displays the configuration file. Sensitive data, such as the credentials, are obfuscated.

                                          customerid: "********"
                                          watchdog:
                                            max_memory_usage_mb: 2048
                                          

                                          Security Considerations

                                          • User-sensitive configuration is obfuscated and not visible through the CLI.

                                          • All the information is read-only. You cannot currently change any configuration by using the Agent console.

                                          • Runs completely inside the agent. It does not use bash or any other Linux terminals to prevent the risk of command injection.

                                          • Runs only via a TLS connection with the Sysdig backend.

                                          Disable Agent Console

                                          This is currently turned on by default. To turn off Agent Console for a particular team:

                                          1. Navigate to Settings > Teams.

                                          2. Select the team that you want to disable Agent Console for.

                                          3. From Additional Permissions, Deselect Agent CLI.

                                          4. Click Save.

                                          To turn it off in your environment, edit the following in the dragent.yaml file:

                                          command_line:
                                            enabled: false