This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Manage Agent Log Levels

Sysdig allows you to configure file log levels for agents globally and granularly.

1 - Change Agent Log Level Globally

The Sysdig agent generates log entries in /opt/draios/logs/draios.log. The agent will rotate the log file when it reaches 10MB in size, keeping the 10 most recent log files archived with a date-stamp appended to the filename.

In order of increasing detail, the log levels available are: [ none | critical| error | warning |notice | info | debug | trace ].

The default level (info) creates an entry for each aggregated metrics transmission to the backend servers, once per second, in addition to entries for any warnings and errors.

Setting the value lower than info may prohibit troubleshooting agent-related issues.

The type and amount of logging can be changed by adding parameters and log level arguments shown below to the agent’s user settings configuration file here:

/opt/draios/etc/dragent.yaml

After editing the dragent.yaml file, restart the agent at the shell with: service dragent restart to affect changes.

Note that dragent.yaml code can be written in both YAML and JSON. The examples below use YAML.

File Log Level

When troubleshooting agent behavior, increase the logging to debug for full detail:

log:
  file_priority: debug

If you wish to reduce log messages going to the /opt/draios/logs/draios.log file, add the log: parameter with one of the following arguments under it and indented two spaces: [ none | error | warning | info | debug | trace ]

log:
  file_priority: error

Container Console Logging

If you are running the containerized agent, you can also reduce container console output by adding the additional parameter console_priority: with the same arguments [ none | error | warning | info | debug | trace ]

log:
  console_priority: warning

Note that troubleshooting a host with less than the default ‘info’ level will be more difficult or not possible. You should revert to ‘info’ when you are done troubleshooting the agent.

A level of ’error’ will generate the fewest log entries, a level of ’trace’ will give the most, ‘info’ is the default if no entry exists.

Examples

Using HELM


helm install ... \
  --set sysdig.settings.log.file_priority=debug \
  --set sysdig.settings.log.console_priority=debug

Using values.yaml

sysdig:
  settings:
    log:
      file_priority: debug
      console_priority: debug

Using dragent.yaml

customerid: 831f3-Your-Access-Key-9401
tags: local:sf,acct:eng,svc:websvr
log:
 file_priority: warning
 console_priority: info

OR

customerid: 831f3-Your-Access-Key-9401
tags: local:sf,acct:eng,svc:websvr
log: { file_priority: debug, console_priority: debug }

Using Docker Run Command

If you are using the “ADDITIONAL_CONF” parameter to start a Docker containerized agent, you would specify this entry in the Docker run command:

-e ADDITIONAL_CONF="log:  { file_priority: error, console_priority: none }"
-e ADDITIONAL_CONF="log:\n  file_priority: error\n  console_priority: none"

Using deamonset.yaml in Kubernetes Infrastructure

When running in a Kubernetes infrastructure (installed using the v1 method, comment in the “ADDITIONAL_CONF” line in the agent sysdig-daemonset.yaml manifest file, and modify as needed:

- name: ADDITIONAL_CONF #OPTIONAL pass additional parameters to the agent
  value: "log:\n file_priority: debug\n console_priority: error"

2 - Manage File Logging for Agent Components

Sysdig Agent provides the ability to set component-wise log levels that override the global file logging level controlled by the file_priority configuration option. The components represent internal software modules and can be found in /opt/draios/logs/draios.log.

By controlling logging at the fine-grained component level, you can avoid excessive logging from certain components in draios.log or enable extra logging from specific components for troubleshooting.

The Agent components can also have an optional feature level logging that can provide a way to control the logging for a particular feature in Sysdig Agent.

To set feature-level or component-level logging:

  1. Determine the agent feature or component you want to set the log level:

    To do so,

    1. Open the /opt/draios/logs/draios.log file.

    2. Copy the component name.

      The format of the log entry is:

      <timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
      

      For example, the given snippet from a sample log file shows log messages from promscrape featture, sdjagent, mountedfs_reader, watchdog_runnable, protobuf_file_emitter, connection_manager, and dragent.

      2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
      2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
      2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
      2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
      2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
      2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
      2020-09-07 18:52:40.065, 27979.27980, Debug,       promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
      2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
      
  2. To set feature-level logging:

    1. Open /opt/draios/etc/dragent.yaml.

    2. Edit the dragent.yaml file and add the desired feature:

      In this example, you are setting the global level to notice and promscrape feature level to info.

      log:
        file_priority: notice
        file_priority_by_component:
          - "promscrape: info"
      

      The log levels specified for feature override global settings.

  3. To set component-level logging:

    1. Open /opt/draios/etc/dragent.yaml.

    2. Edit the dragent.yaml file and add the desired feature:

      In this example, you are setting the global level to notice and promscrape feature level to info, sdjagent, mountedfs_reader component log level to debug, watchdog_runnable component log level to warning and promscrape:prom_emitter component log level to debug.

      log:
        file_priority: notice
        file_priority_by_component:
          - "promscrape: info"
          - "promscrape:prom_emitter: debug"
          - "watchdog_runnable: warning"
          - "sdjagent: debug"
          - "mountedfs_reader: debug" 
      

      The log levels specified for feature override global settings. The log levels specified for component overide feature and global settings.

  4. Restart the agent.

    For example, if you have installed the agent as a service, then run:

    $ service dragent restart
    

3 - Manage Console Logging for Agent Components

Sysdig Agent provides the ability to set component-wise log levels that override the global console logging level controlled by the console_priority configuration option. The components represent internal software modules and can be found in /opt/draios/logs/draios.log.

By controlling logging at the fine-grained component level, you can avoid excessive logging from certain components in draios.log or enable extra logging from specific components for troubleshooting.

Components can also have an optional feature level logging that can provide a way to control the logging for a particular feature in Sysdig Agent.

Configure Logging

To set feature-level or component-level logging:

  1. Determine the agent component you want to set the log level:

    To do so,

    1. Look at the console output.

      If you’re using an orchestrator like Kubernetes, the log viewer facility, such as the kubectl log command, shows the console log output.

    2. Copy the component name.

      The format of the log entry is:

      <timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
      

      For example, the given snippet from a sample log file shows log messages from promscrape featture, sdjagent, mountedfs_reader, watchdog_runnable, protobuf_file_emitter, connection_manager, and dragent.

      2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
      2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
      2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
      2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
      2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
      2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
      2020-09-07 18:52:40.065, 27979.27980, Debug,       promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
      2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
      
  2. To set feature-level logging:

    1. Open /opt/draios/etc/dragent.yaml.

    2. Edit the dragent.yaml file and add the desired feature:

      In this example, you are setting the global level to notice and promscrape feature level to info.

      log:
        console_priority: notice
        console_priority_by_component:
          - "promscrape: info"
      

      The log levels specified for feature override global settings.

  3. To set component-level logging:

    1. Open /opt/draios/etc/dragent.yaml.

    2. Edit the dragent.yaml file and add the desired feature:

      In this example, you are setting the global level to notice and promscrape feature level to info, sdjagent, mountedfs_reader component log level to debug, watchdog_runnable component log level to warning and promscrape:prom_emitter component log level to debug.

      log:
        console_priority: notice
        console_priority_by_component:
          - "promscrape: info"
          - "promscrape:prom_emitter: debug"
          - "watchdog_runnable: warning"
          - "sdjagent: debug"
          - "mountedfs_reader: debug" 
      

      The log levels specified for feature override global settings. The log levels specified for component overide feature and global settings.

  4. Restart the agent.

    For example, if you have installed the agent as a service, then run:

    $ service dragent restart
    

Agent Components

  • analyzer: The logs from this component provide information about events and metrics as they come into the system. These logs assist in basic troubleshooting of event flow.

  • connection_manager: This component logs details about the agent’s connection to the Sysdig backend. These logs help diagnose and troubleshoot connectivity issues.

  • security_mgr: These logs describe the security processing steps the agent is taking. Having these logs assists in understanding what the security side of the agent is doing.

  • infrastructure_state: This component interacts with the orchestration runtime to provide a view of the infrastructure. The logs from this component help troubleshoot orchestration issues and communication with the API server.

  • procfs_parser: The agent uses the procfs parser to gather information about the state of the system. These logs provide insight into the functioning of the agent.

  • dragent: These logs provide data about the core functionality of the agent.

  • process_emitter: This component is used to provide data regarding processes running on a host.

  • k8s_parser: The k8s_parser is used as part of the communication with the Kubernetes API server. These logs help debug communication issues.

  • netsec: These logs provide data about the functioning of the netsec component, which provides topology and endpoint security functionality.

  • protocol_handler: This component logs information about the protobufs the agent sends to the Sysdig backend.

  • k8s_deleg: Kubernetes uses the concept of delegated nodes to help reduce cluster load and manage distributed systems. These logs help with troubleshooting issues within the Kubernetes distributed environment.

  • promscrape: Promscrape allows the agent to send prometheus data as custom metrics.

  • cm_socket: The cm_socket is the low-level networking code used by the connection_manager. These logs work together with the logs from the connection_manager to show the behavior of the network connection between the agent and the backend.

  • secure_audit: Audit is a feature of Sysdig Secure which provides information on system activity such as file and network behavior. These logs help understand the behavior of that feature.

  • memdumper: The memdumper is used to perform back-in-time captures, and logs from this component help troubleshoot any problems which might occur with back-in-time captures.

4 - Change the Agent Log Directory

The Sysdig agent generates log entries in /opt/draios/logs/draios.log. The agent will rotate the log file when it reaches 10MB in size, keeping the 10 most recent log files archived with a date-stamp appended to the filename.

You can change the default location as follows:

log:
  location: new_directory

By default, this location is rooted in the agent install path: /opt/draios/. Therefore, the new log location for the given example would be /opt/draios/new_directory.

You cannot write agent logs outside of the agent install path.

5 - Enable Agent Logs Globally Readable

The Sysdig agent generates log entries in /opt/draios/logs/draios.log. By default, only accounts with superuser credentials can read the agent logs.

To allow all the users access and read the agent logs, use the following configuration:

log:
  globally_readable: true

This option can be combined with the facility to change the agent log location or used independently. For example:

log:
  location: new_directory
  globally_readable: true

Now, all the users can read agent logs from /opt/draios/new_directory/draios.log.

6 - Control Disk Usage by Agent Logs

The Sysdig agent generates log entries in /opt/draios/logs/draios.log. It periodically performs rotation of its own logs.

You can use the following configuration to control the space taken up by agent logs:

  • max_size: Sets a limit to the size of a single agent log file, in megabytes. When the log file reaches this size, a new log file will be created. The old log will be renamed with a timestamp. The default size is 10 megabytes.

  • rotate: The rotate configuration determines how many old log files are kept on the disk. The default is 10 log files.

    When the log file reaches this size, a new log file, draios.log will be created, and the old log will be renamed with a timestamp.

log:
  max_size: 10
  rotate: 10

For example, if the current log file reaches the size limit of 10 megabytes and the number of log files reaches the limit of 10, the oldest will be removed. The last log file will be renamed with a timestamp and added to the list of old log files.

Increasing these values can provide more logs for troubleshooting at the expense of more space.