RSS

Sysdig Agent Release Notes

On this page, you can read the most recent release notes for Sysdig Agent. Review the entries to learn about the latest features, defect fixes, and known issues.

12.18.0 November 22, 2023

Feature Enhancements

Changed Default Matching Strategy for Falco Rules

Sysdig Agent now evaluates an event against all the rules, potentially triggering multiple alerts. In previous versions, the agent stopped evaluating rules after the first match.

To change the matching strategy, see Agent Configuration Library.

Added a Metric to Detect Kubernetes Environment

The Sysdig Agent now indicates whether it is executing within a Kubernetes environment or not as part of the sysdig_agent_info metric.

Added an Option to Show Prometheus Scrape Jobs in Agent Console

Added the option to view all the Prometheus scrape jobs through the agent console with the prometheus scrape_jobs show command.

Report Policy Actions in Kubernetes Events

Sysdig Agent now supports reporting threat detection policy actions in Kubernetes events.

When the agent performs a stop, pause, or kill container action as defined in a rule, the agent will generate a Kubernetes event with the triggering action details and rule name. You can then see why actions were taken directly from the kubectl events, without having to explore the event feed in Sysdig Secure.

If you have deployed agent 12.18+ using Helm, the feature and its associated permissions are enabled by default. If you deploy agents manually, you must set Kubernetes role permissions.

See Agent Configuration Library for details.

Event Forwarding Directly from Sysdig Agent

It is possible to send Runtime Policy Events and Activity Audit events to SIEM platforms and logging tools directly from the agent. This enables event forwarding without exposing the data collection tool on the internet.

See Agent Local Forwarding for details.

Defect Fixes

Added Prometheus Support for FIPS-Enabled ARM Systems

Added Prometheus support for ARM systems running in FIPS mode.

Logging and Warning Messages in Secure Only Mode

Fixed an issue in secure-only mode where some warning log messages were being incorrectly generated.

Retrieve Kubernetes Service Ingresses for LoadBalancer Service Type

Addressed a defect that could hinder the retrieval of Kubernetes service ingresses when the service type is LoadBalancer. This issue had an impact on Cost Advisor.

Fixed overlayfs on Linux Hosts

Fixed overlayfs support on hosts running Linux kernel version 6.5.

Improved Container Runtime API Failure Handling

Added a failsafe to ensure that the container manager doesn’t loop forever trying to read data from the container API.

12.17.1 October 24, 2023

This hotfix is applicable only to Sysdig on-prem deployments; Sysdig SaaS users can disregard Sysdig Agent v12.17.1.

This hotfix release fixes the issue where the agent generates events in large numbers when Legacy Compliance is enabled due to incorrect throttling. With this fix, compliance events will be throttled as they were before 12.17.0.

12.17.0 October 17, 2023

If you are a Sysdig on-prem user, skip v12.17.0 and upgrade to Sysdig Agent v12.17.1. See 12.17.1 October 24 2023 for more information.

Feature Enhancements

Technical Preview of Universal eBPF

Universal eBPF driver is now available in Sysdig Agent as an opt-in feature. The Universal eBPF driver is pre-built and embedded in the agent binary. It does not require kernel headers or to build tools from the target system to function and thereby removes the need for users to build or download probes on supported systems. To learn more about the capability, see Universal eBPF.

Capability for Malware Detection

Sysdig Agent now provides the ability to detect malware and suspicious binary execution by using known bad hashes on hosts and containers.

When a Malware control policy is enabled, the agent computes the hash for every binary execution and checks if the hash matches any of the known malicious hashes. On match, the agent will prevent the execution and generate an event.

Your environment requires Linux kernel v5.0 or beyond for malware detection to work.

This feature is enabled by default. To disable globally on the agent, add the following to the dragent.yaml file:

malware_control:
  enabled: false

To enable the feature for the underlying host node, add the following to the dragent.yaml file:

protections:
  malware_control:
    enable_for_host: true

Use Protocol Buffer to Communicate to Kubernetes API Server

Cointerface uses Google Protocol Buffers as a wire format for communicating with the Kubernetes API server.

Update OpenSSL Library to OpenSSL v3.1 and Include a FIPS-Validated Crypto Module

In light of OpenSSL v1.1.1 reaching end-of-life, this release updates its bundled OpenSSL libraries to v3.1.3.

Additionally, this release bundles a FIPS(Federal Information Processing Standards)-validated OpenSSL crypto module with the agent. Adding the crypto module removes the requirement for user-provided, FIPS-validated OpenSSL shared libraries when the fips_mode configuration parameter is set to true.

This update breaks the agent’s backward compatibility with OpenSSL v1.1.1. If you have configured the openssl_lib parameter, do one of the following:

  • Provide OpenSSL v3.1 shared libraries
  • Remove the parameter and rely on the bundled OpenSSL shared libraries

End of Support for OpenShift v3

Sysdig Agent versions beyond 12.17.0 will no longer be supported on OpenShift 3. v12.17.0 will be the last version supporting OpenShift 3.

Defect Fixes

Prevent Transition During Restarts

The agent will no longer release the Kubernetes delegation lease during tear-down to avoid unwanted transitions during restarts.

Policy Scoping in Fargate Now Respects Agent Labels

Fargate agents will no longer skip agent labels when performing policy scoping.

Display Resolved IPs in the Network Security Policy Egress

The agent uses improved logic to resolve services and endpoints, and therefore, the network communications in some namespaces will no longer be dropped as unresolved.

Use get_mm_exe_file()

A safer version of the Linux kernel API call is used where get_mm_exe_file() is available.

Show Correct Kubernetes Status

Fixed defects in the Kubernetes status reporting. The kube_workload_status_available and kube_workload_status_unavailable metrics now report correct values even when the cluster node count changes and the Kubernetes status reflects the state correctly after the cointerface switches run modes.

Prevent Unintended Agent Restart

A defect was fixed where an invalid message from the backend caused an unintended agent restart.

Store Device Metrics as Expected

A defect was fixed where I/O metrics for devices were not stored.

Display Kubernetes Cluster Association Correctly

A defect was fixed which caused incorrect agent association with Kubernetes clusters on the Sysdig Agents page in the Integrations UI.

Display Correct Timeseries Count in Prometheus Logs

Filtered timeseries counts in Prometheus statistics logs are now reported correctly.

12.16.3 October 15, 2023

This hotfix delivers the following:

  • Addressed the following vulnerabilities:

  • Fixed the kernel null pointer dereference on Linux kernel 6.5. Support for is_exe_upper_layer will be disabled (both kmod and bpf) for those kernels.

12.16.2 September 28, 2023

Improved security for connections between Sysdig Agent and Apache Mesos.

Sysdig Agent primarily connects to Apache Mesos using cURL. When negotiating a TLS (Transport Layer Security) or SSL (Secure Sockets Layer) connection, the Mesos server sends a certificate indicating its identity. cURL verifies the authenticity of the TLS certificate before establishing the connection with the Sysdig Agent. Before this fix, Mesos would not verify the TLS certificate if the connection was over HTTPS.

To estalish a cURL connection to Apache Mesos as of Sysdig Agent v12.16.2:

  • Ensure that the CA cert indicates the Mesos server where the agent will connect. Check that the Mesos server name given in the URL matches the one in the certificate.

12.16.1 September 11, 2023

This hotfix release delivers the following:

12.16.0 August 08, 2023

Feature Enhancements

Default Availability of Process Tree in Sysdig Secure

Process lineage will be available for every event and is enabled by default starting from this version of the agent. The process tree will be visible in the Events detail pane for events related to workloads that are triggered from that point on.

Supports Control Group v2

Control groups v2 (cgroups v2) are now supported in the Sysdig Agent. In particular, the v1 freezer subsystem is no longer mounted when using cgroups v2, as it caused potential compatibility issues in the past.

View Agent Threads for Improved Performance Analysis

The Sysdig Agent threads on Linux x86 platforms have been named to facilitate better analysis of agent performance. Previously, they were named after the default process name, dragent. Now, these threads have descriptive names, with suffixes dr- or dr=. For example, dr-monitor and dr=sinsp_evnt_. The thread name is usually a truncation of the nearest unique function name.

Collects Node Labels

Sysdig Agent can by default collect the node-role.kubernetes.io/* labels set on nodes.

Known Issues

Container Limits to Drift Control

  • For kernel versions below v5.13, Drift Control can monitor up to 128 containers per node.
  • For kernel versions v5.13 or above, modify the container limit using one of the following methods:

    • Open the sysctl -n fs.fanotify.max_user_groups file and set the new value by using sysctl -w fs.fanotify.max_user_groups=<new_limit>.

    • Open the cat /proc/sys/fs/fanotify/max_user_groups file and run echo <new_limit> > /proc/sys/fs/fanotify/max_user_groups.

      Replace <new_limit> with your choice of container limit.

Agent Logs Show Errors for On-Prem Installations in Secure Only Mode

When connecting to an on-prem backend with Secure Only mode, the agent doesn’t connect successfully unless you add the 60s_flush_enable: true configuration under sysdig.settings in the agent configuration file.

Defect Fixes

Removed Compliance Manager Support

Compliance manager functionality has been removed from Sysdig Agent. The feature was no longer supported, but it appeared in a security audit as having a vulnerability. For this reason, the functionality has been dismissed.

Ignores Non-Running Pods for Scraping

The Prometheus k8s-pods job configuration has been modified to drop scrapes from non-running pods.

Enable FIPS-Validated Crypto Module for Agent-to-Backend communication

The agent can now use a FIPS(Federal Information Processing Standards)-validated crypto module for encrypting communication with the Sysdig backend. This feature requires a user-provided, FIPS-validated OpenSSL v1.1.1 shared library to function properly. See the Configuration Library for more information.

Retry Sending Secure Events

The Sysdig Agent now retries sending secure events in cases where the agent disconnects and reconnects.

Adds Missing Health Metrics in Secure Modes

An additional metric is collected in the secure and secure_light modes. The protobuf output for secure and secure_light mode now includes an aggrSamplingRatio aggregation field, weighted to the negotiated metrics interval.

12.15.0 June 28, 2023

Feature Enhancements

Process Tree

This version of the Sysdig Agent adds support in Sysdig Secure for the Process Tree visualization which enriches the Events feed for workload-based events. This feature helps you identify all the processes that led up to the offending process.

To enable this feature:

  1. Add the following configuration to the values.yaml file associated with the sysdig.deploy chart:

     agent:
   sysdig:
     settings:
       enrich_with_process_lineage: true

    You can use the sysdig.settings parameter of the agent subchart to merge this configuration into your existing values.yaml file.

  2. Log in to Sysdig Secure as administrator and select Settings > User Profile. Scroll down to Sysdig Labs and toggle the feature on.

    The process tree will be visible in the Events detail pane for the events related to workloads that are triggered from that point on.

Added Support for Java 7

In Sysdig Agent versions 12.10.0 to 12.14.1, a Java dependency was upgraded to a version that didn’t support Java 7. As a result, those versions cannot run the Java process which collects Java Management Extensions (JMX) metrics on any Java 7 Java Development Kits (JDKs)/Java Runtime Environments (JREs). This release downgrades the dependency back to a version that supports Java 7.

Added Support for Node Cost Metrics

Sysdig Agent now supports node cost metrics when using the thin cointerface.

Building Probes for Airgapped Environments

Effective July 28, 2023, a minor change has been enforced to the process of building probes for airgapped environments. See Airgapped Agent Installation for further details.

Added Sysdig Secure Rule for Detecting Fileless Attacks

Sysdig Secure has added the ability to detect fileless attacks using a new Falco rule on the Sysdig Threat Detection managed policy. See also: SaaS release note.

Vulnerability Fixes

Addressed CVE-2023-0286 by upgrading the OpenSSL version in the agent to 1.1.1t.

Defect Fixes

Metrics Parity Between Secure and Secure Light Modes

The Sysdig Agent will now report the same set of metrics in both secure and secure_light modes, which means that the program metrics in secure mode will also be restricted to the dragent process or container.

Enhanced Execution Time Accounting

Fixed system execution time accounting for certain events that would cause incorrect reporting of agent I/O metrics.

Added Support for s390x for Ubuntu

Recent s390x Linux distributions, including Ubuntu v20.04, require the compiler to support the -march=z13/-mtune=z15 flags when building kernel modules. The gcc version used in agent-kmodule image for the s390x platform has been upgraded to gcc-12, which supports the required flags.

12.14.1 May 16, 2023

This hotfix release provides the following enhancements:

Added Support for Kernel Version 6.3

The kernel module has been updated to support Linux kernel version 6.3.

Fixed Vulnerabilities

Resolved CVE-2023-28840 in promscrape.

Fixed Probe Build Errors on RHEL6

Fixed probe build errors on RHEL6 hosts.

12.14.0 May 08, 2023

Feature Enhancements

Enhanced Console Logging

The console log messages sent to stderr have been restricted to Warning or higher priority only. All the lower priority console log messages are sent to stdout. This reduces noise.

Optimized Filtering

  • The following redundant elements were removed from the Falco rules optimizer:
    • The evt.type/evt.dir fields
    • The evt.arg.res = 0 checks
  • The redundant container.id != host field was removed from conditions while indexing with the rules optimizer.

Improved Drift Detection in Sysdig Secure

With agent 12.14.0, drift detection is improved in Sysdig Secure. Drift detection requires a minimum kernel version of 3.18 and drift prevention requires a minimum kernel version of 5.0.

For the drift feature (both detection and prevention) to work in the 12.14 agent release, set the following in:

  • The agent config file:

    drift_killer:
        enabled: true

  • Helm, add:

    --set agent.sysdig.settings.drift_killer.enabled=true

Added Logging to Detect Incorrect Collector Endpoints

Added detection for invalid HTTP responses on connection.

Enabled Default Scraping of Docker Containers

Sysdig Agent now supports scraping Prometheus metrics from Docker containers by default. Scraping is based on container labels.

Dependencies

Added fmt Library

Added the fmt library to the Agent dependencies. The agent currently does not use this library.

Upgraded Library TBB

The TBB (Intel’s Threading Building Blocks) library has been upgraded to oneTBB v2021.8.0.

Upgraded Boost Library

The Boost library used by Sysdig Agent has been upgraded to v1.81.0.

Known Issues

  • Agent not compatible with GKE Autopilot running Kubernetes v1.26
  • Agent not compatible with Kernel v6.3

Defect Fixes

Restarting the Agent No Longer Causes Premature Process Termination

The SysV init script for RPM-based distributions now takes agent shutdown time into account, avoiding premature SIGKILL.

PID tracking is now enabled for systemd-sysv-generator.

Exclude JVM from Monitoring

Agent can now exclude some Java Virtual Machines (JVMs) from being monitored.

A set of exclusion rules can be defined in the agent’s config. Each rule is a property/pattern pair: when the value of the given Java property matches the pattern, a process of that JVM is excluded from being monitored. For example, the following configuration will exclude OpenJ9-based JVMs from being monitored:

jmx:
  jvm_exclude:
    - property: java.vm.name
      pattern: .+OpenJ9.+

Previously, this functionality was hardcoded to reject OpenJ9, but this is no longer the case. If you observe heap dumps when monitoring OpenJ9, you should add the configuration above to your dragent.yaml file.

Recover from Handshake Errors Between Agent and Collector

Fixed an issue preventing the agent from recovering after a bad protocol handshake.

12.13.0 March 30, 2023

Feature Enhancements

Kernel Support

Sysdig Agent now supports kernel versions 6.2.x.

Version Upgrade for Library Benchmark

Library Benchmark has been updated from version 1.5.0 to 1.7.1.

Fixed Vulnerabilities

Fixed CVE-2022-40897.

Collect PodDisruptionBudget Metrics

Added support for collecting Kubernetes PodDisruptionBudget metrics.

Send Start and Ready Time for Pods

Added support for sending start time and ready time for a pod when configured. For more information, see Customize KSM Collection.

Defect Fixes

Agent No Longer Fails When Customer ID Is Unspecified

Fixed a problem where an agent, stuck in a restart loop due to lack of configured customer ID, would fail to recognize when the configuration was subsequently updated to provide a customer ID.

Agent Retrieves JMX Metrics as Expected

Sysdig agent no longer generates heap dumps while fetching JMX metrics. Agent now performs a check whether the removed JVM is OpenJ9, and in such cases it will not attach to it.

12.12.1 March 12, 2023

This hotfix release provides the following defect fixes:

  • Podman containers running as unprivileged systemd services are detected correctly.
  • Container image metadata is reported correctly with Podman 4.x.

12.12.0 March 02, 2023

Feature Enhancements

Optimize Collecting Runtime Rules

The Falco rules optimizer has been enabled by default. This performs optimizations on the collection of runtime rules in conjunction with system call events to help reduce agent CPU usage.

Defect Fixes

Fixed Vulnerabilities

Fixed the following:

Fixed Proxy Connection

Fixed an issue where a proxy connection could fail if used in conjunction with the agent console.

Fixed nss_compat Records Parsing

The upgrade to v12.11.0 works as expected. The nss_compat records in /etc/passwd are now parsed correctly if data is missing, which fixes the issue of the agent not being ready after an upgrade.

12.11.0 February 13, 2023

Feature Enhancements

Search Container Password and Groups

Container password and groups are now searchable in container terminal shell.

Configurable Live Logs Sessions

Now live_logs sessions start with the last 100 lines by default, instead of 10.

To configure the tail length, edit the dragent.yaml file as follows:

live_logs:
  tail_lines: 200

Replace 200 with the desired number of lines.

Defect Fixes

Removed Proxy Passwords from Logs

The agent logs no longer contain plaintext proxy passwords.

Disable Containerd Events

You can configure Containerd events emission by using the events: >containerd: section in the .yaml configuration.

Enhanced Legacy Delegation

A fallback mechanism has been added to get the agent pod’s namespace. All the pods with label app: sysdig-agent and their namespace are now listed.

Display Correct CPU Utilization for Linux Hosts

Monitor UI now shows correct CPU utilization for the Linux host.

Communicate with Kubernetes Clusters with IPV6 Addresses

The cointerface process continue to communicate with Kubernetes clusters with IPs that only have IPV6 addresses.

Fix Cointerface Process Failure

Fixed a problem in agent 1v2.10.x that could cause the cointerface process to fail when k8s_delegated_nodes was set to 0.

Make CRI Socket Path Searchable in EKS+Bottlerocket Environments

The container runtime interface (CRI) socket path used by EKS+Bottlerocket is now added to the set of paths automatically searched by the agent.

Send Stale Makers for Failed Scrapes

Fixed an issue that could intermittently cause the agent to send invalid Prometheus values instead of stale markers for failed scrapes.

Agent Starts as Expected on Fedora

Fixed a problem of agent startup failure on cloud variants of Fedora v35+ when no kernel headers are available.

12.10.1 December 20, 2022

This hotfix fixes the issues discovered in the YAML tab of Advisor in Sysdig Monitor. Clicking the YAML tab now works as expected and continues to display YAML configuration for pods.

12.10.0 December 15, 2022

Feature Enhancements

Support for Secure Light Modes

A new agent mode, secure_light, has been introduced to provide you with a limited set of secure features. The features supported in this mode are:

Sysdig agents running in secure_light mode consume fewer resources than those running in secure mode.

For more information, see Secure Light.

Add Agent Configuration to Prevent Container Operations

A new agent-level configuration, ignore_container_action, has been added to prevent Sysdig agent from taking potentially disruptive container operations, such as kill, pause, stop, regardless of the policy.

This configuration is disabled by default. To enable it, add the following to the dragent.yaml file:

security:
  ignore_container_action: true

When the configuration is enabled and a policy instructs the agent to perform a container operation, the agent ignores the policy and creates an Info log message explaining the agent did not perform the action because of the configuration.

See also: Manage Threat Detection Policies | Containers

Improved Scope Matching

The scope matching for runtime policies has been improved by using equivalent container labels when corresponding kubernetes labels are temporarily not available.

The following settings determine the behavior. The example shows the default values.

security:
  use-container-labels-mapping: true
  container_labels_map:
    - "kubernetes.pod.name: container.label.io.kubernetes.pod.name"
	- "kubernetes.namespace.name: container.label.io.kubernetes.pod.namespace"

IMDSv2 Support on AWS Deployments

A new agent-level .yaml configuration, imds_version, should be set to 2 on all the deployments that require token-based communication with the Amazon Web Service (AWS) metadata service IMDSv2:

imds_version: 2

To continue using the IMDSv1 style AWS metadata requests, leave the configuration unchanged or set it to 1.

imds_version: 1

Fix Vulnerabilities

  • Updated the Go version used for Promscrape to 1.18.7 to resolve Common Vulneratibilities and Exposure (CVEs).
  • Updated Jackson library to resolve CVE-2022-42003 and CVE-2022-42004.
  • Upgraded snakeyaml to 1.32 in sdjagent to address CVE-2022-38752.

Disabled Superfluous Memory Consumption Checks

Disabled the agent watchdog from checking memory consumption when running in Kubernetes since Kubernetes has its own resource management. If you wish to re-enable the agent watchdog to check memory consumption when running in Kubernetes, set the following config parameter:

watchdog:
   check_memory_for_k8s: true

Report Additional Labels for Cost Advisor

Modified the default Kubernetes label filters to allow the collection of additional labels to identify the instance, region, zone, and the operating system of the nodes. The additional labels help to calculate costs associated with your infrastructure.

Identify Delegated Agents

Added the statsd_dragent_subproc_cointerface_delegated metric to indicate whether the agent is delegated or not.

Improved Retrieval of Container Metadata

Improved fetching container metadata when both Docker and CRI runtimes are available. This reduces problems where runtime policy events have missing container information.

Known Issues

The YAML tab in Advisor in Sysdig Monitor that displays pod structure, similar to a kubectl describe operation, might not work as expected. Clicking the YAML tab can lead to an agent restart, and as a result, a temporary loss of metrics.

As a workaround, disable it in the dragent.yaml file as follows:


k8s_command:
    enabled: false

Defect Fixes

Report all Storage Classes

The agent now reports all storage classes instead of just one. Earlier, the agent only sent one storage class from global_kubernetes in the metrics protobuf even when multiple storage classes existed in the cluster.

Match Group Name and User Name Appropriately in Events

Events now reports group.name and user.name correctly. Previously, there was an issue where root ID got resolved as N/A for containers in some cases.

Container Terminal Shell No Longer Returns N/A

Implemented container password and group lookup to prevent terminal shell in container returning N/A for the user.name.

Generate Command Execution Records for ARM

Fixed an issue with the activity audit where command execution records were not being generated on ARM processor systems, for top-level processes executed within a container, and with no associated TTY.

Reports Labels Correctly on Pod Redeployment

Fixed an issue with promscrape where the agent would report the old pod UID when a pod is redeployed. This led to having all the labels missing from the timeseries scraped from that pod.

Fix JMX Monitoring on Newer JRE Versions

Fixed an issue where JMX monitoring did not work correctly on newer JRE versions due to sdjagent exceptions

12.9.1 November 14, 2022

Defect Fixes

Fixed Legacy Proxy Connection Between Agent and Collector

The legacy mode of the proxy connection between the agent and the collector works as expected. You can continue to configure if need be.

Fix Enriching Prometheus Metrics with Labels Periodically

Fixed an issue where most labels would be dropped from Prometheus metrics every 5 minutes. This issue only affected the Kubelet jobs associated with Prometheus Integrations as well as the custom job configuration declared by the user.

Fix Vulnerabilities

Fixed the following vulnerabilities:

12.9.0 October 11, 2022

Feature Enhancements

Added New KSM Metrics

Sysdig agent now collects the following Kube-State-Metrics (KSM) ingress metrics:

  • kube_ingress_info
  • kube_ingress_labels
  • kube_ingress_created
  • kube_ingress_path
  • kube_ingress_tls

Also, the Sysdig agent collects the following KSM certificate signing request metrics:

  • kube_certificatesigningrequest_created
  • kube_certificatesigningrequest_condition
  • kube_certificatesigningrequest_labels
  • kube_certificatesigningrequest_cert_length

Expanded Node Resource Metrics

The Sysdig agent now sends all the Kubernetes node resource metrics to the Sysdig backend, rather than just CPU, memory, and pods. This allows you to query kube_node_status_capacity and kube_node_status_allocatable node metrics for the following resources.

  • cpu=<core>
  • ephemeral_storage=<byte>
  • pods=<integer>
  • attachable_volumes_*=<byte>
  • hugepages_*=<byte>
  • memory=<byte>

Additionally, the agent now supports a configuration to collect extended resource metrics on a node. To enable the agent to collect the extended resources, add the following to the dragent.yaml file:

k8s_node:
  extended_resources: true

Upgraded Vulnerable Go Packages in Promscrape v1

Upgraded Prometheus version and resolved vulnerabilities in Promscrape v1.

Retry CRI API Calls After Failed Async Attempts

The Sysdig agent now automatically retries querying the CRI API server after a failed attempt, with a backoff timeout strategy. This improves upon the former strategy of trying only once with a configurable delay value (cri:delay).

Added Error Traces when Open SSL Connection Fails

Added new error messages in the agent log to help identify the nature of connection problems in the collector, when they arise.

Report Taint Information for Kubernetes Nodes

The Sysdig agent will now send taint information associated with Kubernetes nodes. This enables you to query node taints using the kube_node_spec_taint metric in Sysdig Monitor.

Known Issues

The s390x architecture image is not available for v12.9.0; therefore, this version of the agent cannot be installed in zLinux. Note that using the latest tag for agent images on zLinux will not work until the next agent version is released.

Defect Fixes

Restarting Agent No Longer Causes Kernel Panic

Fixed an issue in the Sysdig agent’s kernel module that could cause a kernel panic when the agent was restarted.

Support Arbitrary Java Command Names

Added a configuration parameter to allow you to specify the command names to launch Java processes. This helps detect Java processes for JMX metric collection.

For example, if you want the agent to detect a process by the name of jsivm , while still detecting the other commands, add the following to dragent.yaml:

jmx:
  java_commands:
    - java
    - jsvc
    - jsivm

The values specified in dragent.yaml will override the default values, therefore, you need to include the defaults if you want to continue detecting them.

Captures No Longer Corrupted in Certain Hosts

Sysdig Monitor no longer produces corrupted Capture files in certain hosts in a cluster. Previously the Capture files were found corrupted when generated on a host selected from Explore > Hosts & Container.

Report Containers as Expected

Fixed an issue where containers would not be reported if the agent had issues communicating with the Kubernetes API server.

Upgraded psycopg2 Module

Upgraded psycopg2 module to v2.8.6 to fix issue where Postgres AppCheck fails to start due to missing libpq.

Build Kernel Modules on RHEL6

Fixed an issue preventing the kernel module from building on RHEL6 and other kernels of similar vintage.

Report Unschedulable Pods

Fixed an issue where unschedulable pods would not be reported by the agent.

Initialize Agent on Latest Kernels

Previously, the agent failed to initialize on the latest kernels, such as Ubuntu v22.04 and Fedora 35 and 36, with the following error:

gcc: error: unrecognized command-line option '-mharden-sls=all'

This has been fixed.

Disabled the Policy Scope Cache

The scope cache has been disabled by default to prevent it from getting stuck due to longer completion periods for Infra state.

Updated kube-bench and kubectl Binaries

Updated the Go version used for building kubectl and kube-bench binaries to address vulnerabilities.

Fixed Output Message in the Launch Sensitive Mount Container Rule

The Launch Sensitive Mount Container rule in the Suspicious Container Activity 2 policy no longer shows incorrect information in its output.

Fixed Output Fields in Custom Rules

Fixed an issue where not all the required secure event output fields were being generated by the agent.

12.8.1 August 29, 2022

Defect Fixes

Fix Vulnerabilities in Promscrape v1

Upgraded the Prometheus version and resolved vulnerabilities in Promscrape v1.

The agent can now read information on users and groups from /host/etc/passwd and /host/etc/group when the agent is running as a container.

Show Falco Events as Expected

Fixed a bug where the Falco output string for a rule would be cut on the first absent or empty field.

12.8.0 August 02, 2022

Feature Enhancements

Added a New Metric for Retrieving Kubernetes State

Added an internal metric, statsd_dragent_subproc_cointerface_ready, to indicate when the agent has pulled Kubernetes state from the API server.

Read Certificate Chain

Improved the agent’s handling of certificate chains. Previously, the agent would only accept the first certificate in a certificate chain and would attempt to verify all other certificates from the configured certificate store. This behavior is compliant with the TLS specification, but idiomatic usage in the wild requires the agent to accept intermediate certificates provided in the handshake as well. The agent will now accept these certificates when provided.

Falco Rules Optimizer

The optional feature Falco Rules Optimizer is now available. This feature increases the speed of syscalls evaluation against Falco rules by introducing indexing on the rules conditions and by caching partial rule condition evaluations. This feature is available in Sysdig agent, but not in open-source Falco.

To enable the feature:

  • Set falco_optimizer.enabled to true (default value is false).

New Falco Rules Parser

Starting from version 12.8.0, Sysdig agent will use a new Falco rules parser from OSS (open-source software) Falco. The new OSS Falco parser performs stricter grammar parsing and would fail on the following cases:

  • when \n is used instead of , in a list
  • when "[" is present in a rule definition
  • when \034 surrounded by " is present in a rule definition
  • when the or operation is used between lists instead of with the in operator. For example: condition: open_write and fd.filename is (list1 or list2)

When any of the above cases are present in a custom rules file, the agent fails to parse the respective rule and returns the following error:

Error, security_mgr:791: Could not load policies_v2 message:.

In such cases, edit the custom rules to correct or remove unparsable grammar.

Defect Fixes

Process Kubernetes Audit Events as Expected

Agent no longer throw errors while processing Kubernetes audit events when Kubernetes audit rules contain the endswith condition.

Upgraded Go Language Packages

Go language packages have been upgraded to fix vulnerabilities

Fix Vulnerabilities

Fixed the following vulnerabilities with Promscrape v2:

  • CVE-2015-3627
  • CVE-2021-3121
  • CVE-2020-14040
  • CVE-2014-6407
  • CVE-2014-9356
  • CVE-2014-9357
  • CVE-2022-23648
  • CVE-2022-27191
  • CVE-2021-41103
  • CVE-2020-15257
  • CVE-2014-9358
  • CVE-2021-21334
  • CVE-2020-13401
  • CVE-2014-5277
  • CVE-2020-13401
  • CVE-2020-8565
  • CVE-2021-32760
  • CVE-2021-20329
  • CVE-2019-11254
  • CVE-2021-4189
  • CVE-2020-8565
  • CVE-2021-4189
  • CVE-2021-3737
  • CVE-2021-3634
  • CVE-2021-3634
  • CVE-2021-3737
  • CVE-2022-1996

Detect Prometheus Targets Correctly

Fixed a problem that was causing new Prometheus targets to not be detected until an agent restart.

Intermittent Scraping Failure No Longer Causes Missing Metrics

Fixed an issue of missing metrics caused by intermittent metrics scraping failures.

Show Falco Events as Expected

Sysdig agent now throttles redundant secure events for compliance policies reducing the event noise.

Show Username Correctly in Policy Events

Fixed an agent build issue that rendered password and group functions unavailable. Linked the password and group from /host/etc inside the agent container so the username is correctly shown in policy events.

Fixed a Logging Issue in Promscrape v2

Fixed a logging issue with Promscrape v2. Log levels now take effect as expected when passed in with --log.level.

Fixed Agent Behaviour

Fixed an issue that might cause all the agents to behave as delegated.

12.7.1 July 06, 2022

Defect Fixes

Fixed memdump.size Issue

Fixed the memdump.size configuration, which was not accepted earlier.

Fixed Promscrape Crash Issue

Fixed an issue in Promscrape v2 where a node with a large number of pods and multiple containers per pod could crash.

Fixed Issue Affecting Two Agent Modes

Fixed a problem that caused agent subprocesses to be killed in nodriver mode. This affects the custom-metrics-only and monitor_light modes. For more information, see Configure Agent Modes.

12.7.0 June 28, 2022

Feature Enhancements

New Helm Chart

Sysdig released a unified Helm chart, sysdig-deploy with the following benefits:

  • Easier to deploy multiple components with one chart, rather than using multiple separate charts
  • Fewer errors by way of using common configuration for components
  • Auto-detection of certain configurations, including eBPF for Google Kubernetes Enginer (GKE) Contained-Optimized OS (COS) and endpoint region.

We will maintain the old version of Helm chart, sysdig chart for a period of six months. In this period, the Sysdig chart will be updated with new component versions and and defect fixes.

Live Logs

Sysdig Monitor displays Live Logs in Advisor to help you troubleshoot Kubernetes, which is the equivalent of running kubectl logs. Live logs are displayed on-demand and not stored by Sysdig.

Support Prometheus v2.32

Updated Prometheus scraper to version 2.32.

Metrics Collected in Custom Metrics Only Mode

When custom-metrics-only mode is used, no process metrics are collected. Additionally, only the metrics related to resources (for example, CPU and memory) are collected for containers and host.

Known Issues

While the agent is running, you might encounter an error similar to the following:

Error, security_rule:610: Could not parse rule xx from rules json array

The rule number in the error message might change depending on how many rules are defined.

This is a known issue related to failing to parse an experimental rule. The parser will skip this rule and will log the error message as above. The agent performance and policy evaluation will not be affected.

Defect Fixes

Remove Ceph App Checks

Fixed a problem where errors for obsoleted app checks would be shown when Ceph was running on the host.

Disable Timeseries Caching

Removed a configuration option that stopped Prometheus jobs reporting timeseries when the scrape failed temporarily.

Builds eBPF Probes in Bottlerocket

Fixed an issue that prevented eBPF probes from being built by the agent in Bottlerocket Environments.

Reports Infrastructure State Correctly

Fixed an issue where the Sysdig agent would opens a stream to Cointerface even when it is disabled. This resolves the issue of infrastructure state constantly resetting.

Send Only Supported Metrics in Nodriver Mode

Fixed an issue where unused container and process metrics were sent while in nodriver mode.

Change Log Level to DEBUG When Excessive Log Level Occurs

The excessive logging level occurs under specific conditions, for example, a pod whose used memory results in zero. This case seems to be normal for small pods using very little memory. A fix has been provided so that, when these conditions are detected, the log level for the message that is polluting the logs is brought from INFO to DEBUG.

Report Container Resource Limits and Requests Correctly

Fixed an issue where container resource limits and requests would appear as zero when no limit or request was configured.

12.6.0 May 16, 2022

Defect Fixes

Reloading Promscrape v2 No Longer Causes Dropping Scrape Targets

Reloading Promscrape v2 no longer causes some scrape targets to be dropped from sending metrics.

Prevented Duplicate Node Events

Resolved an issue where duplicate events were generated when a Kubernetes node was lost.

Agents Connect to SaaS Backend Through HTTP Proxy on Older Hosts

Fixed an issue related to SSL certificate verification when connecting through an HTTP proxy on an older host OS, such as CentOS 7.

Agent Refreshes Service Account Token as Expected

Connection with the Kubernetes API Server now works as expected. The Kubernetes client is configured to refresh the bearer token.

12.5.0 May 02, 2022

Feature Enhancements

Default Availability of Slim Agent

The agent installation defaults to the slim agent. The slim agent reduces the surface area for potential vulnerabilities as compared to the full agent, which implies increased security for your monitoring environment. For more information, see Agent Installation.

To continue using the regular agent:

  • Set slim.enabled to false in your Helm chart.

Monitoring Kubernetes Resources

Sysdig agent v12.5.0 and above no longer collects the Horizontal Pod Autoscaling (HPA) kube state metrics by default. To enable the agent to collect HPA kube state metrics, you must edit the agent configuration file, dragent.yaml, and include it along with the other resources you would like to collect. For more information, see Enable Kube State Metrics.

Container DriftControl: Detect and Prevent Drift in Container Runtime

Sysdig agent can now detect when a new executable was added to a container after a container has started up. The agent collects when a file was downloaded and made executable. When using prevention mode, the agent can also deny the process from ever running. A policy can also be used to define binaries that should be denied/excluded from being denied if they have been added after the container has started.

See also: Drift Policy

Disable Syscalls for Secure Modes

Switch syscall events are disabled for secure and secure light modes.

Known issues

  • An error message is displayed when the agent detects Ceph and attempts to run an obsoleted app check.
  • The Sysdig agent for ARM can restart when multiple containers are started in rapid succession on the host.

Defect Fixes

Agent on zLinux No Longer Restarts Due to Incorrect Detection of tid Collisions

The agent on s390x architecture (zLinux) has been fixed so the agent does not restart needlessly due to incorrect detection of too many tid collisions.

Reports Correct CronJob Version When Adding CronJob Parents

Fixed an issue causing CronJobs not to be reported as the parents of Job objects.

Agent No Longer Crashes During Abnormal Termination

Fixed an issue causing the agent to crash with a stack backtrace during certain abnormal termination situations.

Slow-Starting JVMs are Terminated Correctly

An incorrect detection of too many tid collisions on s390x architecture (zLinux) will no Longer cause the agent to restart periodically.

Fixed Kubernetes Fetch Issue

Fixed an issue that could prevent Kubernetes events from being correctly fetched.

Disable Watching HorizontalPodAutoscaler

Watching Horizontal Pod Autoscalers has been disabled by default to decrease load on Kubernetes API server. For more information, see Enable Kube State Metrics.

False Positive CVEs for Go Packages No Longer Reported

The Go compiler version has been upgraded to prevent being flagged with false-positive CVEs associated with older Go versions.

Secure Events Reports Correct Cluster Information

Secure events no longer report Kubernetes cluster name default when no cluster exists in the environment.

12.4.0 April 04, 2022

Feature Enhancements

Support for New Architectures

Installing agent on the following architecture are supported:

  • ARM (aarch64)

    aarch64 environments support AWS Graviton.

  • s390x (zLinux)

For more information, see Host Requirements for Agent Installation.

ARM support includes AWS EC2 Graviton platform

Custom-Metrics-Only Mode

A new agent mode, custom-metrics-only, has been introduced. It enables all custom metrics and Kubernetes state metrics but disables all the driver-based metrics.

Prevented Unnecessary Messages

Fixed a case where a message would be sent to reduce CPU usage when no changes were required.

Known Issues

Configure Node Lease to Decrease Resource Consumption

Incorrect configuration of Kubernetes lease can result in elevated memory usage in the Sysdig agent pods as well as increased load on the Kubernetes API server due to multiple agents querying for more information simultaneously. This also results in a significant amount of additional and unnecessary load on the Sysdig backend.

To decrease resource consumption:

  • Upgrade to Sysdig agent 12.5.0 which adapts to the non-optimal Kubernetes configuration.
  • Configure the Kubernetes lease functionality.
    • If you are using Helm, the latest versions of the Sysdig Agent Helm chart defaults to configuring the lease functionality automatically.
    • If you do not use Helm, the DaemonSet and ClusterRole YAML files are available in our gitbub repository.

For further assistance, contact Sysdig Support.

Prevent Periodic Agent Restarts on zLinux

An incorrect detection of too many tid collisions on s390x architecture (zLinux) can cause the agent to restart periodically.

To workaround this issue, set the following configuration option:

watchdog:
  analyzer_tid_collision_check_interval_s: 86400

This reduces the number of restarts to once a day instead of every ten minutes, which is the default value for the above configuration option. The value is in seconds; there are 86,400 seconds in a day.

This issue has been fixed in Sysdig agent v12.5.0.

Defect Fixes

Validate Promscrape Scrape Jobs

Fixed an issue causing errors with scrape jobs. Scrape jobs associated with Promscrape are now validated before scraping the endpoints.

Removed Irrelevant Warning Messages about App Checks

Removed unnecessary warning messages about app checks limits when app checks are disabled.

Slow-Starting JVMs Are No Longer Terminated

Slow starting JVMs can be terminated by sdjagent. For example, -XX:+AlwaysPreTouch with large heaps. This fix introduces additional configuration options to tune the delay between sdjagent detecting a started JVM process and an attempt to connect.

jmx:
  monitor_connect_timeout_ms: 5000
  management_agent_connect_delay_ms: 0

EVE Connector Works as Expected in Kubernetes

Fixed metadata incompatibility in profiling with Kubernetes versions above 1.20.

Name Change to Configuration Parameter

The falcobasline.max_drops_buffer_rate_percentage parameter has been corrected to falcobaseline.max_drops_buffer_rate_percentage. Notice the missing e in falcobasline in falcobasline.max_drops_buffer_rate_percentage. However, backward compatibility is ensured, and therefore, falcobasline.max_drops_buffer_rate_percentage can still be used.

12.3.1 March 03, 2022

Defect Fixes

Noisy Messages Silenced

Removed a kernel message from the driver that could generate spam when the syscall event buffer is full.

12.3.0 February 17, 2022

Feature Enhancements

Binaries Category for Falco Baseline

Added a new category, binaries, to the Falco baselines feature.

Support for Workload Information in Falco Baseline

Added workload information to Kubernetes context for Falco baselines.

Default Monitoring of Kubernetes Resources

The following Kubernetes resources are now monitored by default:

- persistentvolumeclaims
- persistentvolumes
- storageclasses
- horizontalpodautoscalers

Known Issues

IPv6 Addresses Are Saved Incorrectly When Adding Rules

Adding a new rule causes problem saving IPv6 address for both fd.net and fd.ip.

Defect Fixes

Fix Truncated Capture Files

Fixed a problem which caused the agent to generate truncated capture files.

Container Action Pause Work on Kops/GKE Clusters

Fixed the logic that determines the cgroup path for a container in containerd and made the freezer subsystem available to the agent in order to be able to pause and unpause it.

Agent Profiling Works as Expected

High CPU load no longer prevents CPU and memory profiles from being generated in the agent.

Agents Are Not Reset with Signal 11

Large and negative file descriptors are handled correctly so agents are no longer reset with signal 11.

12.2.1 February 07, 2022

Feature Enhancements

Manage Collecting Metadata from Individual Container Engines

Access to individual container engines from within the agent for fetching metadata can now be disabled via agent configuration.

For example, to disable docker, use the following configuration:

container_engines:
  docker: false

Known Issues

The Pause policy action is not working as expected in GKE, Elastic Kubernetes Service (EKS), and Openshift4 environments.

Defect Fixes

Policy Action “Kill” Is Correctly Triggered in GKE Environments

Policy action on GKE with containerd works as expected:

  • The container is stopped if HTTP proxy is enabled.
  • The status of the container is checked upon stop requests. If the status is not CONTAINER_EXITING, termination of the container is attempted with exponential backoff.

Agents Assign Username Correctly for Container Events

Fixed an issue that prevented the proc.name field from extracting the right user from the container started events. This issue was found in agent versions 12.2.0 and above.

12.2.0 January 25, 2022

Feature Enhancements

Improve Install Script to Support eBPF

A new option, bpf or -b is added to the native install script of Sysdig agent to support eBPF.

Enable 10s Flush by Default

By default, the agent collects metrics at 1-second granularity, then aggregates and sends them to the backend in 10-seconds intervals. If you want to use agent versions 12.2.0 or above with the on-prem Sysdig Platform versions below 3.5.0, set the 10s_flush_enable configuration to false to prevent compatibility issues.

The backend in our SaaS deployments continues to enable 10-second flush automatically for all agent versions 10.0.0 or above.

Improved Log Messages

Improved the log messages to report the errors encountered while configuring subprocess_resource_limits.

Handling Incorrect Metric Format

When scraping Prometheus metrics, the agent will set the type to PROMETHEUS_TYPE_INVALID if the metric is exported in an incorrect format or without a specified type. The metric will still be ingested by Sysdig and the query will fallback to gauge.

Known Issues

Processing Secure policy updates in the agent can take longer than it did in the previous releases, and, in some rare scenarios, it causes agent restarts.

Defect Fixes

Fix CVE-2020-29652 in Cointerface

Updated crypto go module to fix CVE-2020-29652.

Promscrape V2 No Longer Crashes on Pods with Multiple Containers

Prevent promscrape_v2 from crashing when a pod has multiple containers.

skip_events_by_type Works as Expected

Fixed an issue in the kernel probe, which prevented the skip_events_by_type feature from correctly filtering events by system call type.

Kubernetes State Is Transmitted as Expected

Fixed an issue where Kubernetes information and metrics would not be sent from the agent. This scenario arose when the agent was deployed in a namespace other than sysdig-agent, and the agent daemonset did not include the podinfo volume.

Agent Successfully Connects to JMX

Fixed an issue where agent wouldn’t connect to JMX on some applications/JVMs. This issue was originally observed on the WebSphere application and Liberty JVM.

Agent Updates Container Status as Expected

Fixed an issue where the agent would not update the container status it first received from the API server. The agent now updates the container statuses as it receives them from the API server.

Check for Invalid Log Level in sdjagent

Fixed an issue where using a log level of none caused sdjagent to crash.

App Checks Run as Expected on Non-Containerized Agent Installations

Fixed an issue preventing app checks running on non-containerized agent installation.

Native Install Doesn’t Support eBPF

Native install prevents insertion of Sysdig probe kernel module when the agent is installed with eBPF by using rpm or deb package.

Prevents Connection Attempts When Agent Encounters Errors

Connection attempts are prevented when the agent encounters errors while handling handshake messages.

Topics in This Section
2021 Archive

2021 Archive of Sysdig Agent release notes.

2020 Archive

2020 Archive of Sysdig Agent release notes.

2019 Archive

2019 Archive of Sysdig Agent release notes.