2023 Archive

2023 Archive of Sysdig Agent release notes.

12.19.0 December 20, 2023

Feature Enhancements

Changed HTTP Health Endpoint to Bind to Localhost

Changed the HTTP health endpoint to only bind to the localhost interface. If you are using Helm, upgrade to the Sysdig Agent Helm Chart v1.18.2 or higher. For more information, see Agent Health.

Export Additional Agent Health Metrics Using Prometheus Exporter

The Sysdig Agent now can use a Prometheus exporter to expose additional agent health metrics. For more information, see Agent Health.

Due to the sensitive nature of some of these metrics, you may want to ensure that the Prometheus exporter endpoints are not exposed outside of your cluster.

Added Profiling Fingerprint Generation to Secure Light Mode

You can now enable Profiling in secure_light mode by setting the falcobaseline.enabled parameter to true in the dragent.yaml or by specifying --set agent.sysdig.settings.falcobaseline.enabled=true if you install the agent via Helm chart.

Modified Audit Tap Message Delivery Policy

Audit Tap messages are now delivered even if they contain only file access records.

Defect Fixes

Improved Health Monitoring for Agent Subprocesses

Health monitoring for agent subprocesses now covers all subprocesses spawned.

Added Socket Timeout for the Proxy Connection to the Collector

Sysdig Agent now utilizes a socket timeout when connected to the collector via proxy. This allows the connection to recover faster without an agent restart when an issue occurs.

Reports Correct Values for Container CPU Usage in Kubernetes v1.26

Resolved an issue that impacted the calculation of CPU usage for containers in Kubernetes v1.26.

Detect App Check Metrics

Sysdig Agent now can successfully detect app check metrics. This fix enables SCM_RIGHTS to transfer file descriptors across all types of processes. Previously, if a file descriptor transferred via SCM_RIGHTS was for a socket serving app check metrics, the agent could not detect and query it for app check metrics. This presented as missing app check metrics after a process reload.

12.18.0 November 22, 2023

Feature Enhancements

Changed Default Matching Strategy for Falco Rules

Sysdig Agent now evaluates an event against all the rules, potentially triggering multiple alerts. In previous versions, the agent stopped evaluating rules after the first match.

To change the matching strategy, see Configure Falco Rule Matching Strategy.

Added a Metric to Detect Kubernetes Environment

The Sysdig Agent now indicates whether it is executing within a Kubernetes environment or not as part of the sysdig_agent_info metric.

Added an Option to Show Prometheus Scrape Jobs in Agent Console

Added the option to view all the Prometheus scrape jobs through the agent console with the prometheus scrape_jobs show command.

Report Policy Actions in Kubernetes Events

Sysdig Agent now supports reporting threat detection policy actions in Kubernetes events.

When the agent performs a stop, pause, or kill container action as defined in a rule, the agent will generate a Kubernetes event with the triggering action details and rule name. You can then see why actions were taken directly from the kubectl events, without having to explore the event feed in Sysdig Secure.

If you have deployed agent 12.18+ using Helm, the feature and its associated permissions are enabled by default. If you deploy agents manually, you must set Kubernetes role permissions.

See Agent Configuration Library for details.

Event Forwarding Directly from Sysdig Agent

It is possible to send Runtime Policy Events and Activity Audit events to SIEM platforms and logging tools directly from the agent. This enables event forwarding without exposing the data collection tool on the internet.

See Agent Local Forwarding for details.

Defect Fixes

Added Prometheus Support for FIPS-Enabled ARM Systems

Added Prometheus support for ARM systems running in FIPS mode.

Logging and Warning Messages in Secure Only Mode

Fixed an issue in secure-only mode where some warning log messages were being incorrectly generated.

Retrieve Kubernetes Service Ingresses for LoadBalancer Service Type

Addressed a defect that could hinder the retrieval of Kubernetes service ingresses when the service type is LoadBalancer. This issue had an impact on Cost Advisor.

Fixed overlayfs on Linux Hosts

Fixed overlayfs support on hosts running Linux kernel version 6.5.

Improved Container Runtime API Failure Handling

Added a failsafe to ensure that the container manager doesn’t loop forever trying to read data from the container API.

12.17.1 October 24, 2023

This hotfix is applicable only to Sysdig on-prem deployments; Sysdig SaaS users can disregard Sysdig Agent v12.17.1.

This hotfix release fixes the issue where the agent generates events in large numbers when Legacy Compliance is enabled due to incorrect throttling. With this fix, compliance events will be throttled as they were before 12.17.0.

12.17.0 October 17, 2023

If you are a Sysdig on-prem user, skip v12.17.0 and upgrade to Sysdig Agent v12.17.1. See 12.17.1 October 24 2023 for more information.

Feature Enhancements

Technical Preview of Universal eBPF

Universal eBPF driver is now available in Sysdig Agent as an opt-in feature. The Universal eBPF driver is pre-built and embedded in the agent binary. It does not require kernel headers or to build tools from the target system to function and thereby removes the need for users to build or download probes on supported systems. To learn more about the capability, see Universal eBPF.

Use Protocol Buffer to Communicate to Kubernetes API Server

Cointerface uses Google Protocol Buffers as a wire format for communicating with the Kubernetes API server.

Update OpenSSL Library to OpenSSL v3.1 and Include a FIPS-Validated Crypto Module

In light of OpenSSL v1.1.1 reaching end-of-life, this release updates its bundled OpenSSL libraries to v3.1.3.

Additionally, this release bundles a FIPS(Federal Information Processing Standards)-validated OpenSSL crypto module with the agent. Adding the crypto module removes the requirement for user-provided, FIPS-validated OpenSSL shared libraries when the fips_mode configuration parameter is set to true.

This update breaks the agent’s backward compatibility with OpenSSL v1.1.1. If you have configured the openssl_lib parameter, do one of the following:

  • Provide OpenSSL v3.1 shared libraries
  • Remove the parameter and rely on the bundled OpenSSL shared libraries

End of Support for OpenShift v3

Sysdig Agent versions beyond 12.17.0 will no longer be supported on OpenShift 3. v12.17.0 will be the last version supporting OpenShift 3.

Defect Fixes

Prevent Transition During Restarts

The agent will no longer release the Kubernetes delegation lease during tear-down to avoid unwanted transitions during restarts.

Policy Scoping in Fargate Now Respects Agent Labels

Fargate agents will no longer skip agent labels when performing policy scoping.

Display Resolved IPs in the Network Security Policy Egress

The agent uses improved logic to resolve services and endpoints, and therefore, the network communications in some namespaces will no longer be dropped as unresolved.

Use get_mm_exe_file()

A safer version of the Linux kernel API call is used where get_mm_exe_file() is available.

Show Correct Kubernetes Status

Fixed defects in the Kubernetes status reporting. The kube_workload_status_available and kube_workload_status_unavailable metrics now report correct values even when the cluster node count changes and the Kubernetes status reflects the state correctly after the cointerface switches run modes.

Prevent Unintended Agent Restart

A defect was fixed where an invalid message from the backend caused an unintended agent restart.

Store Device Metrics as Expected

A defect was fixed where I/O metrics for devices were not stored.

Display Kubernetes Cluster Association Correctly

A defect was fixed which caused incorrect agent association with Kubernetes clusters on the Sysdig Agents page in the Integrations UI.

Display Correct Timeseries Count in Prometheus Logs

Filtered timeseries counts in Prometheus statistics logs are now reported correctly.

12.16.3 October 15, 2023

This hotfix delivers the following:

  • Addressed the following vulnerabilities:

  • Fixed the kernel null pointer dereference on Linux kernel 6.5. Support for is_exe_upper_layer will be disabled (both kmod and bpf) for those kernels.

12.16.2 September 28, 2023

Improved security for connections between Sysdig Agent and Apache Mesos.

Sysdig Agent primarily connects to Apache Mesos using cURL. When negotiating a TLS (Transport Layer Security) or SSL (Secure Sockets Layer) connection, the Mesos server sends a certificate indicating its identity. cURL verifies the authenticity of the TLS certificate before establishing the connection with the Sysdig Agent. Before this fix, Mesos would not verify the TLS certificate if the connection was over HTTPS.

To establish a cURL connection to Apache Mesos as of Sysdig Agent v12.16.2:

  • Ensure that the CA cert indicates the Mesos server where the agent will connect. Check that the Mesos server name given in the URL matches the one in the certificate.

12.16.1 September 11, 2023

This hotfix release delivers the following:

12.16.0 August 08, 2023

Feature Enhancements

Default Availability of Process Tree in Sysdig Secure

Process lineage will be available for every event and is enabled by default starting from this version of the agent. The process tree will be visible in the Events detail pane for events related to workloads that are triggered from that point on.

Supports Control Group v2

Control groups v2 (cgroups v2) are now supported in the Sysdig Agent. In particular, the v1 freezer subsystem is no longer mounted when using cgroups v2, as it caused potential compatibility issues in the past.

View Agent Threads for Improved Performance Analysis

The Sysdig Agent threads on Linux x86 platforms have been named to facilitate better analysis of agent performance. Previously, they were named after the default process name, dragent. Now, these threads have descriptive names, with suffixes dr- or dr=. For example, dr-monitor and dr=sinsp_evnt_. The thread name is usually a truncation of the nearest unique function name.

Collects Node Labels

Sysdig Agent can by default collect the node-role.kubernetes.io/* labels set on nodes.

Known Issues

Container Limits to Drift Control

  • For kernel versions below v5.13, Drift Control can monitor up to 128 containers per node.
  • For kernel versions v5.13 or above, modify the container limit using one of the following methods:

    • Open the sysctl -n fs.fanotify.max_user_groups file and set the new value by using sysctl -w fs.fanotify.max_user_groups=<new_limit>.

    • Open the cat /proc/sys/fs/fanotify/max_user_groups file and run echo <new_limit> > /proc/sys/fs/fanotify/max_user_groups.

      Replace <new_limit> with your choice of container limit.

Agent Logs Show Errors for On-Prem Installations in Secure Only Mode

When connecting to an on-prem backend with Secure Only mode, the agent doesn’t connect successfully unless you add the 60s_flush_enable: true configuration under sysdig.settings in the agent configuration file.

Defect Fixes

Removed Compliance Manager Support

Compliance manager functionality has been removed from Sysdig Agent. The feature was no longer supported, but it appeared in a security audit as having a vulnerability. For this reason, the functionality has been dismissed.

Ignores Non-Running Pods for Scraping

The Prometheus k8s-pods job configuration has been modified to drop scrapes from non-running pods.

Enable FIPS-Validated Crypto Module for Agent-to-Backend communication

The agent can now use a FIPS(Federal Information Processing Standards)-validated crypto module for encrypting communication with the Sysdig backend. This feature requires a user-provided, FIPS-validated OpenSSL v1.1.1 shared library to function properly. See the Configuration Library for more information.

Retry Sending Secure Events

The Sysdig Agent now retries sending secure events in cases where the agent disconnects and reconnects.

Adds Missing Health Metrics in Secure Modes

An additional metric is collected in the secure and secure_light modes. The protobuf output for secure and secure_light mode now includes an aggrSamplingRatio aggregation field, weighted to the negotiated metrics interval.

12.15.0 June 28, 2023

Feature Enhancements

Process Tree

This version of the Sysdig Agent adds support in Sysdig Secure for the Process Tree visualization which enriches the Events feed for workload-based events. This feature helps you identify all the processes that led up to the offending process.

To enable this feature:

  1. Add the following configuration to the values.yaml file associated with the sysdig.deploy chart:

     agent:
   sysdig:
     settings:
       enrich_with_process_lineage: true

    You can use the sysdig.settings parameter of the agent subchart to merge this configuration into your existing values.yaml file.

  2. Log in to Sysdig Secure as administrator and select Settings > User Profile. Scroll down to Sysdig Labs and toggle the feature on.

    The process tree will be visible in the Events detail pane for the events related to workloads that are triggered from that point on.

Added Support for Java 7

In Sysdig Agent versions 12.10.0 to 12.14.1, a Java dependency was upgraded to a version that didn’t support Java 7. As a result, those versions cannot run the Java process which collects Java Management Extensions (JMX) metrics on any Java 7 Java Development Kits (JDKs)/Java Runtime Environments (JREs). This release downgrades the dependency back to a version that supports Java 7.

Added Support for Node Cost Metrics

Sysdig Agent now supports node cost metrics when using the thin cointerface.

Building Probes for Airgapped Environments

Effective July 28, 2023, a minor change has been enforced to the process of building probes for airgapped environments. See Airgapped Agent Installation for further details.

Added Sysdig Secure Rule for Detecting Fileless Attacks

Sysdig Secure has added the ability to detect fileless attacks using a new Falco rule on the Sysdig Threat Detection managed policy. See also: SaaS release note.

Vulnerability Fixes

Addressed CVE-2023-0286 by upgrading the OpenSSL version in the agent to 1.1.1t.

Defect Fixes

Metrics Parity Between Secure and Secure Light Modes

The Sysdig Agent will now report the same set of metrics in both secure and secure_light modes, which means that the program metrics in secure mode will also be restricted to the dragent process or container.

Enhanced Execution Time Accounting

Fixed system execution time accounting for certain events that would cause incorrect reporting of agent I/O metrics.

Added Support for s390x for Ubuntu

Recent s390x Linux distributions, including Ubuntu v20.04, require the compiler to support the -march=z13/-mtune=z15 flags when building kernel modules. The gcc version used in agent-kmodule image for the s390x platform has been upgraded to gcc-12, which supports the required flags.

12.14.1 May 16, 2023

This hotfix release provides the following enhancements:

Added Support for Kernel Version 6.3

The kernel module has been updated to support Linux kernel version 6.3.

Fixed Vulnerabilities

Resolved CVE-2023-28840 in promscrape.

Fixed Probe Build Errors on RHEL6

Fixed probe build errors on RHEL6 hosts.

12.14.0 May 08, 2023

Feature Enhancements

Enhanced Console Logging

The console log messages sent to stderr have been restricted to Warning or higher priority only. All the lower priority console log messages are sent to stdout. This reduces noise.

Optimized Filtering

  • The following redundant elements were removed from the Falco rules optimizer:
    • The evt.type/evt.dir fields
    • The evt.arg.res = 0 checks
  • The redundant container.id != host field was removed from conditions while indexing with the rules optimizer.

Improved Drift Detection in Sysdig Secure

With agent 12.14.0, drift detection is improved in Sysdig Secure. Drift detection requires a minimum kernel version of 3.18 and drift prevention requires a minimum kernel version of 5.0.

For more information, see Enable Drift.

Added Logging to Detect Incorrect Collector Endpoints

Added detection for invalid HTTP responses on connection.

Enabled Default Scraping of Docker Containers

Sysdig Agent now supports scraping Prometheus metrics from Docker containers by default. Scraping is based on container labels.

Dependencies

Added fmt Library

Added the fmt library to the Agent dependencies. The agent currently does not use this library.

Upgraded Library TBB

The TBB (Intel’s Threading Building Blocks) library has been upgraded to oneTBB v2021.8.0.

Upgraded Boost Library

The Boost library used by Sysdig Agent has been upgraded to v1.81.0.

Known Issues

  • Agent not compatible with GKE Autopilot running Kubernetes v1.26
  • Agent not compatible with Kernel v6.3

Defect Fixes

Restarting the Agent No Longer Causes Premature Process Termination

The SysV init script for RPM-based distributions now takes agent shutdown time into account, avoiding premature SIGKILL.

PID tracking is now enabled for systemd-sysv-generator.

Exclude JVM from Monitoring

Agent can now exclude some Java Virtual Machines (JVMs) from being monitored.

A set of exclusion rules can be defined in the agent’s config. Each rule is a property/pattern pair: when the value of the given Java property matches the pattern, a process of that JVM is excluded from being monitored. For example, the following configuration will exclude OpenJ9-based JVMs from being monitored:

jmx:
  jvm_exclude:
    - property: java.vm.name
      pattern: .+OpenJ9.+

Previously, this functionality was hardcoded to reject OpenJ9, but this is no longer the case. If you observe heap dumps when monitoring OpenJ9, you should add the configuration above to your dragent.yaml file.

Recover from Handshake Errors Between Agent and Collector

Fixed an issue preventing the agent from recovering after a bad protocol handshake.

12.13.0 March 30, 2023

Feature Enhancements

Kernel Support

Sysdig Agent now supports kernel versions 6.2.x.

Version Upgrade for Library Benchmark

Library Benchmark has been updated from version 1.5.0 to 1.7.1.

Fixed Vulnerabilities

Fixed CVE-2022-40897.

Collect PodDisruptionBudget Metrics

Added support for collecting Kubernetes PodDisruptionBudget metrics.

Send Start and Ready Time for Pods

Added support for sending start time and ready time for a pod when configured. For more information, see Customize KSM Collection.

Defect Fixes

Agent No Longer Fails When Customer ID Is Unspecified

Fixed a problem where an agent, stuck in a restart loop due to lack of configured customer ID, would fail to recognize when the configuration was subsequently updated to provide a customer ID.

Agent Retrieves JMX Metrics as Expected

Sysdig agent no longer generates heap dumps while fetching JMX metrics. Agent now performs a check whether the removed JVM is OpenJ9, and in such cases it will not attach to it.

12.12.1 March 12, 2023

This hotfix release provides the following defect fixes:

  • Podman containers running as unprivileged systemd services are detected correctly.
  • Container image metadata is reported correctly with Podman 4.x.

12.12.0 March 02, 2023

Feature Enhancements

Optimize Collecting Runtime Rules

The Falco rules optimizer has been enabled by default. This performs optimizations on the collection of runtime rules in conjunction with system call events to help reduce agent CPU usage.

Defect Fixes

Fixed Vulnerabilities

Fixed the following:

Fixed Proxy Connection

Fixed an issue where a proxy connection could fail if used in conjunction with the agent console.

Fixed nss_compat Records Parsing

The upgrade to v12.11.0 works as expected. The nss_compat records in /etc/passwd are now parsed correctly if data is missing, which fixes the issue of the agent not being ready after an upgrade.

12.11.0 February 13, 2023

Feature Enhancements

Search Container Password and Groups

Container password and groups are now searchable in container terminal shell.

Configurable Live Logs Sessions

Now live_logs sessions start with the last 100 lines by default, instead of 10.

To configure the tail length, edit the dragent.yaml file as follows:

live_logs:
  tail_lines: 200

Replace 200 with the desired number of lines.

Instance Metadata Service (IMDS)V2 Support

The agent can now dynamically switch its AWS metadata query version to IMDSv2 if the initial metadata lookup fails with an HTTP 401 error. This retry mechanism will be activated each time the configuration settings are loaded without explicitly setting imds_version to 2 in the dragent.yaml file, as it will result in the same HTTP error on the first subsequent metadata call. For more information on configuration options, see Configuration Library.

Defect Fixes

Removed Proxy Passwords from Logs

The agent logs no longer contain plaintext proxy passwords.

Disable Containerd Events

You can configure Containerd events emission by using the events: >containerd: section in the .yaml configuration.

Enhanced Legacy Delegation

A fallback mechanism has been added to get the agent pod’s namespace. All the pods with label app: sysdig-agent and their namespace are now listed.

Display Correct CPU Utilization for Linux Hosts

Monitor UI now shows correct CPU utilization for the Linux host.

Communicate with Kubernetes Clusters with IPV6 Addresses

The cointerface process continue to communicate with Kubernetes clusters with IPs that only have IPV6 addresses.

Fix Cointerface Process Failure

Fixed a problem in agent 1v2.10.x that could cause the cointerface process to fail when k8s_delegated_nodes was set to 0.

Make CRI Socket Path Searchable in EKS+Bottlerocket Environments

The container runtime interface (CRI) socket path used by EKS+Bottlerocket is now added to the set of paths automatically searched by the agent.

Send Stale Makers for Failed Scrapes

Fixed an issue that could intermittently cause the agent to send invalid Prometheus values instead of stale markers for failed scrapes.

Agent Starts as Expected on Fedora

Fixed a problem of agent startup failure on cloud variants of Fedora v35+ when no kernel headers are available.