Sysdig Agent Release Notes
12.16.1 September 11, 2023
This hotfix release delivers the following:
Addressed critical and high vulnerabilities related to Go v1.19:
Fixed an issue of container detection not working properly for Podman containers:
- on Alpine Linux
- when running the Sysdig Agent in a container
12.16.0 August 08, 2023
Supports Control Group v2
Control groups v2 (cgroups v2) are now supported in the Sysdig Agent. In particular, the v1 freezer subsystem is not mounted when using cgroups v2, which caused potential compatibility issues.
View Agent Threads for Improved Performance Analysis
The Sysdig Agent threads on Linux x86 platforms have been named to facilitate analysis of agent performance better. Previously, they were named after the default process name,
dragent. Now, these threads have descriptive names, with suffixes
dr=. For example,
dr=sinsp_evnt_. The thread name is usually a truncation of the nearest unique function name.
Collects Node Labels
Sysdig Agent can by default collect the
node-role.kubernetes.io/* labels set on nodes.
Container Limits to Drift Control
- For kernel versions below v5.13 Drift Control can monitor up to 128 containers per node.
- For kernel versions v5.13 or above, modify the container limit using one of the following methods:
sysctl -n fs.fanotify.max_user_groupsfile and set the new value by using
sysctl -w fs.fanotify.max_user_groups=<new_limit>.
cat /proc/sys/fs/fanotify/max_user_groupsfile and run
echo <new_limit> > /proc/sys/fs/fanotify/max_user_groups.
<new_limit>with your choice of container limit.
Agent Logs Show Errors for On-Prem Installations in Secure Only Mode
When connecting to an on-prem backend with Secure Only mode, the agent doesn’t connect successfully unless you add the
60s_flush_enable: true configuration under
sysdig.settings in the agent configuration file.
Removed Compliance Manager Support
Compliance manager functionality has been removed from Sysdig Agent. The feature was not supported anymore and yet it appeared in a security audit as having a vulnerability. For these reasons, this functionality has been dismissed.
Ignores Non-Running Pods for Scraping
k8s-pods job configuration has been modified to drop scrapes from non-running pods.
Enables FIPS Mode
The agent can now enable FIPS-compliant(Federal Information Processing Standards) mode even if the whole system isn’t in FIPS-compliant mode.
Resends Unacknowledged Policy Events
Sysdig Agent attempts at resending unacknowledged policy events when the collector disconnects.
Adds Missing Health Metrics in Secure Modes
An additional metric is collected in the
secure_light modes. The protobuf output for
secure_light mode now includes an
aggrSamplingRatio aggregation field, weighted to the negotiated metrics interval.
12.15.0 June 28, 2023
This version of the Sysdig Agent adds support in Sysdig Secure for the Process Tree visualization which enriches the Events feed for workload-based events. This helps with identifying all the processes that led up to the offending process.
To enable this feature:
Add the following configuration to the
values.yamlfile associated with the
agent: sysdig: settings: enrich_with_process_lineage: true
You can use the
sysdig.settingsparameter of the
agentsubchart to merge this configuration into your existing
Log in to Sysdig Secure as administrator and select Settings | Sysdig Labs to toggle the feature on.
The process tree will be visible in the Events detail pane for the events related to workloads that are triggered from that point on.
Added Support for Java 7
In Sysdig Agent versions 12.10.0 to 12.14.1, a Java dependency was upgraded to a version that didn’t support Java 7. As a result, those versions cannot run the Java process which collects JMX metrics on any Java 7 JDKs/JREs. This release downgrades the dependency back to a version that supports Java 7.
Added Support for Node Cost Metrics
Sysdig Agent now supports node cost metrics when using the thin cointerface.
Building Probes for Airgapped Environments
Effective July 28, 2023, a minor change has been enforced to the process of building probes for airgapped environments. See Airgapped Agent Installation for further details.
Added Sysdig Secure Rule for Detecting Fileless Attacks
Sysdig Secure has added the ability to detect fileless attacks using a new Falco rule on the Sysdig Threat Detection managed policy. See also: SaaS release note.
Addressed CVE-2023-0286 by upgrading the OpenSSL version in the agent to 1.1.1t.
Metrics Parity Between Secure and Secure Light Modes
The Sysdig Agent will now report the same set of metrics in both
secure_light modes, which means that the program metrics in
secure mode will also be restricted to the
dragent process or container.
Enhanced Execution Time Accounting
Fixed system execution time accounting for certain events which would cause incorrect reporting of agent I/O metrics.
Support for s390x for Ubuntu
Recent s390x Linux distributions, including Ubuntu v20.04, require the compiler to support the
-march=z13/-mtune=z15 flags when building kernel modules. The
gcc version used in
agent-kmodule image for the s390x platform has been upgraded to
gcc-12, which supports the required flags.
12.14.1 May 16, 2023
This hotfix release provides the following enhancements:
Added Support for Kernel Version 6.3
The kernel module has been updated to support Linux kernel version 6.3.
Resolved CVE-2023-28840 in
Fixed Probe Build Errors on RHEL6
Fixed probe build errors on RHEL6 hosts.
12.14.0 May 08, 2023
Enhanced Console Logging
The console log messages sent to
stderr have been restricted to warning or higher priority only. All the lower priority console log messages are sent to
- When using the Falco rules optimizer, remove the following redundant elements:
evt.arg.res = 0checks
- When using the rules optimizer, remove the redundant
container.id != hostfield from conditions while indexing.
Improved Drift Detection in Sysdig Secure
With agent 12.14.0, drift detection is improved in Sysdig Secure. Drift detection requires a minimum kernel version of 3.18 and drift prevention requires a minimum kernel version of 5.0.
For the drift feature (both detection and prevention) to work in the 12.14 agent release, set the following in:
The agent config file:
drift_killer: enabled: true
Added Logging to Detect Incorrect Collector Endpoints
Added detection for invalid HTTP responses on connection.
Enabled Default Scraping of Docker Containers
Sysdig Agent now supports scraping Prometheus metrics from Docker containers by default. Scraping is based on container labels.
Add fmt Library: Added the fmt library to the Agent dependencies. The agent currently does not use this library.
Upgrade Library TBB: The TBB (Intel’s Threading Building Blocks) library has been upgraded to oneTBB v2021.8.0.
Upgrade Boost Library: The Boost library used by Sysdig Agent has been upgraded to v1.81.0.
- Agent not compatible with GKE Autopilot running Kubernetes v1.26
- Agent not compatible with Kernel v6.3
Restarting the Agent No Longer Causes Premature Process Termination
The SysV init script for RPM-based distributions now takes agent shutdown time into account, avoiding premature SIGKILL.
PID tracking is now enabled for
Exclude JVM from Monitoring
Agent can now exclude some JVM’s from being monitored.
A set of exclusion rules can be defined in the Agent’s config. Each rule is a property/pattern pair: when the value of the given Java property matches the pattern, a process of that JVM is excluded from being monitored. For example, the following configuration will exclude OpenJ9-based JVMs from being monitored:
jmx: jvm_exclude: - property: java.vm.name pattern: .+OpenJ9.+
Previously, this functionality was hardcoded to reject OpenJ9, but this is no longer the case. If you observe heap dumps when monitoring OpenJ9, you should add the configuration above to your dragent.yaml.
Recover from Handshake Errors Between Agent and Collector
Fixed an issue causing the agent not recovering from a bad protocol handshake.
12.13.0 March 30, 2023
Supports kernel versions 6.2.x.
Version Upgrade for Library Benchmark
Library Benchmark has been updated from version 1.5.0 to 1.7.1.
Collect PodDisruptionBudget Metrics
Added support for collecting Kubernetes PodDisruptionBudget metrics.
Send Start and Ready Time for Pods
Added support for sending start time and ready time for a pod when configured. For more information, see Customize KSM Collection.
Agent No Longer Fails When Customer ID Is Unspecified
Fixed a problem where an agent, which is stuck in a restart loop due to lack of configured customer ID, would fail to recognize when the configuration was subsequently updated to provide a customer ID.
Agent Retrieves JMX Metrics as Expected
Sysdig agent no longer generates heap dumps while fetching JMX metrics. Agent now performs a check whether the removed JVM is OpenJ9, and in such case it will not attach to it.
12.12.1 March 12, 2023
This hotfix release provides the following defect fixes:
- Podman containers running as unprivileged systemd services are detected correctly.
- Container image metadata is reported correctly with Podman 4.x.
12.12.0 March 02, 2023
Optimize Collecting Runtime Rules
The Falco rules optimizer has been enabled by default. This performs optimizations on the collection of runtime rules in conjunction with system call events to help reduce agent CPU usage.
Fixed the following:
Fix Proxy Connection
Fixed an issue where proxy connection could fail if used in conjunction with the agent console.
Fix nss_compat Records Parsing
Upgrade to v12.11.0 works as expected. The
nss_compat records in
/etc/passwd are now parsed correctly if data is missing, which fixes the issue of the agent not being ready after an upgrade.
12.11.0 February 13, 2023
Search Container Password and Groups
Container password and groups are now searchable in container terminal shell.
Configurable Live Logs Sessions
live_logs sessions with the last 100 lines by default, instead of 10.
To configure the tail length, edit the
dragent.yaml file as follows:
live_logs: tail_lines: 200
Remove Proxy Passwords from Logs
The agent logs no longer contain plaintext proxy passwords.
Disable Containerd Events
You can configure Containerd events emission by using the
events: >containerd: section in the YAML configuration.
Enhance Legacy Delegation
A fallback mechanism has been added to get the agent pod’s namespace. All the pods with label
app: sysdig-agent and their namespace will be listed now.
Display Correct CPU Utilization for Linux Hosts
Monitor UI now shows correct CPU utilization for the Linux host.
Communicate with Kubernetes Clusters with IPV6 Addresses
The cointerface process continue to communicate with Kubernetes clusters with IPs that only has IPV6 addresses.
Fix Cointerface Process Failure
Fixed a problem in agent 1v2.10.x that could cause the cointerface process to fail when
k8s_delegated_nodes was set to 0.
Make CRI Socket Path Searchable in EKS+Bottlerocket Environments
The CRI socket path used by EKS+Bottlerocket is added to the set of paths automatically searched by the agent.
Send Stale Makers for Failed Scrapes
Fixed an issue that could intermittently cause the agent to send invalid Prometheus values instead of stale markers for failed scrapes.
Agent Starts as Expected on Fedora
Fixed agent startup failure on cloud variants of Fedora v35+ when no kernel headers are available.
12.10.1 December 20, 2022
This hotfix fixes the issues discovered in the YAML tab in Advisor in Sysdig Monitor. Clicking the YAML tab works as expected and continues to display YAML configuration for pods.
12.10.0 December 15, 2022
Support for Secure Light Mode
A new agent mode,
secure_light, has been introduced to provide you with a limited set of secure features. The features that are supported in this mode are:
Sysdig agent running in
secure_light mode consumes fewer resources than that of running in the secure mode.
For more information, see Secure Light.
Add Agent Configuration to Prevent Container Operations
A new agent-level configuration,
ignore_container_action, has been added to prevent Sysdig agent from taking potentially disruptive container operations, such as
stop, regardless of the policy.
To enable this option, add the following to the
security: ignore_container_action: true
This configuration is disabled by default.
When the configuration is enabled and a policy instructs to do a container operation, the agent ignores the policy and creates an
Info log message stating that agent did not take the action because of the configuration.
Improved Scope Matching
The scope matching for runtime policies has been improved by using equivalent container labels when corresponding kubernetes labels are temporarily not available.
The following settings determine the behavior. The example shows the default values.
security: use-container-labels-mapping: true container_labels_map: - "kubernetes.pod.name: container.label.io.kubernetes.pod.name" - "kubernetes.namespace.name: container.label.io.kubernetes.pod.namespace"
IMDSv2 Support on AWS Deployments
A new agent-level configuration,
imds_version, should be set to
2 on all the deployments that require token-based communication with the AWS metadata service (IMDSv2).
To continue using the IMDSv1 style AWS metadata requests, leave the configuration unchanged or set it to 1.
- Updated the Go version used for Promscrape to 1.18.7 to resolve CVEs.
- Updated Jackson library to resolve CVE-2022-42003 and CVE-2022-42004
- Upgraded snakeyaml to 1.32 in sdjagent to address CVE-2022-38752
Disable Checking Memory Consumption
Disabled the agent watchdog from checking memory consumption when running in Kubernetes since Kubernetes has its own resource management. If you wish to re-enable the agent watchdog to check memory consumption when running in Kubernetes, set the following config parameter:
watchdog: check_memory_for_k8s: true
Report Additional Labels for Cost Advisor
Modified the default Kubernetes label filters to allow collecting additional labels to identify the instance, region, zone, and the operating system of the nodes. The additional labels help calculating cost associated with your infrastructure.
Identify Delegated Agents
statsd_dragent_subproc_cointerface_delegated metric to indicate whether the agent is delegated or not.
Improved Retrieval of Container Metadata
Improved fetching container metadata when both Docker and CRI runtimes are available. This reduces problems where runtime policy events have missing container information.
The YAML tab in Advisor in Sysdig Monitor that displays pod structure, similar to a
kubectl describe operation, might not work as expected. Clicking the YAML tab can lead to an agent restart, and as a result, a temporary loss of metrics.
As a workaround, disable it in the
dragent.yaml file as follows:
k8s_command: enabled: false
Report all Storage Classes
The agent now reports all the storage classes instead of just one. Earlier, the agent only sent one storage class from
global_kubernetes in the metrics protobuf even when multiple storage classes exist in the cluster.
Match Group Name and User Name Appropriately in Events
Events now reports
user.name correctly. This was caused by an issue where root ID being resolved as NA for containers in some cases.
Container Terminal Shell No Longer Returns N/A
Implemented container password and group lookup to prevent terminal shell in container returning
N/A for the
Generate Command Execution Records for ARM
Fixed an issue with the activity audit where command execution records not being generated on ARM processor systems, for top-level processes executed within a container, and with no associated TTY.
Reports Labels Correctly on Pod Redeployment
Fixed an issue with
promscrape where the agent would report the old pod
UID when a pod is redeployed. This led to having all the labels missing from the timeseries scraped from that pod.
Fix JMX Monitoring on Newer JRE Versions
Fixed an issue where JMX monitoring did not work correctly on newer JRE versions due to sdjagent exceptions
12.9.1 November 14, 2022
Fix Legacy Proxy Connection Between Agent and Collector
The legacy mode of the proxy connection between the agent and the collector works as expected. You can continue to configure if need be.
Fix Enriching Prometheus Metrics with Labels Periodically
Fixed an issue where most labels would be dropped from Prometheus metrics every 5 minutes. This issue only affects the Kubelet jobs associated with Prometheus Integrations as well as the custom job configuration declared by the user.
Fixed the following vulnerabilities:
12.9.0 October 11, 2022
Add New KSM Metrics
Sysdig agent now collects the following KSM ingress metrics:
Also, the Sysdig agent collects the following KSM certificate signing request metrics:
Send Node Resource Metrics
The Sysdig agent will send all the Kubernetes node resource metrics rather than just CPU, memory, and pods to the Sysdig backend. This allows you to query
kube_node_status_allocatable node metrics for the following resources.
Additionally, the agent now supports collecting extended resource metrics on a node, if configured. To enable the agent to collect the extended resources, add the following to the
k8s_node: extended_resources: true
Upgrade Vulnerable Go Packages in Promscrape V1
Upgraded Prometheus version and resolved vulnerabilities in Promscrape v1.
Retry CRI API Calls After Failed Async Attempts
The Sysdig agent can automatically retry querying the CRI API server, wth a backoff timeout strategy. This improves upon the former strategy of trying only once with a configurable delay value (
Send Node Resource Metrics
The Sysdig agent will send all the Kubernetes node resource metrics rather than just CPU, memory, and pods to the Sysdig backend. This enables you to use Sysdig Monitor to query node resource metrics via the
Add Error Traces when Open SSL Connection Fails
Added a few error messages in the agent log to identify connection problems with the collector.
Report Taint Information for Kubernetes Nodes
The Sysdig agent will send taint information associated with Kubernetes nodes. This enables you to query node taints using the
kube_node_spec_taint metric in Sysdig Monitor.
The s390x architecture image is not available for v12.9.0; therefore, this version of the agent cannot be installed in zLinux. Note that using the
latest tag for agent images on zLinux will not work until the next agent version is released.
Restarting Agent No Longer Causes Kernel Panic
Fixed an issue in the Sysdig agent’s kernel module that could cause a kernel panic when the agent was restarted.
Support Arbitrary Java Command Names
Added a configuration parameter to allow you to specify the command names to launch Java processes. This helps detect Java processes for JMX metric collection.
For example, if you want the agent to detect a process by the name of
jsivm , while still detecting the other commands, you should add the following to
jmx: java_commands: - java - jsvc - jsivm
The values specified in
dragent.yaml will override the default values, therefore, you need to include the defaults if you want to continue detecting them.
Captures Are No Longer Corrupted in Few Hosts
Sysdig Monitor no longer gives corrected Capture files in few hosts in a cluster. Previously the Capture files were found corrupted when generated on the host selected from Explore > Hosts & Container.
Report Containers as Expected
Fixed an issue where containers would not be reported if the agent had issues communicating with the Kubernetes API server.
psycopg2 module to
v2.8.6 to fix issue where Postgres AppCheck fails to start due to missing
Build Kernel Modules on RHEL6
Fixed an issue preventing the kernel module from building on RHEL6 and other kernels of similar vintage.
Stop Reporting Unschedulable Pods
Fixed an issue where unschedulable pods would not be reported by the agent.
Initialize Agent on Latest Kernels
Previously, the agent failed to initialize on latest kernels, such as Ubuntu v22.04 and Fedora 35 and 36, with the following error:
gcc: error: unrecognized command-line option '-mharden-sls=all'
This has been fixed in this release.
Disable the Policy Scope Cache
The scope cache has been disabled by default to prevent it from getting stuck due to longer completion period for Infra state.
Update kube-bench and kubectl Binaries
Updated the golang version used for building
kube-bench binaries to address vulnerabilities.
Show Correct Output Message in the Launch Sensitive Mount Container Rule
Launch Sensitive Mount Container rule in the
Suspicious Container Activity 2 policy no longer shows incorrect information in its output.
Show Required Secure Event Output Fields in Custom Rules
Fixed an issue where not all the required secure event output fields were being generated by the agent.
12.8.1 August 29, 2022
Fix Vulnerabilities in Promscrape V1
Upgraded the Prometheus version and resolved vulnerabilities in Promscrape v1.
Remove Symbolic Link to /etc in the Agent Container
Read information on users and groups from
/host/etc/group when agent is running as a container.
Show Falco Events as Expected
Fixed a problem when the Falco output string for a rule is cut on the first absent or empty field.
12.8.0 August 02, 2022
Add a New Metric to Indicate Retrieving Kubernetes State
Added an internal metric,
statsd_dragent_subproc_cointerface_ready to indicate when the agent has pulled Kubernetes state from the API server.
Read Certificate Chain
Previously, the agent would only accept the first certificate in a cert chain and would attempt to verify all other certificates from the configured certificate store. This behavior is compliant with the TLS specification, but idiomatic usage in the wild requires the agent to accept intermediate certificates provided in the handshake as well. The agent will now accept these certificates if provided.
Falco Rules Optimizer
Now, you can (optionally) enable Falco Rules Optimizer. This feature increases the speed of syscalls evaluation against Falco rules by introducing indexing on the rules conditions and by caching partial rule condition evaluations. This feature is only available in Sysdig agent, but not in open-source Falco. The feature is enabled by setting
true (default value is
New Falco Rules Parser
Starting from version 12.8.0, Sysdig agent uses a new Falco rules parser from OSS Falco. The new OSS Falco parser performs stricter grammar parsing and would fail on the following cases:
\nis used instead of
,in a list
"["is present in a rule definition
"is present in a rule definition
- when or operation between lists is used instead of, with in operator. For example:
condition: open_write and fd.filename is (list1 or list2)
If any of the above cases are present in custom rules file, the agent fails to parse the respective rule and outputs the following error:
Error, security_mgr:791: Could not load policies_v2 message:.
In this case, the custom rules have to be edited to correct or remove unparsable rules
Process Kubernetes Audit Events as Expected
Agent no longer throw errors while processing Kubernetes audit events when Kubernetes audit rules contain the
Upgrade Go Language Packages
Go language packages have been upgraded to fix vulnerabilities
Fixed the following vulnerabilities with Promscrape V2:
Detect Prometheus Targets Correctly
Fixed a problem that was causing new Prometheus targets to not be detected until an agent restart.
Intermittent Scraping Failure No Longer Causes Missing Metrics
Fixed an issue with missing metrics when there are intermittent metrics scraping failures.
Show Falco Events as Expected
Sysdig agent now throttles redundant secure events for compliance policies reducing the event noise.
Show Username Correctly in Policy Events
Fixed an agent build issue that caused password and group functions unvailable and therefore, linked the password and group from `/host/etc’ inside the agent container to have the username correctly shown in policy events.
Fix a Logging Issue in Promscrape V2
Fixed a logging issue with Promscrape v2. Log levels are taking effect as expected when passed in with
Agents No Longer Incorrectly Behave as Delegated
Fixed an issue that might cause all the agents to behave as delegated.
12.7.1 July 06, 2022
Fixed memdump.size Issue
memdump.size configuration, which was not accepted earlier.
Fixed Promscrape Crash Issue
Fixed a crash issue in Promscrape v2 when a node has a large number of pods and multiple containers per pod.
Fixed Issue Affecting Two Agent Modes
Fixed a problem that can cause agent subprocesses to be killed in nodriver mode. This affects the
monitor_light modes. For more information, see Configure Agent Modes.
12.7.0 June 28, 2022
New Helm Chart
Sysdig released a unified Helm chart, sysdig-deploy with the following benefits:
- Easier to deploy multiple components with one chart, rather than using multiple separate charts
- Fewer errors by way of using common configuration for components
- Auto-detection of certain configurations, including eBPF for GKE COS and endpoint region.
We will maintain the old version of Helm chart, sysdig chart for a period of six months. In this period, the Sysdig chart will be updated with new component versions and and defect fixes.
Support Prometheus v2.32
Updated Prometheus scraper to version 2.32.
Metrics Collected in Custom Metrics Only Mode
custom-metrics-only mode is used, no process metrics are collected. Additionally, only the metrics related to resources (CPU, memory) are collected for containers and host.
While the agent is running, you might encounter an error similar to the following:
Error, security_rule:610: Could not parse rule xx from rules json array.
The rule number in the error message might change depending on how many rules are defined.
This is a known issue related to failing to parse an experimental rule. The parser will skip this rule and will log the error message as above. The agent performance and policy evaluation will not be affected.
Remove Ceph App Checks
Fixed a problem where errors for obsoleted app checks would be shown when Ceph was running on the host.
Disable Timeseries Caching
Removed a configuration option that caused Prometheus jobs to not report timeseries if the scrape failed temporarily.
Builds eBPF Probes in Bottlerocket
Fixed an issue that prevented ebpf probes from having built by the agent in Bottlerocket Environments.
Reports Infrastructure State Correctly
Fixed an issue where the Sysdig agent would opens a stream to Cointerface even when it is disabled. This resolves the issue of infrastructure state having reset constantly.
Sends Only Supported Metrics in Nodriver Mode
Fixed an issue where unused container and process metrics were sent while in nodriver mode.
Change Log Level to DEBUG When Excessive Log Level Occurs
The excessive logging level occurs under specific conditions, for example, a pod whose used memory results in zero. This case seems to be normal for small pods using very little memory. A fix has been provided so that, when these conditions are detected, the log level for the message that is polluting the logs is brought from INFO to DEBUG.
Reports Container Resource Limits and Requests Correctly
Fixed an issue where container resource limits and requests would appear as zero when no limit or request was configured.
12.6.0 May 16, 2022
Reloading Promscrape V2 No Longer Causes Dropping Scrape Targets
Reloading Promscrape v2 no longer causes dropping some scrape targets from sending metrics.
Losing Node No Longer Generates Duplicate Node Events
Resolved an issue that caused generating duplicate events when a Kubernetes node is lost.
Agents Connect to SaaS Backend Through HTTP Proxy on Older Hosts
Fixed an issue related to SSL certificate verification when connecting through an HTTP proxy on older host OS, such as CentOS 7.
Agent Refreshes Service Account Token as Expected
Connection with the Kubernetes API Server works as expected. The Kubernetes client is configured to refresh the bearer token.
12.5.0 May 02, 2022
Default Availability of Slim Agent
The agent installation defaults to the slim agent. Slim agent reduces the surface area for potential vulnerabilities as compared to the full agent, which implies increased security for your monitoring environment. For more information, see Agent Installation.
To continue using the regular agent, set
false in your Helm chart.
Monitoring Kubernetes Resources
Sysdig agent v12.5.0 and above no longer collects the HPA kube state metrics by default. To enable the agent to collect HPA kube state metrics, you must edit the agent configuration file,
dragent.yaml, and include it along with the other resources you would like to collect. For more information, see Enable Kube State Metrics.
Container DriftControl: Detect and Prevent Drift in Container Runtime
Sysdig agent can now detect when a new executable was added to a container after a container has started up. The agent collects when a file was downloaded and made executable. When using
prevention mode, the agent can also deny the process from ever running. A policy can also be used to define binaries that should be denied/excluded from being denied if they have been added after the container has started.
See also: Drift Policy
Disable Syscalls for Secure Modes
Switch syscall events are disabled for secure and secure light modes.
- An error message is displayed when the agent detects ceph and attempts to run an obsoleted app check.
- The Sysdig agent for ARM can restart when multiple containers are started in rapid succession on the host.
Agent on zLinux No Longer Restarts Due to Incorrect Detection of tid Colliions
The agent on s390x architecture (zLinux) has been fixed so the agent does not restart needlessly due to incorrect detection of too many tid collisions.
Reports Correct CronJob Version When Adding CronJob Parents
Fixed an issue causing CronJobs to be reported not as the parents of Job objects.
Agent No Longer Crashes During Abnormal Termination
Fixed an issue causing the agent to crash with a stack backtrace during certain abnormal termination situations.
Slow-Starting JVMs are Terminated Correctly
An incorrect detection of too many tid collisions on s390x architecture (zLinux) will no Longer cause the agent to restart periodically.
Kubernetes Events are Collected as Expected
Fixed an issue that could prevent Kubernetes events from being correctly fetched.
Disable Watching HorizontalPodAutoscaler
Watching Horizontal Pod Autoscalers has been disabled by default to decrease load on Kubernetes API server. For more information, see Enable Kube State Metrics.
False Positive CVEs for Go Packages No Longer Reported
The Go compiler version has been upgraded to prevent getting flagged with (false-positive) CVEs associated with older Go versions.
Secure Events Reports Correct Cluster Information
Secure events no longer report Kubernetes cluster name
default when no cluster exists in the environment.
12.4.0 April 04, 2022
Support for New Architectures
Installing agent on the following architecture are supported:
aarch64 environments support AWS Graviton
For more information, see Host Requirements for Agent Installation.
ARM support includes AWS EC2 Graviton platform
A new agent mode,
custom-metrics-only, has been introduced. It enables all custom metrics and Kubernetes state metrics but disables all the driver-based metrics.
Prevent Processing Policy Updates
Prevent processing policy update messages to reduce CPU usage when no changes are required in the agent.
Increased Resource Consumption due to Misconfiguration of Node Lease
Incorrect configuration of Kubernetes lease can result in elevated memory usage in the Sysdig agent pods as well as increased load on the Kubernetes API server due to multiple agents querying for more information simultaneously. This also results in a significant amount of additional and unnecessary load on the Sysdig backend. To resolve this issue,
- Upgrade to Sysdig agent 12.5.0 which adapts to the non-optimal Kubernetes configuration.
- Configure the Kubernetes lease functionality. If you are using Helm, the latest versions of the Sysdig Agent Helm chart defaults to configuring the lease functionality automatically. If you do not use Helm, the DaemonSet and ClusterRole YAML files are available in our gitbub repository. For further assistance, contact Sysdig Support.
Agent Restarts Periodically on zLinux
An incorrect detection of too many tid collisions on s390x architecture (zLinux) can cause the agent to restart periodically. To workaround this issue, set the following configuration option:
watchdog: analyzer_tid_collision_check_interval_s: 86400
This configuration change reduces the number of restarts to once a day instead of every 10 minutes, which is the default value for the above configuration option.
This issue has been fixed in Sysdig agent v12.5.0.
Validate Promscrape Scrape Jobs
Validate scrape jobs associated with Promscrape integration before scraping the endpoints to avoid unnecessary errors with irrelevant scrape jobs.
Remove App Check Warning Messages When App Checks Are Disabled
Remove unnecessary warning messages about app checks limits when app checks are disabled.
Slow-Starting JVMs Are No Longer Terminated
Slow starting JVMs can be terminated by
sdjagent. For example,
-XX:+AlwaysPreTouch with large heaps. This fix introduces additional configuration options to tune the delay between
sdjagent detecting a started JVM process and an attempt to connect.
jmx: monitor_connect_timeout_ms: 5000 management_agent_connect_delay_ms: 0
EVE Connector Works as Expected in Kubernetes
Fixed metadata incompatibility in profiling with Kubernetes versions above 1.20.
Name Change to Configuration Parameter
falcobasline.max_drops_buffer_rate_percentage parameter has been corrected to
falcobaseline.max_drops_buffer_rate_percentage. Notice the missing
falcobasline.max_drops_buffer_rate_percentage. However, the backward compatibility is ensured, and therefore,
falcobasline.max_drops_buffer_rate_percentage can still be used.
12.3.1 March 03, 2022
Noisy Messages Silenced
Removed a kernel message from the driver that could generate spam when the syscall event buffer is full.
12.3.0 February 17, 2022
Binaries Category for Falco Baseline
A new category,
binaries is added to the Falco baselines feature.
Support for Workload Information in Falco Baseline
Add workload information to Kubernetes context for Falco baselines.
Default Monitoring of Kubernetes Resources
The following Kubernetes resources are monitored by default:
- persistentvolumeclaims - persistentvolumes - storageclasses - horizontalpodautoscalers
IPv6 Addresses Are Saved Incorrectly When Adding Rules
Adding a new rule causes problem saving IPv6 address for both
Fix Truncated Capture Files
Fixed a problem which caused the agent to generate truncated capture files.
Container Action Pause Work on Kops/GKE Clusters
Fixed the logic that determines the cgroup path for a container in containerd and made the freezer subsystem available to the agent in order to be able to pause/unpause it.
Agent Profiling Works as Expected
High CPU load no longer prevents generating CPU and memory profiles in the agent.
Agents Are Not Reset with Signal 11
Large and negative file descriptors are handled correctly so agents are no longer reset with signal 11.
12.2.1 February 07, 2022
Manage Collecting Metadata from Individual Container Engines
Access to individual container engines from within the agent for fetching metadata can now be disabled via agent configuration. For example, to disable docker, use the following configuration:
container_engines: docker: false
The Pause policy action is not working as expected in GKE, EKS, and Openshift4 environments.
Policy Action “Kill” Is Correctly Triggered in GKE Environments
Policy action on GKE with containerd works as expected:
- The container is stopped if HTTP proxy is enabled.
- The status of the container is checked upon stop requests. If the status is not
CONTAINER_EXITING, termination of the container is attempted with exponential backoff.
Agents Assign Username Correctly for Container Events
Fixed an issue that prevented the
proc.name field from extracting the right user from the
container started events. This issue was found in agent versions 12.2.0 and above.
12.2.0 January 25, 2022
Improve Install Script to Support eBPF
A new option,
-b is added to the native install script of Sysdig agent to support eBPF.
Enable 10s Flush by Default
By default, the agent collects metrics at 1-second granularity, then aggregates and sends them to the backend in 10-seconds intervals. If you want to use agent versions
12.2.0 or above with the on-prem Sysdig Platform versions below
3.5.0, set the
10s_flush_enable configuration to
false to prevent compatibility issues.
The backend in our SaaS deployments continues to enable 10-second flush automatically for all agent versions 10.0.0 or above.
Improved Log Messages
Improved the log messages to report the errors encountered while configuring
Handling Incorrect Metric Format
When scraping Prometheus metrics, the agent will set the type to
PROMETHEUS_TYPE_INVALID if the metric is exported in an incorrect format or without a specified type. The metric will still be ingested by Sysdig and the query will fallback to gauge.
Processing Secure policy updates in the agent can take longer than it did in the previous releases, and in some rare scenarios, it causes agent restarts.
Fix CVE-2020-29652 in Cointerface
Updated crypto go module to fix CVE-2020-29652.
Promscrape V2 No Longer Crashes on Pods with Multiple Containers
promscrape_v2 from crashing when a pod has multiple containers.
skip_events_by_type Works as Expected
Fixed an issue in the kernel probe, which prevented the
skip_events_by_type feature from correctly filtering events by system call type.
Kubernetes State Is Transmitted as Expected
Fixed an issue where Kubernetes information and metrics would not be sent from the agent. This scenario arose when the agent was deployed in a namespace other than
sysdig-agent, and the agent daemonset did not include the
Agent Successfully Connects to JMX
Fixed an issue where agent wouldn’t connect to JMX on some applications/JVMs. This issue was originally observed on the WebSphere application and Liberty JVM.
Agent Updates Container Status as Expected
Fixed an issue where the agent would not update the container status it first received from the API server. The agent now updates the container statuses as it receives them from the API server.
Check for Invalid Log Level in sdjagent
Fixed an issue where using a log level of
sdjagent to crash.
App Checks Run as Expected on Non-Containerized Agent Installations
Fixed an issue preventing app checks to run on non-containerized agent installation.
Native Install Doesn’t Support eBPF
Native install prevents insertion of Sysdig probe kernel module when the agent is installed with eBPF by using rpm or deb package.
Prevents Connection Attempts When Agent Encounters Errors
Connection attempts are prevented when the agent encounters errors while handling handshake messages.
2021 Archive of Sysdig Agent release notes.
2020 Archive of Sysdig Agent release notes.
2019 Archive of Sysdig Agent release notes.
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.