Sysdig Agent Release Notes

Introduced per-component-level console logging feature. See Manage Console Logging for Agent Components.

Added missing replication controller fields to the aggregator Actions.

Use Kubernetes leases to better control the load on the Kubernetes API Server. This is disabled by default.

Container actions are now properly supported on containerd (CRI-O and other CRI engines that already had support). Actions for unsupported container engines are now properly reported to the Sysdig backend and a warning message is logged in the agent logs.

Introduced a detection and recovery mechanism for hangs during agent shutdown.

Fixed a problem causing the agent to crash after promscrape_v2 is terminated.

The agent tries to fetch the metadata of the AWS instance in which it is running in order to tag metrics generated with the information unique to the AWS instance. If the metadata structure is not as expected, the agent continuously restarts due to an error in fetching such metadata. This issue has been fixed.

Fixed an issue that disabled support for performance profiles in the agent.

agent-kmodule and agent-kmodule-thin can now be used to build eBPF probes.

Start tracking container user information and make that information accessible in container events. These events denote having a container started. This feature works for Docker as well as CRI-O container engines.

Reporting container user information does not work in OpenShift 4.x because it does not provide necessary CRI-O information.

Sysdig supports Agent CLI, a command-line interactive tool, to troubleshoot agents. This tool helps Sysdig support to solve user issues quickly and efficiently. It is currently disabled by default and requires the customer to turn it on.

For more information, see Using the Agent Console

Scraping Prometheus metrics is supported in the following cases:

Use the following:

Terminated pods are no longer collected in order to reduce the load on the Kubernetes API server.

The audit server now by default listens on all the interfaces for Kubernetes audit events. This makes integration with Kubernetes audit events in the agent easier without the need for configuration changes.

The noise-reduction filter for Activity Audit has been improved. All the filtered data is duplicated.

The new CRI-O versions (1.19+, possibly 1.18) now properly report container.image.id.

Demoted logs about duplicate host container_groups from warning to debug level

Fix CVE-2021-28831 in the Slim Agent container.

The readiness probe improvement in version 11.1.2 delayed the transition of the agent pod to a ready state until communication with the Kubernetes API server was established. But this delay could cause a CrashLoopBackoff due to liveness or readiness probes configured with an initialDelaySeconds set to less than 90.

In Agent version 11.1.3 the transition to the ready state does not wait for communication with the Kubernetes API server to be established unless the behavior is enabled via a new configuration option: k8s_wait_before_ready.

The readiness probe improvement in version 11.1.2 delayed the transition of the agent pod to a ready state until communication with the Kubernetes API server was established. But this delay could cause a CrashLoopBackoff due to liveness or readiness probes configured with an initialDelaySeconds set to less than 90.

Workaround. If you are using agent version 11.1.2, set initialDelaySeconds for both liveness and readiness probes to a value that is greater than or equal to 90.

Kubernetes reconnect logic has been improved to automatically backoff (1 min, 2 min, 4 min... 1hr) if the connection is continuously dropped when using Thin Cointerface. This reduces the load that the agent imposes on the Kubernetes API Server in clusters with heavily burdened API servers.

The agent's readiness probe has been improved to not report ready until after the agent connects to the Kubernetes API server. This reduces the load that the agent imposes on the Kubernetes API server when starting up during RollingUpdate.

Fixed an issue where the agent would incorrectly report memory.bytes.used for containers that use more than 4GB.

The runtime policies that have a policy type and capture action are handled as expected.

Agent tags are supported in runtime policy scopes.

Fixed a problem where metric limits were not updated from the defaults. This is unlikely to happen if agents are connected to the SaaS backend.

Fixed a problem in the old Prometheus scraper (used when promscrape is disabled) to ensure that configured tags are properly added to the metrics.

Fixed an issue where short-lived Java processes could cause the Sysdig Agent to stop collecting JMX metrics.

Fixed a problem where the agent would continuously send requests to the Kubernetes API server to query the endpoints API. This occurs when the agent's clusterrole is incorrectly configured. With this fix, the agent will no longer repeat the attempt if it is unable to connect to the Kubernetes API during boot.

The runtime policies are now correctly scoped by kubernetes.cluster.name. The fix in 10.6.0 was incomplete.

Fixed an issue where the agent could lose track of a replicaset and report incomplete metadata.

Thin cointerface reduces the memory required to handle the Kubernetes metadata on both the agent and the Kubernetes API Server. The reduction in memory usage is significant for Kubernetes clusters with a large number of pods (in the range of 10,000 or more) or clusters that heavily use Replication Controllers.

Using this feature returns the same data to the Sysdig backend and does not affect any Sysdig features. The thin cointerface feature is disabled by default.

To enable:

See also: Reduce Memory Consumption in Agent.

Some high-frequency information level log messages are converted to debug level to reduce the volume of messages generated at the default information level.

Per-component file logging capability for an additional set of agent components has been enabled.

For more information, see Manage File Logging for Agent Components.

The number of Prometheus time series ingested has been limited to reduce agent memory consumption. This limit is applied after Prometheus relabeling rules are applied but before the agent's metric filter and metric limit.

Fixed an issue where processes with certain names were improperly aggregated, which in turn caused missing metrics in certain situations.

Fixed an issue that caused the agent's cointerface process to restart continuously while processing kubernetes label selectors.

Fixed a defect in the Thin Cointerface feature which could cause Kubernetes metadata to stop updating. Because Thin Cointerface is turned off by default, the change affects only a small number of users who have this feature turned on.

Kubernetes cronJobs are supported when reporting network communications.

Fixed a race condition that could prevent runtime policies and rules from being loaded properly if multiple messages from the Sysdig backend are received consecutively.

Fixed an issue where Statsd metrics related to compliance would have no associated Kubernetes metadata and were not visible on Cluster Overview.

Filtering long container labels works as expected with no parsing failures or undesirable agent restarts.

Fixed an issue that could cause kubernetes.pod.restart.rate metric to be incorrect.

Fixed a problem that caused scraping Prometheus metrics to fail when another process was listening to the TCP port 9090 on a host interface.

Fixed a problem that caused Statsd metrics to report incorrect values.

Fixed an issue that could cause the environment variable hash associated with the exported processes in audit tap to have an incorrect value.

The sdjagent process in the agent no longer consumes excessive CPU resources.

Sysdig agent no longer supports the old backend message types that were originally deprecated in on-prem release 2.4.0 (August 2019).

Partially load Falco rules in the background to avoid interrupting event processing.

The agent is restarted if a metrics acknowledgment hasn't been received from the Sysdig backend components in 8 minutes. This can happen if networking issues cause the agent to believe it has an active connection when the backend has closed the connection.

Prevents multiple agent services from being launched on the same RHEL-based hosts.

Running the OpenShift Hardening Guide functionality of the Kubernetes Benchmark will now correctly detect master vs worker nodes, and run the appropriate Benchmark tests.

In some rare situations when process creation in the Agent's JMX module failed due to issues caused by resource limits, it could inadvertently stop unrelated processes running on the host. This problem has been fixed.

Python 2.7 has been removed from the agent and agent-slim containers.

Sysdig agent continues to support python 2.7 if installed as a service and the host has python 2.7.

Updated kube-bench to support Kubernetes benchmarks and targets. For a complete list of benchmarks, see Benchmarks (Legacy) .

Configuring metrics expiration time is supported by promscrape.v2 for Prometheus metrics gathered by using Prometheus service discovery.

Add support for scoping policies by kubernetes.cluster.name. The cluster name must still be manually configured by using the configuration option, k8s_cluster_name: <CLUSTER NAME>.

Made kubernetes node matching more reliable for Prometheus Service Discovery by comparing IP addresses as opposed to node names in the default configuration.

Addressed a known vulnerability in the jackson-databind package version 2.9.10.6 by upgrading to version 2.11.3 in agent containers.

Changed the java NoClassDefFoundError class from Error to Info to reduce spamming the logs at the Error level. This happens commonly when the agent attempts to read metrics from a java v11 application which was not started with the com.sun.management.jmxremote option.

Fixed a problem that caused StatsD metrics to be double the expected value.

Removed warning logs about ambiguous source labels when using the Prometheus service discovery with multi-container pods.

Fixed an issue that could potentially cause a slow increase in the agent's memory usage over time when the thin_cointerface_enabled configuration option is enabled.

Improved the default Prometheus configuration for promscrape.v2 to honor pod annotations.

Fixed a problem that could cause a pod to be associated with incorrect deployments.

Sysdig agent to the collector communication can be established via an HTTP or an HTTPS Proxy server.

For more information, see Enable HTTP Proxy for Agents.

A new version of promscrape, promscrape.v2 , has been introduced to offer native Prometheus service discovery capabilities. To support this, a default prometheus.yaml file has been added with Kubernetes pod discovery rules to use when native Prometheus service discovery is enabled. See Enable Prometheus Native Service Discovery for more information.

Sysdig agent now supports secure mode that offers Secure only features. See Secure Mode for more information.

Addressed vulnerabilities reported in the agent and agent-slim containers, including the one for CVE-2017-18640 in a dependency library related to image scanning.

Fixed an issue that caused the agent to hang while handling some types of connection errors. When this issue is encountered, restarting the agent will allow it to reconnect.

Upgrading to Sysdig agent v10.5.0 or higher is strongly recommended to avoid this problem.

Prometheus metrics can now be scraped from endpoints in Docker containers with remapped port numbers.

The agent now starts faster on systems with thousands of processes and hundreds of containers.

The agent logs a warning once in a minute when the Prometheus metric limit is reached.

Fixed a problem that could randomly result in Prometheus metrics not being sent when Prometheus service discovery is enabled.

Fixed a problem that would cause certain app check metrics to be missing when 10-second aggregation in the agent is enabled.

Added a timeout to the handshake protocol between agent and collector.

Fixed a problem that randomly caused JMX metrics to be not collected due to transient errors encountered during the startup of new Java processes.

Fixed a problem that caused the UI to show a pod under an incorrect service if other services exist in different namespaces with the same selectors. This happened when the thin_cointerface_enabled property was set to true.

Fixed the evaluation of secure fast engine syscall rules when the If Not Matching rule is selected.

Fixed a problem that could cause Kubernetes pods to lose association with their deployment or other related resources.

The agent can now scrape Prometheus metrics from the docker containers that expose ports only on specific IP addresses besides the localhost.

The use_forwarder option is now enabled by default. See Collect StatsD Metrics Under Load.

The default value (300) of per-process JMX bean limits can now be changed as follows:

jmx:
  max_per_process_beans: 500

When Statsd is disabled, do not attempt to send metrics related to benchmarks tasks. This also means that benchmarks dashboards will not have data when Statsd is disabled.

A problem that could cause Kubernetes pods to lose association with their deployment or other related resources has been identified in Agent version 10.4.0. A new version, 10.4.1, that will address this problem is currently in development.

Embedded web server for Kubernetes audit events restarts as expected when the agent process is restarted.

Updated the version of the jackson-databind package to fix vulnerabilities discovered in the slim agent v10.3.0

The kube-bench binary that was identified as broken due to the change in the output format has been fixed.

kube-bench that performs the Kubernetes Benchmarks tasks has changed the output format, causing the existing Benchmark tasks to fail in v10.3.0. With this fix, the agent will no longer throw errors related to this issue and the new Kubernetes Benchmark results will appear in the UI as expected.

Fixed an issue with building probes for Linux v5.8.0 kernel.

URL segmentation for metrics has been moved from the default monitor mode to the troubleshooting mode. Due to this change, dashboard panels with per URL metric will show no data. See Additional Metrics Values Available in Troubleshooting.

The Sysdig probe URL is changed to download.sysdig.com.

If the Sysdig probe URL is included in the allow list for outbound firewall access, you must change the endpoints to reflect the new probe location.

The agent now connects to promscrape through a UNIX socket by default as opposed to the TCP port 9876.

The version of kube-bench has been upgraded to 0.2.4. The changes include an additional configuration file path for Hyperkube kube-proxy to support OpenShift.

The kube-bench binary is broken due to the change in the output format and the issue will be fixed in an upcoming release.

kube-bench that performed the Kubernetes Benchmarks tasks changed the output format, causing the existing Benchmark tasks to fail. The new Kubernetes benchmark results will not appear in the UI, and the agent will report errors related to Kubernetes benchmark tasks.

Prometheus metric limits have been modified to ensure that endpoints with fewer timeseries are not affected when another endpoint hits the limit. Reporting of Prometheus timeseries statistics has also been updated.

The calculated Prometheus _count metrics are reported for summaries and histograms even when the _sum values are missing. This feature is not applicable to raw metrics.

A .count metric (which is the rate of change of _count values) and a .avg (which is the average of new samples when _count increases) are calculated for summaries and histograms. Earlier, those .count and .avg metrics are reported only if the raw Prometheus metrics include both  _sum and _count values. In this release, changes have been made such that _sum values are no longer required to calculate Prometheus _count metrics for summaries and histograms.

Fixed an issue pertaining to the reporting of running pod counts for replication controllers, deployments, and ReplicaSets.

Fixed an issue that prevented having Kubernetes jobs segmented by namespace.

Fixed an issue that caused the agent to stall under high load.

Fixed an issue that caused an exception at agent restart while collecting CPU metrics.

Periodic logging of statistics for Prometheus timeseries has been added. When a metric limit is hit, all the timeseries metrics associated with the endpoint are dropped.

Processes with app checks or Prometheus metrics are now included by default in the top processes to be sent to the Sysdig collector.

A variety of performance improvements have been rolled out to accelerate the evaluation of Falco rules and fast engine rules for the common case of events not matching any rules/policies.

The agent has been enhanced to detect JSVC processes as java programs to enable the collection of JMX metrics.

The net.mongodb.* and net.sql.* metrics have been moved from the default monitor mode to the troubleshooting mode. For more information, see Additional Metrics Values Available in Troubleshooting.

The following deprecated App Checks have been removed and will no longer be supported.

Fixed a potential race condition that could occur when receiving multiple policies and related messages from the Sysdig collector at nearly the same time.

The agent no longer runs a built-in set of benchmark tasks. The agent will only run benchmark tasks when configured to do so by a Sysdig Secure backend.

Prometheus metrics from idle processes are no longer dropped even if the target processes are not active enough to be in the top processes. Additionally, the app_checks_always_send parameter, which can force report the idle processes with metrics, now works as expected for metrics gathered by promscrape.

Removed sensitive authentication credentials related to app checks from debug log messages.

Kubernetes events are no longer dropped under some high load conditions.

Fixed a problem that prevented the collection of slab and item stats in the Memcache app checks in certain Python environments.

The following metrics now no longer return incorrect zero values:

Network stats metrics that were moved to the troubleshooting mode in Agent v10.1.0 have been re-enabled by default. The metrics will now be available in the monitor mode, which in turn will enable the network topology by default.

For information on agent modes, see Configure Agent Modes.

Added support for Linux 5.6 kernels.

Added JMX support for Java 11, 12, 13, and 14 JRE. For containerized Java apps with JRE, run the app with the -Dcom.sun.management.jmxremote option.

Added rate limiting configurations to the agent to avoid connection timeouts for metrics and secure messages.

Added a new metric to display the kernel version of the host where the agent is running.

Added container name to the containerd events description. In some rare cases, the container name associated with a containerd event might be unavailable due to metadata lookup delay.

Removed sensitive authentication credentials related to app checks from debug log messages.

The following deprecated app checks will be removed in an upcoming release:

Some metrics related to network and file will not be available by default. You can enable them by editing the dragent.yaml file.

Fixed promscrape to honor the ssl_verify configuration option.

Fixed an issue with the agent-slim container that prevented postgres and pgbouncer app checks from emitting metrics.

Reduced the frequency of a log message to reduce spam and enhanced a statsd related log message to provide more information about incorrectly formatted strings.

Consider only exact matches when linking secure runtime policies to Falco rules to fix this issue.

Fixed calculation of net.bytes.* metrics at the host level when using calico interfaces or VPN tunnels.

Added the ability to run Kubernetes Benchmark Master Programs on additional Kubernetes distributions.

A new process, called promscrape, has been introduced to scrape Prometheus metrics by default. The mechanism, based on the open-source Prometheus, improves compatibility and performance. It also allows per-endpoint metric filtering and relabeling through metric_relabel_configs.

For more information, see Working with Prometheus Metrics.

Added the ability to make draios.log files readable by users other than root. This can be enabled with the following configuration in dragent.yaml.

log:
  globally_readable: true

Added the ability to kill containers as a runtime policy action. See Manage Policies for details.

Fixed the use of the path parameter in Prometheus configuration when using promscrape. With this fix, the configured path is passed to promscrape by the agent when it is set up for a target rule in dragent.yaml.

Prometheus scraping can now be triggered based on service annotations by default.

Added the missing posix-ipc module to the slim agent. This fixed an issue that prevented App Checks from running in the agent-slim container on v9.9.0.

Prometheus scraping metadata is no longer counted toward, or limited by, metric limits when using promscrape.

Fixed a defect that caused percentile metrics to not work properly.

Added the missing Posix module to the slim agent. This fixed an issue that prevented App Checks from running in the agent-slim container on v9.9.0.

Python 3 is the new default Python version for app checks, instead of Python 2. Python 2 can still be used by setting the following option in your dragent.yaml:

python_binary: <path to python 2.7 binary>

For containerized agents, this path will be: /usr/bin/python2.7

The following app checks are deprecated as of 9.9.0:

See also: Integrate Applications (Default App Checks).

Fixed a potential CPU stall on kernels with versions greater >= 4.19 using eBPF probe.

Fixed an issue that caused excessive logging in the agent log file.

Added Openshift Hardening Guide as a benchmark program. It is available as an option for CIS Kubernetes Benchmark.

Added Linux benchmarking as an available benchmark program.

The following metrics are introduced:

For more information, see Metrics Introduced with Agent v9.7.0 and RedisDB Metrics

Fixed the vulnerabilities detected in the agent-slim v9.6.1 image. These issues are related to the python2 and jackson-databind packages. These packages were upgraded to the versions with fixes.

Fixed a defect that prevented app checks from running on hosts that install Python 2.6.