2020 Archive

10.8.0 December 18, 2020

Defect Fixes

Filtering Long Container Labels

Filtering long container labels works as expected with no parsing failures or undesirable agent restarts.

Correct kubernetes.pod.restart.rate Metric

Fixed an issue that could cause kubernetes.pod.restart.rate metric to be incorrect.

Prometheus Metrics With Multiple Process Listening Concurrently

Fixed a problem that caused scraping Prometheus metrics to fail when another process was listening to the TCP port 9090 on a host interface.

StatsD Metrics Reports Correct Value

Fixed a problem that caused Statsd metrics to report incorrect values.

Correct Environment Variable Hash in Audit Tap

Fixed an issue that could cause the environment variable hash associated with the exported processes in audit tap to have an incorrect value.

Improve JMX Availability Check

The sdjagent process in the agent no longer consumes excessive CPU resources.

10.7.0 November 20, 2020

Feature Improvements

Policies and Baselines V1 Messages Are Deprecated

Sysdig agent no longer supports the old backend message types that were originally deprecated in on-prem release 2.4.0 (August 2019).

Load Falco Rules on a Separate Thread

Partially load Falco rules in the background to avoid interrupting event processing.

Workflow for Unacknowledged Metrics

The agent is restarted if a metrics acknowledgment hasn’t been received from the Sysdig backend components in 8 minutes. This can happen if networking issues cause the agent to believe it has an active connection when the backend has closed the connection.

Run Single Agent RPM Per Host

Prevents multiple agent services from being launched on the same RHEL-based hosts.

Known Issues

The host.container.start.count metric acts as a counter metric and its value increases monotonically.

Defect Fixes

OpenShift Hardening Guide Correctly Detects Master and Worker Nodes

Running the OpenShift Hardening Guide functionality of the Kubernetes Benchmark will now correctly detect master vs worker nodes, and run the appropriate Benchmark tests.

Agent No Longer Terminates Non-Agent Processes

In some rare situations when process creation in the Agent’s JMX module failed due to issues caused by resource limits, it could inadvertently stop unrelated processes running on the host. This problem has been fixed.

10.6.0 October 30, 2020

Feature Improvements

Python 2.7 Is No Longer Supported in Agent Containers

Python 2.7 has been removed from the agent and agent-slim containers.

Sysdig agent continues to support python 2.7 if installed as a service and the host has python 2.7.

Kubernetes Benchmarks

Updated kube-bench to support Kubernetes benchmarks and targets. For a complete list of benchmarks, see Benchmarks (Legacy) .

• Kubernetes benchmark 1.6

  • Master

  • Control plane

  • Node

  • etcd

  • Policies

• Google Kubernetes Engine (GKE) Benchmark 1.0

  • Master

  • Control plane

  • Node

  • etcd

  • Policies

  • Managed services

• Amazon Elastic Kubernetes Service (EKS) Benchmark 1.0

  • Control plane

  • Node

  • Policies

  • Managed services

Configuring Prometheus Metric Expiration Time

Configuring metrics expiration time is supported by promscrape.v2 for Prometheus metrics gathered by using Prometheus service discovery.

Support for Scoping Policies by Kubernetes Cluster Name

Add support for scoping policies by kubernetes.cluster.name. The cluster name must still be manually configured by using the configuration option, k8s_cluster_name: <CLUSTER NAME>.

Improved Prometheus Service Discovery

Made kubernetes node matching more reliable for Prometheus Service Discovery by comparing IP addresses as opposed to node names in the default configuration.

Defect Fixes

CVE Fixes

Addressed a known vulnerability in the jackson-databind package version 2.9.10.6 by upgrading to version 2.11.3 in agent containers.

Reduce Severity of NoClassDefFoundError Log from Error to Info

Changed the java NoClassDefFoundError class from Error to Info to reduce spamming the logs at the Error level. This happens commonly when the agent attempts to read metrics from a java v11 application which was not started with the com.sun.management.jmxremote option.

StatsD Metrics No Longer Show Larger Than Expected Values

Fixed a problem that caused StatsD metrics to be double the expected value.

Remove Warning Logs

Removed warning logs about ambiguous source labels when using the Prometheus service discovery with multi-container pods.

10.5.2 October 21, 2020

Defect Fixes

Memory Leak No Longer Occurs in the Agent

Fixed an issue that could potentially cause a slow increase in the agent’s memory usage over time when the thin_cointerface_enabled configuration option is enabled.

10.5.1 October 08, 2020

Feature Improvements

Added New Rules to the Prometheus Configuration to Honor Pod Annotations

Improved the default Prometheus configuration for promscrape.v2 to honor pod annotations.

Known Issues

Logs warning messages in the agent log file when promscrape.v2 is enabled.

Defect Fixes

Pods Are No Longer Associated with Incorrect Deployments

Fixed a problem that could cause a pod to be associated with incorrect deployments.

10.5.0 September 24, 2020

New Features

Enable Communication Between Agent and Collector Through a Proxy Server

Sysdig agent to the collector communication can be established via an HTTP or an HTTPS Proxy server.

For more information, see Enable HTTP Proxy for Agents.

Default Prometheus Configuration File

A new version of promscrape, promscrape.v2 , has been introduced to offer native Prometheus service discovery capabilities. To support this, a default prometheus.yaml file has been added with Kubernetes pod discovery rules to use when native Prometheus service discovery is enabled. See Enable Prometheus Native Service Discovery for more information.

Secure Mode

Sysdig agent now supports secure mode that offers Secure only features. See Secure Mode for more information.

Known Issues

None.

Defect Fixes

CVE Fixes

Addressed vulnerabilities reported in the agent and agent-slim containers, including the one for CVE-2017-18640 in a dependency library related to image scanning.

Agent No Longer Hangs While Handling Connection Errors

Fixed an issue that caused the agent to hang while handling some types of connection errors. When this issue is encountered, restarting the agent will allow it to reconnect.

Upgrading to Sysdig agent v10.5.0 or higher is strongly recommended to avoid this problem.

Scraping Prometheus Endpoints in Docker Containers

Prometheus metrics can now be scraped from endpoints in Docker containers with remapped port numbers.

Prevent Agent Crashes in Large Systems

The agent now starts faster on systems with thousands of processes and hundreds of containers.

Warning for Prometheus Metric Limit

The agent logs a warning once in a minute when the Prometheus metric limit is reached.

Transmitting Prometheus Metrics Works As Expected When Service Discovery Is Enabled

Fixed a problem that could randomly result in Prometheus metrics not being sent when Prometheus service discovery is enabled.

Appcheck Metrics No Longer Go Missing

Fixed a problem that would cause certain app check metrics to be missing when 10-second aggregation in the agent is enabled.

Agent Now Times Out If Connection Attempt to Collector Does Not Work

Added a timeout to the handshake protocol between agent and collector.

Agent Now Collects JMX Metrics from New Process Following a Java Service Restart

Fixed a problem that randomly caused JMX metrics to be not collected due to transient errors encountered during the startup of new Java processes.

Pod to Service Connection

Fixed a problem that caused the UI to show a pod under an incorrect service if other services exist in different namespaces with the same selectors. This happened when the thin_cointerface_enabled property was set to true.

Syscall Fast Rule Triggers as Expected

Fixed the evaluation of secure fast engine syscall rules when the If Not Matching rule is selected.

10.4.1 August 26, 2020

Defect Fixes

Kubernetes Pods No Longer Lose Association with Resources

Fixed a problem that could cause Kubernetes pods to lose association with their deployment or other related resources.

10.4.0 August 19, 2020

New Features

Ability to Scrape Prometheus Metrics from Container IP Addresses

The agent can now scrape Prometheus metrics from the docker containers that expose ports only on specific IP addresses besides the localhost.

Use Forwarder Is Enabled by Default

The use_forwarder option is now enabled by default. See Collect StatsD Metrics Under Load.

Set JMX Limits

The default value (300) of per-process JMX bean limits can now be changed as follows:

jmx:
  max_per_process_beans: 500

Known Issues

Handling Benchmark Task When StatsD Metrics Collection Is Disabled

When Statsd is disabled, do not attempt to send metrics related to benchmarks tasks. This also means that benchmarks dashboards will not have data when Statsd is disabled.

Kubernetes Pods Can Lose Association with Resources

A problem that could cause Kubernetes pods to lose association with their deployment or other related resources has been identified in Agent version 10.4.0. A new version, 10.4.1, that will address this problem is currently in development.

Defect Fixes

Kubernetes Audit Server and Agent Process Restart Congruently

Embedded web server for Kubernetes audit events restarts as expected when the agent process is restarted.

CVE Fixes Related to Slim Agent v10.3.0

Updated the version of the jackson-databind package to fix vulnerabilities discovered in the slim agent v10.3.0

10.3.1 August 06, 2020

Defect Fixes

Kubernetes Benchmark Tasks No Longer Fail

The kube-bench binary that was identified as broken due to the change in the output format has been fixed.

kube-bench that performs the Kubernetes Benchmarks tasks has changed the output format, causing the existing Benchmark tasks to fail in v10.3.0. With this fix, the agent will no longer throw errors related to this issue and the new Kubernetes Benchmark results will appear in the UI as expected.

Probes Works As Expected for v5.8 Kernels

Fixed an issue with building probes for Linux v5.8.0 kernel.

10.3.0 July 28, 2020

New Features and Enhancements

Changes to the Monitor Mode

URL segmentation for metrics has been moved from the default monitor mode to the troubleshooting mode. Due to this change, dashboard panels with per URL metric will show no data. See Additional Metrics Values Available in Troubleshooting.

Sysdig Probe Location Changes

The Sysdig probe URL is changed to download.sysdig.com.

If the Sysdig probe URL is included in the allow list for outbound firewall access, you must change the endpoints to reflect the new probe location.

Agent Connects to Promscrape through UNIX Socket By Default

The agent now connects to promscrape through a UNIX socket by default as opposed to the TCP port 9876.

New Configuration File Paths for Kube Proxy

The version of kube-bench has been upgraded to 0.2.4. The changes include an additional configuration file path for Hyperkube kube-proxy to support OpenShift.

Known Issues

Kubernetes Benchmark Tasks Fail

The kube-bench binary is broken due to the change in the output format and the issue will be fixed in an upcoming release.

kube-bench that performed the Kubernetes Benchmarks tasks changed the output format, causing the existing Benchmark tasks to fail. The new Kubernetes benchmark results will not appear in the UI, and the agent will report errors related to Kubernetes benchmark tasks.

Defect Fixes

EndPoints-Independent Metrics Limits for Prometheus

Prometheus metric limits have been modified to ensure that endpoints with fewer timeseries are not affected when another endpoint hits the limit. Reporting of Prometheus timeseries statistics has also been updated.

Prometheus Count Metrics for Summary and Histogram

The calculated Prometheus _count metrics are reported for summaries and histograms even when the _sum values are missing. This feature is not applicable to raw metrics.

A .count metric (which is the rate of change of _count values) and a .avg (which is the average of new samples when _count increases) are calculated for summaries and histograms. Earlier, those .count and .avg metrics are reported only if the raw Prometheus metrics include both _sum and _count values. In this release, changes have been made such that _sum values are no longer required to calculate Prometheus _count metrics for summaries and histograms.

Reporting Running Pod Counts

Fixed an issue pertaining to the reporting of running pod counts for replication controllers, deployments, and ReplicaSets.

Segmenting Kubernetes Jobs Metrics By Namespace

Fixed an issue that prevented having Kubernetes jobs segmented by namespace.

Agent No Longer Stalls Under High Load

Fixed an issue that caused the agent to stall under high load.

Restarting Agent No Longer Causes Exception

Fixed an issue that caused an exception at agent restart while collecting CPU metrics.

10.2.0 June 25, 2020

New Features and Enhancements

Prometheus Scraping

Periodic logging of statistics for Prometheus timeseries has been added. When a metric limit is hit, all the timeseries metrics associated with the endpoint are dropped.

App Checks and Prometheus Metrics

Processes with app checks or Prometheus metrics are now included by default in the top processes to be sent to the Sysdig collector.

Performance Improvement

A variety of performance improvements have been rolled out to accelerate the evaluation of Falco rules and fast engine rules for the common case of events not matching any rules/policies.

Detect JSVC Processes as Java Programs

The agent has been enhanced to detect JSVC processes as java programs to enable the collection of JMX metrics.

Troubleshooting Metrics Removed from Default Mode

The net.mongodb.* and net.sql.* metrics have been moved from the default monitor mode to the troubleshooting mode. For more information, see Additional Metrics Values Available in Troubleshooting.

Deprecated Metrics

The following deprecated App Checks have been removed and will no longer be supported.

• Network

• RiakCS

• TokuMX

• Ceph

• Gearmand

• Gunicorn

• Kyoto Tycoon

• Teamcity

• Riak

• Solr

• OpenStack

Defect Fixes

Fixed a Race Condition

Fixed a potential race condition that could occur when receiving multiple policies and related messages from the Sysdig collector at nearly the same time.

Benchmark Task Configuration

The agent no longer runs a built-in set of benchmark tasks. The agent will only run benchmark tasks when configured to do so by a Sysdig Secure backend.

Prometheus Metrics From Idle Processes Are No Longer Dropped

Prometheus metrics from idle processes are no longer dropped even if the target processes are not active enough to be in the top processes. Additionally, the app_checks_always_send parameter, which can force report the idle processes with metrics, now works as expected for metrics gathered by promscrape.

Removed Authentication Credentials

Removed sensitive authentication credentials related to app checks from debug log messages.

Kubernetes Events Are No Longer Dropped

Kubernetes events are no longer dropped under some high load conditions.

Memcached App Checks Collects Slabs and Items Stats

Fixed a problem that prevented the collection of slab and item stats in the Memcache app checks in certain Python environments.

Metrics No Longer Report Incorrect Zero Values

The following metrics now no longer return incorrect zero values:

• kubernetes.resourcequota.cpu.requests.hard

• kubernetes.resourcequota.cpu.requests.used

• kubernetes.resourcequota.memory.requests.hard

• kubernetes.resourcequota.memory.requests.used

Agent Automatically Restarts Upon Protocol Mismatch Errors

The agent used to require manual intervention to recover from protocol mismatch errors received from the Sysdig Backend. This error can occur when the agent and Sysdig Backend are not in sync. The agent has been enhanced to automatically restart when this error is encountered, so manual intervention is no longer required.

10.1.1 June 02, 2020

Defect Fixes

Enable Network Topology

Network stats metrics that were moved to the troubleshooting mode in Agent v10.1.0 have been re-enabled by default. The metrics will now be available in the monitor mode, which in turn will enable the network topology by default.

For information on agent modes, see Configure Agent Modes.

10.1.0 June 01, 2020

New Features

Support for Linux v5.6 Kernels

Added support for Linux 5.6 kernels.

JMX Support for Java v11, 12, 13 and 14 JRE

Added JMX support for Java 11, 12, 13, and 14 JRE. For containerized Java apps with JRE, run the app with the -Dcom.sun.management.jmxremote option.

Added Rate Limiting Configurations

Added rate limiting configurations to the agent to avoid connection timeouts for metrics and secure messages.

Added New Metrics

Added a new metric to display the kernel version of the host where the agent is running.

• host.uname

  This metric can be segmented by host.uname.kernel.name, host.uname.kernel.release , and host.uname.kernel.version.

Added Container Name to the Containerd Event Description

Added container name to the containerd events description. In some rare cases, the container name associated with a containerd event might be unavailable due to metadata lookup delay.

Removed Authentication Credentials

Removed sensitive authentication credentials related to app checks from debug log messages.

Removal of Deprecated App Checks

The following deprecated app checks will be removed in an upcoming release:

• Network

• RiakCS

• TokuMX

• Ceph

• Gearmand

• Gunicorn

• Kyoto Tycoon

• Teamcity

• Riak

• Solr

• OpenStack

Enable Removed Metrics

Some metrics related to network and file will not be available by default. You can enable them by editing the dragent.yaml file.

Edit the Configuration File

1. Open the dragent.yaml file.

2. Add the following configuration parameter:

   feature:
     mode: troubleshooting
   

3. Restart the agent.

Removed Metrics in Agent v10.1.0

The following metrics will not be reported by default in agent v10.1. When segmented by a particular label, these metrics will not have some values. The table summarizes the metrics and missing values when they are segmented by a particular label.

Defect Fixes

Promscrape No Longer Breaks Metrics Collection Over HTTPS

Fixed promscrape to honor the ssl_verify configuration option.

Slim Agent Container No Longer Prevents Certain App Checks From Emitting Metrics

Fixed an issue with the agent-slim container that prevented postgres and pgbouncer app checks from emitting metrics.

Reduced the Frequency of Log Messages

Reduced the frequency of a log message to reduce spam and enhanced a statsd related log message to provide more information about incorrectly formatted strings.

Use Exact Rule Names When Adding Rules to Runtime Policies

Consider only exact matches when linking secure runtime policies to Falco rules to fix this issue.

Corrected Calculation of net.bytes.* Metrics

Fixed calculation of net.bytes.* metrics at the host level when using calico interfaces or VPN tunnels.

10.0.0 May 01, 2020

New Features

Kubernetes Benchmark Master Programs

Added the ability to run Kubernetes Benchmark Master Programs on additional Kubernetes distributions.

New Scraping Mechanism for Prometheus

A new process, called promscrape, has been introduced to scrape Prometheus metrics by default. The mechanism, based on the open-source Prometheus, improves compatibility and performance. It also allows per-endpoint metric filtering and relabeling through metric_relabel_configs.

For more information, see Working with Prometheus Metrics.

Non-Root Access to Log Files

Added the ability to make draios.log files readable by users other than root. This can be enabled with the following configuration in dragent.yaml.

log:
  globally_readable: true

New Runtime Policy Action

Added the ability to kill containers as a runtime policy action. See Manage Policies for details.

Defect Fixes

Fixed the Path Parameter Issue in Prometheus Configuration

Fixed the use of the path parameter in Prometheus configuration when using promscrape. With this fix, the configured path is passed to promscrape by the agent when it is set up for a target rule in dragent.yaml.

Service Annotation Based Prometheus Scraping

Prometheus scraping can now be triggered based on service annotations by default.

Added a Missing Module to the agent-slim Container

Added the missing posix-ipc module to the slim agent. This fixed an issue that prevented App Checks from running in the agent-slim container on v9.9.0.

No Metric Limit on Scraped Prometheus Metadata

Prometheus scraping metadata is no longer counted toward, or limited by, metric limits when using promscrape.

Fix for Percentile Metrics

Fixed a defect that caused percentile metrics to not work properly.

9.9.1 April 16, 2020

Defect Fixes

Added the Missing Module to the Slim Agent

Added the missing Posix module to the slim agent. This fixed an issue that prevented App Checks from running in the agent-slim container on v9.9.0.

9.9.0 April 13, 2020

Core Features and Fixes

Python 3 Set as Default and Some App Checks Deprecated

Python 3 is the new default Python version for app checks, instead of Python 2. Python 2 can still be used by setting the following option in your dragent.yaml:

python_binary: <path to python 2.7 binary>

For containerized agents, this path will be:/usr/bin/python2.7

The following app checks are deprecated as of 9.9.0:

• Network

• RiakCS

• TokuMX

• Ceph

• Gearmand

• Gunicorn

• Kyoto Tycoon

• Teamcity

• Riak

• Solr

• Openstack

See Integrate Applications (Default App Checks).

Fixed Kernel Issue when Deploying Agent on GKE

Fixed a potential CPU stall on kernels with versions greater >= 4.19 using eBPF probe.

Fixed Flooded Agent Logs

Fixed an issue that caused excessive logging in the agent log file.

9.8.0 March 31, 2020

Fixes

Metrics Reporting

Fixed an issue in the agent wherein the kubernetes.namespace.pod.desired.count and kubernetes.namespace.pod.available.count metrics were not reporting any values.

HDFS App Check Deprecated

The HDFS (Hadoop Distributed File System) App Check had been deprecated and removed. Users of the HDFS App Check can switch to hdfs_namenode and hdfs_datanode App Checks.

Metric Calculation

Fixed an issue related to calculating the kubernetes.pod.restart.rate metric.

Network Congestion

Isolated the Kubernetes Audit HTTP server from the Audit Event processing path to reduce the chances of slowing down the connections from the Kubernetes API server. This should reduce the likelihood of multiple outstanding connections from the Kubernetes API server.

Certifi Python Module

Added a missing certifi Python module to the agent container.

9.7.0 March 09, 2020

New Features

Support for Openshift Hardening Guide

Added Openshift Hardening Guide as a benchmark program. It is available as an option for CIS Kubernetes Benchmark.

Support for Linux Benchmark

Added Linux benchmarking as an available benchmark program.

New Metrics for Redis and MongoDB App Checks

The following metrics are introduced:

• RedisDB

  • redis.mem.startup

  • redis.mem.overhead

• MongoDB

  • mongodb.tcmalloc.generic.current_allocated_bytes

  • mongodb.tcmalloc.generic.heap_size

  • mongodb.tcmalloc.tcmalloc.aggressive_memory_decommit

  • mongodb.tcmalloc.tcmalloc.central_cache_free_bytes

  • mongodb.tcmalloc.tcmalloc.current_total_thread_cache_bytes

  • mongodb.tcmalloc.tcmalloc.max_total_thread_cache_bytes

  • mongodb.tcmalloc.tcmalloc.pageheap_free_bytes

  • mongodb.tcmalloc.tcmalloc.pageheap_unmapped_bytes

  • mongodb.tcmalloc.tcmalloc.spinlock_total_delay_ns

  • mongodb.tcmalloc.tcmalloc.thread_cache_free_bytes

  • mongodb.tcmalloc.tcmalloc.transfer_cache_free_bytes

Fixes

Slim Agent Vulnerabilities

Fixed the vulnerabilities detected in the agent-slim v9.6.1 image. These issues are related to the python2 and jackson-databind packages. These packages were upgraded to the versions with fixes.

Run App Checks on Hosts with Python 2.6

Fixed a defect that prevented app checks from running on hosts that install Python 2.6.

9.6.1 February 28, 2020

Fixes

Metrics calculation

Fixed an issue that caused an error in the calculation of some metrics such asnet.* in agent version 9.6.0.

Red Hat-based host issue

Fixed an issue that caused the kernel module build associated with agent version 9.6.0 to fail on Red Hat-based hosts.

9.6.0 February 26, 2020

Upgrades

Integrations improved

Added new metrics and configuration options for HAProxy and Consul app check integrations. See HAProxy, and Consul for details.

Fixed a problem with Go app check which caused it to fail with an exception error.

Metrics added

Added Kubernetes metric kubernetes.namespace.pod.running.countto track the number of pods in running state. See Kubernetes Dashboards.

Reduced load on the Kubernetes API server

The version of client-go was updated and now defaults to encoded protobuf messaging instead of JSON to improve performance.

Configuration option new_k8s now enabled by default.

Default collector port changed

The default port for the collector was changed from 6666 to 6443.

Fix for the dynamic back-end configuration of Kubernetes Audit Logging caused some agent deploys to fail

The agent is enhanced to listen on /k8s-auditfor Kubernetes audit events and the path can be configured via the config option security:{k8s_audit_server_path_uris: [path1, path2]}.

Fixes

Prometheus metrics fix

Fixed a problem that inhibited the agent from scraping multiple ports on a single process for Prometheus metrics.

Inaccurate cpu.used reporting fixed

Fixed a problem that caused the agent to erroneously report very high CPU usage in some environments.

9.5.0 January 28, 2020

Fixes and Upgrades

Added new configuration option and metrics for Elasticsearch integrations

In the Elasticsearch app check, the parameter index_stats can be used to collect metrics from individual indices. See Example 4 in Elasticsearch and Elasticsearch Metrics for details.

Added new metrics for NGINX Plus integrations

More than 60 new metrics have been added to the NGINX app check. See NGINX Plus Metrics for details.

Made Go-based event handling the default

See Process Kubernetes Events. As of agent 9.5.0, the default setting for go_k8s_user_events is true and there is no need explicitly to enable it. To switch back to the older events monitoring (C++ based), set the value to false in your agent config (dragent.yaml).

Enhanced log tracing for include/exclude processes filter.

No user action is required; see Include/Exclude Processes to use the filter.

Fixed agent termination issue

Fixed a problem that was causing an internal process within the agent to repeatedly restart.

Improved memory buffer handling

The agent will now auto-disable memdump functionality when the memory buffer is too small.

Agent start/stop improvements on CRI-O and Openshift 4.x

The agent can now correctly perform the pause and stop container actions on clusters running OpenShift 4.x and CRI-O.