Subscribe to the RSS feed for Sysdig Release Notes to stay updated with
the latest product releases. Use your favorite news aggregation apps,
such as RSS Feed
Reader,
to get notified immediately when we post new content.
Add the desired RSS feed URL to your reader and it should notify you
whenever a new release note is available.
Reloading Promscrape V2 No Longer Causes Dropping Scrape Targets
Reloading promscrape v2 no longer causes dropping some scrape targets from sending metrics.
Losing Node No Longer Generates Duplicate Node Events
Resolved an issue that caused generating duplicate events when a Kubernetes node is lost.
Agents Connect to SaaS Backend Through HTTP Proxy on Older Hosts
Fixed an issue related to SSL certificate verification when connecting through an HTTP proxy on older host OS, such as CentOS 7.
Agent Refreshes Service Account Token as Expected
Connection with the Kubernetes API Server works as expected. The Kubernetes client is configured to refresh the bearer token.
12.5.0 May 02, 2022
Feature Enhancements
Default Availability of Slim Agent
The agent installation is defaulted to the slim agent. Slim agent reduces the surface area for potential vulnerabilities as compared to the full agent, which implies increased security for your monitoring environment. For more information, see Agent Installation.
To continue using the regular agent, set slim.enabled to false in your helm chart.
Container DriftControl: Detect and Prevent Drift in Container Runtime
Sysdig agent can now detect when a new executable was added to a container after a container has started up. The agent collects when a file was downloaded and made executable. When using prevention mode, the agent can also deny the process from ever running. A policy can also be used to define binaries that should be denied/excluded from being denied if they have been added after the container has started.
Switch syscall events are disabled for secure and secure light modes.
Known issues
An error message is displayed when the agent detects ceph and attempts to run an obsoleted app check.
The Sysdig agent for ARM can restart when multiple containers are started in rapid succession on the host.
Defect Fixes
Agent on zLinux No Longer Restarts Due to Incorrect Detection of tid Colliions
The agent on s390x architecture (zLinux) has been fixed so the agent does not restart needlessly due to incorrect detection of too many tid collisions.
Reports Correct CronJob Version When Adding CronJob Parents
Fixed an issue causing CronJobs to be reported not as the parents of Job objects.
Agent No Longer Crashes During Abnormal Termination
Fixed an issue causing the agent to crash with a stack backtrace during certain abnormal termination situations.
Slow-Starting JVMs Are Terminated Correctly
An incorrect detection of too many tid collisions on s390x architecture (zLinux) will no Longer cause the agent to restart periodically.
Kubernetes Events Are Collected as Expected
Fixed an issue that could prevent Kubernetes events from being correctly fetched.
Disable Watching HorizontalPodAutoscaler
Watching Horizontal Pod Autoscalers has been disabled by default to decrease load on Kubernetes API server. For more information, see Enable Kube State Metrics.
False Positive CVEs for Go Packages No Longer Reported
The Go compiler version has been upgraded to prevent getting flagged with (false-positive) CVEs associated with older Go versions.
Secure Events Reports Correct Cluster Information
The Secure events no longer report Kubernetes cluster name default when no cluster exists in the environment.
12.4.0 April 04, 2022
Feature Enhancements
Support for New Architectures
Installing agent on the following architecture are supported:
A new agent mode, custom-metrics-only, has been introduced. It enables all custom metrics and Kubernetes state metrics but disables all the driver-based metrics.
Prevent Processing Policy Updates
Prevent processing policy update messages to reduce CPU usage when no changes are required in the agent.
Known Issues
Increased Resource Consumption due to Misconfiguration of Node Lease
Incorrect configuration of Kubernetes lease can result in elevated memory usage in the Sysdig agent pods as well as increased load on the Kubernetes API server due to multiple agents querying for more information simultaneously. This also results in a significant amount of additional and unnecessary load on the Sysdig backend. To resolve this issue,
Upgrade to Sysdig agent 12.5.0 which adapts to the non-optimal Kubernetes configuration.
Configure the Kubernetes lease functionality.
If you are using Helm, the latest versions of the Sysdig Agent Helm chart defaults to configuring the lease functionality automatically.
If you do not use Helm, the DaemonSet and ClusterRole YAML files are available in our gitbub repository. For further assistance, contact Sysdig Support.
Agent Restarts Periodically on zLinux
An incorrect detection of too many tid collisions on s390x architecture (zLinux) can cause the agent to restart periodically. To workaround this issue, set the following configuration option:
This configuration change reduces the number of restarts to once a day instead of every 10 minutes, which is the default value for the above configuration option.
This issue has been fixed in Sysdig agent v12.5.0.
Defect Fixes
Validate Promscrape Scrape Jobs
Validate scrape jobs associated with promscrape integration before scraping the endpoints to avoid unnecessary errors with irrelevant scrape jobs.
Remove App Check Warning Messages When App Checks Are Disabled
Remove unnecessary warning messages about app checks limits when app checks are disabled.
Slow-Starting JVMs Are No Longer Terminated
Slow starting JVMs can be terminated by sdjagent. For example, -XX:+AlwaysPreTouch with large heaps. This fix introduces additional configuration options to tune the delay between sdjagent detecting a started JVM process and an attempt to connect.
Fixed metadata incompatibility in profiling with Kubernetes versions above 1.20.
Name Change to Configuration Parameter
The falcobasline.max_drops_buffer_rate_percentage parameter has been corrected to falcobaseline.max_drops_buffer_rate_percentage. Notice the missing e in falcobasline in falcobasline.max_drops_buffer_rate_percentage. However, the backward compatibility is ensured, and therefore, falcobasline.max_drops_buffer_rate_percentage can still be used.
12.3.1 March 03, 2022
Defect Fixes
Noisy Messages Silenced
Removed a kernel message from the driver that could generate spam when the syscall event buffer is full.
12.3.0 February 17, 2022
Feature Enhancements
Binaries Category for Falco Baseline
A new category, binaries is added to the Falco baselines feature.
Support for Workload Information in Falco Baseline
Add workload information to Kubernetes context for Falco baselines.
Default Monitoring of Kubernetes Resources
The following kubernetes resources are monitored by default:
IPv6 Addresses Are Saved Incorrectly When Adding Rules
Adding a new rule causes problem saving IPv6 address for both fd.net and fd.ip.
Defect Fixes
Fix Truncated Capture Files
Fixed a problem which caused the agent to generate truncated capture files.
Container Action Pause Work on Kops/GKE Clusters
Fixed the logic that determines the cgroup path for a container in containerd and made the freezer subsystem available to the agent in order to be able to pause/unpause it.
Agent Profiling Works as Expected
High CPU load no longer prevents generating CPU and memory profiles in the agent.
Agents Are Not Reset with Signal 11
Large and negative file descriptors are handled correctly so agents are no longer reset with signal 11.
12.2.1 February 07, 2022
Feature Enhancements
Manage Collecting Metadata from Individual Container Engines
Access to individual container engines from within the agent for fetching metadata can now be disabled via agent configuration.
For example, to disable docker, use the following configuration:
container_engines:
docker: false
Known Issues
The Pause policy action is not working as expected in GKE, EKS, and Openshift4 environments.
Defect Fixes
Policy Action “Kill” Is Correctly Triggered in GKE Environments
Policy action on GKE with containerd works as expected:
The container is stopped if HTTP proxy is enabled.
The status of the container is checked upon stop requests. If the status is not CONTAINER_EXITING, termination of the container is attempted with exponential backoff.
Agents Assign Username Correctly for Container Events
Fixed an issue that prevented the proc.name field from extracting the right user from the container started events. This issue was found in agent versions 12.2.0 and above.
12.2.0 January 25, 2022
Feature Enhancements
Improve Install Script to Support eBPF
A new option, bpf or -b is added to the native install script of Sysdig agent to support eBPF.
Enable 10s Flush by Default
By default, the agent collects metrics at 1-second granularity, then aggregates and sends them to the backend in 10-seconds intervals. If you want to use agent versions 12.2.0 or above with the on-prem Sysdig Platform versions below 3.5.0, set the 10s_flush_enable configuration to false to prevent compatibility issues.
The backend in our SaaS deployments continues to enable 10-second flush automatically for all agent versions 10.0.0 or above.
Improved Log Messages
Improved the log messages to report the errors encountered while configuring subprocess_resource_limits.
Known Issues
Processing Secure policy updates in the agent can take longer than it did in the previous releases, and in some rare scenarios, it causes agent restarts.
Defect Fixes
Fix CVE-2020-29652 in Cointerface
Updated crypto go module to fix CVE-2020-29652.
Promscrape V2 No Longer Crashes on Pods with Multiple Containers
Prevent promscrape_v2 from crashing when a pod has multiple containers.
skip_events_by_type Works as Expected
Fixed an issue in the kernel probe, which prevented the skip_events_by_type feature from correctly filtering events by system call type.
Kubernetes State Is Transmitted as Expected
Fixed an issue where Kubernetes information and metrics would not be sent from the agent. This scenario arose when the agent was deployed in a namespace other than sysdig-agent, and the agent daemonset did not include the podinfo volume.
Agent Successfully Connects to JMX
Fixed an issue where agent wouldn’t connect to JMX on some applications/JVMs. This issue was originally observed on the WebSphere application and Liberty JVM.
Agent Updates Container Status as Expected
Fixed an issue where the agent would not update the container status it first received from the API server. The agent now updates the container statuses as it receives them from the API server.
Check for Invalid Log Level in sdjagent
Fixed an issue where using a log level of none caused sdjagent to crash.
App Checks Run as Expected on Non-Containerized Agent Installations
Fixed an issue preventing app checks to run on non-containerized agent installation.
Native Install Doesn’t Support eBPF
Prevents insertion of Sysdig probe kernel module when the agent is installed with eBPF by using rpm or deb package.
Prevents Connection Attempts When Agent Encounters Errors
Connection attempts are prevented when the agent encounters errors while handling handshake messages.
12.1.1 November 22, 2021
Defect Fixes
Falco Action Works as Expected
The kill container Falco action works as expected for containerd in Azure.
12.1.0 November 08, 2021
Feature Enhancements
Ability to Build eBPF Probes for Debian 11 Kernels
The agent container has been enhanced to build probes for Debian 11 kernels.
Prebuilt Probes for Debian 11 Kernels
Prebuilt probes are added for Debian 11 kernels.
Prebuilt Probes for Fedora Kernels
Prebuilt probes are added for latest Fedora kernels.
Ability to Build eBPF Probes for Linux Kernel v5.10
The agent container can now build eBPF probes for Linux kernel version 5.10.
Enhanced Agent Containers for Probes on New Kernels with glibc v2.33
The agent container has been enhanced to build probes for new kernel versions that use glibc v2.33.
File Metrics in Audit Tap
Metrics related to file are included in audit tap.
Promscrape Memory Usage Limit
You can now limit the promscrape memory usage. The default is set to 640 MB. For more information, see Sysdig Agent.
Remove Self-Signed Certificate for Agent to Collector Connection
Self-signed certificate support has been removed for agent connection to the collector. See End of Support.
Defect Fixes
Image Profile Shows Results Correctly
The imageid is reported correctly when using a CRI engine.
Duplicate Environment Variable Hashes No Longer Appear in Audit Tap
The discrepancy between reported environment variables and hash in audit tap has been fixed.
Kubernetes Daemonset and Replicaset Association Works as Expected
Fixed an issue that could invalidate the association between Kubernetes Daemonset and Replicaset.
Agent Updates Prometheus Configurations Correctly
Fixed a problem that was causing Prometheus configurations to be merged incorrectly when certain integrations were updated from the backend.
12.0.4 October 29, 2021
Defect Fixes
Secure Policies Load as Expected
Fixed an issue present in 12.0.3 where Secure policies might not be loaded correctly by the agent.
12.0.3 October 22, 2021
Defect Fixes
Leases Fallback Works as Expected on OpenShift v3
Fixed an issue where Kubernetes clusters that don’t support leases failed to report Kubernetes data due to not falling back to the previous behavior.
Update the Cluster Install Scripts for Leases on OpenShift
Modified the OpenShift agent installer to add the sysdig-agent cluster role and to assign it to the sysdig-agent service account. The new cluster role allows the agent to utilize the coldstart leases.
12.0.2 September 30, 2021
Defect Fixes
Network Security Communication Works As Expected
In some environments Sysdig agents could not send any Network Security (Kubernetes Network Policies) communications upon not completing CIDR auto-discovery. This issue has been fixed.
Agent No Longer Crashes in Orchestrated Environments
Fixed a problem related to a race condition in orchestrated environments, such as OpenShift v3, due to which the agent might crash repeatedly at the agent start.
12.0.1 September 27, 2021
Defect Fixes
OpenShift 4 Clusters Able To Retrieve Metadata Without Leases
Fixed an issue where OpenShift clusters would fail to report Kubernetes data when the agent service-account did not have the permission to create leases. With this fix, the Sysdig agent falls back to the previous behavior to retrieve the metadata.
12.0.0 September 15, 2021
Feature Enhancements
Allow Sysdig Backend to Manage Prometheus Configuration
Allow Sysdig backend to manage Prometheus configuration. For more
information, see the following:
The Agent Console now supports troubleshooting Prometheus configuration.
To support this feature, Agent Console is enabled by default. This helps
both users and Sysdig support to troubleshoot Sysdig agent issues.
Sensitive user configuration is obfuscated and not viewable.
Add Startup Delay to Agent to Kubernetes API Server Connection
Added a delay prior to the agent connecting to the Kubernetes API
server. The delay time is set based on the number of nodes in the
cluster to prevent overloading the API server. This is to support
environments where node leases cannot be used.
Known Issues
None
Defect Fixes
Stale Capture Files No Longer Exhaust Local File System
Prevent incomplete and stale capture files from being left behind and
thereby avoiding storage consumption for such files.
Honor CPU Quotas
Moved the main dragent process to the default cgroup so that CPU
quotas can cover all the agent processes.
Containers Are Detected as Expected
Fixed issue where containers are not detected if SystemdCgroup = true
is not enabled in the containerd configuration.
Report Correct Container Metadata
Fixed a problem that caused some container metadata such as the image
repository and image tag to be reported incorrectly.
Upgrading from 10.8.0 to 11.3.0 No Longer Fails
Provide a http_proxy configuration option to address connection
problems post-OpenSSL upgrade from v11.0 to v11.1.
11.4.1 August 03, 2021
This is a hotfix release.
Defect Fixes
Fixed a problem that broke app checks in agent-slim by adding the
missing dependencies.
11.4.0 July 28, 2021
Feature Enhancements
Probe Builder
The probe builder can now be used to build kernel modules for the Sysdig
agent. It can run on any host with Docker installed, including (with
some preparation) air-gapped hosts.
Promscrape v2 (used when prom_service_discovery is enabled for
Prometheus) has been changed to discover only Kubernetes pods running on
the same node as the agent. This should help reduce the load on the
Kubernetes API servers in large clusters.
Added Missing Fields for Unified Workload Metrics
Added Kubernetes metric fields indicating the availability of daemon
sets (status.numberAvailable, status.numberUnavailable, and
status.updatedNumberScheduled) and replica sets
(status.availableReplicas) to support workload-level metrics (SaaS
only).
Known Issues
App checks in agent-slim don’t work due to missing dependencies. This
problem will be addressed in an upcoming hotfix release.
Defect Fixes
Multiple Hosts No Longer Report the Same Pod
Fixed an issue causing multiple hosts to report the same pod if its UUID
is the same on both hosts.
Duplicate StasD Metrics Are Reported Correctly
Fixed an issue related to handling duplicate StatsD metrics
corresponding to a container that is reported by a host.
Stale Markers Are Sent properly for Dropped Targets
Properly generate stale markers for Prometheus metrics when a scrape
target is no longer available and when using promscrape.v1.
Report a Positive Time Delta Value
Fixed a defect that could result in an invalid file.time.in,
file.time.out, file.time.other, and file.time.total values.
Agent No Longer Crashes When App Check or Prometheus Is Enabled
Fixed a defect that could cause crashing the agent when app checks or
Prometheus is enabled.
Secure Captures No Longer Causes Host Shutdown
Prevent agent restarts caused by apparent stalls encountered in the
sample handler thread.
agent-kmodule and agent-kmodule-thin can now be used to build eBPF
probes.
Replication Controller Fields
Added missing replication controller fields to the aggregator Actions.
Non-Delegated Agents Retrieve Less Data From the API Server
Use Kubernetes leases to better control the load on the Kubernetes API
Server. This is disabled by default.
Defect Fixes
Agent No Longer Generates Core Dumps on Java
Prevents java process core dumps caused by the Sysdig agent while trying
to access /tmp directory.
Support Container Action on Containerd
Container actions are now properly supported on containerd (CRI-O and
other CRI engines that already had support). Actions for unsupported
container engines are now properly reported to the Sysdig backend and a
warning message is logged in the agent logs.
Recovery During Agent Shutdown
Introduced a detection and recovery mechanism for hangs during agent
shutdown.
Promscrape V2 Termination No Longer Causes Agent Crash
Fixed a problem causing the agent to crash after promscrape_v2 is
terminated.
Agent No Longer Restarts in Kubernetes Environment
The agent tries to fetch the metadata of the AWS instance in which it is
running in order to tag metrics generated with the information unique to
the AWS instance. If the metadata structure is not as expected, the
agent continuously restarts due to an error in fetching such metadata.
This issue has been fixed.
Profiling Works as Expected
Fixed an issue that disabled support for performance profiles in the
agent.
11.2.1 May 06, 2021
This is a hotfix release.
Defect Fixes
Report Container User Information
Start tracking container user information and make that information
accessible in container events. These events denote having a container
started. This feature works for Docker as well as CRI-O container
engines.
Reporting container user information does not work in OpenShift 4.x
because it does not provide necessary CRI-O information.
11.2.0 April 26, 2021
Feature Enhancements
Agent CLI
Sysdig supports Agent CLI, a command-line interactive tool, to
troubleshoot agents. This tool helps Sysdig support to solve user issues
quickly and efficiently. It is currently disabled by default and
requires the customer to turn it on.
For IKS, the agent is installed by default using IKS Agent
Script
Use the -af option to install the full agent.
Reduce Load on Kubernetes API Server
Terminated pods are no longer collected in order to reduce the load on
the Kubernetes API server.
Audit Server Listens on All Interfaces
The audit server now by default listens on all the interfaces for
Kubernetes audit events. This makes integration with Kubernetes audit
events in the agent easier without the need for configuration changes.
Improved Noise-Reduction Filter for Activity Audits
The noise-reduction filter for Activity Audit has been improved. All the
filtered data is duplicated.
Defect Fixes
CRI-O Versions Report Correct Image ID
The new CRI-O versions (1.19+, possibly 1.18) now properly report
container.image.id.
Log Level Changes for Duplicate Host Container Groups
Demoted logs about duplicate host container_groups from warning to
debug level
Fix CVE-2021-28831
Fix CVE-2021-28831 in the Slim Agent container.
11.1.3 April 13, 2021
This is a hotfix release.
Defect Fixes
Prevent Agent CrashLoopBackoff Error Caused by Smaller initialDelaySeconds Values
The readiness probe improvement in version 11.1.2 delayed the transition
of the agent pod to a ready state until communication with the
Kubernetes API server was established. But this delay could cause a
CrashLoopBackoff due to liveness or readiness probes configured with an
initialDelaySeconds set to less than 90.
In Agent version 11.1.3 the transition to the ready state does not wait
for communication with the Kubernetes API server to be established
unless the behavior is enabled via a new configuration option:
k8s_wait_before_ready.
11.1.2 March 30, 2021
Known Issues
Prevent Agent CrashLoopBackoff Error Caused by Smaller initialDelaySeconds Values
The readiness probe improvement in version 11.1.2 delayed the transition
of the agent pod to a ready state until communication with the
Kubernetes API server was established. But this delay could cause a
CrashLoopBackoff due to liveness or readiness probes configured with an
initialDelaySeconds set to less than 90.
Workaround
If you are using agent version 11.1.2, set initialDelaySeconds for
both liveness and readiness probes to a value that is greater than or
equal to 90.
Feature Enhancements
Enhanced Connection with Kubernetes API Server
Kubernetes reconnect logic has been improved to automatically backoff (1
min, 2 min, 4 min… 1hr) if the connection is continuously dropped when
using Thin Cointerface. This reduces the load that the agent imposes on
the Kubernetes API Server in clusters with heavily burdened API servers.
Reduced Load on Kubernetes API Server
The agent’s readiness probe has been improved to not report ready until
after the agent connects to the Kubernetes API server. This reduces the
load that the agent imposes on the Kubernetes API server when starting
up during RollingUpdate.
11.1.1 March 26, 2021
Defect Fixes
Agent Reports Memory Usage Accurately for Containers
Fixed an issue where the agent would incorrectly report
memory.bytes.used for containers that use more than 4GB.
Runtime Policies Work as Expected
The runtime policies that have a policy type and capture action are
handled as expected.
11.1.0 March 23, 2021
Defect Fixes
Agent Tags in Policy Scopes
Agent tags are supported in runtime policy scopes.
Metric Limits Are Updated As Expected
Fixed a problem where metric limits were not updated from the defaults.
This is unlikely to happen if agents are connected to the SaaS backend.
Configured Tags in Prometheus Scraper
Fixed a problem in the old Prometheus scraper (used when promscrape is
disabled) to ensure that configured tags are properly added to the
metrics.
JMX Metrics for Short-Lived Java Processes
Fixed an issue where short-lived Java processes could cause the Sysdig
Agent to stop collecting JMX metrics.
Misconfiguration No Longer Leads to Agent Constantly Querying Kubernetes API Server
Fixed a problem where the agent would continuously send requests to the
Kubernetes API server to query the endpoints API. This occurs when the
agent’s clusterrole is incorrectly configured. With this fix, the
agent will no longer repeat the attempt if it is unable to connect to
the Kubernetes API during boot.
Scope Runtime Policies
The runtime policies are now correctly scoped by
kubernetes.cluster.name. The fix in 10.6.0 was incomplete.
Agent Correctly Reports Replicasets
Fixed an issue where the agent could lose track of a replicaset and
report incomplete metadata.
Agent Issues Over HTTP Proxy
Fixed an agent connection issue over plaintext HTTP proxy with
encryption.
Fixed an agent connection issue via HTTP proxy connections over SSL.
11.0.0 February 18, 2021
Feature Enhancements
Thin Cointerface to Reduce Memory Usage
Thin cointerface reduces the memory required to handle the Kubernetes
metadata on both the agent and the Kubernetes API Server. The reduction
in memory usage is significant for Kubernetes clusters with a large
number of pods (in the range of 10,000 or more) or clusters that heavily
use Replication Controllers.
Using this feature returns the same data to the Sysdig backend and does
not affect any Sysdig features. The thin cointerface feature is disabled
by default.
To enable:
Add the following in either the sysdig-agent’s configmap or via
the dragent.yaml file:
Some high-frequency information level log messages are converted to
debug level to reduce the volume of messages generated at the default
information level.
File Logging Capability
Per-component file logging capability for an additional set of agent
components has been enabled.
The number of Prometheus time series ingested has been limited to reduce
agent memory consumption. This limit is applied after Prometheus
relabeling rules are applied but before the agent’s metric filter and
metric limit.
Defect Fixes
Missing Metrics Due to Aggregation in Agent Fixed
Fixed an issue where processes with certain names were improperly
aggregated, which in turn caused missing metrics in certain situations.
Cointerface Fix
Fixed an issue that caused the agent’s cointerface process to restart
continuously while processing kubernetes label selectors.
10.9.1 January 21, 2021
Defect Fixes
Thin Cointerface Works as Expected
Fixed a defect in the Thin Cointerface feature which could cause
Kubernetes metadata to stop updating. Because Thin Cointerface is turned
off by default, the change affects only a small number of users who have
this feature turned on.
10.9.0 January 13, 2021
Feature Improvements
Support for Kubernetes Cronjobs
Kubernetes cronJobs are supported when reporting network communications.
Defect Fixes
Runtime Policies and Rules Are Loaded with No Errors
Fixed a race condition that could prevent runtime policies and rules
from being loaded properly if multiple messages from the Sysdig backend
are received consecutively.
Cluster Overview Displays Compliance Score
Fixed an issue where Statsd metrics related to compliance would have no
associated Kubernetes metadata and were not visible on Cluster
Overview.
10.8.0 December 18, 2020
Defect Fixes
Filtering Long Container Labels
Filtering long container labels works as expected with no parsing
failures or undesirable agent restarts.
Correct kubernetes.pod.restart.rate Metric
Fixed an issue that could cause kubernetes.pod.restart.rate metric to
be incorrect.
Prometheus Metrics With Multiple Process Listening Concurrently
Fixed a problem that caused scraping Prometheus metrics to fail when
another process was listening to the TCP port 9090 on a host interface.
StatsD Metrics Reports Correct Value
Fixed a problem that caused Statsd metrics to report incorrect values.
Correct Environment Variable Hash in Audit Tap
Fixed an issue that could cause the environment variable hash associated
with the exported processes in audit tap to have an incorrect value.
Improve JMX Availability Check
The sdjagent process in the agent no longer consumes excessive CPU
resources.
10.7.0 November 20, 2020
Feature Improvements
Policies and Baselines V1 Messages Are Deprecated
Sysdig agent no longer supports the old backend message types that were
originally deprecated in on-prem release 2.4.0 (August 2019).
Load Falco Rules on a Separate Thread
Partially load Falco rules in the background to avoid interrupting event
processing.
Workflow for Unacknowledged Metrics
The agent is restarted if a metrics acknowledgment hasn’t been received
from the Sysdig backend components in 8 minutes. This can happen if
networking issues cause the agent to believe it has an active connection
when the backend has closed the connection.
Run Single Agent RPM Per Host
Prevents multiple agent services from being launched on the same
RHEL-based hosts.
Known Issues
The host.container.start.count metric acts as a counter metric and its
value increases monotonically.
Defect Fixes
OpenShift Hardening Guide Correctly Detects Master and Worker Nodes
Running the OpenShift Hardening Guide functionality of the Kubernetes
Benchmark will now correctly detect master vs worker nodes, and run the
appropriate Benchmark tests.
Agent No Longer Terminates Non-Agent Processes
In some rare situations when process creation in the Agent’s JMX module
failed due to issues caused by resource limits, it could inadvertently
stop unrelated processes running on the host. This problem has been
fixed.
10.6.0 October 30, 2020
Feature Improvements
Python 2.7 Is No Longer Supported in Agent Containers
Python 2.7 has been removed from the agent and agent-slim containers.
This is a breaking change for users who are using an agent container and
have set the python_binary configuration to /usr/bin/python2.7.
To prevent breaking the setup, do one of the following:
Remove the python_binary configuration option.
Set python_binary to /usr/bin/python3.
Sysdig agent continues to support python 2.7 if installed as a service
and the host has python 2.7.
Kubernetes Benchmarks
Updated kube-bench to support Kubernetes benchmarks and targets. For a
complete list of benchmarks, see Benchmarks (Legacy)
.
Kubernetes benchmark 1.6
Master
Control plane
Node
etcd
Policies
Google Kubernetes Engine (GKE) Benchmark 1.0
Master
Control plane
Node
etcd
Policies
Managed services
Amazon Elastic Kubernetes Service (EKS) Benchmark 1.0
Control plane
Node
Policies
Managed services
Configuring Prometheus Metric Expiration Time
Configuring metrics expiration time is supported by promscrape.v2 for
Prometheus metrics gathered by using Prometheus service discovery.
Support for Scoping Policies by Kubernetes Cluster Name
Add support for scoping policies by kubernetes.cluster.name. The
cluster name must still be manually configured by using the
configuration option, k8s_cluster_name: <CLUSTER NAME>.
Improved Prometheus Service Discovery
Made kubernetes node matching more reliable for Prometheus Service
Discovery by comparing IP addresses as opposed to node names in the
default configuration.
Defect Fixes
CVE Fixes
Addressed a known vulnerability in the jackson-databind package
version 2.9.10.6 by upgrading to version 2.11.3 in agent containers.
Reduce Severity of NoClassDefFoundError Log from Error to Info
Changed the java NoClassDefFoundError class from Error to Info to
reduce spamming the logs at the Error level. This happens commonly when
the agent attempts to read metrics from a java v11 application which was
not started with the com.sun.management.jmxremote option.
StatsD Metrics No Longer Show Larger Than Expected Values
Fixed a problem that caused StatsD metrics to be double the expected
value.
Remove Warning Logs
Removed warning logs about ambiguous source labels when using the
Prometheus service discovery with multi-container pods.
10.5.2 October 21, 2020
Defect Fixes
Memory Leak No Longer Occurs in the Agent
Fixed an issue that could potentially cause a slow increase in the
agent’s memory usage over time when the thin_cointerface_enabled
configuration option is enabled.
10.5.1 October 08, 2020
Feature Improvements
Added New Rules to the Prometheus Configuration to Honor Pod Annotations
Improved the default Prometheus configuration for promscrape.v2 to
honor pod annotations.
Known Issues
Logs warning messages in the agent log file when promscrape.v2 is
enabled.
Defect Fixes
Pods Are No Longer Associated with Incorrect Deployments
Fixed a problem that could cause a pod to be associated with incorrect
deployments.
10.5.0 September 24, 2020
New Features
Enable Communication Between Agent and Collector Through a Proxy Server
Sysdig agent to the collector communication can be established via an
HTTP or an HTTPS Proxy server.
A new version of promscrape, promscrape.v2 , has been introduced to
offer native Prometheus service discovery capabilities. To support this,
a default prometheus.yaml file has been added with Kubernetes pod
discovery rules to use when native Prometheus service discovery is
enabled. See Enable Prometheus Native Service
Discovery for more
information.
Secure Mode
Sysdig agent now supports secure mode that offers Secure only features.
See Secure
Mode
for more information.
Known Issues
None.
Defect Fixes
CVE Fixes
Addressed vulnerabilities reported in the agent and agent-slim
containers, including the one for CVE-2017-18640 in a dependency library
related to image scanning.
Agent No Longer Hangs While Handling Connection Errors
Fixed an issue that caused the agent to hang while handling some types
of connection errors. When this issue is encountered, restarting the
agent will allow it to reconnect.
Upgrading to Sysdig agent v10.5.0 or higher is strongly recommended to
avoid this problem.
Scraping Prometheus Endpoints in Docker Containers
Prometheus metrics can now be scraped from endpoints in Docker
containers with remapped port numbers.
Prevent Agent Crashes in Large Systems
The agent now starts faster on systems with thousands of processes and
hundreds of containers.
Warning for Prometheus Metric Limit
The agent logs a warning once in a minute when the Prometheus metric
limit is reached.
Transmitting Prometheus Metrics Works As Expected When Service Discovery Is Enabled
Fixed a problem that could randomly result in Prometheus metrics not
being sent when Prometheus service discovery is enabled.
Appcheck Metrics No Longer Go Missing
Fixed a problem that would cause certain app check metrics to be missing
when 10-second aggregation in the agent is enabled.
Agent Now Times Out If Connection Attempt to Collector Does Not Work
Added a timeout to the handshake protocol between agent and collector.
Agent Now Collects JMX Metrics from New Process Following a Java Service Restart
Fixed a problem that randomly caused JMX metrics to be not collected due
to transient errors encountered during the startup of new Java
processes.
Pod to Service Connection
Fixed a problem that caused the UI to show a pod under an incorrect
service if other services exist in different namespaces with the same
selectors. This happened when the thin_cointerface_enabled property
was set to true.
Syscall Fast Rule Triggers as Expected
Fixed the evaluation of secure fast engine syscall rules when the
If Not Matching rule is selected.
10.4.1 August 26, 2020
Defect Fixes
Kubernetes Pods No Longer Lose Association with Resources
Fixed a problem that could cause Kubernetes pods to lose association
with their deployment or other related resources.
10.4.0 August 19, 2020
New Features
Ability to Scrape Prometheus Metrics from Container IP Addresses
The agent can now scrape Prometheus metrics from the docker containers
that expose ports only on specific IP addresses besides the localhost.
The default value (300) of per-process JMX bean limits can now be
changed as follows:
jmx:
max_per_process_beans: 500
Known Issues
Handling Benchmark Task When StatsD Metrics Collection Is Disabled
When Statsd is disabled, do not attempt to send metrics related to
benchmarks tasks. This also means that benchmarks dashboards will not
have data when Statsd is disabled.
Kubernetes Pods Can Lose Association with Resources
A problem that could cause Kubernetes pods to lose association with
their deployment or other related resources has been identified in Agent
version 10.4.0. A new version, 10.4.1, that will address this problem is
currently in development.
Defect Fixes
Kubernetes Audit Server and Agent Process Restart Congruently
Embedded web server for Kubernetes audit events restarts as expected
when the agent process is restarted.
CVE Fixes Related to Slim Agent v10.3.0
Updated the version of the jackson-databind package to fix
vulnerabilities discovered in the slim agent v10.3.0
10.3.1 August 06, 2020
Defect Fixes
Kubernetes Benchmark Tasks No Longer Fail
The kube-bench binary that was identified as broken due to the change
in the output format has been fixed.
kube-bench that performs the Kubernetes Benchmarks tasks has changed
the output format, causing the existing Benchmark tasks to fail in
v10.3.0. With this fix, the agent will no longer throw errors related to
this issue and the new Kubernetes Benchmark results will appear in the
UI as expected.
Probes Works As Expected for v5.8 Kernels
Fixed an issue with building probes for Linux v5.8.0 kernel.
10.3.0 July 28, 2020
New Features and Enhancements
Changes to the Monitor Mode
URL segmentation for metrics has been moved from the default monitor
mode to the troubleshooting mode. Due to this change, dashboard panels
with per URL metric will show no data. See Additional Metrics Values
Available in
Troubleshooting.
Sysdig Probe Location Changes
The Sysdig probe URL is changed to download.sysdig.com.
If the Sysdig probe URL is included in the allow list for outbound
firewall access, you must change the endpoints to reflect the new probe
location.
Agent Connects to Promscrape through UNIX Socket By Default
The agent now connects to promscrape through a UNIX socket by default
as opposed to the TCP port 9876.
New Configuration File Paths for Kube Proxy
The version of kube-bench has been upgraded to 0.2.4. The changes
include an additional configuration file path for Hyperkube kube-proxy
to support OpenShift.
Known Issues
Kubernetes Benchmark Tasks Fail
The kube-bench binary is broken due to the change in the output format
and the issue will be fixed in an upcoming release.
kube-bench that performed the Kubernetes Benchmarks tasks changed the
output format, causing the existing Benchmark tasks to fail. The new
Kubernetes benchmark results will not appear in the UI, and the agent
will report errors related to Kubernetes benchmark tasks.
Defect Fixes
EndPoints-Independent Metrics Limits for Prometheus
Prometheus metric limits have been modified to ensure that endpoints
with fewer timeseries are not affected when another endpoint hits the
limit. Reporting of Prometheus timeseries statistics has also been
updated.
Prometheus Count Metrics for Summary and Histogram
The calculated Prometheus _count metrics are reported for summaries
and histograms even when the _sum values are missing. This feature is
not applicable to raw metrics.
A .count metric (which is the rate of change of _count values) and
a .avg (which is the average of new samples when _count increases)
are calculated for summaries and histograms. Earlier,
those .count and .avg metrics are reported only if the raw
Prometheus metrics include both _sum and _count values. In this
release, changes have been made such that _sum values are no longer
required to calculate Prometheus _count metrics for summaries and
histograms.
Reporting Running Pod Counts
Fixed an issue pertaining to the reporting of running pod counts for
replication controllers, deployments, and ReplicaSets.
Segmenting Kubernetes Jobs Metrics By Namespace
Fixed an issue that prevented having Kubernetes jobs segmented by
namespace.
Agent No Longer Stalls Under High Load
Fixed an issue that caused the agent to stall under high load.
Restarting Agent No Longer Causes Exception
Fixed an issue that caused an exception at agent restart while
collecting CPU metrics.
10.2.0 June 25, 2020
New Features and Enhancements
Prometheus Scraping
Periodic logging of statistics for Prometheus timeseries has been added.
When a metric limit is hit, all the timeseries metrics associated with
the endpoint are dropped.
App Checks and Prometheus Metrics
Processes with app checks or Prometheus metrics are now included by
default in the top processes to be sent to the Sysdig collector.
Performance Improvement
A variety of performance improvements have been rolled out to accelerate
the evaluation of Falco rules and fast engine rules for the common case
of events not matching any rules/policies.
Detect JSVC Processes as Java Programs
The agent has been enhanced to detect JSVC processes as java programs to
enable the collection of JMX metrics.
The following deprecated App Checks have been removed and will no longer
be supported.
Network
RiakCS
TokuMX
Ceph
Gearmand
Gunicorn
Kyoto Tycoon
Teamcity
Riak
Solr
OpenStack
Defect Fixes
Fixed a Race Condition
Fixed a potential race condition that could occur when receiving
multiple policies and related messages from the Sysdig collector at
nearly the same time.
Benchmark Task Configuration
The agent no longer runs a built-in set of benchmark tasks. The agent
will only run benchmark tasks when configured to do so by a Sysdig
Secure backend.
Prometheus Metrics From Idle Processes Are No Longer Dropped
Prometheus metrics from idle processes are no longer dropped even if the
target processes are not active enough to be in the top processes.
Additionally, the app_checks_always_send parameter, which can force
report the idle processes with metrics, now works as expected for
metrics gathered by promscrape.
Removed Authentication Credentials
Removed sensitive authentication credentials related to app checks from
debug log messages.
Kubernetes Events Are No Longer Dropped
Kubernetes events are no longer dropped under some high load conditions.
Memcached App Checks Collects Slabs and Items Stats
Fixed a problem that prevented the collection of slab and item stats in
the Memcache app checks in certain Python environments.
Metrics No Longer Report Incorrect Zero Values
The following metrics now no longer return incorrect zero values:
kubernetes.resourcequota.cpu.requests.hard
kubernetes.resourcequota.cpu.requests.used
kubernetes.resourcequota.memory.requests.hard
kubernetes.resourcequota.memory.requests.used
Agent Automatically Restarts Upon Protocol Mismatch Errors
The agent used to require manual intervention to recover from protocol
mismatch errors received from the Sysdig Backend. This error can occur
when the agent and Sysdig Backend are not in sync. The agent has been
enhanced to automatically restart when this error is encountered, so
manual intervention is no longer required.
10.1.1 June 02, 2020
Defect Fixes
Enable Network Topology
Network stats metrics that were moved to the troubleshooting mode in
Agent v10.1.0 have been re-enabled by default. The metrics will now be
available in the monitor mode, which in turn will enable the network
topology by default.
Added JMX support for Java 11, 12, 13, and 14 JRE. For containerized
Java apps with JRE, run the app with the
-Dcom.sun.management.jmxremote option.
Added Rate Limiting Configurations
Added rate limiting configurations to the agent to avoid connection
timeouts for metrics and secure messages.
Added New Metrics
Added a new metric to display the kernel version of the host where the
agent is running.
host.uname
This metric can be segmented by host.uname.kernel.name,
host.uname.kernel.release , and host.uname.kernel.version. For
more information, see
host.uname.
Added Container Name to the Containerd Event Description
Added container name to the containerd events description. In some rare
cases, the container name associated with a containerd event might be
unavailable due to metadata lookup delay.
Removed Authentication Credentials
Removed sensitive authentication credentials related to app checks from
debug log messages.
Removal of Deprecated App Checks
The following deprecated app checks will be removed in an upcoming
release:
Network
RiakCS
TokuMX
Ceph
Gearmand
Gunicorn
Kyoto Tycoon
Teamcity
Riak
Solr
OpenStack
Enable Removed Metrics
Some
metrics
related to network and file will not be available by default. You can
enable them by editing the dragent.yaml file.
Edit the Configuration File
Open the dragent.yaml file.
Add the following configuration parameter:
feature:
mode: troubleshooting
Restart the agent.
Removed Metrics in Agent v10.1.0
The following metrics will not be reported by default in agent v10.1.
When segmented by a particular label, these metrics will not have some
values. The table summarizes the metrics and missing values when they
are segmented by a particular label.
Metrics
Unreported Metrics Values When Segmented by
file.error.total.count
file.name and file.mount labels
file.bytes.total
file.bytes.in
file.bytes.out
file.open.count
file.time.total
host.count
host.error.count
proc.count
proc.start.count
net.bytes.in
net.connection.server, net.connection.direction, net.connection.l4proto and net.connection.clientlabels
net.bytes.out
net.connection.count.total
net.connection.count.in
net.connection.count.out
net.request.count
net.request.count.in
net.request.count.out
net.request.time
net.request.time.in
net.request.time.out
net.bytes.total
Defect Fixes
Promscrape No Longer Breaks Metrics Collection Over HTTPS
Fixed promscrape to honor the ssl_verify configuration option.
Slim Agent Container No Longer Prevents Certain App Checks From Emitting Metrics
Fixed an issue with the agent-slim container that prevented postgres
and pgbouncer app checks from emitting metrics.
Reduced the Frequency of Log Messages
Reduced the frequency of a log message to reduce spam and enhanced a
statsd related log message to provide more information about
incorrectly formatted strings.
Use Exact Rule Names When Adding Rules to Runtime Policies
Consider only exact matches when linking secure runtime policies to
Falco rules to fix this issue.
Corrected Calculation of net.bytes.* Metrics
Fixed calculation of net.bytes.* metrics at the host level when using
calico interfaces or VPN tunnels.
10.0.0 May 01, 2020
New Features
Kubernetes Benchmark Master Programs
Added the ability to run Kubernetes Benchmark Master Programs on
additional Kubernetes distributions.
New Scraping Mechanism for Prometheus
A new process, called promscrape, has been introduced to scrape
Prometheus metrics by default. The mechanism, based on the open-source
Prometheus, improves compatibility and performance. It also allows
per-endpoint metric filtering and relabeling through
metric_relabel_configs.
Added the ability to make draios.log files readable by users other
than root. This can be enabled with the following configuration in
dragent.yaml.
log:
globally_readable: true
New Runtime Policy Action
Added the ability to kill containers as a runtime policy action. See
Manage Policies for
details.
Defect Fixes
Fixed the Path Parameter Issue in Prometheus Configuration
Fixed the use of the path parameter in Prometheus configuration when
using promscrape. With this fix, the configured path is passed to
promscrape by the agent when it is set up for a target rule in
dragent.yaml.
Service Annotation Based Prometheus Scraping
Prometheus scraping can now be triggered based on service annotations by
default.
Added a Missing Module to the agent-slim Container
Added the missing posix-ipc module to the slim agent. This fixed an
issue that prevented App Checks from running in the agent-slim
container on v9.9.0.
No Metric Limit on Scraped Prometheus Metadata
Prometheus scraping metadata is no longer counted toward, or limited by,
metric limits when using promscrape.
Fix for Percentile Metrics
Fixed a defect that caused percentile metrics to not work properly.
9.9.1 April 16, 2020
Defect Fixes
Added the Missing Module to the Slim Agent
Added the missing Posix module to the slim agent. This fixed an issue
that prevented App Checks from running in the agent-slim container on
v9.9.0.
9.9.0 April 13, 2020
Core Features and Fixes
Python 3 Set as Default and Some App Checks Deprecated
Python 3 is the new default Python version for app checks, instead of
Python 2. Python 2 can still be used by setting the following option in
your dragent.yaml:
python_binary: <path to python 2.7 binary>
For containerized agents, this path will be: /usr/bin/python2.7
The following app checks are deprecated as of 9.9.0:
Change the URL in the whitelisting firewall/proxy setting to reflect
https://download.sysdig.com/. Otherwise, the agent install on Linux
will fail.
Fixes
Metrics Reporting
Fixed an issue in the agent wherein the
kubernetes.namespace.pod.desired.count and
kubernetes.namespace.pod.available.count metrics were not reporting
any values.
HDFS App Check Deprecated
The HDFS (Hadoop Distributed File System) App Check had been deprecated
and removed. Users of the HDFS App Check can switch to hdfs_namenode
and hdfs_datanode App Checks.
Metric Calculation
Fixed an issue related to calculating the kubernetes.pod.restart.rate
metric.
Network Congestion
Isolated the Kubernetes Audit HTTP server from the Audit Event
processing path to reduce the chances of slowing down the connections
from the Kubernetes API server. This should reduce the likelihood of
multiple outstanding connections from the Kubernetes API server.
Certifi Python Module
Added a missing certifi Python module to the agent container.
9.7.0 March 09, 2020
New Features
Support for Openshift Hardening Guide
Added Openshift Hardening Guide as a benchmark program. It is available
as an option for CIS Kubernetes Benchmark.
Support for Linux Benchmark
Added Linux benchmarking as an available benchmark program.
Fixed the vulnerabilities detected in the agent-slim v9.6.1 image.
These issues are related to the python2 and jackson-databind
packages. These packages were upgraded to the versions with fixes.
Run App Checks on Hosts with Python 2.6
Fixed a defect that prevented app checks from running on hosts that
install Python 2.6.
9.6.1 February 28, 2020
Fixes
Metrics calculation
Fixed an issue that caused an error in the calculation of some metrics
such as net.* in agent version 9.6.0.
Red Hat-based host issue
Fixed an issue that caused the kernel module build associated with agent
version 9.6.0 to fail on Red Hat-based hosts.
9.6.0 February 26, 2020
Upgrades
Integrations improved
Added new metrics and configuration options for HAProxy and
Consul app check integrations. See
HAProxy, HAProxy
Metrics, and
Consul for details.
Fixed a problem
Go
app check which caused it to fail with an exception error.
Metrics added
Added Kubernetes metric kubernetes.namespace.pod.running.count to
track the number of pods in running state. See Kubernetes
Dashboards.
Reduced load on the Kubernetes API server
The version of client-go was updated and now
defaults to encoded protobuf messaging instead of JSON to improve
performance.
Configuration optionnew_k8snow enabled by default.
Default collector port changed
The default port for the collector was changed from 6666 to 6443.
This could affect your firewall port settings; you may want to review
them before upgrading the agent.
Fix for the dynamic back-end configuration of Kubernetes Audit
Logging caused some agent
deploys to fail
The agent is enhanced to listen on /k8s-audit for Kubernetes audit
events and the path can be configured via the config option
security:{k8s_audit_server_path_uris: [path1, path2]}.
Fixes
Prometheus metrics fix
Fixed a problem that inhibited the agent from scraping multiple ports on
a single process for Prometheus metrics.
Inaccurate cpu.used reporting fixed
Fixed a problem that caused the agent to erroneously report very high
CPU usage in some environments.
9.5.0 January 28, 2020
Note that the versioning scheme for agent releases has been updated with
this release. Previous versions used the format
0.<version number><hotfix>, such as 0.94.0.
Sysdig is aligning version numbers to the rest of the product. The new
version number reflects the maturity of the Agent software over the last
several years. Going forward, all Agent versions will be numbered as
Major.minor.hotfix.
We encourage users to be on the latest version of the Agent. Starting
with the next release of the Agent, we will support n-3 versions
back based on the minor number. For example, if the next release is v
9.6.0, we will support n-3 versions back, e.g to 9.3.0 (old version
scheme = 0.93.0).
Fixes and Upgrades
Added new configuration option and metrics for Elasticsearch
integrations
In the Elasticsearch app check, the parameter index_stats can be used
to collect metrics from individual indices. See Example 4 in
Elasticsearch and
Elasticsearch Metrics for
details.
Added new metrics for NGINX Plus integrations
More than 60 new metrics have been added to the NGINX app check. See
NGINX Plus Metrics for
details.
Made Go-based event handling the default
See Process Kubernetes
Events. As of agent 9.5.0,
the default setting for go_k8s_user_events is true and there is no
need explicitly to enable it. To switch back to the older events
monitoring (C++ based), set the value to false in your agent config
(dragent.yaml).
Enhanced log tracing for include/exclude processes filter.
Fixed a problem that was causing an internal process within the agent to
repeatedly restart.
Improved memory buffer handling
The agent will now auto-disable memdump functionality when the memory
buffer is too small.
Agent start/stop improvements on CRI-O and Openshift 4.x
The agent can now correctly perform the pause and stop container actions
on clusters running OpenShift 4.x and CRI-O.
0.94.0 December 20, 2019
Fixes and Upgrades
Fixed issue in the agent install scripts
The agent install scripts have been updated to mount /etc/modprobe.d
from the host into the agent container. This prevents a problem where
the agent loaded drivers that were excluded from the host.
Added user events for additional resource types
Added events monitoring for statefulsets, services, and horizontal pod
auto-scalers (HPAs) when the Golang-based events monitoring feature is
enabled. To enable, see Process Kubernetes
Events.
Added regex support for Kafka integrations
Added regex capability for consumer groups and topics in Apache Kafka
configurations. See Example 6 in Apache
Kafka.
Increased the Prometheus max_tags default value
The Prometheus max_tags configuration has been increased from 20 to
40.
Made change to guarantee support for older cpuset configurations.
Changed CRIO cpuset calculations to use the configured cpuset.cpus
value instead of cpuset.effective_cpus. This guarantees support on
older cpuset configurations.
Corrected an issue that resulted in the suffix “_total” to be
stripped to Prometheus counter metric names.
0.93.1 November 25, 2019
Fixes and Updates
Fixed installation issue on native RHEL 7.x installs
The agent installer script has been updated to refer to an updated epel
repository.
Improved JMX metrics reporting
Fixed an issue when retrieving JMX metrics which could result in missing
samples.
(Sysdig Secure): Improvement in Kubernetes Audit events
Fixed runtime policy scopes for Kubernetes audit events.
(Sysdig Secure) Fixed audit event exception
The system now catches JSON object-type exceptions when parsing
Kubernetes audit events.
Improved error message
Improved the error message reported when the Sysdig agent cannot find a
pre-installed kernel header or cannot download a sysdigcloud-probe.
Performance improvement in dragent logging
0.93.0.1 November 15, 2019
Fixes
Fixed issue with Prometheus metrics names
Corrected a problem that resulted in the suffix _total to be removed
from Prometheus counter metric names.
0.93.0 November 6, 2019
New Features
Mask the customer ID in log files
The Customer ID is no longer output in the agent log, to avoid
inadvertent exposure when sharing of log files.
Kubernetes role node label included by default
The kubernetes.node.label.kubernetes.io/role label is available by
default
Update Kubernetes API used, in order to expand support of Kubernetes
v1.16
Replaced usage of the extensions/v1beta1 Kubernetes API with apps/v1
in the agent. This is required for supporting Kubernetes v1.16 using the
agent’s legacy Kubernetes integration (when new_k8s is not enabled).
Introduced a new config option in ElasticSearch app check
Introduced a new config option to generate cluster-wide primary shard
metrics from a master node: pshard_stats_master_node_only. See
Elasticsearch (Example 3).
Enhanced Postgresql app check
The Postgres app check has been enhanced to provide new metrics and
examples. See PostgreSQL.
Agent preparation for upcoming Policy Advisor feature in Sysdig
Secure
The agent will support new Rules generated by Sysdig’s Kubernetes Policy
Advisor. This agent is the minimum version required to use the upcoming
feature.
Updates and Fixes
Improved system events handling for Ubuntu 19.10
On kernels 5.1 and newer, some syscall events were incorrectly dropped.
This has been fixed.
Stopped Kubernetes pause containers (pods) from being reported
Fixed an issue where Kubernetes pause containers were also showing up in
Kubentes events. This fix filters them out from the events being
reported.
Fixed rare issue on OpenShift
Fixed an issue where, in a rare case, a dropped event could cause a
kernel deadlock and crash the node.
Fixed issue preventing kernel module creation for Debian Buster
This change adds support for building the Sysdig Monitor agent kernel
module for Debian Buster.
Improved event timestamp in Kubernetes
This fix ensures that user events get the correct timestamp with
Kubernetes v1.16 when thego_k8s_user_events option is set to true.
Updated Kubernetes API used, in order to expand support of Kubernetes
v1.16
In dragent.yaml, the Kubernetes API extensions/v1beta1 is updated to
apps/v1. This enables agent support for Kubernetes v1.16 even when the
new_k8s option is set to false.
Fixed a Kubernetes event reporting issue
Fixed an issue with Kubernetes Events where the host MAC scope was not
populated correctly, resulting in not showing up on the dashboard.
Improved Kubernetes events handling from delegated agents
When using go_k8s_user_events, kubernetes events from non-delegated
agents are no longer sent.
Eliminated legacy “BASELINES” message
Stopped processing legacy BASELINES messages from the backend collector.
Performance improvement at startup
The agent now defers initialization of Secure-related components
slightly to reduce excess resource usage at startup.
0.92.3 October 7, 2019
Updates and Fixed Issues
Included Example of a Prometheus Matching Rule Using HTTPS
The Sysdig agent will use HTTPS for scraping when target’s annotation
has “kuberentes.pod.annotation.prometheus.io/scheme: https”.
Kubernetes versions older than 1.9 no longer supported.
The Sysdig agent has replaced the use of the extensions/v1beta1
Kubernetes API with apps/v1.
Included Example of a Prometheus Matching Rule Using HTTPS
The Sysdig agent will use HTTPS for scraping when target’s annotation
has “kuberentes.pod.annotation.prometheus.io/scheme: https”.
The RabbitMQ app check has a new config option: filter_by_node
Without this option, each node reports cluster-wide information (as
presented by rabbitmq itself). This option makes it easier to view the
metrics in the UI by removing redundant information reported by
individual nodes. See
RabbitMQ for details.
0.92.2 September 26, 2019
New Features
Asynchronous metadata collection for CRI-O and containerd
The collection of container metadata from CRI-based runtimes was
previously synchronous with other agent tasks.
**Prioritize and filter how process metrics are reported in Sysdig
Monitor. **
In addition to filtering data by container, it is also possible to
filter independently by process. Broadly speaking, this refinement helps
ensure that relevant data is reported while noise is
reduced. See Include/Exclude
Processes for details.
As of this version, App Checks on hosts with Python 2.6 will no longer
be supported.
Fixed Issues
**Fix for Agent termination during resource discovery from the
Kubernetes API Server **
Fixed an issue where the Agent stopped and shut down if there an
error occurred during resource discovery from the Kubernetes API
Server. This fix simply reports the error and continues with the
discovered resources.
Fix for Kubernetes delegation error
Fixed an issue that caused Kubernetes delegation to not work after
the cointerface process restarts following a crash.
Fix for accounting Network errors
Network-related errors are now correctly accounted for instead of
being treated as file-open errors.
New Prometheus Client Version
Updated prometheus_client to version 0.7.1. This should result in
improved performance while ingesting Prometheus metrics.
Fix for dropping StatsD Metrics
A defect in earlier versions of Sysdig Monitor with the
statsd.use_forwarderoption could drop some StatsD metrics from
containers. This change resolves that problem; the agent will begin
fetching metrics from containers 10 seconds after first identifying
that the container exists. The 10 second delay allows containers to
start StatsD servers within their network namespaces if they choose.
The timeout can be overridden using the
statsd.container_server_creation_delay_s option, which specifies
the delay in seconds.
Fixed resource metrics for CRI-O containers
The following metrics reporting correctly in the Monitor
UI: memory.limit.bytes, memory.limit.used.percent, and cpu.quota.used.percent.
The CRI extra_queries option now enabled by default. See Runtime
Support: CRI-O and
Containerd for
details.
Sysdig Secure
**Fix for enlarging Sysdig Capture **
Fixed an issue where a Sysdig capture would grow endlessly if a
security policy was set to Capture 0 seconds after an event.
Fix for processing system events
Fixed problem where gettimeofday syscall was called in compliance
code while processing system events. This could potentially cause
performance problems in Linux distros that called down to the kernel
for gettimeofday responses, such as some versions of Amazon Linux.
Sysdig Platform
New RPM dependency
Changed RPM dependency to Python 2 to support installation on
RHEL 8.
0.92.1 August 16, 2019
Fixed Issues
Sysdig Monitor
Fixed issue with cluster name in Monitor UI
Cluster name was being populated incorrectly for Kubernetes event
scopes.
Fixed Kubernetes events issue
Fixed Kubernetes event collection issue that occurred when using the
go_k8s_user_events option. This option was introduced in agent
version 0.91.
Sysdig Platform
RHEL 7.7 and 8.0+ support The kernel module now builds for RHEL
7.7 and 8.0+
Fixed issue with StatsD metrics collection limits Some versions
of the Sysdig agent allowed fewer than the configured number of
StatsD metrics because Sysdig Secure-related StatsD metrics were
counted towards the configured limit.
This change corrects that behavior so that the configured limit
applies only to StatsD metrics that do not originate from Sysdig
components.
Sysdig Secure
Fixed a profiling-related issue that impacts Sysdig Secure 2.4
Sysdig Secure 2.4 will include a new Profiling feature, and 0.92.1
fixes a bug where profiling could remain disabled after periods of
high load. In order to use Profiling, it is required to upgrade to
agent 0.92.1 or higher.
0.92 August 7, 2019
New Features
Preparatory enhancements for upcoming Sysdig Secure Policy Editor
Although the feature UI will not be released until version 2.4.0, Sysdig
encourages all users of Sysdig Secure to upgrade to agent 0.92 in
preparation for the new Policy Editor feature. Agent 0.92 will accept
policies messages from both the current backend as well as a backend
that supports the new policy editor.
Ability to compress metrics data for internal transfer
With app checks integrations, when the volume of metrics data collected
was too large to send over the agent’s internal queue, app checks could
fail. This problem is solved by introducing an option to compress app
checks metrics data, which reduces the internal load. See Compress
Metrics Data for details on
how to enable this option.
Fixed Issues
Sysdig Monitor
Fix for occasionally dropped metrics In earlier releases of Sysdig
Monitor, the agent sometimes failed to parse metrics containing negative
values for some fields.
This change updates the behavior to drop fields that have unsupported
negative values, and to generate a log message when such fields are
encountered.
Sysdig Platform
Fix for MySQL versions 8.0.14+
Fixed a bug that caused the MySQL app check to fail with an error.
Fixed agent crash issue exposed by recent Linux kernels
Affected kernels include the 5.2.x line, 5.1.8+, and 4.19.49+.
Fixed a bug in HTTP parserIn the (uncommon) situation where
absoluteURI is used in the Request-URI, fixed a bug that was causing
a faulty URL.
0.91 July 17, 2019
New Features
Improved securityRemoved obsolete and vulnerable Python
2.6-compatible libraries from Docker images.
More efficient Kubernetes event handling.
The agent has added functionality to allow more efficient processing of
Kubernetes user events.
Reduced CPU usage on Kubernetes clusters Extended performance
optimizations for processing Kubernetes Services, which will reduce
agent CPU usage in large clusters.
Fixed issue with Prometheus metrics gathering intervals
The agent will now respect the configured interval for scraping
Prometheus metrics from remote endpoints, as opposed to doing it
every second.
Fixed limit/requests calculations for init containers
Fixed memory calculations for Kubernetes init container limits and
requests
Improved Healthcheck monitoringAgent has improved ability to
detect commands identified as a part of Kubernetes
Liveness/Readiness Probes, in addition to Docker Health Checks.
Improved error messaging
Warning messages for container group inconsistencies were demoted to
debug level, as they are harmless and do not need to clutter the
error reporting stream.
Fixed issue with container “incomplete” reporting status
Starting with version 0.90.0, the agent would report containers for
which it had not yet fetched metadata as “incomplete.” This would
then propagate to the Monitor UI. This restores the behavior where
the agent leaves the unknown fields unset.
Resolved REST server issue
Fixed problem where an enabled port would respond to HTTP requests
when not desired.
Fixed issue with StatsD metrics collection
Previous versions of the Sysdig agent, when configured to use the
StatsD fowarder ({{statsd.use_forwarder: true}}) truncated
messages that it received from containers to 2048 bytes, resulting
in the potential for dropped and corrupted metrics. This change
resolves that problem. See details under StatsD
Integration.
It is recommended to follow upgrade best practices:
Keep upgrades current
Test upgrades in a non-mission-critical or staging environment
before rolling into production.
The instrumented task should no longer be blocked from starting when using DEBUG logging with log-forwarding enabled, and better error messages have been added for failures when log forwarding.
Fixed Termination Error
Instrumentation tasks now terminate correctly on fatal errors and trigger the ECS restart policy.
Cleaned Up Serverless Agent Metadata
Redundancies in the serverless agent metadata (labels and tags) were corrected:
AWS-related metadata are grouped below aws.* tags
Container-related metadata are grouped below container.* tags
Custom tags are grouped below agent.* tags
New Features
New Container-Based Installer
The Serverless Agent 3.0.0 provides a new container-based installer to simplify the deployment of the instrumentation & orchestration stacks. (Serverless Agent 3.0.0 supports the existing command-line-based installer as well.) See also: AWS Fargate Serverless Agents.
2.3.0 March 15, 2022
Defect Fixes
Container Metadata Now Automatically Provided to Avoid Errors
The following metadata values are now automatically passed by serverless agents:
*value is always provided in same way
**value depends on how the image is referred to when deploying the instrumented container, i.e. repo:tagvsrepo@digest.
Example:
:latest
When specifying an image such as falcosecurity/event-generator:latest
the metadata configuration =:
Fixed Display Problem in Insights Composite View for Fargate Events
Secure events from the Fargate serverless agent are now correctly labeled with Account ID and Region, allowiing them be grouped correctly in the Insights Composite view.
Fixed Occassional Problem with Starting Instrumented Tasks
Added retry and fallback logic to avoid restarts when a log-forwarding endpoint isn’t present.
Manual Instrumentation of Workload Agents
Improved documentation for manual instrumentation of workload agents, including handling logs.
2.2.0 December 2, 2021
Defect Fixes
Fixed Workload Agent Start Issue
The system no longer allows the workload agent to connect to the orchestrator agent if policies have not been loaded. This prevents the workload from starting without policies in place in the event of network disruption.
New Features
Easier Setup of Alternative Port for Orchestrator
Because the 6667 port is hardcoded in multiple places in the orchestrator CTF, users who needed to assign a different port to the orchestrator agent faced a cumbersome process. The orchestrator port can now be configured via either SYSDIG_ORCHESTRATOR_PORT (default) or the SysdigOrchestratorAgentPort (new) parameter in the CloudFormation template.
Instrumentation Logs Collected Separately from Workload Logs
Fargate instrumentation logs are by default collected in a separate log group, which is created when installing the CFN instrumentation macro.
2.1.0 September 27, 2021
Defect Fixes
Fixed Task Stall Issue
Fixed a memory leak in the Serverless Agent instrumentation that could cause the instrumented task to stall. The problem is more likely to be encountered when a large number of captures are generated in quick succession.
Resolved an Agent Error when Reading File Descriptors
Reduced the log level of a benign warning message to debug.
2.0.0 July 7, 2021
New Features
Captures Available
Announcing the availability of the
Captures feature in
Fargate.
Defect Fixes
Fixed/Enabled Policy Scoping on Instrumented Fargate Tasks
At this time, only container-related scope labels such as container.id
or container.name are supported.
Delay Event Source Startup by Default
The system now waits for policies to be available before launching the
instrumented task, to fully secure workloads
Fixed Exit Codes for Faulty Workloads
The exit codes of the instrumented tasks are now faithfully propagated.
Better Handling of cmd and entrypoint Errors
Log more informative errors when cmd and/or entrypoint are not
available for serverless agent instrumentation.
Fixed S3 Bucket Error
Fixed an issue in the serverless agent installer that caused a failure
while attempting to create an S3 bucket in us-east-1 region.
1.0.1 April 15, 2021
Segmentation Fault Error Fixed
Fixed a problem that caused a segmentation fault error inside a Fargate
task due to Sysdig instrumentation.
Container Definition Fields Now Support Complex Values
Added support for complex values inside Name and Image fields of the
container definition. See also the ECS Task Definition
docs
from Amazon.
March 15, 2021: Serverless Agents Introduced
Sysdig Serverless Agent 1.0.0 for Fargate ECS
The “container-as-a-service” serverless environment calls for new
agent models, and Sysdig provides them. Whereas in ECS, users still
manage the underlying instances, with AWS Fargate the host is never
visible and users simply run their workloads. And while this model is
convenient, it can introduce risk as many people leave the containers
unattended, without monitoring security events within that can
exfiltrate secrets, compromise business data, impact performance, and
increase their AWS costs. In addition, it is not possible to install a
standard agent in an environment where you do not have access to a host.
For these reasons, Sysdig has introduced a new “serverless agent” model
that can be deployed in these container-based cloud environments. The
first implementation is for Fargate (ECS).
Sysdig will be rolling out security features on the serverless agent
over time. In v1.0.0, users will see:
Runtime Policies and Rules
Secure Events
To obtain secure event information and the associated Falco policies and
rules in the Sysdig Secure UI from a Fargate environment, users install
the serverless agent using a CloudFormation Template. Then log in to
Sysdig Secure and review the events in the UI.
The dates shown are for the initial release of a feature. The feature
may not be rolled out to all regions concurrently. The availability of
a feature in a particular region will depend on scheduling.
Supported Web Browsers
Sysdig supports, tests, and verifies the latest versions of Chrome and
Firefox.
Other browsers may also work, but are not tested in the same way.
June 7, 2022
Enhanced Metric and Label Selection
The metric and label selectors in Dashboards and Metrics Explorer have been improved to provide easier search and find what you are looking for.
Improvements include:
Suggested labels now show only relevant labels for a selected metric.
Displays 500 labels by default for a selected metric. Previously it was 50.
Supports inline editing of metric and label names.
Provides improved search relevancy.
Dashboard enhancements
Translate Form-Query to PromQL
You no longer require advanced Prometheus knowledge to build complex PromQL queries in Sysdig Monitor. With single click, you can translate form query to PromQL, and build PromQL-based dashboards in no time. For more information, see Build PromQL Panels from Form Query.
PromQL Support for Toplist
Toplist panels support running PromQL queries.
Multi-Query Support for Stacked Area Charts
Timechart now supports visualizing multiple queries as stacked areas in the same y-axis.
With this feature, it’s easier to visualize and compare sparse metrics.
Lazy Loading of Dashboard Panels
Dashboards now supports lazy loading panels. Lazy loading greatly reduces the initial page loading time by only loading panels once they become visible on screen.
Monitoring Integrations
Integrations
Added the following integrations:
Fluentd
NTP
Improved CoreDNS Prometheus job to be detected in IKS clusters
Changed troubleshooting metrics in some integrations for metrics inside the filter of the Prometheus job
Dashboards and Alerts
Added the following templates for dashboard and alert:
Fluentd
NTP
Changed OOTB K8s dashboards to use “is” vs “in” scoping to improve performance.
Changed the following dashboards:
Cluster/Namespace Available Resources
Cluster Capacity Planning
Pod Rightsizing & Workload Capacity Optimization
Pod Scheduling Troubleshooting
Kubernetes HPA
Added the containers with limits/requests only in certain panels in the Cluster Capacity Planing dashboard
Limited the use of the label job to some panels in the Kubernetes CoreDNS dashboard
Exporters
Added support for CA files in ElasticSearch exporter Helm chart
Removed duplicated securityContext in ElasticSearch exporter Helm chart
Changed the ElasticSearch wizard and Helm chart to use secrets for URL of the ElasticSearch server
Bumped Helm chart repository version to include NTP exporter and fixes in Elasticsearch
The following Exporter images for NTP exporter have been added:
quay.io/repository/sysdig/ntp-exporter:v2.0.3
quay.io/repository/sysdig/ntp-exporter:v2.0.3-ubi
New version of grok exporter with security updates:
quay.io/sysdig/grok-exporter:v1.0.2
quay.io/sysdig/grok-exporter:v1.0.2-ubi
May 23, 2022
Custom Roles
A custom role is an admin-defined role that allows Sysdig administrators to bundle a set of permissions and assign those permissions to individual users or teams. Custom roles allow for finer-grained definition beyond the standard out-of-the-box Sysdig Roles. Once defined, a custom role can be assigned to any user inside a particular team, and also be configured as the default role for new users in that team. For more information, see Custom Roles.
The addition of custom roles into the platform is transparent, meaning that standard roles and assignments that already exist will not experience any changes.
May 4, 2022
Sysdig Platform Audit
We are glad to announce that Sysdig Platform now supports the capability of tracking, logging, and reporting on all changes in the system.
Track all activities on the API level
Retention period: 90 days
Simple API for retrieving audit information (no UI)
Events Forwarding support to be included in the near future (to be announced)
We would like to announce that Sysdig Monitor and Secure now allow you to define a Login Message that will be presented to all users. Added to boost Sysdig compliance/enterprise readiness, requested originally by the IRS.
Users are not allowed to access the system until they acknowledge the message
One login banner per account
Only Admin users can enable/update the message
Single banner for both Monitor and Secure (for Platform customers)
Advisor brings your metrics, alerts, and events into a focused and curated view to help you operate and troubleshoot Kubernetes infrastructure. To help you solve problems faster, over time, Advisor will surface your infrastructure issues that you should pay attention to. For more information, see Advisor.
Metrics Explorer
Metrics Explorer has been rebuilt from the ground up to focus on advanced metric exploration and querying.
Improvements to Metrics Explorer include:
Simple querying that builds PromQL queries under the hood. Metrics Explorer is the easiest way to build PromQL queries.
Graph multiple metrics at once for correlation. For example, CPU usage vs Kubernetes limits.
Queries are ungrouped by default, showing the individual time series for a metric. This allows you to spot any problems faster. For example, 1 of 50 Cassandra nodes with high pending compactions. Instead of segmenting, you now group by one or more labels, for example, workload, pod, and container.
When selecting a scope in the tree, only those metrics that are applicable to that entity are displayed.
Metrics are now more logically categorized by metric namespace (prefix).
Resolution has been improved. For example a 1-hour view now shows 10-seconds data. Additionally, the concept of time re-alignment has been removed.
Collapsible main menu: Allows you to toggle the visibility of menu options. The collapsible left-hand navigation prevents long lists from displaying by default and gives you a clear structure that is easy to scan and locate.
Hoverable sub-menu: With each module that has additional menu options, hover over the respective module to quickly navigate.
New Menu Option for Integrations
A dedicated Integrations menu option provides an easy way to access both inbound and outbound integrations with Sysdig.
Inbound: Access Monitoring Integrations quickly and understand which applications and services are running. You can also manage your AWS Account and review the Sysdig agent installation.
Outbound: Manage the Notification Channels and S3 Capture Storage.
Revamped User Menu
The User menu provides the following:
Option to efficiently switch between Sysdig Teams.
Access Management to the Administrator.
Sysdig API Tokens to the authenticated user.
Documentation and What’s new links
The Settings sub-menu link is provided to review all the available options for the current user.
January 26, 2022
Support for PVC Metrics
Contact your Sysdig representative or Sysdig Support to enable PVC metrics in your environment.
With Sysdig agent v12.2.0 or above installed in your monitoring environment, Sysdig Monitor can help you surveil your Kubernetes PV/PVCs objects. Use the PVC dashboard and alert templates to get an insight into your PV usage, such as disk usage, inodes, storage latency, errors, and so on.
Sysdig provides the following new troubleshooting metrics:
kube_workload_pod_status_phase
kube_workload_pod_status_reason
kube_pod_status_unschedulable
kube_pod_container_status_waiting
kube_pod_container_status_waiting_reason
kube_pod_container_status_terminated
kube_pod_container_status_terminated_reason
These metrics give insights into why pods are stuck or crashing (CrashLoopBackOff, OOMKilled, DeadlineExceeded etc.). To support this:
The Kubernetes Alerts Library has been updated to provide additional alerts for errors such as CrashLoopBackOff.
New panels has been added to the Kubernetes Workload Status & Performance dashboard.
In environments running older versions of Sysdig agent, the Kubernetes Dashboards will display a banner prompting you to upgrade to agent v12.2.0 or above for these metrics to be automatically collected.
origin/main
December 17, 2021
Update on Log4j Vulnerability (CVE-2021-44228)
Sysdig confirms that all services that compose Sysdig’s Cloud Platform running Apache’s vulnerable Log4j library have been patched to 2.16. We have not detected any successful attempts at exploitation of this attack vector during that time window.
December 15, 2021
Update on Log4j Vulnerability (CVE-2021-44228)
The sysdig agent does not include the Log4j library
Sysdig is using an alternative framework for logging, called Logback. The logback framework isn’t vulnerable to this issue.
Sysdig components include a log4j library in our standard distribution that was vulnerable. This library is included for compatibility reasons only, is not used for primary logging, and our security team has determined we are not vulnerable based on our application architecture and existing mitigating controls.
Sysdig can confirm that all services that compose Sysdig’s Cloud Platform running Apache’s vulnerable Log4j library have been patched to the latest version or adds additional mitigating controls suggested by vendors. We have not detected any successful attempts at exploitation of this attack vector during that time window.
Details regarding upgrades
We:
explicitly set commonsLog4jVersion = 2.15.0
update all of log4j-to-slf4j, log4j-api, and log4j-core to version 2.15.0
December 12, 2021
A Statement on Log4j Vulnerability (CVE-2021-44228)
Security researchers recently disclosed the vulnerability CVE-2021-44228 in Apache’s log4j, which is a common Java-based library used for logging purposes
Sysdig is using an alternative framework for logging called Logback. The logback framework isn’t vulnerable to this issue.
Sysdig components include a log4j library in our standard distribution that appears to be vulnerable. It has been confirmed that this library is included for compatibility reasons only and is not used for primary logging. As a result this should not pose any risks.
Patches will be provided to upgrade the log4j libraries that are included for compatibility reasons.
If you have any questions or concerns, please reach out to your Sysdig contact.
October 12, 2021
Expose Custom Data on Webhook Notification
Sysdig gives you the ability to specify custom data and attach it to the alert notification. For more information, see Configure a Webhook Channel.
Prometheus Recording Rules
Sysdig now supports Prometheus recording rules for metric aggregation and querying. To enable this feature in your environment, contact Sysdig Support.
Sysdig gives you the ability to determine the granularity of data collected by Prometheus Remote Write to which team members will have the visibility. You can specify what data team members can see by specifying tag/value expressions for the metrics. The drop-down defaults to “is”, but can be changed to “is not”, “in”, “contains”, and so on. Complex policies can be created by clicking drop-down to create AND chains of several expressions.
You can view the saved team scope by hovering on the corresponding team from the User menu.
Enhanced User Experience for Monitor Integrations
The UI for Monitoring Integrations has been enhanced to include guided configuration for exporters.
For each integration, you can use the wizard to specify the required information and you will be provided with a single-line command to install the exporter in your cluster. You no longer have to see the documentation or the given exporter source code to guess the name of the variable to configure the credentials of your database or the SSL certificate in the connection string.
If you want to deploy it through your CI/CD pipeline and you cannot run commands directly in production, you also have the option to generate the manifests to upload to your repository. If you prefer a package management approach, you also have the option to use Helm charts for the Monitoring Integrations.
After deploying an exporter, you can see whether it is working on the wizard. Sysdig Monitor automatically detects the metrics arriving in your account and associates them to the workload. This way, it is easy to visually detect the applications that are correctly reporting metrics and those that need some attention.
Ability to edit dashboard and panel name inline in the Panel Editor.
Ability to add dashboard template to favorites.
Moved the legacy dashboard templates to Deprecated section
Supports RabbitMQ Integration. Configure it using the Monitoring Integrations.
Added new dashboard templates for the following:
Fargate Usage
Go applications
Sysdig Admission Controller
RabbitMQ Integrations
Kubernetes Controller Manager
Kubernetes Scheduler
CoreDNS
Alert Enhancements
Added new alerts for RabbitMQ and CoreDNS integrations and for Go applications.
August 10, 2021
Monitoring Integrations
Sysdig discovers the services running in your environment and gives you
visibility into deeper application performance and health telemetry by
configuring a managed Monitoring Integration through
PromCat. You can easily view which services you
can configure an integration for, check the status of existing
integrations, and leverage curated content in the alerts
library and out-of-the-box
dashboards.
The Alerts Library in Sysdig Monitor gives you a recommended list of
alerts to configure based on the services running in your
infrastructure. The curated content from the Sysdig removes the need for
guessing which alerts to configure and takes you from zero to full
monitoring coverage faster.
The usability of the Alert page has been enhanced to include:
Ability to create and edit alert groups based on the service that
they are representing. The alerts created from alert templates will
have groups automatically assigned to them.
Efficient visual cues to see alert activities and identity the
alerts that are not resolved. A bell icon next to an alert indicates
that it has not been resolved. Alerts that are active over the past
two weeks will have an event chart under the Activities Over Last
Two Weeks column and an event feed on the alert details slider.
We have introduced several improvements to the out-of-the-box Kubernetes
dashboards:
Workload dashboards are refreshed with relevant status and golden
signals.
Improved UX with panel location and color code.
Some workflows are simplified to make it easier for beginners in
Kubernetes
Improved capacity planning capabilities.
Text boxes are easier to read and locate near the relevant panels.
July 19, 2021
Customized Session Expiration
Session expiration is the amount of time a user can remain idle before
the session is automatically ended or expired. After the session
expires, the user must log in to the Sysdig application again.
Sysdig now gives you the ability to make a shorter or longer idle
session expiration for Sysdig applications. When a user browser is idle
for a certain period of time, they will get automatically logged out.
For more information, see Configure Customized Session
Expiration.
Enhanced Session Logout
To offer superior user security, the logout procedure has been enhanced.
When the users log out of a Sysdig application, they will be
automatically be logged out of both Monitor and Secure applications.
June 01, 2021
PromQL Library
We have compiled a list of PromQL queries to give you one-click insights
into the health and performance of your infrastructure. The library also
includes a PromQL 101 category to give you hands-on exposure to PromQL.
For more information, see PromQL
Library.
Prometheus Remote Write
Sysdig supports ingesting metrics from Prometheus servers by
using remote_write capabilities.
In Sysdig terminology, the remote endpoints that can read Prometheus
metrics are known as Prometheus Remote Write. Prometheus Remote Write
does not require the Sysdig agent to be installed in the Prometheus
environment. This facility expands Sysdig monitoring capability beyond
Kubernetes and regular Linux kernels to environments where the Sysdig
agent cannot be installed.
The dark appearance, known as Dark Mode, is available in Sysdig
applications.
Sysdig can now automatically match your OS preferences. Available in
Sysdig platform on-premises, or in SaaS in the US East and rolling out
globally. For more information, see Configure Theme
Preference.
Improved Dashboard Templates
The following Dashboard templates have been enhanced to display the data
better, return improved results, and add golden signals.
Kubernetes
Application
Ngnix
Ceph
Ngnix Ingress
ElasticSearch
Redis
May 10, 2021
Silencing Alert Notifications
Sysdig Monitor allows you to silence alert notifications for a given
scope for a predefined amount of time, and schedule silence in advance.
When silenced, the alert will still be triggered and posted on the
Events feed and in the graph overlays but will indicate it has been
silenced. The types of notification channels you can use are Email,
Slack, and Amazon SNS.
You will be notified 30 minutes before the start time and 30 minutes
before the end time of a silence window. You will also be able to easily
extend or end an active silence. To access the feature, navigate to
Alerts > Silence on the Monitor UI.
Sysdig Monitor now supports two new labels, kubernetes.workload.name
and kubernetes.workload.type which can be used for scoping Dashboards
and configuring Gropings.
Earlier, each type of object (deployment, replicaset, statefulset, etc.)
was unique, and in turn, you needed to use different types of Kubernetes
Dashboards and a different Grouping resulting in
n/a
, where distinct types of Kubernetes objects are listed.
Ensure there is sufficient capacity in a cluster to deploy a new
application.
Application Status&Overview
Understand the status of applications (workloads) running in a cluster
by monitoring performance, pod health, and resource usage
Cluster Capacity Planning
Monitor the capacity of Kubernetes clusters ensuring they’re correctly
sized to support new applications when they’re deployed.
Container Resource Usage&Troubleshooting
Understand the performance of the different containers running in pods
across your infrastructure and identify any that are behaving
anomalously.
Node Status&Overview
Monitor the health, resource usage, and network statistics for nodes
running in clusters
Pod Rightsizing&Capacity Optimization
Optimize your infrastructure and better control cluster spend by
ensuring pods are sized correctly. Understand if you can free up
resources by reducing memory and/or CPU requests.
Pod Scheduling Troubleshooting
If a pod cannot be scheduled due to insufficient resources, use this
dashboard to identify where the resource bottleneck is.
Pod Status&Overview
Monitor the health, resource usage, and network statistics for pods
running as part of workloads.
April 26, 2021
Extended Label Set
Running PromQL queries is now smoother and faster with the extended
label set. The extended label set is created by augmenting the incoming
data with the rich metadata associated with your infrastructure and
making it available in PromQL. You now no longer have to write complex
queries in order to troubleshooting infrastructure issues or building
dashboards and alerts. For more information, see Run PromQL Queries
Faster with Extended Label
Set.
Configuring S3-compatible storage, such as Minio or IBM Cloud Object
Storage, for your Sysdig captures is now supported on Sysdig Monitor.
The capability can be turned on by configuring the system appropriately,
as given in (SaaS) Configure Custom S3 Storage
Endpoint.
Webhook Channel Enhancements
Sysdig supports the following on a Webhook channel integration:
Insecure connections: You now have the ability to skip the TLS
verification.
Custom headers: If your Webhook integrations require additional
headers or data you can append to the alert format by using a custom
header on the UI. This option is in addition to the existing API
facility to add custom headers programmatically.
If your environment has both LogDNA and Sysdig, you can view relevant
LogDNA Alerts as Events in Sysdig. These Sysdig Events behave like any
other type of Events in Sysdig They will be overlaid on Sysdig graphs,
listed in the Event Feed, and can be used to create an Alert in the
Sysdig Platform. The link provided in the Event Details redirects
you to the LogDNA Platform, in case further investigation is needed. For
more information, see LogDNA
Events.
March 03, 2021
PromQL Query Explorer
PromQL Explorer helps you understand metrics and their labels and
values, and create queries faster before using them in Dashboards and
Alerts.
PromQL can be used not only with metrics collected from Prometheus
endpoints but also with Sysdig native metrics collected by the agent.
For more information, see PromQL Query
Explorer.
Sysdig supports SAML Single Logout. This feature enables you to
configure automatic logout from the Identity Provider when users log out
of Sysdig. This feature is currently available for SaaS regions US-West
and EU-Central. For more information, see Configure SAML Single
Logout.
Enhanced Dashboard Scope Session
When returning to a previously visited Dashboard the UI retains your
last used scope.
February 05, 2021
Import Prometheus Alert Rules
You have now the ability to import Prometheus alert rules into Sysdig
Monitor. The ease of YAML import makes it significantly convenient to
tap into Prometheus ecosystem resources, such as
promcat.io.
Sysdig Monitor interface has been enhanced to provide the following
capabilities:
Edit dashboard scopes in a panel editor.
Set a dashboard template as the team entry point.
January 05, 2021
Improved Alerts
The Alert interface has been improved to allow faster browsing and
easier management. For more information, see
Alerts.
December 16, 2020
Statement RE: Solarwinds and Sysdig’s Security
We have seen requests for statements regarding tooling in the wake of
the Solarwinds and related compromises. Sysdig does not use these tools
internally. To maintain a secure SDLC process for own product we use
Sysdig Secure as well as source code analysis tools. We also maintain
our own branch of key OSS components to ensure software is fully vetted
before it’s delivered to customers.
November 19, 2020
Explore Workflow Enhancements
The Explore interface has been improved to allow faster
troubleshooting.
You are now launched directly into the drill-down view when you navigate
to Explore. You will still be able to group and navigate your
infrastructure by using the hierarchical scope tree.
The new Grouping editor helps you create and manage your
infrastructure groupings.
Administrators have now the ability to transfer dashboard ownership to
another user. For more information, see Transfer Dashboard
Ownership.
Enhancements for Navigating Dashboards
You can now pin the dashboard menu to the sidebar in the Sysdig Monitor
UI. Pinning makes it easier to navigate and browse different Dashboards.
In addition, the Dashboard interface has been enhanced to retain your
preference for open or closed categories to help you locate the desired
items faster.
October 22, 2020
Visualizing Missing Data on Dashboards
Dashboards now show null or missing data values as gaps instead of zero.
Optionally, missing data can be displayed as a dotted or solid line in
both Form-based and PromQL panels. StatsD metrics will continue to show
null values as zero unless overridden by the settings. For more
information, see Display Missing
Data.
Time Navigation in Events Feed
You can now browse and find historic events easily by using time
navigation.
Zooming Out Dashboards
You now have the ability to zoom out Dashboards. This feature doubles
the selected timeframe by 2x for a better context surrounding a problem
when troubleshooting an incident.
July 27, 2020
Sysdig Essentials
We have introduced a new product tier, Sysdig Essentials. This tier
includes everything required to achieve the five essential requirements
for practicing Secure DevOps:
With the introduction of Essentials, It’s also easier to get started
with a trial program and manage your Sysdig
subscription.
Learn the difference between Essentials and Enterprise, including
pricing and features, at Pricing.
Rebranded Login Page
The login page has been updated with the Sysdig Kraken and the new logo.
Sysdig Monitor Enhancements
Hosts Overview
To complement Sysdig Kubernetes Overviews, Hosts Overview has been
released. Host Overview provides a unified view of the performance and
health of physical hosts in your infrastructure.
New and Improved Empty States
A number of different splash screens have been introduced to guide you
through getting up and running with features across the application.
Sysdig Platform Enhancements
SAML Single Sign-On
The initial email to the following types of users will take them
directly to the Single-Sign-On URL, and not the registration page.
SAML SSO Users
The users that are invited to the platform (as opposed to having
them automatically created via Sysdig on-demand provisioning for
SSO)
Earlier, landing on the registration page was confusing to users because
they had to set up their initial password.
June 17, 2020
This 3.2.6 release focuses on the general availability of New Dashboard
with a rich set of features and enhancements. Learn more about the
release from the blog post, New and improved
dashboards .
New Dashboards is GA
Sysdig Monitor offers a new version of dashboards. Its improved editing
experience provides you with more flexibility and the new set of
functionalities offers additional ways to visualize and consume your
Sysdig data.
Features and Enhancements
Improved User Experience
The New Dashboard offers a more fluid, natural dashboard building
experience. The UI has been redesigned to introduce two types of
panels—form-based and PromQL-based— to make visualizing your metrics
effortless. Use a PromQL-based panel to build dashboards for Prometheus
raw metrics and custom metrics. The form-based panel for legacy queries.
For more information, see About the Dashboard
UI.
Dashboard Sharing
You can now share your dashboard with members within your Sysdig team or
share it across teams with fine-grained access controls. Define who
should be able to see the dashboards and what level of access they
should be granted: view only or collaborator with edit privileges. For
more information, see Sharing New
Dashboards
Time Series Name Templating
Customize the time series names on the legend on the panel editor by
using the labels associated with Prometheus metrics and segments to gain
context faster. For more information, see Create a New
Panel.
Multi-Metric, Multi-Segmentation Options
Configure multiple queries within a single panel, and configure each
query with multiple segmentation and scoping options. Individual queries
can be customized to render as a line or stacked area. For more
information, see Using
PromQL.
Event Overlay
Contextualize metrics and understand the “why” faster with a unified
view of both metrics and events. Configure event overlay to display
events from Kubernetes environments as well as alert events, and any
other events ingested using Sysdig’s open REST API. For more
information, see Display Dashboard Specific
Events.
Dashboard Templates
You can quickly view your infrastructure through the lens of one of
Sysdig’s curated dashboards, or use it as a base to start building your
own. You can find dashboard templates for managing Kubernetes capacity
and health, hosts and server performance, applications and services
telemetry, and the security posture of your infrastructure with data fed
from Sysdig Secure. See Dashboard
Templates to learn more.
Mapping Values to Text
Instantly understand what’s going on by mapping number panel values to
text. If you have a metric that returns 1 for up, and 0 for down, map
those values to “UP” and “DOWN” respectively. By defining thresholds and
mapping to text, you don’t need to be concerned about the values. This
is critically valuable when dashboards are shared between team members.
For more information, see
Text.
Granular Axes and Legend Controls
You have more flexibility when customizing the axes, as well as better
support for time series with long names. You can now configure the
legend by toggling its visibility and moving it to the bottom of the
panel. See About the Dashboard
UI.
Major Changes
Significant changes have been introduced to enhance the usability of the
existing functionalities. Review the changes before you explore the
functionalities.
Topology Maps
Topology maps are no longer available in Dashboard. Access Topology
maps through Explore, as you explore your microservices and
Kubernetes applications.
Dashboard Wizard
My Dashboards are no longer accessible in Explore. Additionally,
Dashboard Wizard has been removed. Instead, the concept of
Templates has been introduced in Dashboards to help you get
started with a library of templates addressing key use cases.
Histogram and Summary Metric Type
Histogram and summary metrics are no longer supported in the Histogram
panel type. You can continue to use them within Explore. If you have
enabled PromQL, we encourage you to use Prometheus functions for
visualizing histograms.
Use the new Prometheus histograms with the histogram_quantile metrics
on a time-series graph.
APIs and Integrations
API endpoints for the legacy dashboards (v2) will soon be deprecated. If
you are directly integrating into the API, please contact Sysdig for
guidance. Additionally, our Python SDK and CLI have been updated to
support the new dashboards APIs.
PromQL Support
PromQL support for querying Prometheus metrics has been rolled out to a
subset of Sysdig Monitor users. See Using
PromQL.
Intelligent $__interval
Use $__interval within a PromQL query and Sysdig will intelligently
use the most appropriate sampling depending on the time range you have
selected. This configuration ensures that we balance providing access to
the most granular data available while downsampling when you select a
long time range to panels load as fast as possible.
Scope variables
Configure scope variables at the dashboard level to quickly filter
metrics based on cluster, namespace, workload, and more. When using
PromQL queries, the scope can be injected by using dynamic variables.
This configuration is significant when troubleshooting as it allows you
to switch context quickly without reconfiguring queries.
Smart Autocompletion and Syntax Highlighting
Autocomplete suggests metrics, operators, and functions, while syntax
highlighting helps keep you on the right path and helps highlight
problems within a PromQL query. This is invaluable in dynamic
environments and allows you to craft the right queries faster.
Configurable Default Team Role
You can now define the default user role to apply when a new member is
added to the team. The Admin can change this default on a per-team
basis. See also: Create a
Team.
RBAC and Team Assignment for Notification Channels
Previously, notification channels in Sysdig Secure and Monitor were
treated as global entities, visible and editable for most users of the
platform regardless of team configurations.
We are enhancing the management and RBAC controls in the following ways:
Notification channels can now be “global” or limited to a particular
team
Global channels can be managed by admins and can be viewed/used by
other roles, while team-limited channels are available only to team
members
Team Manager , Advanced User, and Service Manager (Secure)
roles can create/update/delete team-scoped notification channels,
they can also read and use the global ones
Standard and View Only roles can read team-limited and global
notification channels
Admins will be able to create global notification channels and
migrate channels from “global” to “team-limited”, and also from one
team to another.
The Get Started page provides the key steps to ensure that you are
getting the most value out of Sysdig Monitor. We’ll update this page
with new steps as we add new features to Sysdig Monitor.
The Get Started page also serves as a linking page for:
Documentation
Release Notes
The Sysdig Blog
Self-Paced Training
Support
You can access the page at any time by clicking the rocketship icon in
the left navigation bar. See Getting Started with Sysdig
Monitor.Getting
Started with Sysdig Monitor
AWS Role Delegation
Sysdig Monitor can now utilize the Amazon Web Service (AWS) AssumeRole
functionality and discover cloud assets, grab CloudWatch metrics from
your AWS account, and use custom S3 bucket for storing captures. Upon
integrating with an AWS role, you can delegate access to AWS resources
that are not associated with your Sysdig AWS account.
Role delegation is an alternative to the existing integration method
using the access keys. This method is considered secure as sharing
developer access keys with third-parties is not recommended by Amazon.
Default dashboards (Overview and Services dashboards) are now available
for Istio v1.5 in addition to the existing ones for Istio v1.0.
November 21, 2019
Overview Is GA
Overview is now generally available. Overview leverages Sysdig’s unified
Kubernetes data platform to monitor, secure, and troubleshoot your
Kubernetes clusters and workloads.
Cluster Overview
Major highlights of Overview GA include but are not limited to:
Multi-cloud view of the health, risk, and capacity of your
Kubernetes infrastructure— a single pane of glass for Kubernetes
Clusters, Nodes, Namespaces, and Workloads across a multi- and
hybrid-cloud environment. You can easily filter by any of these
entities and view associated events and health data. View the
infrastructure organized by Clusters, Nodes, Workloads
Shows metrics prioritized by event count and severity, allowing you
to get to the root cause of the problem faster.
Drill down to Dashboards for instant insights.
To learn about the capabilities of the Overview feature, see
Overview.
Beta Features: Prometheus and New Dashboards
Introducing Prometheus and New Dashboards available in Beta. Contact
sales@sysdig.com to join the Beta Program.
[BETA] Prometheus Capabilities
Sysdig now supports native Prometheus time series ingestion. Run
Prometheus queries inside Sysdig Monitor and create visualization by
using the new Beta Dashboards that support it. This enables you to use
Sysdig Monitor as a standard Prometheus data source for other
visualization tools, such as Grafana. For more information, see Using
PromQL.
PromQL Dashboard
With this support, Prometheus and Sysdig metrics can now be supported in
regular Prometheus expressions.
PromQL Dashboard
[BETA] New Dashboards
Sysdig Monitor provides an enhanced New Dashboard to use with
Prometheus. For more information, see
Dashboards.
New Dashboards
The New Dashboards offer:
Flexibility to position the Legend.
Ability to run multiple queries.
Inherit the Dashboard scope to individual panels.
Multi-select items in the Legend to narrow down the lines you want
to focus on. Use command-click on Mac and Control-click on non-Mac
machines.
Features new query types: Form-based and PromQL expressions with the
easy toggling facility.
Enhanced auto-layout with the ability to re-position panels.
To access the New Dashboards:
Click the Dashboards tab on the left navigation panel.
Click Add Dashboard (+)
Click Beta Dashboards.
Enhanced Out-of-the-box Dashboards
In an attempt to improve the Dashboards experience, the following
changes have been introduced:
The following Dashboards are added:
Kubernetes Cluster Overview: Provides nodes and workloads
availability and highlights the high-level health of your Clusters.
It also summarizes resources consumption (CPU, memory) across Nodes
and Namespaces to pinpoint possible anomalies and node disk
utilization
Kubernetes Node Overview: Provides availability of the Nodes,
indicating potential issues reported by Kubernetes; a summary of
resource (CPU and Memory) allocation and utilization, as well as
Network and Disk utilization.
Kubernetes Namespace Overview: Provides a high-level summary of
availability, and resource allocation and utilization across all the
Workloads in the selected Namespace.
Kubernetes Deployment Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods for each Workload.
Kubernetes StatefulSet Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods for each StatefulSet.
Kubernetes DaemonSet Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods.
Kubernetes Job Overview: Provides a detailed summary of job
status, completion trend, pod restarts, as well as resource
allocation and utilization across pods.
Kubernetes ReplicaSet Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods for each ReplicaSet.
Kubernetes Pod Overview: Provides a detailed summary of pod
status, pod restarts, and resource allocation and utilization in a
selected pod.
Kubernetes Workloads CPU Usage and Allocation: Helps you verify
that CPU requests are properly configured and actual utilization is
expected.
Kubernetes Workloads Memory Usage and Allocation: Helps you
verify that memory requests are properly configured and actual
utilization is expected.
Kubernetes CPU Allocation Optimization: Helps you verify that
infrastructure resources are available for future needs and are not
wasted.
Kubernetes Memory Allocation Optimization: Helps you verify that
infrastructure resources are available for future needs and are not
wasted.
The following Dashboards are retained:
Health Overview (applicable to all the objects in the environment)
Horizontal Pod Autoscaler (the default Dashboard when selecting an
HPA)
Resource Quota
Service Health (the default dashboard when selecting a service)
The Sysdig Monitor UI displays n/a in several scenarios associated
with labeling. The Explore UI has now been enhanced to add a tooltip for
n/a to help you understand the scenario. See The Meaning of
n/a
for more information.
Filtering Events by Scope
Events are now filtered by Scope to show the most relevant Events in
Explore and Dashboards. This is an extension of the existing
Event Scope functionality.
You can toggle between showing Event feed from the entire infrastructure
and only from the particular scope you are interested in within the
infrastructure. Event scoping for Dashboards and Explore is
enabled by default.
By default, Events are filtered to show only the relevant ones. However,
you can turn the filtering off and see Events from the complete scope.
To do so:
Click the Dashboard Settings (three dots) icon and select
Events.
Use the toggle button to turn off Filter events by dashboard
Scope.
Click Save.
Similarly, you can filter Events by Scope in Explore.
Known Issues
Time Chart may encounter some response time delays
Not all the functionality from the existing dashboards will be
available in the new dashboards. The following functionalities are
not yet fully functional or not yet available:
Gauge chart
Text Panel
Top Chart
Table
October 11, 2019
Ability to “Favorite” a Dashboard
Users can click the star icon to mark a “Favorite” dashboard, which will
then be listed under “My Favorites” in the Dashboard view.
Enhancement: Additional Metrics Segmentation
This change enables Sysdig Monitor to segment metrics file.bytes.in
and file.bytes.out by file.mount and file.name.
Sysdig’s documentation platform has been upgraded and moved to
docs.sysdig.com.
Improvements include:
Look and feel: Updated to match the rest of the Sysdig branding
Search: Enhanced search speed, accuracy, and ease
Structure and content: Enhancements to content have been added
and are being continuously updated
Feedback: Buttons on each page enable users to communicate
directly with the documentation team.
August 14, 2019
New Default Kubernetes Grouping
Groupings for Kubernetes have been modified. This updated Grouping is
available to new teams. Default groupings are immutable–-they cannot be
modified or deleted other than by copying. Modifying a copy is allowed.
New Groupings:
Clusters and Nodes
(cluster.name > node.name > pod.name > container.name)
The ability to customize the subject and body of alert notifications
with variables has been extended to Event notifications. Event titles
and notification messages are in sync in the following cases:
The format of metric units are the same for the following:
The CPU and Memory metrics for Host and Container.
Kube-state CPU and Memory metrics.
Introducing the same format now makes the comparison of those metrics
easier on a chart.
Container Segmentation
Sysdig now supports segmenting all net.* metrics at container or pod
level by low level net.* dimensions, such as net.http.url or
net.http.status.code. Container-based teams now display segmentations
for net.http.* metrics as expected. The net.http.url and
net.http.status.codes are displayed if you select a container-based team
as it does for a host-based team for the same cluster.
Display Instance Name
Instance name in the Sysdig Monitor UI is now visible during creating
and editing it. Instance names are displayed right below the username in
the user dialog for switching teams.
Default Dashboard for Cluster and Node Capacity
Kubernetes Cluster and Node Capacity Dashboard has been refreshed to add
actual usage of CPU and Memory compared to Requests, Limits and
Allocatable capacity.
Aggregation for Kubernetes Nodes Health
Aggregation method has been refreshed for Kubernetes Node metrics. The
Kubernetes Node Health dashboard has been updated with metric
aggregations that are ‘summed’ across all containers running on the node
to reflect accurate node level data.
July 11, 2019
Enhanced Dashboard Menu
The Dashboard menu features a drawer-style popover that displays a list
of Dashboards you own and those shared by your team. With the popover
menu, you can add new Dashboards and search for existing ones. Click a
Dashboard name to access the relevant Dashboard page where you can
continue with the regular Dashboard settings.
Customize Alert Notification Template
Sysdig Monitor alerts now provide an option to customize the messages
that are sent with alert notifications in email and other channels, such
as Pagerduty and Webhook.
Use the Alert Editor to input dynamic variables, such as hostname, or a
hyperlink, and to add custom messages in plain text to the notifications
for intended recipients. You can modify both the subject and the body of
the alert notification with a hyperlink or a variable. For example, you
can add an agent id or a link to a Dashboard to the message. This can
help provide context for troubleshooting the errors that triggered the
alert.
Sysdig Monitor can now collect Prometheus metrics from remote endpoints
with minimal configuration.
Remote endpoints (remote hosts) refer to hosts where the Sysdig agent
cannot be deployed, e.g., a Kubernetes master node on managed Kubernetes
services such as GKE and EKS, where user workload cannot be deployed. To
enable remote scraping on such hosts, simply identify an agent to
perform the scraping and declare the endpoint configurations in the
agent configuration file.
The collected Prometheus metrics are reported under and associated with
the agent that performed the scraping, rather than with a process. For
more information, see Collecting Prometheus Metrics from Remote
Hosts .
Enhancements to Kafka AppCheck
Kafka integrations can now support authentication and SSL/TLS. If
authentication or SSL/TLS are enabled in Kafka, see Apache Kafka
Example 5 for how to enable
configuration details on the Sysdig side.
Two New Metrics for Accurate Pod Counts
Two new Kubernetes metrics, kubernetes.namespace.pod.desired.count and
kubernetes.namespace.pod.available.count, have been added at the
Namespace level to track desired and available pod counts.
You may also want to review the update log for Falco rules used in the
Policy Editor: Falco Rules
Changelog.
Dates shown are for the initial release of a feature. The feature may
not be be rolled out to all regions concurrently. Availability of a
feature in a particular region will depend on scheduling.
Supported Web Browsers
Sysdig supports, tests, and verifies the latest versions of Chrome and
Firefox.
Other browsers may also work, but are not tested in the same way.
June 23, 2022
New Secure Event Forwarder Integrations: Elasticsearch & Microsoft Sentinel
Two new integrations have been released for Sysdig Secure’s Event Forwarder functionality:
A custom role is an admin-defined role that allows Sysdig administrators to bundle a set of permissions and assign those permissions to individual users or teams. Custom roles allow for finer-grained definition beyond the standard out-of-the-box Sysdig Roles. Once defined, a custom role can be assigned to any user inside a particular team, and also be configured as the default role for new users in that team. For more information, see Custom Roles.
The addition of custom roles into the platform is transparent, meaning that standard roles and assignments that already exist will not experience any changes.
May 19, 2022
Menu Option to Display New and/or Old Scanning Interfaces
To facilitate a smooth transition from the Legacy Scanning Engine to the new Sysdig Secure Vulnerability Management, the Settings Menu now provides options for displaying the UI for the new, legacy, or both scanning engines.
Safe and transparent: This is a non-intrusive change; regardless of how you have the current New Vulnerabilities engine toggle set, the Sysdig Secure navigation menu will not be modified without explicit user intervention. And the toggles will only alter the user interface and not impact the function or running of the engine itself.
Policy Advisor was built during a time when PodSecurityPolicies (PSPs) were the only way to add Security Policies to a Kubernetes workload. PSPs have now been deprecated in Kubernetes 1.21, released more than a year ago.
origin
May 17, 2022
Runtime Scanner 1.0.3 Released
Optimized requests performed on the Kubernetes API
We would like to announce that Sysdig Monitor and Secure now allow you to define a Login Message that will be presented to all users. Added to boost Sysdig compliance/enterprise readiness, requested originally by the IRS.
Users are not allowed to access the system until they acknowledge the message
One login banner per customer
Only Admin users can enable/update the message
Single banner for both Monitor and Secure (for Platform customers)
This release marks the general availability (GA) of the Secure Insights feature. Some of the changes introduced include:
Better support for Azure events
AWS IAM permission integration
Bug fixes for policy tuner flow
Limit for displaying events in a time range removed
May 2, 2022
DriftControl Policies: Detect and Prevent Drift in Container Runtime
Sysdig agent can now detect when a new executable was added to a container after a container has started up. The agent collects when a file was downloaded and made executable. When using prevention mode, the agent can also deny the process from ever running. A policy can also be used to define binaries that should be denied/excluded from being denied if they have been added after the container has started.
Sysdig is pleased to announce the New Vulnerability Management engine, a major upgrade to the vulnerability and image scanning functionality for the Sysdig Secure product.
Major Highlights
Scanning times have been drastically reduced: 8x faster on average!
Additional data for vulnerabilities and remediation
CVSS scores and metrics: Network Attack Vector, Privileges required, etc.
Flagging of publicly available code Exploits
Suggested package fix version
Risk spotlight Focus on the vulnerabilities that Sysdig detects in active packages at runtime.
This is a new filter that only shows CVEs with active packages, to save time browsing infrastructure and to help focus on high-impact CVEs
“Generate now” immediate scheduling directly available from the UI
Flexible policies that can be attached to the different runtime and security contexts
How to Move to the New Scanning Engine
The new vulnerability management engine uses a different data storage, API, host components, and user interfaces than the legacy scanning.
Contact your Sysdig representative; she/he will guide you through the process of migrating your subscription and vulnerability management configuration to the new engine.
The following components have been upgraded to the listed versions with bug fixes and security updates:
node-image-analyzer:0.1.16
secure-inline-scan:2.4.9
host-analyzer:0.1.6
The latest Helm chart includes these versions for Node Image Analyzer and Host Analyzer. Follow the usual process to upgrade the inline scanner.
March 3, 2022
New CIEM Features
User Risk Labels
Risk Labels are now surfaced to highlight insecure attributes for specific Users and Roles. They are listed within the Users & Roles page and within the User Details tab of a specific user.
Trend Charts in Overview
Time charts are now available within the Overview tab of Identity and Access. These help to visualize your Permission trends over time for Users, Policies, and Resources.
CSV Report Export
All of the pages within Identity and Access can now be exported as a CSV file. Select the Download CSV button found at the top right corner of all pages.
Effective Permission Calculation
AWS supports different types of policies to limit permissions on different scopes. Sysdig has added support for calculating effective permissions based on permission boundaries and organization level service control policy (SCP). This gives additional context when viewing permissions set on identities. For example, an identity that has been given administrator level identity policy will be limited in overall permissions if there is a permission boundary policy attached to it.
CIEM Data in Insights
Within the Cloud Activity and User Activity views in Insights, there is now an Identity and Access tab. This will help investigative flows to understand the context from an IAM perspective.
March 1, 2022
New: Data Sources Instrumentation
On the Data Sources > Managed Kubernetes page: For unconnected clusters, Sysdig has added quick instrumentation instructions using the known details about the cluster, such as the cloud account, region, and cluster name.
February 10, 2022
Improved Usability with New Navigation
Sysdig’s new navigation improves the usability of the left-hand navigation for faster and easier navigation of where you’re trying to go.
Hoverable Sub-Menu: With each module that has additional menu options, hover over the respective module to quickly navigate.
Collapsible Main Menu: Save space with the collapsible left-hand navigation.
New Menu Option: Integrations
A dedicated Integrations menu option provides an easy way to access both inbound and outbound integrations.
Inbound:
Access the Cloud Accounts page to quickly understand which applications and services are running, and where the Sysdig agent is installed.
Access Managed Kubernetes to get a catalog for all the managed Kubernetes clusters in your environment. The status shown is connected/unconnected based on whether the agent is installed or not.
3rd Party: Manage your Git Integrations
Outbound: Manage your Event Forwarding, Notification Channels, and S3 Capture Storage
3rd Party: Manage your Git Integrations
Revamped User Menu
Now all the settings options are collected and exposed in one mega menu. Find the right page before navigating away from where you are.
February 2, 2022
Enhanced Unified Filter for Event Feed
The Sysdig Secure Event Feed is getting a new unified filtering experience, available now for SaaS accounts.
Easily toggle from the original to the cleaner, simpler enhanced version, where you will find:
Unified scopes, free text and any other filterable/searchable attributes on a single lean bar
Autocomplete on keys and values
Autocomplete/suggest operands
One-click quick filtering directly from the list of displayed elements
Saved filters in various formats– no more retyping common filter expressions
We are pleased to announce a rework of our Compliance and Benchmarking capabilities. This change brings a number of improvements:
Compliance and Benchmark tasks are now scheduled, managed, and generate reports in an updated and unified interface, including simpler pathways to remediation and easier-to-navigate reports.
The logic used to check individual controls now checks for events signalling control failures, as well as ensuring the correct Runtime rules are configured to detect these events. This leads to a more comprehensive audit that captures activity as well as configuration.
New compliance standards and platforms: added
For workload, AWS, GCP, and Azure:
NIST 800-82 Rev2
For workload and AWS:
Fedramp (workload and AWS only)
HITRUST CSF 9.4.2 (workload and AWS only)
For GCP and Azure
GDPR
HIPAA
ISO 27001:2003
NIST 800-53 Rev4
NIST 800-53 Rev5
NIST 800-171
NIST 800-190
PCI / DSS v3.2.1
SOC 2
Prerequisites
Agent version >=12.0.4
If necessary, install or upgrade your agent to the appropriate version.
NOTE: If you are upgrading from an earlier version of Sysdig Secure, your existing compliance and benchmark records will be migrated to the new version and retained on the same schedule as before.
Infrastructure-as-Code (IaC) is an important part of today’s cloud-native infrastructure. We at Sysdig know that the earlier you identify possible posture issues, the better off you are.
The new feature allows you to integrate Kubernetes IaC checks into your Git pipeline.
With just a few clicks, the standard compliance checks will be integrated into the Pull Request (PR) flow and alert developers when they create violations of the policy before they merge.
Supportability & Requirements
The new capability will use either an application or a webhook in your respective git provider.
Github - Github Application
Gitlab - Webhook
Azure DevOps - Webhook
Bitbucket - Webhook
For each provider you can define the repos and folders to protect, as well as branches on which to perform the evaluation.
Sysdig confirms that all services that compose Sysdig’s Cloud Platform running Apache’s vulnerable Log4j library have been patched to 2.16. We have not detected any successful attempts at exploitation of this attack vector during that time window.
December 15, 2021
Update on Log4j Vulnerability (CVE-2021-44228)
The sysdig agent does not include the Log4j library
Sysdig is using an alternative framework for logging called Logback. The logback framework isn’t vulnerable to this issue.
Sysdig components include a log4j library in our standard distribution that was vulnerable. This library is included for compatibility reasons only, is not used for primary logging, and our security team has determined we are not vulnerable based on our application architecture and existing mitigating controls.
Sysdig can confirm that all services that compose Sysdig’s Cloud Platform running Apache’s vulnerable Log4j library have been patched to the latest version or adds additional mitigating controls suggested by vendors. We have not detected any successful attempts at exploitation of this attack vector during that time window.
Details regarding upgrades
We:
explicitly set commonsLog4jVersion = 2.15.0
update all of log4j-to-slf4j, log4j-api, and log4j-core to version 2.15.0
December 15, 2021
Sysdig Secure for Azure
Sysdig is pleased to announce the general availability of the Sysdig Secure for cloud capabilities for Azure.
Cloud Security Posture Management (CSPM): Based on CIS benchmarks tailored for your assets
Cloud Threat Detection: Identify threats in your Azure environment using Falco rules for Azure
Event Hub: Fully managed, real-time data ingestion service that’s simple, trusted, and scalable
Image Vulnerability Scanning: Automatic vulnerability scanning of images pushed to Azure Container Registry and images executed on Azure Container Instance Group.
A Statement on Log4j vulnerability (CVE-2021-44228)
Security researchers recently disclosed the vulnerability CVE-2021-44228 in Apache’s log4j, which is a common Java-based library used for logging purposes
Sysdig is using an alternative framework for logging called Logback. The logback framework isn’t vulnerable to this issue.
Sysdig components include a log4j library in our standard distribution that appears to be vulnerable. It has been confirmed that this library is included for compatibility reasons only and is not used for primary logging. As a result this should not pose any risks.
Patches will be provided to upgrade the log4j libraries that are included for compatibility reasons.
If you have any questions or concerns, please reach out to your Sysdig contact.
December 1, 2021
Image Analyzer 0.1.15 Inline Scanner 2.4.8 Released
Fixed support for COPY, USER, and other instructions when the image is built using buildkit
November 5, 2021
Improved Handling of Forwarded Benchmark Events
Forwarded benchmark events now include AWS tags as key-value pairs (rather than a flattened string), making them easier to consume.
November 2, 2021
Inline Scanner 2.4.7 Update
Requirements
libseccomp >= 2.3.3 (on the Host/JenkinsWorker - where the docker command is executed)
docker version > v18.05.0-ce
Fixes
Fixed support for COPY, USER, and other instructions when image is built.
October 27, 2021
Cloud Infrastructure Entitlements Management (CIEM) for AWS
Sysdig Secure has added Permissions and Entitlements Management functionality. You can find it under Posture menu tab.
By combining the CIEM capabilities announced today with Sysdig’s existing capabilities, Sysdig customers can proactively prevent cloud permissions risks, scan for vulnerabilities and misconfigurations, and detect and respond to attacks across container and cloud environments.
Gain visibility into all cloud identities and their privileges: get a comprehensive view into access permissions across all AWS users and services
Enforce least privilege: eliminate excessive permissions by applying least-privilege policies to users and services with automatically generated IAM policies. Sysdig proposes policies based on analyzing which entitlements are granted versus which are actually used.
Simplify audit of access controls to meet compliance requirements: use reports for regular access reviews to evaluate active and inactive user permissions and activity.
Learn more about Sysdig CIEM capabilities • https://sysdig.com/use-cases/ciem-cloud-infrastructure-entitlements-management/)
October 26, 2021
New Secure Event Forwarder Integrations: Google Chronicle, Google Pub/Sub & Amazon SQS
An extended set of output data integrations has been added to Sysdig Secure’s Event Forwarder functionality, in particular:
Integration with Google Chronicle.
NOTE: Only Runtime policy events are available as data to send at this moment.
Integration with Google Pub/Sub and Amazon SQS, which can be used as temporary storage that will adapt the EFO push behaviour into a data pull endpoint.
Sysdig is pleased to announce the general availability of the Sysdig Secure for cloud capabilities for GCP.
Cloud Security Posture Management (CSPM): Based on CIS benchmarks tailored for your assets
Cloud Threat Detection: Identify threats in your GCP environment using Falco rules for GCP
Audit Logs: Google Security Command Center integration to forward threats identified by Falco rules.
Image Vulnerability Scanning: Automatic vulnerability scanning of images pushed to Google Container Registry, Google Artifact Registry and images executed on Google Cloud Run.
Chronicle Integration: Events forwarding to Google Chronicle.
Installation via GCP Marketplace: You can install Sysdig from the GCP marketplace and pay using the payment method and credit of your GCP account.
Sysdig Secure is developing a new scanning engine that will deliver major improvements, additional capabilities, and scanning-centric workflows.
The first iteration is available to test and provides:
Much faster scantimes: 4x to 10x faster initial image analysis
Extended vulnerability data, including CVSS scores, vectors containing the full exploitability data, availability of an associated public exploit, etc.
Inline scanner available as a stand-alone binary, no longer requires spawning a container
Better remediation advice, including ‘Which packages are the worst offenders in my image? Considering all the possible fix versions, which one should I apply?’
Improved, more intuitive user experience, with faster response times
Important: The new engine is still on “Preview” phase.
This means:
Not suitable for production. There is no forward compatibility guarantee for the data or configuration (yet)
Testing the new scanning preview will NOT affect any existing scanning workflows that leverage the current scanning backend. It is safe to enable the preview on any account.
Additional fundamental components are still in development; they will be released in an upcoming version.
To test the new engine, simply enable the flag under Settings >User Profile>Sysdig Labs.
In Sysdig Secure, the Scheduled Reports for Scanning now display additional vulnerability metadata for both runtime and registry reports.
Specifically:
Disclosure date: Time when the vulnerability information was registered in the feed
Solution date: Time when the fix version for this vulnerability (if any) was registered in the feed
To avoid breaking compatibility with existing reports and external instrumentation, these fields will only be available for newly created reports; existing Scheduled reports (even if they are modified and saved again) will not contain these columns.
September 8, 2021
New and Updated Compliance Standards
Sysdig Secure has added three new compliance standards and updated
another. See also:
Compliance
Updates to PCI DSS v3.2.1 Compliance for Workload
We have implemented some changes to the PCI DSS v3.2.1 for workload
compliance checks. The control coverage for PCI is now: 1.1.2, 1.1.3,
1.1.5, 1.1.6.b, 2.2, 2.2.a, 2.2.1, 2.2.2, 2.4, 2.6, 4.1, 6.1, 6.2,
6.4.2, 6.5.1, 6.5.6, 6.5.8, 7.2.3, 10.1, 10.2, 10.2.1, 10.2.5, 10.2.7,
10.5.5, 11.5.1
Checks added:
Check for Network Security enabled added to controls 1.1.2, 1.1.3
and 1.1.5
Check for Kubernetes audit enabled added to controls 4.1, 6.4.2 and
6.5.8
Rules added:
Rule
Outbound or Inbound Traffic not to Authorized Server Process and Port
added to control 2.2.1
Rule Attach to cluster-admin Role added to controls 7.2.3 and
10.5.5
Rules EphemeralContainers Created and
Terminal shell in container added to controls 10.1 and 10.2.1
Rules ClusterRole With Pod Exec Created ,
ClusterRole With Wildcard Created and
ClusterRole With Write Privileges Created added to control 10.2
Rule Launch Privileged Container added to control 10.2.5
Rules Container Drift Detected (chmod) and
Container Drift Detected (open+create) added to control 11.5.1
Rules removed:
Rule All K8s Audit Events rule removed from controls 10.1, 10.2,
10.2.1, 10.2.7
New PCI DSS v3.2.1 Compliance for AWS
The PCI Quick
Reference describes
the full range of controls required to pass a PCI 3.2 audit. In this
release, Sysdig Secure will add the following controls.
The AWS Well Architected Framework whitepaper
defines best practices to build secure, high-performing, resilient, and
efficient infrastructure for applications and workloads.
For AWS protection, Sysdig Secure will check the following sections:
AutoScaling.1, CloudTrail.1, Config.1, EC2.6, CloudTrail.2, DMS.1,
EC2.1, EC2.2, EC2.3, ES.1, IAM.1, IAM.2, IAM.4, IAM.5, IAM.6, IAM.7,
Lambda.2, GuardDuty.1
New NIST 800-171 rev2 Compliance
The National Institute of Standards and Technology (NIST) Special
Publication 800-171
rev2
describes the full range of controls required to pass a NIST 800-171
audit. It provides agencies with recommended security requirements for
protecting the confidentiality of Controlled Unclassified Information
(CUI) when the information is resident in nonfederal systems and
organizations.
For AWS protection, Sysdig Secure will check the following
sections:3.1.1, 3.1.2, 3.1.3, 3.3.1, 3.3.2, 3.3.7, 3.5.7, 3.5.8, 3.14.6,
3.14.7
September 2, 2021
New Terraform Onboarding Options for Secure for cloud
Users can now onboard Sysdig Secure for cloud with their AWS accounts
(single or organizational) using Terraform. See the feature
description and the
deployment/onboarding
instructions.
Updated base image to get updated security fixes (July 2021)
Added retry mechanism when pulling images from registries
Added --write-json PATH option to permit storing json log to file
Fixed Malware scan fails when image has not read the permissions on
files
Fixed failure in getting images for registries that do not support
tag listing
July 27, 2021
Admission Controller with Kubernetes Audit (k8s_audit Falco rules)
Today we announce the general availability of the Kubernetes Audit
functionality as part of the Sysdig Secure Admission Controller.
Background:
Kubernetes admission
controllers
provide operators the ability to validate and/or mutate incoming API
requests. Admission controllers are a core functionality of Kubernetes,
and many are enabled by
default.
Sysdig Secure has long provided Kubernetes API security using
k8s_audit Falco rules to create policies against Kubernetes audit
logs. However, there have been some complications:
Many Kubernetes distros are opinionated in the way to collect and
access logs, some using dynamic backends (deprecated in Kubernetes
1.19, but still available in OCP up to 4.3), while more vanilla
approaches use webhooks, and cloud providers require a bridge to
collect logs via their own logging streams.
Distros diverging from Falco:
With OCP 4.4+, we had no clear way to collect and validate audit
logs against our Falco rules.
The Solution?
Tap directly in the Kubernetes API request via Admission Controllers and
use the existing k8s_audit rules our customers have relied on for so
long. See the installation
instructions.
July 2, 2021
Inline Scanner 2.4.3 Released
Change:
Updated base image to get updated security fixes (June 2021).
Fixed:
Fixed incorrect version detection for Apache Struts 2 packages which
was leading to false positives.
July 1, 2021
Node Image Analyzer v0.1.13 Released
Version
0.1.13
of the Node Image Analyzer has been released.
This release comes with the following improvements:
Fixed:
Fixed a GKE- and ContainerID-specific bug where the node image
analyzer couldn’t scan the image due to missing blobs
Implemented a few-second pause at startup to allow for Istio
sidecars to complete the initialization before creating connections
New
We use the Universal Base Image
(UBI)
Sysdig-approved image as the base, in order to ensure the highest
patch level approved by our security team.
June 23, 2021
Enhancements to Compliance Module
Terminology note: Compliance standards are scoped to different
platforms depending on the specific security rules they include,
Broadly, these are divided into:
Workload types: Including any Falco rules for kernel system
calls, Falco rules for Kubernetes audit logs, host benchmarks, and
security features that affect hosts, containers, and kubernetes
clusters
AWS/cloud type: Falco rules for CloudTrail and Cloud Custodian
rules on Amazon Web Services
Sysdig navigation just got a facelift. To help our Sysdig Secure users
navigate easily, we:
Added the new menu item
Network (previously
found under the Policies menu), and
Grouped Activity
Audit +
Captures into
Investigation to better describe the use-case it helps users
resolve.
Activity Audit
The Activity Audit module also got several interface and user experience
improvements:
Runtime scope moved to the top to align with other Secure interfaces
and allow more space for activity data
Activity types (network, file, kubectl, command) can now be
filtered directly from the graph using the legend
Attributes of the displayed elements can be filtered directly from
the list, without displaying the side detail panel
June 4, 2021
Kubernetes Network Security: New Configuration and Improved User Experience
Sysdig’s Kubernetes Network Policy tool
has been updated to include
additional fine-tuning configurations and an improved user experience.
Additional Configuration Panel
Workload Labels: Depending on your workload labelling policy,
some labels may not be relevant for generating a KNP policy. Use the
additional config to include/exclude a particular set of labels per
cluster/namespace to declutter your UI and the resulting policy.
Unresolved IP Configuration: Now it is possible to label raw IPs
that are not mapping to your Kubernetes/OpenShift entities, i.e.
external cloud provider services, so these labels will be
automatically applied to the topology and ingress / egress tables.
Cluster CIDR configuration: If the CIDR configuration is not
automatically detected by the agent, you can now directly configure
internal subnets per cluster using the Sysdig interface.
Improved UX
Topology map: Additional information pop-up when hovering over a
network connection or a network node, such as server process,
source, destination, and more.
Unresolved IP filtering: In the ingress and egress tables, by
type or using free text search.
Additionally, Network
is now presented as a top-level item in the Sysdig Secure navigation.
May 27, 2021
Falco Policy Tuner - Beta
Sysdig is now releasing a managed
version of the standalone
Falco Tuner.
Previously, you had to run the tuner in your local environment, print
suggestions, and manually update a rule with those suggestions. The new
feature runs in the background and automatically tunes noisy rules and
false positives. To streamline the creation of these exceptions, we’ve
created a new object within Falco called exceptions.
Note: To enable the tuner, Admin access rights to Sysdig Secure are
required.
Feature Enhancement: Falco Exceptions
Previously, exceptions were created using and not conditions inside a
Falco rule, e.g.
- rule: Write below binary dir
...
condition: >
bin_dir and evt.dir = < and open_write
and not package_mgmt_procs
and not exe_running_docker_save
and not python_running_get_pip
and not python_running_ms_oms
and not user_known_write_below_binary_dir_activities
....
However, this process can be unwieldy and can result in unintended
behavior. The new format, using exceptions, looks like this:
- rule: Write below binary dir
...
condition: bin_dir and evt.dir = < and open_write
....
exceptions:
- name: package_mgmt_procs
fields: proc.name
comps: in
values: package_mgmt_binaries # list of known binaries
...
HIPAA (Health Insurance
Portability and Accountability Act)
See also:Compliance for
information about the specific controls Sysdig covers for each security
standard.
Inline Scanner v2.4.1 Released
Version
2.4.1
of the inline scanner container has been released. See also: Integrate
with CI/CD Tools.
Fixes:
Updated ClamAV version to 0.103.2 to avoid end-of-life problems
present in the former version, such as failure in updating the
antivirus database
Additional type Descriptor Forwarding Activity Audit through Event Forwarder
The JSON payload when sending Activity Audit
elements through the Event
Forwarder will now contain an additional field name: type. This
describes the type of the entry, respectively: command, connection,
fileaccess, or kubernetes.
New and Improved Host OS and Container Scanning Tools
We at Sysdig are working hard to improve your security posture and
compliance experience. As part of this commitment we are implementing a
new framework to generate host benchmark results, introducing host
scanning, and making backend improvements to the image scanning
mechanism.
Installation Steps
The new features require a new component to be installed called the
Node Analyzer. We’ve provided an installation
script to automate the
installation or to upgrade an existing Node Image Analyzer daemonset, if
applicable.
Once you’ve installed or updated the components, the UI will
automatically show Host Scanning and new Benchmarks functionality
(Legacy Benchmarks can
still be accessed.)
Automatically scan images if they have not been scanned
April 29, 2021
New Scan Results Page Layout
We have reorganized the visual layout of the Scan Results summaries to
clearly distinguish policy evaluation from vulnerability matching and to
better summarize the information.
Improvements include:
Vulnerabilities and Policies are now two different sections in the
UI
Vulnerability match update time is displayed to further distinguish
from the Policy Evaluation time
Policy breakdown is collapsed by default to reduce cognitive load
Re-evaluate policies button is now located in the impacted section
only, as opposed to whole page
Apart from the vulnerability update time, the data remains unchanged
from previous versions
Added HTTP_PROXY and HTTPS_PROXY environment variables support
for malware scanning mode. This is required if you want to retrieve
the malware database inline behind a proxy.
Added support for.dockercfgrepository auth method, accessible
via the--registry-auth-dockercfgCLI flag.
Fixes:
Now using HTTP1.1 by default to bypass a cURL bug.
Provided fix for an error when using the docker-daemon storage type
with a docker UID different than 1000.
March 30, 2021
Sysdig Secure for cloud
Sysdig Secure for cloud is
available with Cloud Risk Insights for AWS, Cloud Security Posture
Management based on Cloud Custodian for AWS and multi-cloud threat
detection for AWS using Falco.
What’s Included in this release:
Insights: a
powerful new visualization tool for threat detection, investigation,
and risk prioritization, to help identify compliance anomalies and
ongoing threats to your environment. With Insights, all findings
generated by Sysdig across both workload and cloud environments are
aggregated into a visual platform that streamlines threat detection
and forensic analysis.
Cloud Security Posture Management with AWS Benchmarks:The AWS CIS Benchmarks
assessment evaluates your AWS services against the benchmark
requirements and returns the results and remediation activities you
need to fix misconfigurations in your cloud environment.
We’ve included several UI improvements to provide additional details
such as: control descriptions, affected resources, failing assets,
and guided remediation steps, both manual and CLI-based.
Manage cloud posture with a daily run of CIS Benchmarks
Detect threats with out-of-the-box CloudTrail detection rules based
on Falco
Scan containers (ECR/Fargate scanning) automatically and within your
cloud environment for upto 250 images a month
March 24, 2021
Image Scanning Reports v3 [BETA]
The Image Scanning Reports feature has been thoroughly updated and has
moved from a synchronous model to an asynchronous mode, in which you
schedule the reports you need and then receive them through your normal
notification channels (email, Slack, webhook.). The new version also
includes:
A preview function to check report structure in the UI
A more advanced query builder
Extended set of data columns (i.e. CVSS base score and vector) and
extended set of available filters (i.e. package type)
Reporting v3 supports two different types or reports:
Vulnerability report: Containing vulnerability, package and
image data
I.e. Vulnerabilities in my runtime with Severity ≥ High, a Fix
available and not included in a vuln exception list.
Policy report: Containing scanning policies and evaluated images
data
I.e. Images in my internal registry failing the “NIST” scanning
policy.
You need to enable this feature from the Sysdig Labs setting on the
User
Profile
page.
Sysdig Secure has introduced Policy Types– a separation of policies
into logical groups, based on the sources used in the policy engine.
When creating a policy, you choose a type and then only the relevant
scopes and container actions will be presented.
We have also introduced a new policy type to support threat
detection with AWS
CloudTrail rules.
New table design to offer additional visual feedback and reduce data
redundancy, plus additional vulnerability data.
New functionality:
Individual vulnerabilities can now be clicked to display additional
information in a side panel:
The vulnerability feed source that was used for the matching
A description of the vulnerability
March 15, 2021
Sysdig Serverless Agent 1.0.0 for Fargate ECS
The “container-as-a-service” serverless environment calls for new
agent models, and Sysdig provides them. Whereas in ECS, users still
manage the underlying instances, with AWS Fargate the host is never
visible and users simply run their workloads. And while this model is
convenient, it can introduce risk as many people leave the containers
unattended, without monitoring security events within that can
exfiltrate secrets, compromise business data, impact performance, and
increase their AWS costs. In addition, it is not possible to install a
standard agent in an environment where you do not have access to a host.
For these reasons, Sysdig has introduced a new “serverless agent” model
that can be deployed in these container-based cloud environments. The
first implementation is for Fargate (ECS).
Sysdig will be rolling out security features on the serverless agent
over time. In v1.0.0, users will see:
Runtime Policies and Rules
Secure Events
To obtain secure event information and the associated Falco policies and
rules in the Sysdig Secure UI from a Fargate environment, users install
the serverless agent using a CloudFormation Template. Then log in to
Sysdig Secure and review the events in the UI.
The “Commands Audit” feature was deprecated in favor of Activity
Audit in November 2019.
This feature will be completely removed from the SaaS product April
2021 (next month).
Sysdig agent version
0.93+,
released in November 2019, is required by the Activity Audit
feature.
The “Policy Events” feature was deprecated in favor of the new
Events feed in
June 2020. This feature will be completely removed from the SaaS
product April 2021 (next month).
Usability improvements that display the policies in which a rule is
used, from both the Rules Library list and the Rule Detail view. See
Manage Rules for details.
March 2, 2021
Regulatory Compliance for SOC 2, NIST 800-53 rev4 and rev5
Three new compliance standards have been added to Sysdig compliance
feature: SOC 2, NIST 800-53 rev4 and NIST 800-53 rev5.
The compliance validator now also includes new checks for the following
features: Admission Controller, Network Security Policies and Node
Analyzer.
See the Compliance
documentation for usage details and the controls implemented.
February 23, 2021
Windows Scanning Released
A beta version of the Windows Scanning Inspector has been released. This
is a new feature from Sysdig for scanning Windows containers.
This is a standalone scanning engine. There is no centralized UI,
management, or historical data. These features are planned for a future
release.
Identify Windows container image vulnerabilities from:
Windows OS CVEs
Windows or Linux hosts
Reports in JSON and PDF
Policy support
Severity
Fix available
Days since fixed
UI-Based Admission Controller Released
Kubernetes’ admission controllers help you define and customize which
requests are allowed on your cluster. An admission controller intercepts
and processes requests to the Kubernetes API prior to persistence of the
object, but after the request is authenticated and authorized.
Sysdig’s Admission Controller (UI-based) builds upon Kubernetes and
enhances the capacity of the image scanner to check images for Common
Vulnerabilities and Exposures (CVEs), misconfigurations, outdated
images, etc., elevating the scan policies from detection to actual
prevention. Container images that do not fulfill the configured
admission policies will be rejected from the cluster before being
assigned to a node and allowed to run.
Granular admission policies: Defining a global policy per cluster,
but also at the level of particular namespaces or image paths (i.e.
registries) Registry and repository whitelist
Only allow images that pass the scanning evaluation criteria
Only allow images that have been evaluated recently
Only allow images that have been scanned before creation is
requested to Kubernetes
With the addition of CronJob support, communication is aggregated to the
CronJob (scheduler) level, rather than the Job. Therefore, when
administrators review the activity in the Network Security Policy menu,
they will see the higher-level CronJobs listed, and not an excess number
of individual Job entries.
This update also adds support for
Weave and
Cilium CNIs on top of
Calico
support.
Malware Detection during Inline Image Analysis
As part of the inline scanner version 2.3.1
release,
malware scanning was added as a configurable detection that can be
performed during inline analysis.
The default behavior if this feature is enabled and malware is found is
to consider the scanning failed, report malware details, and abort
analysis:
Registry Credentials: Support for Multiple Credentials
Sysdig Secure now supports assigning multiple credentials to the same
registry depending on the relative internal registry path that is used
to pull the image.
A wildcard can be added to the end of the path, indicating that any
image located under the partial path inside the registry (/rg-2-1er in
the example) will use the registry credentials configured here. This
additional flexibility is useful, for example, for IBM registries which
can have a different set of permissions depending on the namespace.
Version
2.3 of
the Inline Scanner has been released.
Fixes:
Avoid prefixing the image names with localbuild when not strictly
necessary
New:
Improved version detection for specific software packages:
logback, SpringFramework and Tomcat Java
Allow setting of openssl security level via OPENSSL_SECLEVEL env
var to support old certificates
More robust image ID identifier, avoiding unnecessary image re-scans
along the container lifecycle
Added malware detection feature
February 4, 2021
Enhanced Activity Audit Filters
We have improved the noise-reduction filter for the Activity Audit
feature in Sysdig Secure.
The feed will now automatically filter out duplicate entries with a high
number of occurrences. No information is lost, as the filtered noise is
only duplications of entries in the feed.
A sudden reduction in the number of Activity Audit entries per time slot
is expected as a result of this filter.
January 28, 2021
Node Image Analyzer v0.1.9 Released
Version 0.1.9
of the Node
Image Analyzer has been released.
This release comes with the following improvements:
Fixes:
Fixed an issue that prevented some images from being processed on
GKE clusters using Docker and Containerd
Fixed an issue that prevented some images that don’t have full tags
from being processed on OpenShift
Improved version detection for Logback, SpringFramework and Tomcat
Java packages
Fixed an issue that resulted in the image analyzer crashing without
a proper error message when an incorrect Docker socket path was
provided
We have seen requests for statements regarding tooling in the wake of
the Solarwinds and related compromises. Sysdig does not use these tools
internally. To maintain a secure SDLC process for own product we use
Sysdig Secure as well as source code analysis tools. We also maintain
our own branch of key OSS components to ensure software is fully vetted
before it’s delivered to customers.
December 14, 2020
Perform Image Scanning as a GitHub Action
A new version
of
the Sysdig Secure inline scanning action has been released. This
Github action allows you to perform image analysis on locally built
container images.
The action uses the new secure-inline-scan 2.x, which provides better
performance and more input options. See Inline Scan 2.2 Release
Note.
The action provides the following benefits:
Image evaluation results can be consumed using the Sysdig Secure UI
or locally as check-run annotations.
Support for SARIF report output.
This provides native integration with Github’s code scanning, for
example: executing the
codeql-action/upload-sarif
action.
The JSON format for the runtime policy events has been upgraded to
include full scope information, rule labels, and a single-line
representation for the event field’s keys and values.
To preserve backwards compatibility with existing integrations, the
former JSON format is still available (and used by default on
migration).
From the Event Forwarder page, under “Data to Send,” the old JSON format
is labeled “Policy Events (Legacy)” and the new one as “Runtime Policy
Events.”
Kubernetes-Native Network Security with Sysdig Secure (Beta)
A new feature has been added to Sysdig Secure for authoring and refining
Kubernetes network policies (KNPs) that:
Automatically extracts the connection information, by observing the
cluster networks and microservices communications
Offers a visual flow to fine-tune the Kubernetes network policies,
incorporating the user’s adjustments
Automatically generates the KNP YAML to be applied, without
requiring previous Kubernetes policy knowledge from the user.
As soon as the feature is enabled, the Sysdig agent starts collecting
and processing application communications, which are then enriched using
Kubernetes metadata and presented in two different ways:
Topology maps: a visual representation of the network flow
between the Kubernetes entities (Services, Deployments,
StatefulSets, DaemonSets, Jobs)
Ingress / Egress tables: for additional detail on each
inbound/outbound communication and policy tuning.
Once the user has finished editing the desired policy, Sysdig will
automatically compute the associated KNP YAML:
Enforcement is delegated to the Kubernetes control plane, favoring
policy-as-code and avoiding direct tampering with cluster
communications
Allow-only approach ensures that any communication which is not
explicitly allowed by the policy will be forbidden
Prerequisites
Sysdig agent version 10.7+
Supported Orchestrator Distributions and CNI Plugins:
Vanilla Kubernetes (kops, kube-admin) using Calico
OpenShift 4.x using OVS
Google GKE using Calico
Amazon EKS using Calico
Rancher Kubernetes using Calico
Please contact us to enable this feature for your Sysdig Secure
accounts.
Sysdig Inline scanner can be used stand-alone or as a step inside a
CI/CD pipeline (Jenkins, Tekton, CircleCI, etc). In the upcoming weeks,
we will update the different integrations to provide out-of-the-box
support for the 2.0 version.
Using the Kubernetes benchmark, we now provide customer-selected
benchmark checks for GKE and EKS (rather than just the Kubernetes
default).
October 9, 2020
Regulatory Compliance Control Validation & PCI Checks
A new feature has been added to Sysdig Secure for checking controls from
various compliance standards. For the first release, we provide checks
against specific controls in PCI 3.2. Future releases will include
SOC2, NIST-800-53, and more. See also:
Compliance in Sysdig
documentation.
Compliance Validator and Reports
The validator checks many Sysdig Secure features, including: image
scanning policies, Falco runtime policies and rules, scheduled benchmark
testing, Admission Controller, Network Security Policies, Node Image
Analyzer, and more. Over time we will add new compliance coverage.
Disclaimer: Sysdig cannot check all controls within a framework, such
as those related to physical security.
This feature is a beta release. A Sysdig Secure admin must enable it
from the Sysdig Labs interface under Settings.
PCI Control Details
The PCI Quick Reference
describes
the full range of controls required to pass a PCI 3.2 audit. In this
release, Sysdig Secure will check the following subset:
Authentication methods: Basic authentication, Bearer Token, and
Signature Header
Custom headers defined by the user to accommodate any additional
parameter required on the receiving end
September 29, 2020
Vulnerability Exceptions Handling Enhanced
The Vulnerability Exceptions feature in Sysdig Secure has been
redesigned and enhanced.
It now offers:
Additional vulnerability and feed context
Precise mapping between images and their associated exceptions
A better exception management lifecycle
Multiple vulnerability lists, which can be flexibly assigned to
different image sets (or just a particular image), using the
scanning policy assignments
Additional information displayed to improve team awareness and
security context
Vulnerability description
User-defined notes
Vulnerability feed info, with severities and links as provided
per feed
Configurable expiration dates:
An exception is automatically disabled when the expiration date
is met
Day resolution, all times relative to 0:00 UTC
Enhanced workflow integration with the “Scan results” page for an
individual image, with the ability to quickly append a flagged
vulnerability to a list.
Migration: The exception and evaluation behavior in the current
environment will be maintained after the feature upgrade. In particular:
Pre-existing vulnerability exceptions will be migrated to the
“Default exceptions list”
The “Default exceptions list” will be assigned to every pre-existing
policy assignment
All the pre-existing vulnerability exceptions expiration date will
be set to “Never.”
AWS Threat Detection using CloudTrail and Sysdig Secure
Sysdig is happy to announce the general availability of a CloudFormation
Template that will deploy a cloud-native operational security engine. By
leveraging AWS CloudTrail and the Falco language, you can detect any
unexpected or unwanted behavior in your AWS accounts.
Sysdig Cloud Connector leverages AWS CloudTrail as the source of truth
for enabling governance, compliance, operational auditing, and risk
auditing for your AWS account.
Every API action over your infrastructure resources is recorded as a set
of CloudTrail entries. Once the integration is deployed in your
infrastructure, the Sysdig Cloud Connector can analyze these entries in
real-time and provide AWS threat detection by filtering them against a
flexible set of security rules.
Example detection rules included in this release:
Attach a user to an Administrator Policy
Create an HTTP Target Group without SSL
Deactivate MFA for user access
Delete S3 bucket encryption
Sysdig Cloud Connector provides several notification options, including
sending security findings to AWS CloudWatch and AWS Security Hub. When
configured, you can consume the security events without leaving your
cloud console.
Sysdig is pleased to announce the general availability of a new
integration leveraging the Sysdig Inline Scanning capabilities to
automatically analyze the base images used for any task created using
AWS Elastic Container Service (ECS or Fargate).
The only mandatory parameter is the Sysdig API token.
Inline scanning living inside your AWS account means improved
security:
No need to expose or configure private AWS registries
Only image metadata is sent to Sysdig Secure, not the actual
image contents
No sensitive information ever leaves your AWS account
An ephemeral task will be spawned to analyze each discovered
images, in parallel
Each time you deploy a new task in AWS ECS/Fargate, an EventBridge
event will be triggered and a lambda function will parse which
images need to be analyzed by the CodeBuild pipeline job.
Fully automated
Scan results and scanning policies are still controlled from a
single security governance point using Sysdig Secure
Node Image Analyzer Version 0.1.3
This version adds support for running the node image analyzer in
Kubernetes environments with containerd, such as Google Kubernetes
Engine configured with cos_containerd. See also: Scan Running
Images.
July 29, 2020
Replacing RHSA Advisories with CVE Advisories
In new images scanned, RHSA advisories will be replaced with CVE
advisories. The results for existing images will be updated in the
background over the next week.
This change provides better matches for CVEs that are not yet fixed or
will not be fixed since those do not yet have RHSAs. It also makes the
CVE the match key rather than RHSA for more consistent whitelisting and
policy handling compared to other distros.
Scanning Adapter Available for Harbor
The Sysdig Secure Harbor Scanner Adapter enables Harbor to use Sysdig
Secure scanning engine to analyze the container images managed by the
platform.
This adapter also provides a service that translates the Harbor scanning
API requests into Sysdig Secure API calls, allowing Harbor to retrieve
vulnerability reports and additional information from the scanning
adapter. This will be presented transparently in the Harbor UI to the
user.
The scanning adapter supports two operation modes:
Backend Scanning: Image scanning happens in the Sysdig Secure
backend
Inline Scanning: Image scanning happens in the infrastructure
where Harbor is hosted
To learn more about this integration, read the
documentation.
July 28, 2020
Captures Filter on the Policies Page
Policies can now be
filtered to display if a capture is associated with an active or
inactive policy.
Image Exclusion on Policy Events
Users often want to tune policy
events. We’ve added a
button on the event detail that will add an exclusion to a specific
container.image.repo for the policy that triggered the event. Once
that exclusion is applied to the scope, policies will no longer fire for
that container.image.repo.
July 26, 2020
Sysdig Essentials
We have introduced a new product tier, Sysdig Essentials. This tier
includes everything required to achieve the five essential requirements
for practicing Secure DevOps:
With the introduction of Essentials, It’s also easier to get started
with a trial program and manage your Sysdig
subscription.
Learn the difference between Essentials and Enterprise, including
pricing and features, at Pricing.
Sysdig Platform Enhancements
SAML Single Sign-On
The initial email to the following types of users will take them
directly to the Single-Sign-On URL, and not the registration page.
SAML SSO Users
The users that are invited to the platform (as opposed to having
them automatically created via Sysdig on-demand provisioning for
SSO)
Earlier, landing on the registration page was confusing to users because
they had to set up their initial password.
Rebranded Login Page
The login page has been updated with the Sysdig Kraken and the new logo.
June 29, 2020
New Sysdig Secure Overview Page
The Sysdig Secure Overview provides an at-a-glance view of the critical
areas of your security posture.
Scoping
Panels can be scoped by Cluster or Namespace. The scope will update
all panels that are displaying run-time data and the corresponding
drill-down views.
Panels
Build Time - Images Scanned: Image scan results for all static
image scans
Sysdig Secure’s Event Forwarder Now Supports IBM Cloud Pak for Multicloud Management and IBM QRadar
IBM Cloud Pak for Multicloud Management centralizes visibility,
governance, and automation for containerized workloads across clusters
and clouds into a single dashboard.
You can now forward security events to an IBM MCM instance by accessing
the Settings > Event Forwarding menu and selecting IBM MCM from the
dropdown:
IBM QRadar Security Information and Event Management (SIEM) helps
security teams accurately detect and prioritize threats across the
enterprise and provides intelligent insights that enable teams to
respond quickly to reduce the impact of incidents.
You can now forward security events to an IBM QRadar instance by
accessing the Settings > Event Forwarding menu and selecting IBM
QRadar from the dropdown.
The Events feed in Sysdig Secure (formerly called Policy Events) has
been redesigned, both visually and functionally.
Apart from the styling and user experience improvements, these are the
major new features and use cases
Advanced Filtering
We are deprecating the grouping/clustering of events present in the old
version in favor of a much more powerful set of filtering capabilities:
Severity filters: Presented as quick buttons at the top,
supporting multi-select
Attribute filters: Provide a simplified syntax to filter events
by the attributes they contain. For example
ruleType="Falco - Syscall" or image.repo!="sysdig/agent"
Open the event details side panel to find quick filtering
widgets to include or exclude the attribute values associated
with the displayed event
Event type selector: Supports runtime scanning alerts on top of
policy runtime events (see section below), with an easy
multi-selector in the UI.
Free text search: Allows you to search the event titles and
scope label values. I.e. Terminal shell in or my-k8s-cluster.
New scope selector: Allows for additional selector logic (in,
not in, contains, startswith, etc), improving the scoping
flexibility over earlier versions. This scope selector also provides
scope variables, allowing you to quickly switch between, for
example, Kubernetes namespaces without having to edit the panel
scope.
All these filters can be combined additively to further refine your
search.
The backend architecture, filtering, and UX have been designed to
accommodate additional types of security events that will be pushed to
the Event Feed in the future, upgrading the interface from a
policy-runtime-centric experience to a full security center control
panel.
Additional Event Details
Policy runtime events: These now display the rule that was fired
together with the rule labels. You can use the quick filters mentioned
above to further refine the search.
Richer scope: Every security event now displays all the scope labels
retrieved for the event, not just those configured in the scope
selector.
Events in the old and new format will be stored separately:
No event or event data will be lost during the transition
Events that were registered before the new feed is deployed can be
browsed using the old feed interface, which is available from the
burger menu in the top-right corner
Events that happen after the new feed is deployed will appear in the
new event feed
Eventually, all events within the retention period will be present
in the new interface, at which point the version switcher will
disappear
June 17, 2020
Menu Update
The ordering of the side menu has been changed.
Image Scanning Updates
The image scanning navigation bar has changed.
The side menu is reorganized into Analyze and Configure sections
Analyze: Different areas of scanning that allow users to
view scan results
Configure: The areas of scanning that involve the setup of
the application
Whitelist terminology with CVEs has been removed.
“CVE whitelist” is now CVE Exceptions.
Team, Role, and Channel Updates
A variety of enhancements have been added to the team, role, and
notification channel options.
Now a new role, Service Manager, is also available in Secure. It has
the same permissions as the Standard User, plus the ability to invite
existing users to the team and manage the notifications channels
assigned to the team. See Team-Based Roles and
Privileges
Configurable Default Team Role
You can now define the default user role to apply when a new member is
added to the team. The Admin can change this default on a per-team
basis. See also: Create a
Team.
RBAC and Team Assignment for Notification Channels
Previously, notification channels in Sysdig Secure and Monitor were
treated as global entities, visible and editable for most users of the
platform regardless of team configurations.
We are enhancing the management and RBAC controls in the following ways:
Notification channels can now be “global” or limited to a particular
team
Global channels can be managed by admins and can be viewed/used by
other roles, while team-limited channels are available only to team
members
Team Manager , Advanced User, and Service Manager (Secure)
roles can create/update/delete team-scoped notification channels,
they can also read and use the global ones
Standard and View Only roles can read team-limited and global
notification channels
Admins will be able to create global notification channels and
migrate channels from “global” to “team-limited”, and also from one
team to another.
An additional tool for evaluating and admitting images is now available.
Sysdig Admission Controller
Sysdig’s Admission Controller (UI-based) combines the Sysdig Secure
image scanner with a policy language to evaluate scan results and the
admission context, providing great flexibility in the admission
decision. It also provides the first line of defense against image-based
security threats.
By using Kubernetes API extensions to perform image scanning and other
security checks on admission, we cover a major threat-prevention and
hardening use case: “Only the images that are explicitly approved will
be allowed to run on my cluster”.
The admission decision relies not only on the image name and tag but
also on additional context from the admission review, including
namespace, pod metadata, etc.
Features
Registry and repository whitelist / blacklist
Global and per-namespace admission configuration
Configurable pre-scan and post-scan behavior, i.e.:
Accept only the images that pass the
scan (default)
Directly reject non-whitelisted registries / repos, without
scanning
Accept the image even if it doesn’t pass the scan
Do not accept any image that hasn’t been scanned already
Pod mutation: image tag is replaced by digest to prevent TOCTOU
(Time of Check, Time of Use) issue if the tag is updated between the
scan and the pod scheduling
We’ve added VulnDB as an
additional 3rd-party vulnerability source to improve Sysdig’s coverage
in non-OS package vulnerabilities.
In addition, a new page is available for each VULNDB-linked advisory. It
lists the CVEs and details about the Common Vulnerability Scoring System
(CVSS) scores and external references.
Added Automatic Image Scanning using Node Analyzer
The (node) image analyzer (NIA) provides the capability to scan images
as soon as they start running on hosts where the analyzer is installed.
It is typically installed alongside the Sysdig agent
container.
This component was introduced to reduce dependencies on analyzing images
within the Sysdig backend (SaaS or On-prem). Some advantages include:
Sharing credentials with the Sysdig backend in order to pull images
is not required
Sharing the image content and potentially code with the Sysdig
backend is not required; only metadata will be sent out
Opening a network route to allow the Sysdig backend to reach the
user’s registries is not required
If you have run the single line agent install with the –image-analyzer
flag, then this component is already running in your infrastructure.
The feature is available for Kubernetes environments.
Two new scanning integrations are available for CI/CD pipelines. Sysdig
provides:
A reference implementation with Tekton Pipelines (prototype)
A fully supported integration with Amazon Elastic Container Registry
(ECR) for triggering auto-scans from the registry
Integrating Secure Image Scanning with Tekton Pipelines
Tekton Pipelines
allow you to implement CI/CD workflows using a highly modular,
cloud-native approach that:
Uses containers as the building blocks for individual tasks
Runs directly on Kubernetes/OpenShift without requiring a dedicated
infrastructure
Uses tasks that are purely declarative and described using their own
CRD, making them easily composable and reusable
Sysdig’s reference implementation details the prototype task to invoke
Sysdig Secure image scanning as a pluggable step in your CI/CD pipeline
with just a YAML file:
Leveraging Tekton integration with the orchestration layer, you can
retrieve the image scanning policy evaluation and state (pass/fail)
directly from the logs of the task pod.
Sysdig image scanner integration is deployed as a CloudFormation
template that listens to ECR registry events and uses AWS resources to
streamline the image scanning process.
ECR itself will trigger the scan, no need for your CI/CD pipelines
to actively pull from the registry
Deployed in a few clicks, you just provide basic configuration
parameters such as the Sysdig API token or the Sysdig backend URL
No need to configure registry scanning credentials on the Sysdig
Secure side
This integration offers two different operation modes
Inline scanning:
Scanning will be performed inside an AWS CodeBuild pipeline
allocating ephemeral resources
No need to configure any registry credentials for Sysdig Secure
No need to expose your ECR registry to the Sysdig Secure backend
Sysdig Secure will not retrieve the image contents, only the
metadata that is required to perform the policy evaluation
Backend scanning:
Sysdig Secure will retrieve the full image contents in order to
perform the scan
Your ECR registry must be reachable by the Sysdig Secure backend
Registry credentials are required, but they are pushed automatically
by a lambda function, no need for manual configuration
April 9, 2020
Updates to Default Rules and Policies
The following changes have been made to default Policies in Sysdig
Secure, and to default Falco rules:
New rule tags added that map Falco rules to PCI and NIST controls
New default policies added specifically for PIC/NIST compliance
Tuning modifications for:
Write below etc
Write below root
Change thread namespace
Run shell untrusted
Detect outbound connections to common miner pool ports
The Get Started page provides the key steps to ensure users are getting
the most value out of Sysdig Secure. We’ll update this page with new
steps as we add new features to Sysdig Secure.
The Get Started page also serves as a linking page for:
Documentation
Release Notes
The Sysdig Blog
Self-Paced Training
Support
Users can access the page at any time by clicking the rocketship in the
side menu.
Sysdig Agents can run the Independent Linux benchmark against the
underlying host where the agent is installed. The Linux benchmark can be
scheduled to run at a chosen interval in your environment and emits
results and metrics about the status of the tests.
Openshift Hardening Guide
The Openshift hardening guide implements configuration checks run by the
agent against Openshift environments.
Note: This is supported for 3.x versions of Openshift. When Openshift
releases a hardening guide for 4.x versions, we will update the
configuration checks.
Captures can be Routed to Specific Storage Locations
As a user, you may have different S3 buckets where you’d like to store
Sysdig captures, based on the environment where the policy event was
triggered. New options are available for deciding what storage option
you’d like to use for each policy event.
Feeds Status Page Added
It’s useful to understand the last time the feeds were updated,
especially in self-hosted environments. The Feeds Status page shows the
different vulnerability feeds we integrate with, their feed group (often
the distro version), the time of the last sync, and how many CVE records
are present in the feed group.
Inline Scanning Reporting Improvements and Documentation
This script from
SysdigLabs is useful for performing image analysis on locally built
container images and posts. The only dependency for this script is
access to docker-engine, Sysdig Secure endpoint (with the API token) and
network connectivity to post image analysis results.
Here are examples of using the inline scanner in different pipelines:
will generate a PDF artifact that is available for developers to consume
in the pipeline.
February 6, 2020
Data Retention Limits for Scan Results
Use this feature to set limits on how long image scan metadata is
stored, either by tags or days. This removes stale data and helps keep
scan results easy to read.
We’ve extended our test and documentation coverage for various
Kubernetes audit log integrations. This integration enables Sysdig
Secure to use Kubernetes audit log data for Falco rules, activity audit,
and to test the impact of Pod Security Policies.
In image scanning reports, the vulnerability comparison feature allows
users to compare two different tags within the same repo to see which
vulnerabilities are new or have been fixed in version X compared to
version Y.
This allows developers easily to compare the latest image to a previous
version to easily report on which vulnerabilities have been addressed
and which are new.
Sysdig Secure’s Activity Audit now supports a new data source element:
File activity.
Sysdig agent version 9.5.0+ is required to enable this new data source.
You can now filter the audit trail by file type or specific
file attributes:
File name
Directory
Command (used to access the file)
Access mode
File activity is also visible in the time-series graph at the top
(pink color):
Activity Audit will capture non-read file operations executed by
interactive commands
January 27, 2020
RBAC Capability Available in Sysdig Secure
The new role-based access control (RBAC) model available in Sysdig
Secure allows you to define the access privileges granted to each user
in a Sysdig Secure team.
Besides the Admin role, which has full access and belongs to every
team, there are four roles that can be assigned when adding a user to a
team. (Note that the role names are the same in Monitor and Secure, but
the privileges differ slightly. Users must be assigned Monitor team
roles and Secure team roles separately.)
View Only: Read access to every Secure feature within the team
scope. A View Only user cannot modify runtime policies, image
scanning policies, or any other content.
Standard User: Can push container images to the scanning queue
and view the image scanning reports. Standard Users can also display
the runtime security events within the team scope. They cannot
access the Benchmarks, Activity Audit. or Policy definition sections
of the product.
Advanced User: Can access every Sysdig Secure feature within the
team scope in read and write mode. Advanced Users can create,
delete, or update runtime policies, image scanning policies or any
other content. The Advanced User cannot manage other users.
Team Manager: Same permissions as the Advanced User + ability to
add/delete team members or change team member permissions.
Team Managers only have user administration rights within the
specific team(s) for which they are designated Managers.
The Captures function in Sysdig Secure has a new look and the following
usability improvements:
Bulk deletion of capture files
Ability to see whether a capture was triggered manually or by a
policy
Search across all capture files
November 13, 2019
Activity Audit (Beta)
The Activity Audit in Sysdig Secure allows you to browse a live stream
of activity from your Kubernetes containers and nodes. Audit takes the
highly detailed data from syscalls and Kubernetes audit logs captured at
the agent level, and makes it always-on, searchable, and indexed against
your cloud-native assets.
This stream includes executed commands, network activity, and kubectl
exec requests to the Kubernetes API. The Activity Audit allows users to
view different data sources in-depth for monitoring, troubleshooting,
diagnostics, or to meet regulatory controls (SOC2, NIST, PCI, etc).
Flexible filtering and scoping to help you focus on what’s relevant:
Filters allow you to search, sort, and surface meaningful data and
connections as they are needed. You can filter by data source type, data
source attributes (like command name or Kubernetes user) and dynamic
Kubernetes scope
Automatically trace a kubectl exec session : The built-in trace
functionality allows you to isolate and trace a kubectl exec access to
a pod, automatically correlating the original Kubernetes user and IP
that accessed the pod with the activity that was performed during the
interactive session, including commands and network connections.
Kubernetes Policy Advisor (Beta)
With the Kubernetes Policy Advisor, Sysdig Secure auto-generates Pod
Security Policies (PSPs) to significantly decrease the time spent
configuring Kubernetes Policies. Strict security policies reduce risk,
but can also break applications. Sysdig tests the impact of pod security
policies through simulations, enabling teams to adjust misconfigurations
before shifting to production. There are three main features that
comprise the Kubernetes Policy Advisor:
Auto generation: Sysdig Secure can parse any Kubernetes yaml file
that includes a pod spec to generate a tailor-made PSP based on the
configuration.
Simulations: Start a simulation of the auto-generated PSP or any
user-inputted PSP to see what pods would have been blocked from running
if this PSP had been actively applied to the cluster.
Events and tuning: Each pod/activity that would have violated the
PSP will generate an event. Within the event details, users can see
information about potential modifications they may need to make to the
policy or the pod configuration.
Image Scanning Improvement
Support for images based on Google distro-less OS, including detection
of base OS/version and installed OS dpkg packages.
November 4, 2019
Scanning Improvements
New Scanning Rules
File attributes can now be verified as part of the image scan analysis.
A specific file can be validated against a node or sha256 hash.
Scale Improvements to Scanning Reporting
No query conditions are required as part of the Package and Policy
Queries.
October 10, 2019
In-Line Scanning
Images can now be analyzed locally before they are pushed to a registry.
This has a couple key benefits to users.
Images can be analyzed before they’re pushed to a registry and
reduce registry cost
Customers using the Sysdig Secure SaaS offering don’t need to expose
their registry to our SaaS for images to be scanned
For openshift customers the in-lince scan option can be integrated
into the S2I process to scan images without needing to expose a
local cluster registry via a route
The Sysdig CLI provides an easy way to interact with the cli via the
command line. Read more
here.
Usage
Run it without parameters to get a list of all the commands.
$ sdc-cli
Usage: sdc-cli [OPTIONS] COMMAND [ARGS]...
You can provide the monitor/secure tokens by the SDC_MONITOR_TOKEN and
SDC_SECURE_TOKEN environment variables.
Options:
-c, --config TEXT Uses the provided file as a config file. If the config
file is not provided, it will be searched at
~/.config/sdc-cli/config.yml and /etc/sdc-cli/config.yml.
-e, --env TEXT Uses a preconfigured environment in the config file. If
it's not provided, it will use the 'main' environment or
retrieve it from the env var SDC_ENV.
--json Output raw API JSON
--version Show the version and exit.
--help Show this message and exit.
Commands:
alert Sysdig Monitor alert operations
backup Backup operations
capture Sysdig capture operations
command Sysdig Secure commands audit operations
compliance Sysdig Secure compliance operations
dashboard Sysdig Monitor dashboard operations
event Sysdig Monitor events operations
policy Sysdig Secure policy operations
scanning Scanning operations
settings Settings operations
profile Profile operations
New Package Reports
Package name/version are now grouped together to provide easy parsing of
all CVE’s associated with a package and the images using that package.
Sept 24, 2019
New Trigger Parameters for CVSS Score
Image Vulnerabilities can now be evaluated against their
CVSS (Common Vulnerabilities Scoring
System) score. If a vulnerability is =, <;>, <=, or >= to a specific
score, then the rule can trigger a warn/stop action.
Sept 18, 2019
Time Ranges Updated
The default time range options have been updated in Sysdig Secure.
The default time ranges are now set to:
10 Minutes
30 Minutes
1 HR
6 HRs
1 Day
3 Days
To look at a custom window of time, use the manual time window.
Sysdig Secure Summary Dashboard in Sysdig Monitor
Sysdig Monitor includes default dashboards that provide metrics about
number of agents installed, active policies, events that have occurred,
and the policies that have triggered them. Use these dashboards to
identify trends, report on coverage, or facilitate the tuning process.
Aug 12, 2019
Policy Editor
*Please upgrade to an agent version 0.92.0 or greater
This UX overhaul brings three major improvements for every Sysdig Secure
user:
Runtime policies can import any number of security rules. You can
scope the security policy using container, cloud and Kubernetes
metadata.
Tighter Falco integration, directly from the web UI. You will be
able to define a new trigger condition or append to the list of
forbidden external IPs just clicking on the rule.
A more structured way to group, classify and lookup rules,
following the standard Cloud native procedure: tags and labels.
Rules Library
Visualize your runtime rules properties in just a glance:
Where this rule comes from (Published By). The security team
can instantly recognize whether a rule came from a specific Sysdig
update, from a custom rules file created within the organization or
from an external rules source (like the Falco community rules).
When was the last time it was updated (Last Updated). You can
use this information to audit your rules or if you schedule periodic
updates, to confirm when last happened.
Rule tags: An effective method for organizing your rules. You
can use these tags to describe the targeted entity (host, k8s,
process), the compliance
standard
it belongs to
(MITRE,
PCI, CIS Kubernetes) or any other criteria you want to use to
annotate your rules.
Falco Lists
Easily browse, append, and re-use lists to create new rules. Lists can
also be updated directly via API if users want to add existing feeds of
malicious domains, or IPs.
Falco Macros
Easily browse, append, and re-use macros to create new rules.
Image Scanning - View Scan Results
Scan Results Page - The existing repositories page has been renamed
“Scan Results” this page also includes new capabilities to filter based
on where the images are deployed, and to easily browse/expand the
different repositories to see the image:tag’s that were evaluated and
their results
Whitelist labels available in vulnerabilities view - If a
vulnerability has been added to a whitelist then that status is
reflected in the Vulnerability report within the scan results.
Image Scanning Reports
Please contact Sysdig Support to enable this feature
The reports feature allows users to query the contents of a scan against
a static or run-time scope to generate a report that shows the risk,
exposure, or components of an image.
Use cases could include:
A new CVE has been announced, let me find all the running images in
my US East Cluster that are exposed to that CVE
Show me all images within my Google Container registry that have the
tag prod and have a vulnerability with a fix that’s more than 30
days old
Show me all images with a high severity vulnerability with a fix
that are running in my billing namespace
Types of Scanning Reports
There are three types of queries in the image scanning Reports:
Vulnerability Query Type
This report returns rows of vulnerabilities mapped to packages within
images in a static or run-time scope. In the example above we can see
the two images that are actively running in my environment now that have
the CVE - CVE-2017-8831
Package Query Type
This report shows all images actively running in my environment that
have a version of the bash package. It also shows if multiple images are
running the same package name & version and if there are any CVE’s
associated.
Policy Reports
Policy reports show all the policy evaluations that have occured,
whether or not they passed or failed, and the reason why an image may
have passed or failed. Reasons for passing or failing could be because
of, whitelists, blacklists, or just a standard policy evaluation.
July 12, 2019
Minor Improvements
Compliance Dashboards in MonitorLink from Sysdig Secure now
defaults to a 90-day view, to give users better visibility into how
their posture is changing over time.
Image ScanningNegligible vulnerabilities are now also shown as
part of the scan results summary.
June 27, 2019
Image Scanning: New Trigger Options
New Image Analyzed - Send notifications to different channels
when images with a particular registry, repo, tag are scanned.
Some users implement these type of alerts for implementing
workflows for image promotion, i.e.
“Push an image from staging to prod registry after a webhook is
sent that the image was scanned and it passed.”
CVE Update - Be notified whenever a vulnerability is added,
updated, or removed from an image within a registry.
Repository Alerts
Receive alerts about activity and changes that occur within your
registry. See Manage Scanning
Alerts.
Slack Notifications
Sample output of a CVE alert:
Sample output of an image-analyzed alert:
5 -
Sysdig On-Premises Release Notes
You may also want to review the update log for Falco rules used in the
Sysdig Secure Policy Editor. See Falco Rules
Changelog.
Oversight Services Now Offered for All Installs and Upgrades
As part of our continued focus on our customers, we are now offering
oversight services for all on-premise installs and upgrades. Your
Technical Account Manager (TAM), in conjunction with our support
organization and Professional Services [where applicable], will work
with you to:
Assess your environment to ensure it is configured correctly
Review your infrastructure to validate the appropriate storage
capacities are available
Review and provide recommendations for backing up your Sysdig data
Work with you to ensure our teams are ready to assist you during the
install and upgrade process
Provide the software for the install
Be available during the process to ensure a successful deployment
You can always review the process in the documentation on
GitHub (v. 3.6.0+) or
the standard docs site
(for older versions).
If you are a new customer looking to explore Sysdig, please head over
here to sign up for a trial on
our SaaS Platform. Alternatively, you can contact us
here.
Sysdig supports, tests, and verifies the latest versions of Chrome and
Firefox.
Other browsers may also work, but are not tested in the same way.
5.1.2 Hotfix May 2022
Upgrade Process
Supported Upgrades From: 4.0.x, 5.0.x
Secure Feature: Reporting
Added the Run Now and Download(s) menu items
Bugs
Fixed an “Unable to load latest task result” bug when accessing compliance benchmarks results
5.1.1 Hotfix May 2022
Upgrade Process
Supported Upgrades From: 4.0.x, 5.0.x
Sysdig Platform
Added the RelayState parameter optional for SAML configuration
Upgraded the Spring Framework to version 5.2.20 in the sysdig-backend container
Monitor
Added the ability to choose regions with Capture Storage.
Installer Improvements
Fixed an issue with MultiAZ GCP/GKE platforms that would prevent Elasticsearch from starting.
Fixed an ingress permissions issue when upgrading from 5.0.4 to 5.1.0 that would result in the Sysdig UI generating a 404 Not Found error.
Fixed an installer bug when cloudProvider.name was set and cloudProvider.region was not set.
Fixed a Kafka/Zookeeper statefulset naming issue when installing or upgrading Sysdig on-premise
Bugs
Monitor Alert re-notification messages now provide the latest metric value instead of the metric value at time of triggering.
Fixed a Runtime scan page issue not displaying image results based on specific Team scopes.
Release 5.0.5 Hotfix for CVE-2022-22965
This hotfix upgrades the Spring Framework to version 5.2.20 in the sysdig-backend container.
Release 5.1.0 March 2022
Upgrade Process
Supported Upgrades From: 4.0.x, 5.0.x
For the full supportability matrix, see the Release
Notes on
Github. There you will also find important
Install instructions.
Sysdig Platform
Installer Improvements
Kubernetes versions 1.22 and 1.23 are now supported.
An optional cronjob for the falco-rules-installer, which runs once a month, can now be created through the Installer values file.
Users operating their own ingress controller, such as Rancher, are no longer need to manually create Ingress Objects Go HTTP APIs. Note that the Collector uses TCP and will need external configuration.
The Installer now has a pre-flight check to verify the kubectl and Kubernetes versions of the cluster with the context provided by the user.
Secure
API Docs
API documentation for Sysdig Secure are now enabled by default.
Bugs
Fixed an issue with Secure Events not displaying the correct number of events in the dashboard.
Fixed an issue that prevented Rapid Response being enabled with a Secure Team created with LDAP.
Fixed a network issue that would sometimes occur during an upgrade which would cause PostgreSQL to timeout.
Fixed an issue when the nats-streaming-init container failed to start due to permission problem when storageClassProvisioner is set to hostPath.
Fixed a Compliance Database Password issue during upgrades from on-prem 4.0.x to on-prem 5.0.x
Fixed an issue with the StatefulSet definition when upgrading from 4.0.x to 5.0.x on a Kubernetes cluster prior to 1.18.x
Release 4.0.7/5.0.4 Hotfix for CVE-2021-44228 in Apache’s log4j (3.6.4, 4.0.7, 5.0.4)
The patch relese upgrades all components that compose Sysdig’s Platform running Apache’s vulnerable Log4j library to 2.16.
Note on ElasticSearch: This is using Log4j v2.11.1. An additional JVM parameter has been added through the Installer in accordance with the recommendations from Elastic. In addition, the impacted class from the Log4j library has been removed completely. Security scanners may still list this as vulnerable but in this case it will be a false positive. Elastic currently does not offer a way to fully remove or upgrade this component.
Release 4.0.6/5.0.3 Hotfix for CVE-2021-44228 in Apache’s log4j (3.6.3, 4.0.6, 5.0.3)
Security researchers recently disclosed the vulnerability CVE-2021-44228 in Apache’s log4j, which is a common Java-based library used for logging purposes
Sysdig is using an alternative framework for logging called Logback. The logback framework isn’t vulnerable to this issue.
Sysdig components include a log4j library in our standard distribution that was vulnerable. This library is included for compatibility reasons only, is not used for primary logging, and our security team has determined we are not vulnerable based on our application architecture and existing mitigating controls.
We have released a patch version of our self hosted-software which upgrades the vulnerable version of log4j or adds additional mitigating controls suggested by vendors.
3.6.3
4.0.6
5.0.3
Please reach out to support or the customer success team for assistance with your upgrade.
For the full supportability matrix, see the Release
Notes on
Github. There you will also find important
Install instructions.
Fixes
Fixed Scheduled Reports not displaying last evaluation date field
Fixed an issue in 4.0.x Scheduled Reports when scanning Red Hat images, causing vulnerabilities missing related Red-Hat advisory (RHSA) to not be displayed
Version 5.0.0 does not yet support Kubernetes 1.22.
Upgrade Process
**Supported Upgrades From: **4.0.x
For the full supportability matrix, see the Release
Notes on
Github. There you will also find important
Install
and
Upgrade
instructions.
Sysdig Platform
Define S3 Bucket Path for Storing Captures
Sysdig Platform users can now define a custom path in the S3 bucket they
are using for storing captures. This is useful to those who want to
reuse a certain bucket used for other purposes or send captures from
different installations to the same S3 bucket. For more information, see
(On-Prem) Configure Custom S3
Endpoint.(On-Prem)
Configure Custom S3 Endpoint
Insecure connections: You now have the ability to skip the TLS
verification.
Custom headers: If your Webhook integrations require additional
headers or data you can append to the alert format by using a custom
header on the UI. This option is in addition to the existing API
facility to add custom headers programmatically.
S3-Compatible Storage for Capture Files
Configuring S3-compatible storage, such as Minio or IBM Cloud Object
Storage, for your Sysdig captures is now supported on Sysdig Monitor.
The capability can be turned on by configuring the system appropriately,
as given in (SaaS) Configure Custom S3 Storage
Endpoint.
The dark appearance, known as Dark Mode, is available in Sysdig
applications.
Sysdig can now automatically match your OS preferences. Available in
Sysdig platform on-premises, or in SaaS in the US East and rolling out
globally. For more information, see Configure Theme
Preference.
Customized Session Expiration
Session expiration is the amount of time a user can remain idle before
the session is automatically ended or expired. After the session
expires, the user must log in to the Sysdig application again.
Sysdig now gives you the ability to make a shorter or longer idle
session expiration for Sysdig applications. When a user browser is idle
for a certain period of time, they will get automatically logged out.
For more information, see Configure Customized Session
Expiration.
Sysdig Monitor
Workload Label
Sysdig Monitor now supports two new labels, kubernetes.workload.name
and kubernetes.workload.type which can be used for scoping Dashboards
and configuring Gropings.
Earlier, each type of object (deployment, replicaset, statefulset, etc.)
was unique, and in turn, you needed to use different types of Kubernetes
Dashboards and a different Grouping resulting in
n/a
, where distinct types of Kubernetes objects are listed.
Sysdig Monitor allows you to silence alert notifications for a given
scope for a predefined amount of time, and schedule silence in advance.
When silenced, the alert will still be triggered and posted on the
Events feed and in the graph overlays but will indicate it has been
silenced. The types of notification channels you can use are Email,
Slack, and Amazon SNS.
You will be notified 30 minutes before the start time and 30 minutes
before the end time of a silence window. You will also be able to easily
extend or end an active silence. To access the feature, navigate to
Alerts > Silence on the Monitor UI.
Sysdig Secure for cloud is
available with Cloud Risk Insights for AWS, Cloud Security Posture
Management based on Cloud Custodian for AWS and multi-cloud threat
detection for AWS using Falco.
What’s Included in this release:
Insights: a
powerful new visualization tool for threat detection, investigation,
and risk prioritization, to help identify compliance anomalies and
ongoing threats to your environment. With Insights, all findings
generated by Sysdig across both workload and cloud environments are
aggregated into a visual platform that streamlines threat detection
and forensic analysis.
Cloud Security Posture Management with AWS Benchmarks:The AWS CIS Benchmarks
assessment evaluates your AWS services against the benchmark
requirements and returns the results and remediation activities you
need to fix misconfigurations in your cloud environment.
We’ve included several UI improvements to provide additional details
such as: control descriptions, affected resources, failing assets,
and guided remediation steps, both manual and CLI-based.
Sysdig is now releasing a managed
version of the standalone
Falco Tuner.
Previously, you had to run the tuner in your local environment, print
suggestions, and manually update a rule with those suggestions. The new
feature runs in the background and automatically tunes noisy rules and
false positives. To streamline the creation of these exceptions, we’ve
created a new object within Falco called exceptions.
Note: To enable the tuner, Admin access rights to Sysdig Secure are
required.
Feature Enhancement: Falco Exceptions
Previously, exceptions were created using and not conditions inside a
Falco rule, e.g.
- rule: Write below binary dir
...
condition: >
bin_dir and evt.dir = < and open_write
and not package_mgmt_procs
and not exe_running_docker_save
and not python_running_get_pip
and not python_running_ms_oms
and not user_known_write_below_binary_dir_activities
....
However, this process can be unwieldy and can result in unintended
behavior. The new format, using exceptions, looks like this:
- rule: Write below binary dir
...
condition: bin_dir and evt.dir = < and open_write
....
exceptions:
- name: package_mgmt_procs
fields: proc.name
comps: in
values: package_mgmt_binaries # list of known binaries
...
We’ve added the ability to identify and add exceptions using the Policy
Tuner in the Insights module. Now you can receive policy tuning
recommendations directly within the Insights view, enhancing usability,
ease, and refinement of results.
We have reorganized the visual layout of the Scan Results summaries to
clearly distinguish policy evaluation from vulnerability matching and to
better summarize the information.
Improvements include:
Vulnerabilities and Policies are now two different sections in the
UI
Vulnerability match update time is displayed to further distinguish
from the Policy Evaluation time
Policy breakdown is collapsed by default to reduce cognitive load
Re-evaluate policies button is now located in the impacted section
only, as opposed to whole page
Apart from the vulnerability update time, the data remains unchanged
from previous versions
New and Improved Host OS and Container Scanning Tools
We at Sysdig are working hard to improve your security posture and
compliance experience. As part of this commitment we are implementing a
new framework to generate host benchmark results, introducing host
scanning, and making backend improvements to the image scanning
mechanism.
Installation Steps
The new features require a new component to be installed called the
Node Analyzer. We’ve provided an installation
script to automate the
installation or to upgrade an existing Node Image Analyzer daemonset, if
applicable.
Once you’ve installed or updated the components, the UI will
automatically show Host Scanning and new Benchmarks functionality
(Legacy Benchmarks can
still be accessed.)
Automatically scan images if they have not been scanned
Kubernetes Network Security: New Configuration and Improved User Experience
Sysdig’s Kubernetes Network Policy tool
has been updated to include
additional fine-tuning configurations and an improved user experience.
Additional Configuration Panel
Workload Labels: Depending on your workload labelling policy,
some labels may not be relevant for generating a KNP policy. Use the
additional config to include/exclude a particular set of labels per
cluster/namespace to declutter your UI and the resulting policy.
Unresolved IP Configuration: Now it is possible to label raw IPs
that are not mapping to your Kubernetes/OpenShift entities, i.e.
external cloud provider services, so these labels will be
automatically applied to the topology and ingress / egress tables.
Cluster CIDR configuration: If the CIDR configuration is not
automatically detected by the agent, you can now directly configure
internal subnets per cluster using the Sysdig interface.
Improved UX
Topology map: Additional information pop-up when hovering over a
network connection or a network node, such as server process,
source, destination, and more.
Unresolved IP filtering: In the ingress and egress tables, by
type or using free text search.
Additionally, Network
is now presented as a top-level item in the Sysdig Secure navigation.
Activity Audit Improved
The Activity Audit user
interface was enhanced as follows:
Activity Audit entry point moved under the Investigate menu
Trace feature, used for kube exec, is now also available for
parent commands
The filter selector is also available in-line, with no need to open
the detail view
Lateral Tree view removed and replaced with the Scope menu above, in
alignment with the Event panel
Alert Notification Channel for Microsoft Teams
Microsoft Teams is now available as an Alert Notification Channel in
Sysdig Secure for Runtime Policies. See also: Manage
Policies
Internal Scanning Date Improvements
Scanning policies have improved the reliability of the
Max days since creation and Max days since fix rule gate parameters.
The information is now included in the inline-scan JSON report and
available in the Jenkins plugin.
Reporting Improved with Multi-Select Option
Added the option to select multiple policies and multiple package types
as part of a scheduled scanning report.
For the full supportability matrix, see the Release
Notes
on Github. Other upgrade notes are maintained in the GitHub
upgrade
folder.
Installation Instructions
Full installation instructions for Kubernetes environments:
here.
Defect Fixes
Inline Scanning Fix for Sysdig Secure
Fixed an issue when scanning long Java manifest files that caused the
scan to fail.
LDAP Improvements for Sysdig Platform
Fixed an issue with the LDAP sync Job running out of shared memory. The
LDAP sync will no longer stop if it encounters an intermittent issue or
error, but will allow the sync to complete.
4.0.2 June 29, 2021
This release is a hot-fix only release for Sysdig Secure features.
Upgrade Process
Supported Upgrades From: 3.6.2, 4.0.0, 4.0.1.
For the full supportability matrix, see the Release
Notes on
Github.
Improvements
CSV Runtime Reports
The runtime labels that were described in a single CSV column (JSON
encoded) will now be represented using one column per label.
If the same vulnerability, same package, same image is found in
several runtime contexts, the CSV will separate each runtime context
in a separate row, instead of building a JSON array with several
objects nested.
Fixed Incorrect Fingerprinting Causing False Positives in Scanning
Fixed incorrect version detection for Apache Struts 2 packages leading
to false positives.
Fixed Metadata Retrieval Issue in Scanning
Fixed incorrect metadata retrieval for corner cases when imageIDs are
associated with several digests.
Improved Memory Usage
Reduced Redis memory consumed by scanning by optimizing the usage of the
scanning API cache.
Fixed Subscription Alert Entries
Fixed scanning alerts triggers for images discovered via the Node Image
Analyzer or Inline Scan container.
Readable Filenames for Scanning Reports
The scheduled scanning reports now generate report files named after the
report name i.e. my-daily-critical-vulns-2021-05-04.zip
Release 4.0.1 May 05, 2021
This release is a hotfix-only release for Sysdig Secure features.
Upgrade Process
Supported Upgrades From: 3.6.2
For the full supportability matrix, see the Release
Notes on
Github.
Improvements
Improved RHEL Vulnerability Matching
The RedHat OVAL source feed
interpretation and the
matching algorithm have been improved to handle special RedHat packages
versioning rules. This should effectively translate into fewer false
positives and more accurate fix versions for RH-based packages.
Defect Fixes
Security Fix
A SQL injection vulnerability discovered in 4.0.0 has been fixed in
4.0.1.
Scan Results
The vulnerability list on the UI shows a different number of
vulnerabilities as compared to the summary PDF report for the same
image. This issue has been fixed as part of Improved RHEL Vulnerability
Matching.
Secure Audit Reporting Errors
Secure Audit Reporting displayed intermittent errors for custom agent
versions. Fixed the agent version parsing to correctly assess feature
support.
Release 4.0.0 April 06, 2021
Upgrade Process
Supported Upgrades From: 3.6.2
For the full supportability matrix, see the Release
Notes
on Github.
Migrating MySQL to PostgreSQL
For consolidation and to meet higher performance requirements, upgrading
to v4.0.0 from v3.x.x involves migrating MySQL to the PostgreSQL
database. The migration process is seamless and no user intervention is
expected. For more information, see Migration
Documentation
on Github.
Review the Upgrade and other files within the version-specific GitHub folder for additional information.
Replicated
Not supported on 4.0.0
Deprecations
Deprecating “Scan Image” Reaction in Alerts
When setting up runtime alerts in previous versions, there was an option
to trigger “scan image” when an unscanned image was detected. This has
been deprecated in the UI in favor of the Node Image
Analyzer,
which is bundled by default with the Sysdig agent as an additional
container per node.
An issue was detected in an earlier version where large SAML metadata
could not be saved due to limits in the database field size. This issue
is now fixed and Sysdig now supports large SAML metadata.
Single Sign-On for Monitor and Secure
When a user logs in to Sysdig products successively, a confusing error
message related to SAML was displayed if:
If both Secure and Monitor have been configured with SSO.
The Create User on login feature has been turned on for both
products.
This issue is fixed with this release.
When a user created in one product logs in to another, and if the
Create user on login feature is turned on, no error message is
thrown. The user is added to the appropriate team in the product and can
log in to the other.
Sysdig Platform
Monitor UI Displays On-Prem License Information
The on-prem license information is now displayed on the Monitor UI.
Additionally, users will be warned of imminent license expiration on the
UI.
Changes to Auditing Sysdig Platform Activities
Due to the changes in the underlying database (PostgreSQL instead of
MySQL), the existing Sysdig auditing data will be dropped when
performing the upgrade from 3.x to 4.0 on-premise version. The audit
data is not migrated due to the potentially large size of the table,
which could prolong the upgrade process. The data remains available in
the MySQL database. If you require the data, do the following:
Before upgrading, dump the audit_events table from MySQL.
When the upgrade is completed, import the data back to the new
database if you desire.
Contact your Sysdig contact for details on how to perform this
operation.
Sysdig Monitor
Improved Alerts
The Alert interface has been improved to allow faster browsing and
easier management. For more information, see
Alerts.
Explore Workflow Enhancements
The Explore interface has been improved to allow faster
troubleshooting.
You are now launched directly into the drill-down view when you navigate
to Explore. You will still be able to group and navigate your
infrastructure by using the hierarchical scope tree.
The new Grouping editor helps you create and manage your
infrastructure groupings.
Dashboards now show null or missing data values as gaps instead of zero.
Optionally, missing data can be displayed as a dotted or solid line in
both Form-based and PromQL panels. StatsD metrics will continue to show
null values as zero unless overridden by the settings. For more
information, see Display Missing
Data.
Host Overview
To complement Sysdig Kubernetes Overviews, Hosts Overview has been
released.
Host Overview provides a unified view of the performance and health of physical hosts
in your infrastructure.Hosts
Overview
Sysdig Secure
Serverless Agent Preview Feature
The 1.0.x serverless agent is supported as a preview feature with Sysdig
Platform 4.0. Note that there is no guarantee of forward or backwards
compatibility with this preview release.
Sysdig Serverless Agent 1.0.0 for Fargate ECS
The “container-as-a-service” serverless environment calls for new
agent models, and Sysdig provides them. Whereas in ECS, users still
manage the underlying instances, with AWS Fargate the host is never
visible and users simply run their workloads. And while this model is
convenient, it can introduce risk as many people leave the containers
unattended, without monitoring security events within that can
exfiltrate secrets, compromise business data, impact performance, and
increase their AWS costs. In addition, it is not possible to install a
standard agent in an environment where you do not have access to a host.
For these reasons, Sysdig has introduced a new “serverless agent” model
that can be deployed in these container-based cloud environments. The
first implementation is for Fargate (ECS).
Sysdig will be rolling out security features on the serverless agent
over time. In v1.0.0, users will see:
Runtime Policies and Rules
Secure Events
To obtain secure event information and the associated Falco policies and
rules in the Sysdig Secure UI from a Fargate environment, users install
the serverless agent using a CloudFormation Template. Then log in to
Sysdig Secure and review the events in the UI.
Kubernetes-Native Network Security with Sysdig Secure (Beta)
A new feature has been added to Sysdig Secure for authoring and refining
Kubernetes network policies (KNPs) that:
Automatically extracts the connection information, by observing the
cluster networks and microservices communications
Offers a visual flow to fine-tune the Kubernetes network policies,
incorporating the user’s adjustments
Automatically generates the KNP YAML to be applied, without
requiring previous Kubernetes policy knowledge from the user.
As soon as the feature is enabled, the Sysdig agent starts collecting
and processing application communications, which are then enriched using
Kubernetes metadata and presented in two different ways:
Topology maps: a visual representation of the network flow
between the Kubernetes entities (Services, Deployments,
StatefulSets, DaemonSets, Jobs)
Ingress / Egress tables: for additional detail on each
inbound/outbound communication and policy tuning.
Once the user has finished editing the desired policy, Sysdig will
automatically compute the associated KNP YAML:
Enforcement is delegated to the Kubernetes control plane, favoring
policy-as-code and avoiding direct tampering with cluster
communications
Allow-only approach ensures that any communication which is not
explicitly allowed by the policy will be forbidden
Prerequisites
Sysdig agent version 10.7+
Supported Orchestrator Distributions and CNI Plugins:
Vanilla Kubernetes (kops, kube-admin) using Calico
OpenShift 4.x using OVS
Google GKE using Calico
Amazon EKS using Calico
Rancher Kubernetes using Calico
Please contact us to enable this feature for your Sysdig Secure
accounts.
With the addition of CronJob support, communication is aggregated to the
CronJob (scheduler) level, rather than the Job. Therefore, when
administrators review the activity in the Network Security Policy menu,
they will see the higher-level CronJobs listed, and not an excess number
of individual Job entries.
This update also adds support for
Weave and
Cilium CNIs on top of
Calico
support.
New Product: Rapid Response
Rapid Response is an Endpoint Detection and Response (EDR) solution
built for cloud-native workloads, which gives security engineers the
ability to respond to incidents directly via a remote shell. The shell
uses the underlying host tooling already installed, such as kubectl,
Docker commands, cloud CLIs, etc. Users can also mount their own scripts
to use any familiar tooling.
Rapid Response requires a component installed on the host machine. This
component provides end-to-end encrypted communication using a passphrase
only your team knows. The Rapid Response feature is disabled by default
and can only be accessed to teams that have the feature enabled. Admins
can see all user activity, including access to audit logs, and can
initiate a rapid response session. Advanced users can view only their
own user activity, including their audit logs, and can initiate a rapid
response session.
The Image Scanning Reports feature has been thoroughly updated and has
moved from a synchronous model to an asynchronous mode, in which you
schedule the reports you need and then receive them through your normal
notification channels (email, Slack, webhook.). The new version also
includes:
A preview function to check report structure in the UI
A more advanced query builder
Extended set of data columns (i.e. CVSS base score and vector) and
extended set of available filters (i.e. package type)
Reporting v3 supports two different types or reports:
Vulnerability report: Containing vulnerability, package and
image data
I.e. Vulnerabilities in my runtime with Severity ≥ High, a Fix
available and not included in a vuln exception list.
Policy report: Containing scanning policies and evaluated images
data
I.e. Images in my internal registry failing the “NIST” scanning
policy.
You need to enable this feature from the Sysdig Labs setting on the
User
Profile
page.
Kubernetes’ admission controllers help you define and customize which
requests are allowed on your cluster. An admission controller intercepts
and processes requests to the Kubernetes API prior to persistence of the
object, but after the request is authenticated and authorized.
Sysdig’s Admission Controller (UI-based) builds upon Kubernetes and
enhances the capacity of the image scanner to check images for Common
Vulnerabilities and Exposures (CVEs), misconfigurations, outdated
images, etc., elevating the scan policies from detection to actual
prevention. Container images that do not fulfill the configured
admission policies will be rejected from the cluster before being
assigned to a node and allowed to run.
Granular admission policies: Defining a global policy per cluster,
but also at the level of particular namespaces or image paths (i.e.
registries) Registry and repository whitelist
Only allow images that pass the scanning evaluation criteria
Only allow images that have been evaluated recently
Only allow images that have been scanned before creation is
requested to Kubernetes
A new cloud compliance standard has been added to the Sysdig compliance
feature - CIS AWS Benchmark. This assessment is based on an
open-source engine - Cloud Custodian - and is an initial release of
Sysdig Cloud Security Posture Management (CSPM) engine. This first
Sysdig cloud compliance standard will be followed by additional security
compliance and regulatory standards for GCP, IBM Cloud and Azure.
The CIS AWS Benchmarks assessment evaluates your AWS services against
the benchmark requirements and returns the results and remediation
activities you need to fix misconfigurations in your cloud environment.
We’ve also included several UI improvements to provide additional
details such as: control descriptions, affected resources, failing
assets, and guided remediation steps, both manual and CLI-based when
available.
The JSON format for the runtime policy events has been upgraded to
include full scope information, rule labels, and a single-line
representation for the event field’s keys and values.
To preserve backwards compatibility with existing integrations, the
former JSON format is still available (and used by default on
migration).
From the Event Forwarder page, under “Data to Send,” the old JSON format
is labeled “Policy Events (Legacy)” and the new one as “Runtime Policy
Events.”
We have added introductory splash screens throughout the product to help
you get started when using a feature for the first time.
UI Improvement on Rules Library and Rule Details
Usability improvement so you can see in which policies a rule is used,
from both the Rules Library list and the Rule Detail view. See Manage
Rules for details.
The “Commands Audit” feature was deprecated in favor of Activity
Audit in November 2019.
This feature will be completely removed from the On-prem
distribution in version 4.1.
Sysdig agent version
9.5.0+,
released in January 2020, is required by the Activity Audit feature.
The “Policy Events” feature was deprecated in favor of the new
Events feed in
June 2020. This feature will be completely removed from the On-prem
distribution in version 4.1.
Identify Windows container image vulnerabilities from:
Windows OS CVEs
Windows or Linux hosts
Reports in JSON and PDF
Policy support
Severity
Fix available
Days since fixed
Malware Detection during Inline Image Analysis
As part of the inline scanner version 2.3.1
release,
malware scanning was added as a configurable detection that can be
performed during inline analysis.
The default behavior if this feature is enabled and malware is found is
to consider the scanning failed, report malware details, and abort
analysis:
In some cases, including alerts with very large scopes and some
others, email notifications were not sent due to a bug in the email
renderer. This issue has been fixed.
Fixed Kubernetes metadata display delay
In 3.6.0 and 3.6.1 releases, upon connecting an agent, it would take
1h for Kubernetes metadata to appear. With this bug fixed, the
metadata is displayed a couple of minutes after connecting the
agent.
Fixed dashboard display error when switching teams
When the user switched teams, the dashboard menu was not displayed
and required the user to reload the application. This has been
fixed.
Improvements to the security setup of our Intercom integrations
We have improved the security of the Sysdig Intercom integration, as
in some cases, the conversations could leak between different users.
Fix to Activity Audit Janitor
Fixed an Activity Audit Janitor error that stopped the AA clean-up
process when a particular set of Sysdig Secure features were not
enabled.
Improvements
Increased Decimal Precision from 4 to 6
With this release, we increased the decimal precision from 4 to 6
decimal places. This feature is mostly useful for customers using
Prometheus metrics, as by convention, the metrics for time are given in
seconds in Prometheus exporters, which does not work well for low
numbers (for example - latencies in microseconds).
New Runtime Policy Events JSON Format
The JSON format for the runtime policy events has been upgraded to
include full scope information, rule labels, and a single-line
representation for the event field’s keys and values.
To preserve backwards compatibility with existing integrations, the
former JSON format is still available (and used by default on
migration).
From the Event Forwarder page, under “Data to Send,” the old JSON format
is labeled “Policy Events (Legacy)” and the new one as “Runtime Policy
Events.”
Sysdig Platform v 3.5.3 has been tested and qualified against the same
components as in v.
3.5.1.
Supported Upgrade from: 3.5.1, 3.2.x, 3.0
Bug Fixes
Sysdig Platform
Fixed email notifications error
In some cases, including alerts with very large scopes and some
others, email notifications were not sent due to a bug in the email
renderer. This issue has been fixed.
Improvements to the security setup of our Intercom integrations
We have improved the security of the Sysdig Intercom integration, as
in some cases, the conversations could leak between different users.
Sysdig Secure
Events Forwarder improvement
Fixed a crash condition in the Events Forwarder service stemming
from a microservices connectivity issue.
Release 3.6.1 November 23, 2020
Oversight Services Now Offered for All Installs and Upgrades
As part of our continued focus on our customers, we are now offering
oversight services for all on-premise installs and upgrades. Your
Technical Account Manager (TAM), in conjunction with our support
organization and Professional Services [where applicable], will work
with you to:
Assess your environment to ensure it is configured correctly
Review your infrastructure to validate the appropriate storage
capacities are available
Review and provide recommendations for backing up your Sysdig data
Work with you to ensure our teams are ready to assist you during the
install and upgrade process
Provide the software for the install
Be available during the process to ensure a successful deployment
You can always review the process in the documentation on
GitHub (v. 3.6.0+) or
the standard docs site
(for older versions).
If you are a new customer looking to explore Sysdig, please head over
here to sign up for a trial on
our SaaS Platform. Alternatively, you can contact us
here.
Upgrade Process
Supportability Matrix
Sysdig Platform has been tested and qualified against the following.
* Note that as of this release, there are no upgrades for Replicated
installations.
Review the Upgrade and other files within the version-specific GitHub folder for additional information.
Replicated
No Replicated release from 3.6.0 forward.
Sysdig Secure
The following improvements were introduced in release 3.6.1:
Node Image Analyzer: Scan “Repo-less” Images
Added support to scan images that lack a Repo tag, such as OpenShift 4.x
distribution images.
Audit Tap Forwarding: Fixed Splunk Event Timestamp Metadata
The format of the “time” field included in the Splunk event metadata for
forwarded Audit Tap events is now increased to millisecond granularity.
Fixed False Positives on Java Libraries Related to log4j
Fixed an issue that resulted in log4j-jboss-logmanager and
log4j-1.2-api being incorrectly detected as log4j, possibly
generating vulnerability false positives.
This component is independent of the Sysdig Platform version you are
running–it can be used with Sysdig On-Prem version 3.6.1 and with
earlier versions.
Inline Scanner 2.1 includes the following enhancements:
NEW
Added ability to analyze scratch-based images
FIXES
Fixed a bug retrieving the PDF output for previously- scanned images
Addressed several vulnerabilities found in the inline scanner
container
Oversight Services Now Offered for All Installs and Upgrades
As part of our continued focus on our customers, we are now offering
oversight services for all on-premise installs and upgrades. Your
Technical Account Manager (TAM), in conjunction with our support
organization and Professional Services [where applicable], will work
with you to:
Assess your environment to ensure it is configured correctly
Review your infrastructure to validate the appropriate storage
capacities are available
Review and provide recommendations for backing up your Sysdig data
Work with you to ensure our teams are ready to assist you during the
install and upgrade process
Provide the software for the install
Be available during the process to ensure a successful deployment
You can always review the process in the documentation on
GitHub (v. 3.6.0+) or
the standard docs site
(for older versions).
If you are a new customer looking to explore Sysdig, please head over
here to sign up for a trial on
our SaaS Platform. Alternatively, you can contact us
here.
Upgrade Process
Supportability Matrix
Sysdig Platform has been tested and qualified against the following.
* Note that as of this release, there are no upgrades for Replicated
installations.
With this release, you can define a period of interactive-session
expiration, so that when a user is idle for a defined period of time,
the session terminates. This helps enterprises with strict security and
compliance requirements comply with relevant security controls, such as
NIST or PCI-DSS
8.1.8
.
Currently, this feature is available for on-premises only and is
configured per installation.
With this release, we have enhanced the performance for listing and
search for users on both Settings>Users and Settings>Teams
pages. We now support tens of thousands of users comfortably.
LDAP: Search for users by both username and email address
For enterprises using LDAP, this release enables search on both
username and user email address in the Settings > Users and
Settings > Teams pages. Users are listed by name but can be searched
by email as well.
LDAP: Default team role respected
This fix ensures that when LDAP users are created upon login, the
default user role for the team is respected.
Inline Scanner 2.0
A new version
of the Sysdig inline scanner script has been released.
Major improvements:
The inline analysis container doesn’t need to spawn any additional
containers
This removes the requirement for the Docker client,
docker-in-docker, etc.
This enables usage in environments where docker-in-docker is not
feasible or hard to instrument (e.g., Tekton).
Additional analysis workflows and formats:
Added support to analyze a docker archive
A .tar.gz file containing the image, i.e. the output from a
“docker save”
Sysdig Inline scanner can be used stand-alone or as a step inside a
CI/CD pipeline (Jenkins, Tekton, CircleCI, etc). In the upcoming weeks,
we will update the different integrations to provide out-of-the-box
support for the 2.0 version.
Sysdig Secure
Regulatory Compliance Control Validation & PCI Checks
A new feature has been added to Sysdig Secure for checking controls from
various compliance standards. For the first release, we provide checks
against specific controls in PCI 3.2. Future releases will include
SOC2, NIST-800-53, and more. See also:
Compliance in Sysdig
documentation.
Compliance Validator and Reports
The validator checks many Sysdig Secure features, including: image
scanning policies, Falco runtime policies and rules, scheduled benchmark
testing, Admission Controller, Network Security Policies, Node Image
Analyzer, and more. Over time we will add new compliance coverage.
Disclaimer: Sysdig cannot check all controls within a framework, such
as those related to physical security.
This feature is a beta release. A Sysdig Secure admin must enable it
from the Sysdig Labs interface under Settings.
PCI Control Details
The PCI Quick Reference
describes
the full range of controls required to pass a PCI 3.2 audit. In this
release, Sysdig Secure will check the following subset:
In new images scanned, RHSA advisories will be replaced with CVE
advisories.
Benchmarks support for Kubernetes Benchmark 1.6
Kubernetes Bench upgraded to version 1.6
Using the Kubernetes benchmark, we now provide customer-selected
benchmark checks for GKE and EKS (rather than just the Kubernetes
default).
Vulnerability Exceptions Handling Enhanced
The Vulnerability Exceptions feature in Sysdig Secure has been
redesigned and enhanced.
It now offers:
Additional vulnerability and feed context
Precise mapping between images and their associated exceptions
A better exception management lifecycle
Multiple vulnerability lists, which can be flexibly assigned to
different image sets (or just a particular image), using the
scanning policy assignments
Additional information displayed to improve team awareness and
security context
Vulnerability description
User-defined notes
Vulnerability feed info, with severities and links as provided
per feed
Configurable expiration dates:
An exception is automatically disabled when the expiration date
is met
Day resolution, all times relative to 0:00 UTC
Enhanced workflow integration with the “Scan results” page for an
individual image, with the ability to quickly append a flagged
vulnerability to a list.
Migration: The exception and evaluation behavior in the current
environment will be maintained after the feature upgrade. In particular:
Pre-existing vulnerability exceptions will be migrated to the
“Default exceptions list”
The “Default exceptions list” will be assigned to every pre-existing
policy assignment
All the pre-existing vulnerability exceptions expiration date will
be set to “Never.”
Authentication methods: Basic authentication, Bearer Token, and
Signature Header
Custom headers defined by the user to accommodate any additional
parameter required on the receiving end
Image Exclusion on Policy Events
Users often want to tune policy
events. We’ve added a
button on the event detail that will add an exclusion to a specific
container.image.repo for the policy that triggered the event. Once
that exclusion is applied to the scope, policies will no longer fire for
that container.image.repo.
Captures Filter on the Policies Page
Policies can now be
filtered to display if a capture is associated with an active or
inactive policy.
Quick Menu to Captures from Runtime Events
For runtime policy events that have an associated capture, we now offer
a contextual menu for performing quick actions over the event capture,
rather than a simple link to the Captures interface. You can:
Additionally, if the event is scoped to a particular container, Sysdig
Inspect will automatically filter the displayed information to the scope
of that Container ID.
Image Scan Results Page Redesigned to Improve Load Times & User Experience
The user interface is cleaned up, reorganized, and provides the
following functional improvements:
Load times are significantly decreased because the last known
evaluation for the image is automatically fetched
View the latest evaluation time directly in the scan summary
Evaluated at
Use the new Re-evaluate button to fetch current data if
desired
View the image origin/reporting mechanism in the new “Added By”
field.
Possible values are: Sysdig Secure UI, Node Image Analyzer,
API, Sysdig Inline Scanner, or Scanning alert.
Copy the Image Digest and Image ID to the clipboard using a quick
pop-up panel.
You can now browse and find historic events easily by using time
navigation.
Zooming Out Dashboards
You now have the ability to zoom out Dashboards. This feature doubles
the selected timeframe for a better context surrounding a problem when
troubleshooting an incident.
Release 3.5.1 August 24, 2020
NOTE: Version 3.5.1 includes a fix for vulnerabilities that were
detected in version 3.5.0. It is recommended to skip version 3.5.0 and
install version 3.5.1 instead. As of this release, all on-premises
installs and upgrades include oversight services from Sysdig support.
Oversight Services Now Offered for All Installs and Upgrades
As part of our continued focus on our customers, we are now offering
oversight services for all on-premise installs and upgrades. Your
Technical Account Manager (TAM), in conjunction with our support
organization and Professional Services [where applicable], will work
with you to:
Assess your environment to ensure it is configured correctly
Review your infrastructure to validate the appropriate storage
capacities are available
Review and provide recommendations for backing up your Sysdig data
Work with you to ensure our teams are ready to assist you during the
install and upgrade process
Provide the software for the install
Be available during the process to ensure a successful deployment
You can always review the process in the documentation on
GitHub (v. 3.6.0+) or
the standard docs site
(for older versions).
If you are a new customer looking to explore Sysdig, please head over
here to sign up for a trial on
our SaaS Platform. Alternatively, you can contact us
here.
Upgrade Process
Sysdig Platform has been tested and qualified against the following:
Supported Upgrade From
3.5.0, 3.2.x, 3.0
Platform
Version
Vanilla Kubernetes
1.13.4, 14.10, 1.15.12, 1.16.13, 1.17.9, 1.18.6
OpenShift
4.4 –> 1.17.1+1aa1c48
GKE
1.14.10-gke.36
EKS
v1.17.7-eks-bffbac
Rancher
v2.3.3
IBM
Unqualified
PKS
Unqualified
Agent
Version
sysdig/agent
10.2.0
Components
Replicated TBD
Kubernetes with Statefulsets
Redis
n/a
4.0.12
MySQL
n/a
5.6.44
MySQL HA*
n/a
8.0.16 (see note)
ElasticSearch
n/a
5.6.16
Cassandra
n/a
release_version: 2.1.21
cql_version: 3.2.1
RDS
n/a
8.0.16
Postgres (image scanning)*
n/a
12.3 (see note)
Anchore (image scanning)
n/a
0.6.1
NATS Exporter
n/a
0.6.0.1
NATS Streaming
n/a
0.17.0.1
HA-Proxy
n/a
1.9.15
*MySQL8: You can use MySQL8 for non-HA setups using the flag
useMySQL8: true
* Postgres: Upgrading to 3.5.0 will also involve an automatic
Postgres version upgrade from 10.6.x to 12.x. Depending on your database
size, the upgrade could take some time. See Postgres Version Update
v10.x to
12.x
for details.
We no longer point to ancho.re for feeds update but
to[ https://api.sysdigcloud.com/api/scanning-feeds/v1/feeds](http:// https://api.sysdigcloud.com/api/scanning-feeds/v1/feeds).
This could require a change to your firewall rules, as an exception to
your proxy for ancho.re would impact the feeds update.
Sysdig Secure
Note that the Secure Overview is not available with Replicated
installations.
New Sysdig Secure Overview Page
The Sysdig Secure Overview provides an at-a-glance view of the critical
areas of your security posture.
Scoping
Panels can be scoped by Cluster or Namespace. The scope will update
all panels that are displaying run-time data and the corresponding
drill-down views.
Panels
Build Time - Images Scanned: Image scan results for all static
image scans
The Get Started page provides the key steps to ensure users are getting
the most value out of Sysdig Secure. We’ll update this page with new
steps as we add new features to Sysdig Secure.
The Get Started page also serves as a linking page for:
Documentation
Release Notes
The Sysdig Blog
Self-Paced Training
Support
Users can access the page at any time by clicking the rocketship in the
side menu.
It’s useful to understand the last time the feeds were updated,
especially in self-hosted environments. The Feeds Status page shows the
different vulnerability feeds we integrate with, their feed group (often
the distro version), the time of the last sync, and how many CVE records
are present in the feed group.
The Events feed in Sysdig Secure (formerly called Policy Events) has
been redesigned, both visually and functionally.
Apart from the styling and user experience improvements, these are the
major new features and use cases
Advanced Filtering
We are deprecating the grouping/clustering of events present in the old
version in favor of a much more powerful set of filtering capabilities:
Severity filters: Presented as quick buttons at the top,
supporting multi-select
Attribute filters: Provide a simplified syntax to filter events
by the attributes they contain. For example
ruleType="Falco - Syscall" or image.repo!="sysdig/agent"
Open the event details side panel to find quick filtering
widgets to include or exclude the attribute values associated
with the displayed event
Event type selector: Supports runtime scanning alerts on top of
policy runtime events (see section below), with an easy
multi-selector in the UI.
Free text search: Allows you to search the event titles and
scope label values. I.e. Terminal shell in or my-k8s-cluster.
New scope selector: Allows for additional selector logic (in,
not in, contains, startswith, etc), improving the scoping
flexibility over earlier versions. This scope selector also provides
scope variables, allowing you to quickly switch between, for
example, Kubernetes namespaces without having to edit the panel
scope.
All these filters can be combined additively to further refine your
search.
The backend architecture, filtering, and UX have been designed to
accommodate additional types of security events that will be pushed to
the Event Feed in the future, upgrading the interface from a
policy-runtime-centric experience to a full security center control
panel.
Additional Event Details
Policy runtime events: These now display the rule that was fired
together with the rule labels. You can use the quick filters mentioned
above to further refine the search.
Richer scope: Every security event now displays all the scope labels
retrieved for the event, not just those configured in the scope
selector.
Events in the old and new format will be stored separately:
No event or event data will be lost during the transition
Events that were registered before the new feed is deployed can be
browsed using the old feed interface, which is available from the
burger menu in the top-right corner
Events that happen after the new feed is deployed will appear in the
new event feed
Eventually, all events within the retention period will be present
in the new interface, at which point the version switcher will
disappear
Team, Role, and Channel Updates
A variety of enhancements have been added to the team, role, and
notification channel options.
Now a new role, Service Manager, is also available in Secure. It has
the same permissions as the Standard User, plus the ability to invite
existing users to the team and manage the notifications channels
assigned to the team. See Team-Based Roles and
Privileges
Configurable Default Team Role
You can now define the default user role to apply when a new member is
added to the team. The Admin can change this default on a per-team
basis. See also: Create a
Team.
RBAC and Team Assignment for Notification Channels
Previously, notification channels in Sysdig Secure and Monitor were
treated as global entities, visible and editable for most users of the
platform regardless of team configurations.
We are enhancing the management and RBAC controls in the following ways:
Notification channels can now be “global” or limited to a particular
team
Global channels can be managed by admins and can be viewed/used by
other roles, while team-limited channels are available only to team
members
Team Manager , Advanced User, and Service Manager (Secure)
roles can create/update/delete team-scoped notification channels,
they can also read and use the global ones
Standard and View Only roles can read team-limited and global
notification channels
Admins will be able to create global notification channels and
migrate channels from “global” to “team-limited”, and also from one
team to another.
The side menu is reorganized into Analyze and Configure sections
Analyze: Different areas of scanning that allow users to
view scan results
Configure: The areas of scanning that involve the setup of
the application
Whitelist terminology with CVEs has been removed.
“CVE whitelist” is now CVE Exceptions.
CLI-Based Admission Controller for Image Scanning
An additional tool for evaluating and admitting images is now available.
Sysdig Admission Controller
Sysdig’s Admission Controller (UI-based) combines the Sysdig Secure
image scanner with a policy language to evaluate scan results and the
admission context, providing great flexibility in the admission
decision. It also provides the first line of defense against image-based
security threats.
By using Kubernetes API extensions to perform image scanning and other
security checks on admission, we cover a major threat-prevention and
hardening use case: “Only the images that are explicitly approved will
be allowed to run on my cluster”.
The admission decision relies not only on the image name and tag but
also on additional context from the admission review, including
namespace, pod metadata, etc.
Features
Registry and repository whitelist / blacklist
Global and per-namespace admission configuration
Configurable pre-scan and post-scan behavior, i.e.:
Accept only the images that pass the
scan (default)
Directly reject non-whitelisted registries / repos, without
scanning
Accept the image even if it doesn’t pass the scan
Do not accept any image that hasn’t been scanned already
Pod mutation: image tag is replaced by digest to prevent TOCTOU
(Time of Check, Time of Use) issue if the tag is updated between the
scan and the pod scheduling
Added Automatic Image Scanning using Node Analyzer
The (node) image analyzer (NIA) provides the capability to scan images
as soon as they start running on hosts where the analyzer is installed.
It is typically installed alongside the Sysdig agent
container.
This component was introduced to reduce dependencies on analyzing images
within the Sysdig backend (SaaS or On-prem). Some advantages include:
Sharing credentials with the Sysdig backend in order to pull images
is not required
Sharing the image content and potentially code with the Sysdig
backend is not required; only metadata will be sent out
Opening a network route to allow the Sysdig backend to reach the
user’s registries is not required
If you have run the single line agent install with the –image-analyzer
flag, then this component is already running in your infrastructure.
The feature is available for Kubernetes environments.
Two new scanning integrations are available for CI/CD pipelines. Sysdig
provides:
A reference implementation with Tekton Pipelines (prototype)
A fully supported integration with Amazon Elastic Container Registry
(ECR) for triggering auto-scans from the registry
Integrating Secure Image Scanning with Tekton Pipelines
Tekton Pipelines
allow you to implement CI/CD workflows using a highly modular,
cloud-native approach that:
Uses containers as the building blocks for individual tasks
Runs directly on Kubernetes/OpenShift without requiring a dedicated
infrastructure
Uses tasks that are purely declarative and described using their own
CRD, making them easily composable and reusable
Sysdig’s reference implementation details the prototype task to invoke
Sysdig Secure image scanning as a pluggable step in your CI/CD pipeline
with just a YAML file:
Leveraging Tekton integration with the orchestration layer, you can
retrieve the image scanning policy evaluation and state (pass/fail)
directly from the logs of the task pod.
Sysdig image scanner integration is deployed as a CloudFormation
template that listens to ECR registry events and uses AWS resources to
streamline the image scanning process.
ECR itself will trigger the scan, no need for your CI/CD pipelines
to actively pull from the registry
Deployed in a few clicks, you just provide basic configuration
parameters such as the Sysdig API token or the Sysdig backend URL
No need to configure registry scanning credentials on the Sysdig
Secure side
This integration offers two different operation modes
Inline scanning:
Scanning will be performed inside an AWS CodeBuild pipeline
allocating ephemeral resources
No need to configure any registry credentials for Sysdig Secure
No need to expose your ECR registry to the Sysdig Secure backend
Sysdig Secure will not retrieve the image contents, only the
metadata that is required to perform the policy evaluation
Backend scanning:
Sysdig Secure will retrieve the full image contents in order to
perform the scan
Your ECR registry must be reachable by the Sysdig Secure backend
Registry credentials are required, but they are pushed automatically
by a lambda function, no need for manual configuration
Updated Inline Scan Script
Added header values for import API for better supportability.
Upgraded to Anchore engine v0.6.1.
Use docker:dind instead of ubuntu for the base image. This
reduces the image size and speeds up downloading.
Inline Scanning Reporting Improvements and Documentation
This script from
SysdigLabs is useful for performing image analysis on locally built
container images and posts. The only dependency for this script is
access to docker-engine, Sysdig Secure endpoint (with the API token) and
network connectivity to post image analysis results.
Here are examples of using the inline scanner in different pipelines:
We’ve added VulnDB as an
additional 3rd-party vulnerability source to improve Sysdig’s coverage
in non-OS package vulnerabilities.
In addition, a new page is available for each VULNDB-linked advisory. It
lists the CVEs and details about the Common Vulnerability Scoring System
(CVSS) scores and external references.
Sysdig Agents can run the Independent Linux benchmark against the
underlying host where the agent is installed. The Linux benchmark can be
scheduled to run at a chosen interval in your environment and emits
results and metrics about the status of the tests.
Openshift Hardening Guide
The Openshift hardening guide implements configuration checks run by the
agent against Openshift environments.
Note: This is supported for 3.x versions of Openshift. When Openshift
releases a hardening guide for 4.x versions, we will update the
configuration checks.
Captures can be Routed to Specific Storage Locations
As a user, you may have different S3 buckets where you’d like to store
Sysdig captures, based on the environment where the policy event was
triggered. New options are available for deciding what storage option
you’d like to use for each policy event.
Sysdig Monitor
New Dashboards is GA
Sysdig Monitor offers a new version of dashboards. Its improved editing
experience provides you with more flexibility and the new set of
functionalities offers additional ways to visualize and consume your
Sysdig data.
Features and Enhancements
Improved User Experience
The New Dashboard offers a more fluid, natural dashboard building
experience. For more information, see About the Dashboard
UI.
Dashboard Sharing
You can now share your dashboard with members within your Sysdig team or
share it across teams with fine-grained access controls. Define who
should be able to see the dashboards and what level of access they
should be granted: view only or collaborator with edit privileges. For
more information, see Sharing New
Dashboards
Time Series Name Templating
Customize the time series names on the legend on the panel editor by
using the labels associated with Prometheus metrics and segments to gain
context faster. For more information, see Create a New
Panel.
Multi-Metric, Multi-Segmentation Options
Configure multiple queries within a single panel, and configure each
query with multiple segmentation and scoping options. Individual queries
can be customized to render as a line or stacked area. For more
information, see Create a New
Panel.
Event Overlay
Contextualize metrics and understand the “why” faster with a unified
view of both metrics and events. Configure event overlay to display
events from Kubernetes environments as well as alert events, and any
other events ingested using Sysdig’s open REST API. For more
information, see Display Dashboard Specific
Events.
Dashboard Templates
You can quickly view your infrastructure through the lens of one of
Sysdig’s curated dashboards, or use it as a base to start building your
own. You can find dashboard templates for managing Kubernetes capacity
and health, hosts and server performance, applications and services
telemetry, and the security posture of your infrastructure with data fed
from Sysdig Secure. See Dashboard
Templates to learn more.
Mapping Values to Text
Instantly understand what’s going on by mapping number panel values to
text. If you have a metric that returns 1 for up, and 0 for down, map
those values to “UP” and “DOWN” respectively. By defining thresholds and
mapping to text, you don’t need to be concerned about the values. This
is critically valuable when dashboards are shared between team members.
For more information, see
Text.
Granular Axes and Legend Controls
You have more flexibility when customizing the axes, as well as better
support for time series with long names. You can now configure the
legend by toggling its visibility and moving it to the bottom of the
panel. See About the Dashboard
UI.
Major Changes
Significant changes have been introduced to enhance the usability of the
existing functionalities. Review the changes before you explore the
functionalities.
Topology Maps
Topology maps are no longer available in Dashboard. Access Topology
maps through Explore, as you explore your microservices and
Kubernetes applications.
Dashboard Wizard
My Dashboards are no longer accessible in Explore. Additionally,
Dashboard Wizard has been removed. Instead, the concept of
Templates has been introduced in Dashboards to help you get
started with a library of templates addressing key use cases.
Histogram and Summary Metric Type
Histogram and summary metrics are no longer supported in the Histogram
panel type. You can continue to use them within Explore.
APIs and Integrations
API endpoints for the legacy dashboards (v2) will soon be deprecated. If
you are directly integrating into the API, please contact Sysdig for
guidance. Additionally, our Python SDK and CLI have been updated to
support the new dashboards APIs.
Sysdig Monitor Rebranding
The Monitor app has been refreshed with new logos and icons. The
navigation pane has been re-organized. The Explore tab is moved
below Dashboards.
The New Get Started Page
The Get Started page provides the key steps to ensure that you are
getting the most value out of Sysdig Monitor. We’ll update this page
with new steps as we add new features to Sysdig Monitor.
The Get Started page also serves as a linking page for:
Documentation
Release Notes
The Sysdig Blog
Self-Paced Training
Support
You can access the page at any time by clicking the rocketship icon in
the left navigation bar. See Getting Started with Sysdig
Monitor.Getting
Started with Sysdig Monitor
RBAC and Team Assignment for Notification Channels
Previously, notification channels in Sysdig Secure and Monitor were
treated as global entities, visible and editable for most users of the
platform regardless of team configurations.
We are enhancing the management and RBAC controls in the following ways:
Notification channels can now be “global” or limited to a particular
team
Global channels can be managed by admins and can be viewed/used by
other roles, while team-limited channels are available only to team
members
Team Manager , Advanced User, and Service Manager (Secure)
roles can create/update/delete team-scoped notification channels,
they can also read and use the global ones
Standard and View Only roles can read team-limited and global
notification channels
Admins will be able to create global notification channels and
migrate channels from “global” to “team-limited”, and also from one
team to another.
Sysdig Monitor can now utilize the Amazon Web Service (AWS) AssumeRole
functionality and discover cloud assets, grab CloudWatch metrics from
your AWS account, and use custom S3 bucket for storing captures. Upon
integrating with an AWS role, you can delegate access to AWS resources
that are not associated with your Sysdig AWS account.
Role delegation is an alternative to the existing integration method
using the access keys. This method is considered secure as sharing
developer access keys with third-parties is not recommended by Amazon.
You can now define the default user role to apply when a new member is
added to the team. The Admin can change this default on a per-team
basis. See also: Create a
Team.
Default Dashboards for Istio 1.5
Default dashboards (Overview and Services dashboards) are now available
for Istio v1.5 in addition to the existing ones for Istio v1.0.
Release 3.2.2, June 11, 2020
This is a hotfix release for Benchmarks. See Defect
Fixes
for details.
Upgrade Process
Sysdig Platform has been tested and qualified against the following:
Problem: On a cluster running Kubernetes v1.12 or later versions with
Sysdig agent v9.7.0 or later versions, the CIS Kubernetes benchmark
result could not be interpreted, resulting in an infinite spinner
displayed in the UI.
Resolution: Sysdig agents v9.7.0 or later versions can now be used with
Kubernetes v1.12 or later versions. The CIS Kubernetes versions included
are 1.3, 1.4, and 1.5.
Sysdig Monitor
This release contains no new features or defect fixes.
Sysdig Platform
This release contains no new features or defect fixes.
Release 3.2.1-Onprem (Replicated Only), March 23, 2020
This is a hotfix release that enforces a minimum Replicated Console
version to include a necessary security patch. This release contains no
new Sysdig functionality and is not a required upgrade.
Use of release 3.2.1-onprem requires first upgrading your Replicated
Console to version 2.42.4 or newer.
Release 3.2.0, March 04, 2020
Upgrade Process
Sysdig Platform has been tested and qualified against the following:
Use this feature to set limits on how long image scan metadata is
stored, either by tags or days. This removes stale data and helps keep
scan results easy to read.
The new role-based access control (RBAC) model available in Sysdig
Secure allows you to define the access privileges granted to each user
in a Sysdig Secure team.
Besides the Admin role, which has full access and belongs to every
team, there are four roles that can be assigned when adding a user to a
team. (Note that the role names are the same in Monitor and Secure, but
the privileges differ slightly. Users must be assigned Monitor team
roles and Secure team roles separately.)
View Only: Read access to every Secure feature within the team
scope. A View Only user cannot modify runtime policies, image
scanning policies, or any other content.
Standard User: Can push container images to the scanning queue
and view the image scanning reports. Standard Users can also display
the runtime security events within the team scope. They cannot
access the Benchmarks, Activity Audit. or Policy definition sections
of the product.
Advanced User: Can access every Sysdig Secure feature within the
team scope in read and write mode. Advanced Users can create,
delete, or update runtime policies, image scanning policies or any
other content. The Advanced User cannot manage other users.
Team Manager: Same permissions as the Advanced User + ability to
add/delete team members or change team member permissions.
Team Managers only have user administration rights within the
specific team(s) for which they are designated Managers.
In image scanning reports, the vulnerability comparison feature allows
users to compare two different tags within the same repo to see which
vulnerabilities are new or have been fixed in version X compared to
version Y.
This allows developers easily to compare the latest image to a previous
version to easily report on which vulnerabilities have been addressed
and which are new.
The Captures function in Sysdig Secure has a new look and the following
usability improvements:
Bulk deletion of capture files
Ability to see whether a capture was triggered manually or by a
policy
Search across all capture files
File Data Source Support for Activity Audit
Sysdig Secure’s Activity Audit now supports a new data source element:
File activity.
Sysdig agent version 9.5.0+ is required to enable this new data source.
You can now filter the activity by file type or specific
file attributes:
File name
Directory
Command (used to access the file)
Access mode
File activity is also visible in the time-series graph at the top
(pink color):
Activity Audit will capture non-read file operations executed by
interactive commands
Sysdig Monitor
This release contains various bug fixes and improvements. There are no
new features in v3.2.0.
Sysdig Platform
S3-Compatible Storage for Capture Files
Configuring S3-compatible storage (such as Minio or IBM Cloud Object
Storage) for your Sysdig captures is now supported on Sysdig Platform
on-prem deployments. The capability can be turned on by configuring the
system appropriately, as given in (On-Prem) Configure Custom S3
Endpoint.
Release 3.0.0, December 19, 2019
Upgrade Process
Sysdig Platform has been tested and qualified against the following:
The Activity Audit in Sysdig Secure allows you to browse a live stream
of activity from your Kubernetes containers and nodes. Audit takes the
highly detailed data from syscalls and Kubernetes audit logs captured at
the agent level, and makes it always-on, searchable, and indexed against
your cloud-native assets.
This stream includes executed commands, network activity, and kubectl
exec requests to the Kubernetes API. The Activity Audit allows users to
view different data sources in-depth for monitoring, troubleshooting,
diagnostics, or to meet regulatory controls (SOC2, NIST, PCI, etc).
Flexible filtering and scoping to help you focus on what’s relevant:
Filters allow you to search, sort, and surface meaningful data and
connections as they are needed. You can filter by data source type, data
source attributes (like command name or Kubernetes user) and dynamic
Kubernetes scope
Automatically trace a kubectl exec session : The built-in trace
functionality allows you to isolate and trace a kubectl exec access to
a pod, automatically correlating the original Kubernetes user and IP
that accessed the pod with the activity that was performed during the
interactive session, including commands and network connections.
Activity Audit is a Preview Beta feature. Contact your customer success
manager to learn more about rolling out this feature.
Kubernetes Policy Advisor (Beta)
With the Kubernetes Policy Advisor, Sysdig Secure auto-generates Pod
Security Policies (PSPs) to significantly decrease the time spent
configuring Kubernetes Policies. Strict security policies reduce risk,
but can also break applications. Sysdig tests the impact of pod security
policies through simulations, enabling teams to adjust misconfigurations
before shifting to production. There are three main features that
comprise the Kubernetes Policy Advisor:
Auto generation: Sysdig Secure can parse any Kubernetes yaml file
that includes a pod spec to generate a tailor-made PSP based on the
configuration.
Simulations: Start a simulation of the auto-generated PSP or any
user-inputted PSP to see what pods would have been blocked from running
if this PSP had been actively applied to the cluster.
Events and tuning: Each pod/activity that would have violated the
PSP will generate an event. Within the event details, users can see
information about potential modifications they may need to make to the
policy or the pod configuration.
Scanning Improvements
New Scanning Rules
File attributes can now be verified as part of the image scan analysis.
A specific file can be validated against a node or sha256 hash.
Scale Improvements to Scanning Reporting
No query conditions are required as part of the Package and Policy
Queries.
Google Distro-less OS
Support for images based on Google distro-less OS, including detection
of base OS/version and installed OS dpkg packages.
Sysdig Monitor
Overview Is GA
Overview is now generally available. Overview leverages Sysdig’s unified
Kubernetes data platform to monitor, secure, and troubleshoot your
Kubernetes clusters and workloads.
Please contact your Sysdig Technical Account Manager or email
support to enable Overview for on-premises
environments.
Cluster Overview
Major highlights of Overview GA include but are not limited to:
Multi-cloud view of the health, risk, and capacity of your
Kubernetes infrastructure— a single pane of glass for Kubernetes
Clusters, Nodes, Namespaces, and Workloads across a multi- and
hybrid-cloud environment. You can easily filter by any of these
entities and view associated events and health data. View the
infrastructure organized by Clusters, Nodes, Workloads
Shows metrics prioritized by event count and severity, allowing you
to get to the root cause of the problem faster.
Drill down to Dashboards for instant insights.
To learn about the capabilities of the Overview feature, see
Overview.
Enhanced Out-of-the-box Dashboards
In an attempt to improve the Dashboards experience, the following
changes have been introduced:
The following Dashboards are added:
Kubernetes Cluster Overview: Provides nodes and workloads
availability and highlights the high-level health of your Clusters.
It also summarizes resources consumption (CPU, memory) across Nodes
and Namespaces to pinpoint possible anomalies and node disk
utilization
Kubernetes Node Overview: Provides availability of the Nodes,
indicating potential issues reported by Kubernetes; a summary of
resource (CPU and Memory) allocation and utilization, as well as
Network and Disk utilization.
Kubernetes Namespace Overview: Provides a high-level summary of
availability, and resource allocation and utilization across all the
Workloads in the selected Namespace.
Kubernetes Deployment Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods for each Workload.
Kubernetes StatefulSet Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods for each StatefulSet.
Kubernetes DaemonSet Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods.
Kubernetes Job Overview: Provides a detailed summary of job
status, completion trend, pod restarts, as well as resource
allocation and utilization across pods.
Kubernetes ReplicaSet Overview: Provides a detailed summary of
pod status, pod restarts, as well as resource allocation and
utilization across pods for each ReplicaSet.
Kubernetes Pod Overview: Provides a detailed summary of pod
status, pod restarts, and resource allocation and utilization in a
selected pod.
Kubernetes Workloads CPU Usage and Allocation: Helps you verify
that CPU requests are properly configured and actual utilization is
expected.
Kubernetes Workloads Memory Usage and Allocation: Helps you
verify that memory requests are properly configured and actual
utilization is expected.
Kubernetes CPU Allocation Optimization: Helps you verify that
infrastructure resources are available for future needs and are not
wasted.
Kubernetes Memory Allocation Optimization: Helps you verify that
infrastructure resources are available for future needs and are not
wasted.
The following Dashboards are retained:
Health Overview (applicable to all the objects in the environment)
Horizontal Pod Autoscaler (the default Dashboard when selecting an
HPA)
Resource Quota
Service Health (the default dashboard when selecting a service)
Events are now filtered by Scope to show the most relevant Events in
Explore and Dashboards. This is an extension of the existing
Event Scope
functionality. You can toggle between showing Event feed from the entire
infrastructure and only from the particular scope you are interested in
within the infrastructure. Event scoping for Dashboards and
Explore is enabled by default.
By default, Events are filtered to show only the relevant ones. However,
you can turn the filtering off and see Events from the complete scope.
To do so:
Click the Dashboard Settings (three dots) icon and select
Events.
Use the toggle button to turn off Filter events by dashboard
Scope.
Click Save.
Similarly, you can filter Events by Scope in Explore.
What’s n/a?
The Sysdig Monitor UI displays n/a in several scenarios associated
with labeling. The Explore UI has now been enhanced to add a tooltip for
n/a to help you understand the scenario. See The Meaning of
n/a
for more information.
Release 2.5.0, October 29, 2019
Upgrade Process
Kubernetes and OpenShift environments upgrade to 2.5.0 using the new
installer tool (see below).
Supported Upgrade Path: 2.3.0, 2.4.1
Sysdig Platform
New Installer Tool for Kubernetes/OpenShift Environments
With this release, Sysdig platforms can be installed and upgraded using
a semi-automated installer tool that greatly simplifies the installation
process. Available for Kubernetes and OpenShift environments.
Sysdig’s documentation platform has been upgraded and moved to
docs.sysdig.com.
Improvements include:
Look and feel: Updated to match the rest of the Sysdig branding
Search: Enhanced search speed, accuracy, and ease
Structure and content: Enhancements to content have been added
and are being continuously updated
Feedback: Buttons on each page enable users to communicate
directly with the documentation team.
Sysdig CLI
The Sysdig CLI provides an easy way to interact with the cli via the
command line. Read more
here.
Usage:
Run it without parameters to get a list of all the commands.
$ sdc-cli
Usage: sdc-cli [OPTIONS] COMMAND [ARGS]...
You can provide the monitor/secure tokens by the SDC_MONITOR_TOKEN and
SDC_SECURE_TOKEN environment variables.
Options:
-c, --config TEXT Uses the provided file as a config file. If the config
file is not provided, it will be searched at
~/.config/sdc-cli/config.yml and /etc/sdc-cli/config.yml.
-e, --env TEXT Uses a preconfigured environment in the config file. If
it's not provided, it will use the 'main' environment or
retrieve it from the env var SDC_ENV.
--json Output raw API JSON
--version Show the version and exit.
--help Show this message and exit.
Commands:
alert Sysdig Monitor alert operations
backup Backup operations
capture Sysdig capture operations
command Sysdig Secure commands audit operations
compliance Sysdig Secure compliance operations
dashboard Sysdig Monitor dashboard operations
event Sysdig Monitor events operations
policy Sysdig Secure policy operations
scanning Scanning operations
settings Settings operations
profile Profile operations
Sysdig Monitor
Ability to “Favorite” a Dashboard
Users can click the star icon to mark a “Favorite” dashboard, which will
then be listed under “My Favorites” in the Dashboard view.
Sysdig Secure
In-Line Scanning
Images can now be analyzed locally before they are pushed to a registry.
This has a few key benefits to users.
Images can be analyzed before they’re pushed to a registry and
reduce registry cost
Customers using the Sysdig Secure SaaS offering don’t need to expose
their registry to our SaaS for images to be scanned
For OpenShift users, the in-lince scan option can be integrated into
the S2I process to scan images without needing to expose a local
cluster registry via a route
Package name/version are now grouped together to provide easy parsing of
all CVE’s associated with a package and the images using that package.
New Trigger Parameters for CVSS Score
Image Vulnerabilities can now be evaluated against their
CVSS (Common Vulnerabilities Scoring
System) score. If a vulnerability is =, <;>, <=, or >= to a specific
score, then the rule can trigger a warn/stop action.
Time Ranges Updated
The default time range options have been updated in Sysdig Secure.
The default time ranges are now set to:
10 Minutes
30 Minutes
1 HR
6 HRs
1 Day
3 Days
To look at a custom window of time, use the manual time window.
Sysdig Secure Summary Dashboard in Sysdig Monitor
Sysdig Monitor includes default dashboards that provide metrics about
number of agents installed, active policies, events that have occurred,
and the policies that have triggered them. Use these dashboards to
identify trends, report on coverage, or facilitate the tuning process.
Secure Authentication for Cassandra and Elasticsearch on Replicated
Cassandra and Elasticsearch datastores now have an extra layer of
security on Replicated. Sysdig Replicated install allows you to enable
authentication and secure communication between Sysdig backend
components and the Elasticsearch or Cassandra datastores. For more
information, see Install with
Replicated.
[BETA] Audit Logging
The following APIs have been introduced to support administrators to
view a log of user activities and modifications to the components in the
system:
AppAttributes
AuditEvents
Audit logs stand for chronologically cataloged events to provide a
history of operational actions and to mitigate challenges. The ability
to trace an event back to its origin provides proof of compliance,
operational integrity, and protection from unsolicited use. For more
information, see [BETA] Auditing Sysdig Platform
Activities.
Known Issues
If you want to use Audit logging and have MySQL in your Kubernetes HA
environment, run kubectl -n sysdigcloud delete pod -l role=worker to
ensure Audit logging works as expected. This issue is observed only in
Kubernetes HA environments.
Sysdig Monitor
New Default Kubernetes Grouping
Groupings for Kubernetes have been modified. This updated Grouping is
available to new teams. Default groupings are immutable–-they cannot be
modified or deleted other than by copying. Modifying a copy is allowed.
New Groupings:
Clusters and Nodes
(cluster.name > node.name > pod.name > container.name)
The format of metric units are the same for the following:
The CPU and Memory metrics for Host and Container.
Kube-state CPU and Memory metrics.
Introducing the same format now makes the comparison of those metrics
easier on a chart.
Container Segmentation
Sysdig now supports segmenting all net.* metrics at container or pod
level by low level net.* dimensions, such as net.http.url or
net.http.status.code. Container-based teams now display segmentations
for net.http.* metrics as expected. The net.http.url and
net.http.status.codes are displayed if you select a container-based team
as it does for a host-based team for the same cluster.
Enhanced Event Notification
The ability to customize the subject and body of alert notifications
with variables has been extended to Event notifications. Event titles
and notification messages are in sync in the following cases:
Kubernetes Cluster and Node Capacity Dashboard has been refreshed to add
actual usage of CPU and Memory compared to Requests, Limits and
Allocatable capacity.
Aggregation for Kubernetes Nodes Health
Aggregation method has been refreshed for Kubernetes Node metrics. The
Kubernetes Node Health dashboard has been updated with metric
aggregations that are ‘summed’ across all containers running on the node
to reflect accurate node level data.
Bug Fixes
Export CSV/JSON was missing columns, not all data was exported as
expected. All columns from the dashboard should exist in the
exported output.
All data and columns are is now exported as expected.
Sysdig Secure
Policy Editor
*Please upgrade to an agent version 0.92.0 or greater
This UX overhaul brings three major improvements for every Sysdig Secure
user:
Runtime policies can import any number of security rules. You can
scope the security policy using container, cloud and Kubernetes
metadata.
Tighter Falco integration, directly from the web UI. You will be
able to define a new trigger condition or append to the list of
forbidden external IPs just clicking on the rule.
A more structured way to group, classify and lookup rules,
following the standard Cloud native procedure: tags and labels.
Rules Library
Visualize your runtime rules properties in just a glance:
Where this rule comes from (Published By). The security team
can instantly recognize whether a rule came from a specific Sysdig
update, from a custom rules file created within the organization or
from an external rules source (like the Falco community rules).
When was the last time it was updated (Last Updated). You can
use this information to audit your rules or if you schedule periodic
updates, to confirm when last happened.
Rule tags: An effective method for organizing your rules. You
can use these tags to describe the targeted entity (host, k8s,
process), the compliance
standard
it belongs to
(MITRE,
PCI, CIS Kubernetes) or any other criteria you want to use to
annotate your rules.
Falco Lists
Easily browse, append, and re-use lists to create new rules. Lists can
also be updated directly via API if users want to add existing feeds of
malicious domains, or IPs.
Falco Macros
Easily browse, append, and re-use macros to create new rules.
Image Scanning Reports
Please contact Sysdig Support to enable this feature
The reports feature allows users to query the contents of a scan against
a static or run-time scope to generate a report that shows the risk,
exposure, or components of an image.
Use cases could include:
A new CVE has been announced, let me find all the running images in
my US East Cluster that are exposed to that CVE
Show me all images within my Google Container registry that have the
tag prod and have a vulnerability with a fix that’s more than 30
days old
Show me all images with a high severity vulnerability with a fix
that are running in my billing namespace
Image Scanning - View Scan Results
Scan Results Page - The existing repositories page has been renamed
“Scan Results” this page also includes new capabilities to filter based
on where the images are deployed, and to easily browse/expand the
different repositories to see the image:tag’s that were evaluated and
their results
Whitelist labels available in vulnerabilities view - If a
vulnerability has been added to a whitelist then that status is
reflected in the Vulnerability report within the scan results.
Event Forwarding
Sysdig Secure can forward policy
events to tools like Splunk
or events can be forwarded via syslog as an easy way to send policy
events to any downstream SIEM.
Option to Secure Elasticsearch and Cassandra (Kubernetes only)
It is now possible to secure Elasticsearch and the Cassandra DB with
password authentication and/or SSL/TLS protection.
Sysdig Monitor
Enhanced Dashboard Menu
The Dashboard menu features a drawer-style popover that displays
on-demand to provide maximum real estate for your Dashboards. The menu
displays an alphabetical list of Dashboards you own and those shared by
your team. With the popover menu, you can add new Dashboards and search
for existing ones. Click a Dashboard name to access the relevant
Dashboard page where you can continue with the regular Dashboard
settings.
Customize Alert Notification Template
Sysdig Monitor alerts now provide an option to customize the messages
that are sent with alert notifications in email and other channels, such
as Pagerduty and Webhook.
Use the Alert Editor to input dynamic variables, such as hostname, or a
hyperlink, and to add custom messages in plain text to the notifications
for intended recipients. You can modify both the subject and the body of
the alert notification with a hyperlink or a variable. For example, you
can add an agent id or a link to a Dashboard to the message. This can
help provide context for troubleshooting the errors that triggered the
alert.
Sysdig Monitor can now collect Prometheus metrics from remote endpoints
with minimal configuration.
Remote endpoints (remote hosts) refer to hosts where the Sysdig agent
cannot be deployed, e.g., a Kubernetes master node on managed Kubernetes
services such as GKE and EKS, where user workload cannot be deployed. To
enable remote scraping on such hosts, simply identify an agent to
perform the scraping and declare the endpoint configurations in the
agent configuration file.
The collected Prometheus metrics are reported under and associated with
the agent that performed the scraping, rather than with a process. See
Collecting Prometheus Metrics from Remote
Hosts for details
Enhancements to Kafka App Check
Kafka integrations can now support authentication and SSL/TLS. If the
authentication or SSL/TLS are enabled in Kafka, see Apache Kafka
Example 5 for how to enable
configuration details on the Sysdig side.
Two New Metrics for Accurate Pod Counts
Two new Kubernetes metrics, kubernetes.namespace.pod.desired.count and
kubernetes.namespace.pod.available.count, have been added at the
Namespace level to track desired and available pod counts.
Sysdig Secure
Image Scanning: New Trigger Options
New Image Analyzed - Send notifications to different channels
when images with a particular registry, repo, tag are scanned.
Some users implement these type of alerts for implementing
workflows for image promotion, i.e.
“Push an image from staging to prod registry after a webhook is
sent that the image was scanned and it passed.”
CVE Update - Be notified whenever a vulnerability is added,
updated, or removed from an image within a registry.
Repository Alerts
Receive alerts about activity and changes that occur within your
registry. See Manage Scanning
Alerts.
Slack Notifications
Sample output of a CVE alert:
Sample output of an image-analyzed alert:
Image Scanning: Policies - New rule parameter available
A new field: Max days since creation is now available. This allows
users to only take Stop or Warn actions if a vulnerability has been
in the feed for a certain number of days.
For example: Only stop a build if an image has a high-severity CVE with
a fix, and the CVE is more than 30 days old.
Image Scanning: Policy Assignments - New compliance audits available
Policy assignments now support the ability to add audit policies to
provide a second step of validation of container images. Additional
audit policies evaluate images against Dockerfile Best Practices, PCI,
and NIST 800-190. These Audit policies have “Warn” actions set by
default and are intended to validate compliance/audit use cases and not
cause CI/CD builds to fail.
Updated Menu Navigation in Sysdig Secure
The top-menu navigation has been replaced by a context-sensitive
drawer-style side navigation bar.
Image Scanning: Scan Results Redesign
Scan results have been expanded to help users get a better idea about
the policy evaluation status and vulnerabilities present in an image.
This new version of scan results allows the user to
Get a breakdown of the different OS/Non-OS Critical, High, Medium,
Low CVEs present in the image
See the different policies the image has been evaluated against
See which specific rules have triggered the most stop/warn actions
and identify areas needing attention
A breakdown of the evaluation result has been added to give users a
better idea about what has triggered warn/stop actions as part of the
evaluation.
In this case, we can look at the Dockerfile Best Practice policy to see
the image
Has an effective user of root
Doesn’t include a Healthcheck
Uses apt-get upgrade as part of a Run instruction
Includes an ADD instruction
The Vulnerabilities section also now supports enhanced sorting and
filtering by severity level and whether or not a fix is available.
Image Scanning: PDF Reports
PDF reports, which include a summary of the policy evaluation and all
vulnerabilities present in the image, can be downloaded from the
console.
Bug Fixes
Explore display fix
Fixed an issue where, when the Explore Table had no columns
configured, the Explore view showed an error.
Enable/disable alerts fix
Fixed a problem where users were unable to toggle alerts.
Event posting fix
Fixed an issue where events posted in Slack did not appear in the
event stream. Now they do.
Monitor Spotlight fix
Fixed issue where Monitor Spotlight incorrectly alerted to update
On-Premise releases all the time. Update alert now turns on only
when an update is actually available.
Improved access to kube-state metrics
Teams based on ‘hosts’ (e.g., scoped by agent.tag.* ) will now
have access to all host and container data, including kube-state
metrics and dashboards. In previous versions, kube-state metrics
were not available for host-based teams.
Release 2435, July 24, 2019
Release 2435 replaces version 2172, 2266 and 2304 which were
released on May 28, 2019, June 17, 2019 and June 21, 2019. If you
installed 2172, 2266 or 2304, upgrade to 2435.
(Note that if you installed 2172, 2266 or 2304, please upgrade to 2435.
Otherwise, skip 2172, 2266 and 2304.)
Important Note Regarding Dashboard Migration V1 > V2
If you are upgrading from a previous version, the Dashboards will be
upgraded from V1 to V2. The process requires 20-30 minutes on large
systems, and the environment remains live throughout the rolling
upgrade.
DO NOT create or delete dashboards during the upgrade. After upgrading,
if you have saved v1 dashboards previously and need to upload them to
the v2 environment, see Migrate Saved Dashboards from V1 to V2.
Sysdig Platform Fix
Custom certificates fix
Fixed an install issue caused when using custom certificates.
Release 2304, June 21, 2019
Release 2304 replaces version 2172 and 2266 which were released on
May 28, 2019 and June 17, 2019. If you installed 2172 or 2266, upgrade
to 2304.
(Note that if you installed 2172 or 2266, please upgrade to 2304.
Otherwise, skip 2172 and 2266.)
Important Note Regarding Dashboard Migration V1 > V2
If you are upgrading from a previous version, the Dashboards will be
upgraded from V1 to V2. The process requires 20-30 minutes on large
systems, and the environment remains live throughout the rolling
upgrade.
DO NOT create or delete dashboards during the upgrade. After upgrading,
if you have saved v1 dashboards previously and need to upload them to
the v2 environment, see Migrate Saved Dashboards from V1 to V2.
Architecture Change in the Containers
In previous releases, there was a single backend container which ran
several processes.
As of version 2266, the processes have been divided into unique
containers, following container best practices.
Updated an underlying tool (Jedis 2.9.1) to Jedis 2.9.3, to address a
bug in the connection pool.
Sysdig Monitor
Manage Notification Frequency for Alerts
Users now have the ability to specify how often they want to be reminded
about an alert if the event is unresolved. Available under ‘Notify’
section of the alert configuration screen. See
Alerts.
Advanced Scope Selection
The scope editor (for dashboards, alerts, teams, etc.) has added
improved granularity, intelligent scope restriction, and the ability to
add custom values on-the-fly. The editor now restricts the scope of the
selection for subsequent filters by rendering values that are specific
to the selected label. The values that are only relevant to the previous
selection are displayed. For more information, see Dashboard
Scope.
Ability to Choose Unit of Metric
Sysdig Monitor now automatically detects the type of input and scale for
custom metrics. Earlier, custom metrics were marked as numbers on both
Explore and Dashboard UI. The UI now supports custom unit scale for
custom metrics. The supported units are byte, percent, and time. This
enhancement simplifies the mapping of units of measurement with that of
integrated application metrics, such as Prometheus. For more
information, see Editing the Unit
Scale.
Kubernetes Horizontal Pod Autoscaling (HPA) metrics
Support for the following HPA metrics has been introduced:
kubernetes.hpa.replicas.min, kubernetes.hpa.replicas.max, kubernetes.hpa.replicas.current, and kubernetes.hpa.replicas.desired. For more information, see Resource
Usage.
Expose Dashboard Scope in URL
The Dashboard URL can include scope parameters, including scope
variables. Users can now share the URL with non-Sysdig Monitor users and
allow them to collaborate on dashboard scope. Collaborators with a valid
link can change the scope parameters without having to sign in. They can
edit either on the UI or in the URL. For more information, see Share a
Dashboard.
Sysdig Secure
Image Scanning: Policy Assignments
Policy assignments allow you to specify where your image scanning
policies are applied. A policy assignment can include a Registry,
Repository, Tag combination and has full wildcard support for each of
those fields.
Policy assignments are evaluated in descending order, so be sure to
specify the most important policies first.
Examples
To evaluate all images with a “Prod” tag with your Example Prod
Image Policy, use the assignment: */*/Prod
To evaluate all images from gcr.io with an
Example Google Policy, use the assignment: gcr.io/*/*
The recommended way to run an image registry for an OpenShift cluster is
to run it locally. The Sysdig agent will detect the internal registry
names, but for the Anchore engine to pull and scan the image it needs
access to the internal registry itself. There can now set this path in
the Registries UI. See Manage Registry
Credentials.
Compliance: Custom Report Filters
When running CIS benchmark tests, you can filter your view of the
results to show only high-priority items or selected controls.
Improved metric aggregation defaults in Explore window
When a metric is first selected on the Explore page, the time and
group aggregation will be pre-populated with the most reasonable
choice, rather than average/average.
Topology view fixes: Implemented fixes for proper loading of
Topology panels in public dashboards, and proper “group by” and
‘scope" Topology Views.
See Visualizing Metrics using Topology View.
Non-root user security enhancements
Added changes to permit running Sysdig applications as non-root
user.
Image scanning fix in Sysdig Secure
Bug fix in the Jenkins plugin used to scan images in Sysdig Secure.
Release 1929, April 12, 2019
This release supports upgrades from
1149, 1245, 1402 (1511), 1586 (1630), 1765
New Features
Sysdig Platform
CRI-O Support
Sysdig on Kubernetes now provides support for
CRI-O, an implementation of the Kubernetes
Container Runtime Interface (CRI).
CRI-O container runtimes can be identified by the symbol beside the
entry in the Explore table:
Customize Data Retention Times using Sysdig REST API
The Sysdig platform has predefined data retention settings determined by
license plan. Using the Sysdig REST API, it is possible to configure
separate retention times (up to plan limit).
Sysdig Secure allows users to manage CVEs and images that may impact
builds by defining them as globally trusted or blacklisted. See Manage
Vulnerability Exceptions and Global
Lists for more information.
Kubernetes Audit Logging
Sysdig Secure allows users to create Falco security rules based on a
stream of Kubernetes audit events, integrating Kubernetes audit logging
with the Sysdig Agent. This allows users to track changes made to the
cluster, and send alerts where necessary. See Kubernetes Audit
Logging for more
information.
Enhancements
Manual PagerDuty Notification Channel Setup
Sysdig has expanded the PagerDuty notification channel configuration
process to allow users that have a team role of Manager, but a user role
of Team Responder or lower, to manually configure the channel settings
in order to add new channels. See PagerDuty
Notifications for more
details.
Agent Installation Changes
The default agent installation instructions in the UI have been updated
to ensure all agents use SSL. If SSL is not required, the following JVM
parameter will need to be set in the backend:
Anchore issue that caused scanning to hang when adding a registry
An issue occurred where scanning stopped functioning when adding a new
image scanning registry to an environment. This was caused by a bug
found in the Anchore open-source engine. This on-premises release
includes the approved workaround patch that corrects the issue. The next
release of the Anchore open-source engine will include the full fix.
Scanning service degradation due to orphaned services
An issue occurred in systems with substantial churn where the event
system became overloaded/flooded with orphaned service events, resulting
in service and performance degradation. This was caused by the Anchore
engine emitting an event each time it found a service that was
down/orphaned. This issue has been resolved.
Images with host/port component weren’t flagged with the correct analysis
An issue occurred where images with a host/port component were not
flagged correctly, resulting in them showing as unscanned. This was
caused by a bug in the scanning backend and has now been resolved.
Scan alert e-mail
An issue occurred in on-premises version 1765, where email alerts for
scanning results directed users to an internal Sysdig environment,
rather than their own. This has been corrected.
Some panels in self-monitored dashboards not working
An issue occurred where some panels in the Self-Monitored default
dashboards were not displaying data correctly, because of an error in
the default dashboard configuration file. This error has been corrected.
Relocated “Control Plane” from Default Dashboard in Explore
Kubernetes Control Plane Health dashboard has relocated to the
Dashboards module. This dashboard allows users to monitor the health of
Kubernetes master components
(kube-apiserver, etcd, kube-scheduler, kube-controller-manager). The
Kubernetes Control Plane health dashboard has been removed from the list
of default dashboards available under Resource Usage.
ElasticSearch on Replicated Restarts into Split Brain
When a customer restarted their Replicated environment, ElasticSearch
sometimes came up in a split-brain scenario (generally 2 + 1). This
issue has now been addressed.
Install code lines for Sysdig Agent corrected
On the Agent Installation page of the Sysdig UI, the supplied install
strings for Docker and Linux were incorrect and would not work “out of
the box” for a Replicated deployment. This issue has been addressed.
Upgrade Process for Sysdig in Kubernetes Environments
If you are running Sysdig Secure in OpenShift OR if you are running more
than 400 agents, please contact Sysdig
Support before upgrading.
If you are running Sysdig in Kubernetes, then the upgrade process
for this release is comprised of two parts:
Run the migration script:
This accommodates the backend transition to a different library for
communicating with the database.
Perform the Upgrade:
For Sysdig Monitor Only: If you have not licensed Sysdig Secure
and run only Sysdig Monitor, use the Basic Upgrade instructions.
For Sysdig Platform (including Secure): If you have licensed
both Sysdig Monitor and Sysdig Secure, you must follow the v1765
Upgrade (Kubernetes) instructions. These steps add the components
necessary to run the Scanning feature.
New Features
Sysdig Platform
Containerd Support
The Sysdig agent will automatically detect
containerd metadata, as well as any Docker
metadata, in your environment. Note that you must have agent version
0.88.1 or higher. See the agent install
instructions for details.
If you are upgrading from an earlier version of the agent, note that you
must also download the latest sysdig-agent-daemonset-v2.yamlfrom
GitHub for containerd functionality.
Sysdig Monitor
Improved Notification Channels Configuration
A newly redesigned notification channels page under settings has been
implemented. For more information, see Set Up Notification
Channels.
New Kubernetes Dashboards
Added two new default Kubernetes dashboards to help users monitor
Cluster / Node health and Namespace health. The dashboards are available
under the default dashboard list in Explore.
Sysdig Secure
Improved Registry Credential UI
The user interface for adding registry credentials has been redesigned
to improve user experience and add new configuration functionality. See
Registries.
Event Forwarding
Sysdig Secure policy events can now be forwarded to Splunk. See Event
Forwarding.
New Scanning Policies
New scanning policies have been added for compliance use cases and best
practices, interpreting NIST 800-190 and PCI controls to detect
misconfigured images.
Remediation Information
Remediation information has been added to assist in solving non-passing
test results, in order to bring an environment into compliance. See
Remediation Information.
Identify the Kubernetes Master Node
A new label has been added to the Compliance task results page to assist
in identifying the Kubernetes master node. See Identify the Kubernetes
Master Node.
Run a Compliance Task Manually
Users can now choose to run a compliance task immediately, rather than
scheduling a task for later. See Run a Benchmark Test
Manually.
The Intercom button has been moved from the bottom right corner of the
Sysdig Monitor UI to the bottom left to facilitate a better user
experience, as the previous location interfered with other UI elements.
It can now be found below the Help, Spotlight, and User menus.
Bug Fixes
The following issues have been fixed in this release:
Dashboard data display issue
An issue occurred when users in a team scoped by container tried to
access a dashboard. While building the read requests, the correct team
filters were used, but the write request incorrectly set the domain to
host instead of container, resulting in the backend not reading the data
correctly. This issue has been resolved.
AWS data display issue
For some AWS queries, data displayed incorrectly because the backend
could not determine the AWS resource type being queried, so the
aws.resource.type metadata was added to the request scope.
Assign User to Team in Secure
In some cases, users could not be added to Sysdig Secure teams, because
of a backend issue that occurred when loading the list of available
users to add to a team. This has been resolved.
Release 1630 Hotfix, January 31, 2019
This release supports upgrades from: 1149. 1245, 1402, 1511, and 1586.
Performance Issues
A performance issue was found when creating snapshots for large number
of teams and large number of custom metrics. This issue has been fixed.
Release 1586, January 21, 2019
This release supports upgrades from: 1149. 1245, 1402, and 1511.
New Features
Sysdig Monitor
New Events Feed
A redesigned Events Feed is now available. The new design unifies all of
your infrastructure-related events, alerts, and other activity in a
single view to help you quickly identify critical issues that need your
attention. For more information, refer to the
Events documentation.
New Topology is now GA
The new topology map functionality in Sysdig Monitor has moved from a
labs feature to full general availability. It features a redesigned
layout and enhanced interaction model to provide insight into
dependencies with drill-down to the container-process level.
Authentication UI
Administrators can now configure single sign-on authentication methods
(LDAP, SAML, OpenID, Google OAuth) via the Sysdig Monitor UI. For more
information, refer to the Authentication and Authorization (On-Prem
Options) documentation.
Enhancements
New Metrics
An additional metric (kubernetes.pod.restart.rate) has been added to
show the number of pod restarts since the last check.
Kubernetes Groupings
In previous releases, the default Kubernetes groupings used
kubernetes.cluster.id. This has been changed to
kubernetes.cluster.name to improve user experience.
Java Virtual Machine (JVM)
The JVM flag -UseContainerSupport has been disabled for performance
reasons.
Alert Delay at Startup
Sysdig alert jobs begin immediately at start-up. However, in instances
where Sysdig goes down unexpectedly, or without proper shutdown/startup
procedures implemented, data can be missing, triggering alert
notifications.
A start-up delay in alert jobs can be configured in on-premises
environments, by setting the draios.alerts.startupDelay parameter
during the installation process. The parameter requires a duration
value; the example below shows a duration of 10 minutes:
draios.alerts.startupDelay=10m
This parameter can be configured for either Replicated or Kubernetes
environments:
For Replicated environments, add the parameter to the Sysdig
application JVM options list. For more information, refer to the
Install Using the Replicated
GUI
documentation.
For Kubernetes environments, add the parameter to the
sysdigcloud.jvm.worker.options parameter in the configmap. For more
information, refer to the Sysdig Install with Kubernetes
1.9+
documentation.
Sysdig Secure
Compliance (Benchmarks)
CIS compliance benchmarks now support customizable schedules, using
a selection of intervals, days, and times, for different compliance
tasks to execute on.
Users can now download individual compliance results as a CSV file.
For more information, refer to the Download Task
Results documentation.
The Compliance scheduling page now displays when the next compliance
test will run.
An error log is now displayed when a compliance test fails.
Users can now search the list of compliance tests by hostname.
Bug Fixes
Mesos.*percent metrics do not currently have ‘%’ as a selectable unit scale
Mesos.*percent metrics did not include percentage as an option for the
metric unit scale. This has been corrected in the backend.
Split brain in Elasticsearch when launching Kubernetes HA env
A bug in the Elasticsearch container configuration created the potential
for the nodes to fail to discover all of the members of their cluster at
start-up. This resulted in a “split-brain” in the Elasticsearch cluster,
where nodes created multiple separate clusters, instead of a single
cohesive cluster.
The configuration of the container was re-tooled to allow the Kubernetes
cluster to expose the existence of the pods to their peers before they
finish starting up, and the cluster pods will now be aware of all of the
cluster members at start-up.
Release 1511 Hotfix, January 8, 2019
Issue: Better Handle Unknown Container Runtimes
In previous releases, snapshot jobs would fail if data for computing
aggregations for Kubernetes pods from unsupported container runtimes was
present. Containers in unknown runtimes are now skipped when computing
these aggregations to circumvent the error.
These containers are still present, and the metrics can be seen in
non-kubernetes contexts, as well as some Kubernetes contexts. (For
Kubernetes contexts, they are listed as null).
Issue: JVM Settings Fix
Prior to JVM update 191, the JVM
was not container-aware, and used system-level resources for
auto-configuration. Update 191 changed this behavior to use container
values instead. Sysdig has now updated the default settings in order to
use system-level resources for auto-configuration.
Users who want to fix the issue, but do not want to upgrade to the new
Sysdig hotfix, need to update the JVM settings in either the
config.yaml or the Replicated console, by adding the
-XX:-UseContainerSupport flag.
Release 1472, December 13, 2018
Tuned the configuration of metrics rollups to handle high-scale environments
Release 1402 December 3, 2018
Sysdig Monitor
Global silence alerts for scheduled downtime
Administrators can now temporarily disable alert events to mute
notifications during planned downtime or maintenance. The new feature
also supports sending a downtime notification to selected channels.
Access the new capability via Settings > Notification Channels. See
Disable or Delete a Notification
Channel.
Dashboard Templating
New dashboard templating enables users to create and configure a fixed
dashboard that enables alternating between multiple scope variables.
Users can assign custom names for labels and choose to set fixed or
variable label selection values.
Integration with AWS IAM role to grant permissions
New support for Amazon Web Services IAM roles grants permissions via IAM
to applications running on Amazon.
The Users and Teams settings pages have been updated to improve
performance and now feature a streamlined full-page edit layout. See
Manage Teams and Roles.
Sysdig Secure
CIS Compliance Checks
The ability to schedule CIS compliance tasks for the agent to run on
your infrastructure is now available.
These tasks will generate metrics that are available in Sysdig Monitor
and reports that are available in Sysdig Secure.
Bug Fixes
Several minor enhancements to improve performance and usability.
Release 1245 November 05, 2018
Please skip this release and install 1402 instead.
Enhanced connection tracking features
Security updates
Backend updates to address security vulnerabilities.
Teams functionality is now available in Sysdig Secure.
Caching on image scanning run-time page for performance
improvements.
Various bug fixes and improvements
Release 1149 September 14, 2018
Prerequisites
Your on-premises Sysdig installation MUST be running release v1091
before you can upgrade to this release v1149. Please upgrade to
v1091
before proceeding.
Unified Events table and migration tool (Required before upgrade)
A change was introduced in how events are indexed and stored in the
Sysdig platform. In prior versions, the three types of events were
stored in three separate indexes based on their different sources. After
migration and upgrade are complete, they will be combined in one index.
Before upgrading to v1149 it is necessary to run a Unified Events
migration tool.
Sysdig Agent Crash custom event
Generates a custom event if a Sysdig agent crash is experienced.
Node Ready alert reset
Enables transition of a notification from active => ok for a down
node (NodeNotReady) when the node with the same scope becomes ready
again (NodeReady).
Improved Mesos/Marathon label handling
Improved handling of Mesos/Marathon labeling to ensure proper display of
containers within the Sysdig UI.
Various bug fixes and improvements.
Release 1091 August 16, 2018
Component updates and CVE patches
Delivers minor-minor upgrades and CVE patches for all 3rd party
components in Replicated install. The Kubernetes install includes a
major upgrade for MySQL from 5.6.34 to 8.0.11. Please see product
README
for upgrade guidance and details.
StatefulSets for Kubernetes deployment
Provides StatefulSet option for select Redis and MySQL with Kubernetes.
Please see product
README
for usage eligibility and further details.
New ‘Standard User’ role and RBAC changes
Introduces new ‘Standard User’ role for developers that includes edit
access to dashboards, alerts, events but NO access to Explore. Renames
‘Edit user’ role to ‘Advanced user’ and ‘Read only’ role to ‘View only’.
See Manage Teams and Roles
for details.
Team scoping performance improvement
When creating or editing teams, the first 30 labels and tags are
displayed with the ability to search for additional options.
Multi-select alerts and bulk actions
New checkboxes on the alerts page enable selection of multiple alerts
for bulk actions.
Kubernetes Node Ready alert
A new alert provides notification when a Kubernetes node is not ready.
Default alert level is ‘warning’ (user-configurable).
Release 987 July 11, 2018
Solr dashboards update
Modifications to default Solr dashboard
Metrics aggregation fix
Fixed an issue with metrics aggregation
Release 963 June 26, 2018
LDAP enhancements
Enabling and disabling of LDAP authentication is now performed via
API configuration rather than Replicated console or K8S ConfigMap.
See LDAP for details.
An option has been added to allow chasing of
referrals
during LDAP authentication. See the
documentation for
details.
HTTPS enforcement
Sysdig is now enforcing HTTPS connectivity and using secure cookies.
With this change, we have disabled TLS v1.0. Users should modify any
scripts and/or applications to use HTTPS and TLS v1.2 for uninterrupted
operation.
Text Panels
You can now add text panels to your dashboards to provide additional
information. Text panels can be used as title headers or to provide
additional context that you would like to communicate. Features limited
markdown support .
Multiple segments for a single metric
You can now add up to five different segments for a given metric in
time-series and stacked area panels.
Default entry point
Admins can now set a default entry point for a team to simplify the
onboarding process. This determines the first page users see when they
start the application (e.g., a specific dashboard, settings, etc.).
Default Istio dashboards
Sysdig provides out of the box dashboards for monitoring Istio using
Prometheus exporters.
Test notification channels
New test function lets you pre-test your notification channels such as
email, Slack, PagerDuty, etc.
Copy and share groupings
Copy and share unique groupings with all of your teams.
Icon labels
New icon labels appear on hover to clarify underlying function for
users.
Alert on rate of change
Introducing a new ‘rate of change’ math function for metrics. Now you
can alert by the rate at which a metric changes vs. a static threshold.
For example, a default alert: Rate of change of disk usage alerts you
if your disk usage increases more than x% in a day.
Release 925 June 10, 2018
Solr dashboards improvement
Increased number of segments for Solr default dashboard panels
Public dashboards fix
Fixed an issue that caused errors when loading public dashboards due to
missing metrics
Enhancements to improve Sysdig Monitor response time during login.
Release 893 May 9, 2018
Daily metric rollup fix
Fixed an issue caused during daily metric rollup due to Cassandra-14092.
Release 892 May 2, 2018
Various bug fixes and improvements.
Release 890 April 30, 2018
New default ports for API/Collector containers (Replicated)
New default TCP ports are exposed from Sysdig backend API/collector
containers to the host level in Replicated-based installs. Read this
support
article for
info on avoiding possible port conflicts.
‘SSO CA certificate in PEM format’ option
Replicated-based installs using SSO that access their IDP via SSL/TLS
and need to import a CA certificate for Sysdig to trust the connection
can now do using the SSO CA certificate in PEM formatoption. This is
available under the ‘Advanced’ section of the ‘Settings’ tab in the
Admin console. Kubernetes-based installs can do the equivalent as
described in this
README.
LDAP settings changes
LDAP authentication settings are now configured via the Sysdig Platform
Admin API. Environments running releases pre-890 will have their LDAP
settings automatically migrated to the new API endpoints automatically
when upgrading to 890.
New UI design
Our new user interface provides a more modern framework for interacting
with the product. Navigation is re-oriented from a top-of-screen menu to
an icon-driven left side panel, providing more space for viewing your
metrics and dashboards. Click here for a quick video
introduction!.
Alert on rate of change
Introducing a new ‘rate of change’ math function for metrics. Now you
can alert by the rate at which a metric changes vs. a static threshold.
For example, a default alert: Rate of change of disk usage alerts you
if your disk usage increases more than x% in a day.
Support for Prometheus histogram metrics
Sysdig Monitor can now ingest a Prometheus histogram metric
type and
visualize them in a chart to show the distribution of specific metrics.
Link to Grafana plugin
Did you know you can add Sysdig as a Grafana data source? To help you
get started visualizing Sysdig-collected metrics in Grafana, we’ve added
a Grafana Plugin link to the help menu that takes you to the setup
instructions.
Revised alerting with Kubernetes metrics
Alert configuration settings for Kubernetes metrics now limit scope and
segmentation based on the metric that is selected to allow for more
accurate alerting. Check out our support page
for more
details.
Compare-to for timeseries
In your time series line charts you can now compare time-shifting
metrics to easily spot trends and anomalies. With compare-to for time
series you can configure and observe how one or more metrics have
changed since a previous time (e.g., 1 hour ago or 2 days ago).
‘Compare to’ for number panels
Metric number panels now feature a configurable ‘Compare to’ function to
display the change in measurement since a previous time frame. Provides
insight into the increase or decrease of metrics over time.
New Metrics for CPU Core Usage
We’ve added cpu.cores.used and cpu.cores.used.percent that align
with the way Kubernetes exposes CPU usage. Now you can compare values
using kube-state-metrics such as kubernetes.node.capacity.cpuCores,
kubernetes.pod.resourceLimits.cpuCores in order to determine if
resources are oversubscribed. These metrics are also key for capacity
planning and chargeback calculations.
Improved documentation for CPU metrics
The Sysdig Monitor Metrics
Dictionary now features
updated CPU metrics descriptions to provide more insight into each
available metric.
Resizable columns
The UI now allows columns to be resized for all tables in the
application including alerts, events, teams, and users.
Suggest Mode
Suggest mode auto-selects only the relevant dashboards and metrics,
hiding any inapplicable views. This is now the normal mode of operation.
The turn on/off option is no longer available.
Redesigned login screen
We’ve put a new, more modern face on the Sysdig Monitor login screen.
Release 858 April 12, 2018
Captures and Sysdig Inspect fix
Upgrades the open source sysdig version in on-prem build to resolve
sysdig capture and Sysdig Inspect compatibility issue.
Customers running version 693 and above can upgrade directly to release
858.
Release 800 March 13, 2018
New Explore design
We’ve redesigned Sysdig Monitor’s Explore page to give you extra screen
space to view your killer dashboards and metrics. The new vertical
layout helps you see more and get to what you need faster.
Golden Signals dashboards
New Service Golden Signal dashboards provide out-of-the-box metrics that
developers need when launching and monitoring a service or app. Includes
slowest transactions, latency, request volume, error rates, and most
requested URLs.
Spotlight
Want a simple way to quickly see what matters most in your environment?
Spotlight helps you quickly discover, detect, and optimize your
infrastructure and services. A Spotlight health check shows you new
integrations, infrastructure, app, and agent status, and more
at-a-glance.
Export table data as JSON/CSV
You can now download table data in JSON or CSV format for offline
viewing and analysis.
UI updates
We’ve simplified the dashboard panel copy function and added a duplicate
panel option in menu. We’ve also redesigned the dropdowns in the
top-right header including making it easier to quickly see and select
your teams.
Additional items
Various bug fixes and improvements including:
Performance and stability fixes for metrics
Fix for issue with ElasticSearch migration
Configurable program retention by customer (default limit 12)
Fix for migrations using BE mapper – now use dedicated customer
mapper.
Release 760 February 23, 2018
Explore grouping and scoping enhancements
We’ve massively simplified grouping and scopes. Our new approach gives
you better, more precise data - with less chance of invalid groupings
(e.g. Kubernetes deployment > hostname). Have questions? Watch this
video, read this
article, or
contact Customer Success and we’ll analyze your account for you!
kube-state-metrics
Sysdig Monitor now collects kube-state-metrics for monitoring and
alerting on the state of Kubernetes
objects. New dashboards provide visibility of metrics for nodes,
namespaces, services, daemonSets, jobs, replicaSets and pods. Requires
update to the Sysdig agent version 0.77.0 or higher.
Public URL dashboards
Ever want to share a killer dashboard with a colleague who is not a
Sysdig Monitor user? Now you can! Just pick, click, and send your URL.
Team Manager role
We’ve introduced a new ‘Team Manager’ role that provides the privilege
to add, delete, and modify team users as well as grant read or edit
access.
Proxy support for outgoing HTTP/HTTPS connections
You can now configure outgoing HTTP/HTTPS connections to be made via
proxy. Supports outgoing web connections to support notification
channels, PagerDuty, Slack, Amazon SNS, VictorOps, OpsGenie, WebHooks,
AWS CloudWatch data gathering. Read more
here.
Suggest mode enabled by default
Last year we introduced suggest mode – available in ‘Settings>Sysdig
Labs’ – as a way to boost your efficiency by showing only the views,
metrics, and grouping presets applicable to your environment. This
option has proven so popular that it is now enabled by default.
Custom headers for webhooks
When using webhooks, typically used to pass authentication credentials,
you can now add custom headers to pass along additional details with an
outgoing request.
Rename of Admin team to Monitor Operations
As part of the broader Sysdig Platform initiative, ‘Admin Team’ within
Sysdig Monitor is now renamed to ‘Monitor Operations.’ The Monitor
Operations team will continue to behave the same as the previous Admin
team:
The Monitor Operations team cannot be deleted.
Monitor Operations users have full visibility to all resources.
To change settings for any team, admins must switch to the Monitor
Operations team.
Support for JMX metrics from Java 9
Sysdig Monitor now supports JMX monitoring for Java 9 applications. To
enable collection of Java 9 metrics, update to the latest Sysdig Agent.
For more details, review the Sysdig Agent
changelogs.
Introducing read-only users
Users can have different roles for each of the teams they belong to,
either ‘Read user’ or ‘Edit user’. A read user can only use the app in
read-only mode, with no permission to create/edit/delete dashboards,
alerts, etc while the edit user is allowed to make those changes. This
is a per team role defined by Admin users.
Memcached default dashboard
A new default dashboard has been added to the Explore page where you can
see the most important Memcached performance monitoring metrics:
connections, commands, get hits/misses, evictions, etc.
Python client changes: Team/User configs
Changes to support Role Based Access Control (RBAC) modify how ‘Teams’
and ‘User’ configurations are stored and modified via the API. This
affects the functionality of the Python client. If you currently have
scripts that use these methods, click here
for details
on how to upgrade your Python client and make the necessary changes to
your scripts.
Release 722 January 8, 2018
CPU usage host-level segmentation
CPU usage at host level can now be segmented by CPU core.
AWS and Cloudwatch improvements
Enabled more reliable AWS metadata by separating AWS metadata from
Cloudwatch metrics
Additional items
Various bug fixes and improvements.
It is recommended to follow upgrade best practices:
Keep upgrades current
Test upgrades in a non-mission-critical or staging environment
before rolling into production.
5.1 -
Sysdig On-Premises Release Support
Sysdig On-Premises releases are versioned and labeled as [Major Version].[Minor Version].[Patch Number] and a build number. On-Premises releases are categorized as:
Major
Minor
Patch
A major release is defined as having significant changes to the application, such as changes in architecture, addition of a component or service, features, or infrastructure components.
A minor release typically includes functionality or feature enhancements, new features, UX improvements, etc.
A patch release is created when an issue is identified, either in the field or internally, that requires an immediate fix. These typically include bugs, or patches for newly discovered vulnerabilities.
Supported Releases
Customers who run Sysdig on premise are encouraged to stay up to date with our latest releases. This ensures the most hardened code base, infrastructure components, vulnerability patches, and new features.
Sysdig provides support (in accordance with our on prem Support Services Policy) for the most recent major version (n) and one version prior (n -1). Once a release has become unsupported, Sysdig will continue to support that release for a period of three months. This allows our customers a window of planning for an upgrade to a supported version (n or n-1).
Additionally, Sysdig will work to patch the known and impacting vulnerabilities for the most recent version (n) as of the on-premise build date. This will include critical, high, medium, and low CVE severities.
For critical and high vulnerabilities that are not fixed with the latest build, Sysdig will provide a Security Advisory document that details the impact exposure. This can include false-positives or benign vulnerabilities. For vulnerabilities of grave consequence (such as Log4J), Sysdig may provide hot-fixes for the most recent version (n) and the version prior (n -1). All customers are encouraged to keep their environments to n and n-1 versions.
Version
Supported Until
EOS Date
5.x (Latest Version)
7.x
3 Months after 7.x release date
4.x
6.x
3 Months after 6.x release date
3.x
Obsolete
2.x
Obsolete
If you have questions regarding Sysdig’s product supportability policy, please Contact Support.
6 -
Falco Rules Changelog
Falco rules are used in the Sysdig Secure Policy
Editor.
Commit Date
Rule Notes
Version of the Falco Rules Installer (On-Prem)
June 24, 2022
Rule Changes
Added additional exceptions to older agent versions to aid in addressing false positives:
Create Symlink Over Sensitive Files
Execution of binary using ld-linux
Run shell untrusted
Modified the following macros:
truncate_shell_history
modify_shell_history
Extended the condition of the rule, Detect crypto miners using the Stratum protocol , to improve detection capabilites
New rules created:
Launch malicious container image
GCP Suspicious IP Inbound Request
GCP Allow Public Access to Bucket
GCP KMS Schedule Key Deletion
GCP Create DLP Job
GCP Delete DLP Job
GCP Update DLP Job
GCP Paused DLP Job
Updated IoCs Ruleset with new findings.
Default Policy Changes
Added the following rule to the default policy, IoCs Malware Activity: Launch malicious container image
Added the following rules to the default policy, Sysdig GCP Best Practices:
GCP Suspicious IP Inbound Request
GCP Allow Public Access to Bucket
GCP KMS Schedule Key Deletion
GCP Delete DLP Job
GCP Paused DLP Job
0.75.0
June 17, 2022
Rule Changes
Added the following new rules:
AWS Suspicious IP Inbound Request
eBPF Program Loaded into Kernel
Modified the following rules:
Symlink over Sensitive Files
Container Drift rules (with new exceptions)
Updated the macro: sysdig_commercial_images. It now contains two new KSPM images.
Add the new macro ti_anon_ips for Tor source IPs.
Updated IoCs Ruleset with new findings.
Default Policy Changes
Added the new rule, AWS Suspicious IP Inbound Request to the Sysdig AWS Best Practices policy.
Added the new rule, eBPF Program Loaded into Kernel to the Suspicious Container Activity policy.
0.74.3
June 03, 2022
Rule Changes
Added a new rule: Suspicious Java Child Processes
Updated the package_mgmt_procs macro to detect package management process with python
Updated some exceptions in the rule,Change thread namespace
Updated IoCs Ruleset with new findings.
Default Policy Changes
Added the new rule, Suspicious Java Child Processes,to the IoCs Malware Activity
0.72.0
May 26, 2022
Rule Changes
Added the following new rules:
Reconnaissance attempt to find SUID binaries
Suspicious Home Directory Creation
Modified exceptions to reduce noise:
Change thread namespace
Contact cloud metadata service from container
DB program spawned process
K8s ConfigMap Created
K8s ConfigMap Deleted
K8s Serviceaccount Created
Netcat Remote Code Execution in Container
Privileged Shell Spawned Inside Container
Set Setuid or Setgid bit
System ClusterRole Modified/Deleted
Write below monitored dir
Write below root
Updated IoCs Ruleset with new findings.
Default Policy Changes
Added the following new policies:
Reconnaissance attempt to find SUID binaries
Suspicious Home Directory Creation
0.70.3
May 20, 2022
Rule Changes
Added additional exceptions to the following rules to aid in addressing false positives:
Set Setuid or Setgid bit
Execution from /tmp
Fixed the condition of the following rules:
Execution from /tmp
Execution from /dev/shm
Updated IoCs Ruleset with new findings.
0.69.0
May 13, 2022
Rule Changes
Added additional exceptions to older agent versions to aid in addressing false positives:
Run shell untrusted
Launch Privileged Container
Container Run as Root User
Write below root
Write below rpm database
DB program spawned process
Privileged Shell Spawned Inside Container
Launch Suspicious Network Tool in Container
Remove Bulk Data from Disk
Set Setuid or Setgid bit
Packet socket created in container
Execution from /tmp
Created the new rule, Possible Backdoor using BPF. This rule triggers if process was seen attaching a BPF filter on a network socket, this could indicate packet sniffing for use in a backdoor such as BPFDoor. Network diagnostic tools may also trigger this rule.
Created the new rule, Execution of binary using ld-linux. This method can be used to execute programs that do not have the exec bit set and may possibly evade detection measures.
Fixed the condition of the following rules:
Write below binary dir
Set Setuid or Setgid bit
Updated IoCs Ruleset with new findings
Default Policy Changes
Added the new rule, Possible Backdoor using BPF to the Notable Network Activity policy.
Write below binary dir
Added the new rule, Execution of binary using ld-linux to the IoCs Malware Activity policy.
0.68.1
May 6, 2022
Rule Changes
Added additional exceptions to older agent versions to aid in addressing false positives:
Modify binary dirs
Redirect STDOUT/STDIN to Network Connection in Container
Container Run as Root User
Execution from /tmp
Created the new rule Tampering with Security Software in Container. This rule detects common techniques by
threat actors to disable runtime security software.
Created the new rule Detect outbound connections to TOR Entry Nodes. This rule detects when clients reach the
TOR network through its entry nodes. NOTE: This is an EXPERIMENTAL rule and only contains a subset of TOR entry
nodes. It will be improved upon in the future.
Fixed the condition of the following rule: Execution from /tmp
Updated IoCs Ruleset with new findings.
Default Policy Changes
Moved the rule Redirect STDOUT/STDIN to Network Connection in Container to the Notable Container Activity default policy
Added the new rule Tampering with Security Software in Container to the Suspicious Container Activity default policy
Added the new rule Detect outbound connections to TOR Entry Nodes to the IoCs Malware Activity default policy
0.67.1
April 28, 2022
Rule Changes
Added new rule file, threat_intelligence_feed.yaml
, with lists and macros directly updated by Sysdig Threat Research Team.
Updated the following list: sysdig_commercial_images
Updated IoCs Ruleset with new findings.
Updated Falco rules conditions:
Execution from /tmp
Execution from /dev/shm
Network Connection outside Local Subnet
Added additional exceptions to aid in addressing false positives:
Execution from /tmp
Create Symlink Over Sensitive Files
Change thread namespace
DB program spawned process
Suspicious Cron Modification
0.66.1
April 21, 2022
Rule Changes
Added a new AWS Cloudtrail rule:
Create RDS DB Instance with Public Access
Added the following Falco rules:
Base64-encoded Shell Script Execution
Execution from /dev/shm
Added additional exceptions to aid in addressing false positives:
Service Account Created in Kube Namespace
K8s Serviceaccount Created
Modified to add a list of malicious IPs:
Outbound Connection to C2 Servers
Updated IoCs Ruleset with new findings
Default Policy Changes
Added the following:
Base64-encoded Shell Script Execution
Execution from /dev/shm
Moved to enabled policy: Outbound Connection to C2 Servers
0. 65.1
April 18, 2022
Rule Changes
Added additional exceptions to the following rules to aid in addressing false positives:
Change thread namespace
Create Symlink Over Sensitive Files
Container Run as Root User
DB program spawned process
Privileged Shell Spawned Inside Container
Run shell untrusted
Set Setuid or Setgid bit
Write below etc
0.65.0
April 17, 2022
Rule Changes
Added additional exceptions to the following rules to aid in addressing false positives: Privileged Shell Spawned Inside Container
0.64.1
April 15, 2022
Rule Changes
Added additional exceptions to the following rules to aid in addressing false positives:
Packet socket created in container
Change thread namespace
Run shell untrusted
Container Run as Root User
Created the new rule Base64-encoded Python Script Execution. This rule detects base64-encoded Python scripts on command line arguments. Base64 can be used to encode binary data for transfer to ASCII-only command lines. Attackers can leverage this technique in various exploits to load shellcode and evade detection.
Fixed the output of the following rules:
K8s Serviceaccount Created
K8s Serviceaccount Deleted
Updated IoCs Ruleset with new findings
Rule Changes
Added the Base64-encoded Python Script Execution rule to the IoCs Malware Activity default policy
Added the Launch Ingress Remote File Copy Tools in Container rule to the Notable Container Activity default policy
Created the new default policy, Known Exploit Detection. This policy embedes the rules that can identify potential exploits of well-known CVEs.
0.64.0
April 12, 2022
Rule Changes
Added additional exceptions to the following rules to aid in addressing false positives:
Schedule Cron Jobs
Set Setuid or Setgid bit
Create Symlink Over Sensitive Files
Disable the Unprivileged Delegation of Page Faults Handling to a Userspace Process rule removing its condition.
0.63.0
April 09, 2022
Rule Changes
Updated the following rules:
Simple output changes to the Detect outbound connections to common miner pool portscode> rule.
Updated priority and included additional cron paths for the Create Symlink Over Sensitive Files rule.
Updated IoCs Ruleset with new findings
The following new rules have been introduced.
Privileged Shell Spawned Inside Container. This rule detects a root shell being opened by a compromised process for interaction by the attack
Debugfs Launched in Privileged Container. This rule detects file system debugger, debugfs, launched inside a privileged container which might lead to container escape.
Mount Launched in Privileged Container. This rule detects file system mount occurrence inside a privileged container which might lead to container escape.
Unprivileged Delegation of Page Faults Handling to a Userspace Process. This rule detects a successful unprivileged userfaultfd syscall which might act as an attack primitive to exploit other bugs
Launch Ingress Remote File Copy Tools in Container. This rule detects ingress remote file copy tools launched in a container. For example, curl and wget.
Suspicious Cron Modification. This rule detects direct writes to cron job files.
Default Policy Changes
Policy: Notable Filesystem Changes
added the Suspicious Cron Modification rule.
Policy: Suspicious Container Activity
Added the Debugfs Launched in Privileged Container rule.
Added the Unprivileged Delegation of Page Faults Handling to a Userspace Process rule.
Policy: Suspicious Lateral Movement Activity to Cloud
Added the Mount Launched in Privileged Container rule.
Policy: Unexpected Spawned Processes
Added the Privileged Shell Spawned Inside Container rule.
0.62.1
April 06, 2022
Rule Changes
Reduce noise for the rulesWrite below monitored dir and write below etc by adding additional exceptions.
0.62.0
March 18, 2022
Rule Changes
Updated the Launch Root User Container condition rule.
Updated the following lists to address false positive :
miner_domains
allowed_k8s_users
Updated some exceptions in the Schedule Cron Jobs rule.
Created the sssd_writing_krb macro from the new release of OSS Falco.
Updated IoCs Ruleset with new findings.
Updated the following macros based on the changes in Falco OS:
modify_shell_history
truncate_shell_history
write_etc_common
Default Policy Changes
TheIoCs Malware Activity policy has been updated.
Malicious filenames writtenadded.
Malicious process detected removed.
Removed some rules from Notable Filesystem Changes policy:
Write below etc
Write below root
Write below rpm database
Write below binary dir
Removed one rule from the Notable Container Activity policy: Change thread namespace
0.59.2
March 10, 2022
Rule Changes
Exclude ptp and dp from the Change thread namespacerule.
Exclude self from the K8s Serviceaccount Created rule.
Exclude known cron writers from the Schedule Cron Jobs rule.
Updated the IoCs Ruleset with new findings.
0.58.1
March 06, 2022
Rule Changes
Add additional exceptions to aid in addressing false positive for rules:
Schedule Cron Jobs
Non sudo setuid
Launch Privileged Container
K8s Serviceaccount Created
Updated the following macros baed on the changes in Falco OS:aws_eks_core_images
Updated IoCs Ruleset with new findings.
0.57.2
March 03, 2022
Rule Changes
Fixed exception to aid in addressing false positives for rules:
Contact K8S API Server From Container
0.56.5
March 01, 2022
Rule Changes
Update rule: DB program spawned process
Create macro:pgbackrest_info_childs
0.56.4
February 18, 2022
Rule Changes
Add additional exceptions to older agent versions to aid in addressing false positive for rules:
Modify Shell Configuration File
Modify Shell Configuration File
Write below etc
Write below rpm database
DB program spawned process
Clear Log Activities
Launch Root User Container
Updated the following macros based on the changes in Falco OS:
containerd_shell_modify
tanium_client_running_python
postgres_running_pgbackrest
proc_file_suffix
known_redirect_procs
Updated the following lists to address false positives:
known_setuid_binaries
known_k8s_api_programs
gke_trusted_images_launch_root_list
Updated IoCs Ruleset with new findings.
0.55.2
February 10, 2022
Rule Changes
Add additional exceptions to older agent versions to aid in addressing false positive for rules:
Change thread namespace
Write below rpm database
Write below root
Clear Log Activities
Launch Root User Container
Updated the following macros based on the changes in Falco OS:
parent_python_running_sdchecks
python_running_sdchecks
exe_sysdig
tanium_client_running_python
sysdig_dragent
trusted_logging_images
Updated the following lists based on the changes in Falco OS:
sysdig_commercial_images
allowed_dev_files
user_known_chmod_applications
miner_domains
Updated IoCs Ruleset with new findings.
0.54.3
February 07, 2022
Rule Changes
Add additional exceptions to older agent versions to aid in addressing false positive for rules: Launch Root User Container
0.53.4
February 04, 2022
Rule Changes
Add additional exceptions to older agent versions to aid in addressing false positive for rules:
Modify Shell Configuration File
Write below etc
Write below root
Read sensitive file trusted after startup
Change thread namespace
Launch Suspicious Network Tool in Container
Redirect STDOUT/STDIN to Network Connection in Container
Updated the following macros based on the changes in Falco OS:
spawned_process
sensitive_mount
Updated the following lists based on the changes in Falco OS:
falco_hostnetwork_images
deb_binaries
known_sa_list
falco_sensitive_mount_images
Updated the following lists to address false positives:
db_server_binaries
user_known_chmod_applications
Updated IoCs Ruleset with new findings.
0.53.3
January 29, 2022
Rule Changes
Add additional exceptions to older agent versions to aid in addressing false positives for rules:Write below etc.
Updated IoCs Ruleset with new findings.
Add new rules:
Modify ld.so.preload
Polkit Local Privilege Escalation Vulnerability(CVE-2021-4034)
0.52.0
January 21, 2022
Rule Changes
Updated IoCs Ruleset with new findings.
0.51.1
January 14, 2022
Rule Changes
Created a new AWS Rule: Read Object through AWSSupportServiceRolePolicy Assumed Role.
Updated tags for AWS Rule:AWS Command Executed on Unused Region.
Updated tags for the following GCP Rules:
GCP Invitation Sent to Non-corporate Account
GCP Create User-managed Service Account Key
GCP Create GCP-managed Service Account Key
GCP Create Cloud Function Not Using Latest Runtime
GCP Set Bucket IAM Policy
GCP Create Bucket
0.50.5
December 16, 2021
Rule Changes
Add a new rule:Malicious C2 IPs or domains exploiting log4j: detect connections with malicious IPs involved in log4j exploitation.
Updated IoCs Ruleset with new findings
0.49.2
January 03, 2022
Rule Changes
Created a new AWS Rule: Read Object through AWSSupportServiceRolePolicy Assumed Role.
Updated tags for AWS Rule:AWS Command Executed on Unused Region.
Updated tags for the following GCP Rules:
GCP Invitation Sent to Non-corporate Account
GCP Create User-managed Service Account Key
GCP Create GCP-managed Service Account Key
GCP Create Cloud Function Not Using Latest Runtime
GCP Set Bucket IAM Policy
GCP Create Bucket
0.48.0
December 06, 2021
Rule Changes
Add a new rule:Find AWS Credentials: Find or grep AWS credentials in host or container.
Add additional exceptions formats to aid in addressing false positives for rules: K8s ConfigMap Deleted.
Updated IoCs Ruleset with new findings
0.46.2
November 30, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false positives for rules: Create Sensitive Mount Pod.
Updated IoCs Ruleset with new findings
0.46.0
November 22, 2021
Rule Changes
Created a new GCP Rule: GCP Create Cloud Function
Create following Azure Rules:
Azure Remember MFA for User Access on Devices
Azure Users Can Consent to Apps Accessing Company Data on Their Behalf
Azure Deactivate MFA for User Access
Azure Container ACL Modified
Add additional exceptions formats to aid in addressing false positives for rules:
Modify Shell Configuration File
Launch Privileged Container
Container Run as Root Users
Updated IoCs Ruleset with new findings
Updated AWS, Azure,and GCP tags
0.45.1
November 16, 2021
Rule Changes
Updated IoCs Ruleset with new findings.
0.44.1
November 15, 2021
Rule Changes
Added new rule for AWS Cloudtrail: Create Lambda Function Using Unsupported Runtime
Modified rule for AWS Cloudtrail:Run Instances with Non-standard Imagenow checks the image ID from aws.ec2.imageID instead of getting this value from respondeElements/instanceSet/items using jevt
0.44.0
November 11, 2021
Rule Changes
Added new tags to the following rules:
GCP Delete Resources from the PCI Blueprint Environment
GCP Create KMS Key Without Rotation
GCP Remove KMS Key Rotation
GCP Delete DNS Zone
GCP Delete GKE Node Pool
GCP Delete Router
GCP Delete GKE IAM Role
GCP Delete VPC Network
GCP Delete GKE Subnetwork
0.43.2
November 5, 2021
Rule Changes
Added new tags to existent rules for MITRE and NIST categories.
0.43.1
October 29, 2021
Rule Changes
Added new tags to the following rules:
Modify RDS Snapshot Attribute
Modify Image Attribute
Modify Snapshot Attribute
Detect outbound connections to common miner pool ports
Detect crypto miners using the Stratum protocol
Updated Malware IoCs with the new findings.
0.42.0
October 20, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false positives for rules:
Modify Shell Configuration File
Run shell untrusted
Launch Sensitive Mount Container
Outbound or Inbound Traffic not to Authorized Server Process and Port
Create Sensitive Mount Pod
Create NodePort Service
Attach/Exec Pod
Service Account Created in Kube Namespace
System ClusterRole Modified/Deleted
Default Policy Changes
Lowered Severity to INFO for the following policies:
All K8s User Modifications
All K8s Object Modifications
0.41.0
October 11, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false positives for rules:
Modify binary dirs
Clear Log Activities
Remove Bulk Data from Disk
Create HostNetwork Pod
Launch Suspicious Network Tool in Container
Added three new Falco rules to detect Malware:
Malicious IPs or domains detected on command line
Malicious binary detected
Malicious process detected
Default Policy Changes
Added New Policy IoCs Malware Activity
0.40.0
October 07, 2021
Rule Changes
Changed inbound_outbound macro condition.
Add additional exceptions formats to aid in addressing false positives for rules:
Write below etc
Read sensitive file untrusted
Search Private Keys or Passwords
Disallowed K8s User
K8s Deployment Created
K8s Deployment Deleted
K8s Service Created
K8s Service Deleted
K8s ConfigMap Created
K8s ConfigMap Deleted
K8s Namespace Created
K8s Namespace Deleted
K8s Serviceaccount Created
K8s Serviceaccount Deleted
K8s Role/Clusterrole Created
K8s Role/Clusterrole Deleted
K8s Role/Clusterrolebinding Created
K8s Role/Clusterrolebinding Deleted
0.39.0
September 23, 2021
Rule Changes
Changed net_miner_pool macro used in the Detect outbound connections to common miner pool ports rule.
0.37.1
September 21, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false positives for rules: Non sudo setuid rule.
0.37.0
August 26, 2021
Rule Changes
Added the following rules:
Console Login Through Assume Role
AWS Command Executed by Untrusted User
Console Login Success
Console Login Success From Untrusted IP
Delete AWS user
Remove AWS User from Group
Put Object in Watched Bucket
Read Object in Watched Bucket
Added new lists:
trusted_aws_users
watched_buckets
Updated rules:
Console Login Without MFA now does not fire on assumed role
Console Root Login Without MFA now does not fire on assumed role
Add AWS User to Group now outputs the user added to the group
0.36.0
POSTPONED August 20, 2021
POSTPONED Rule Changes
Added a new rule: Unprivileged Delegation of Page Faults Handling to a Userspace Process
Update the list:
sysdig_commercial_images
falco_hostnetwork_images
Updated the macro: interactive macro updated checking the tty state. The interactive session is with proc.tty != 0
POSTPONED 0.35.0
August 13, 2021
Rule Changes
Added additional exceptions formats to aid in addressing false positive for the rules:
Launch Package Management Process in Container
Terminal shell in container
The docker client is executed in a container
Updated the list: sysdig_commercial_images
Updated the macro: interactive macro updated checking the tty state. The interactive session is with proc.tty != 0
0.34.0
August 02, 2021
Rules Changes
Add additional exceptions formats to aid in addressing false positive for rules:
DB program spawned process Rule
Change thread namespace
The docker client is executed in a container
Launch Suspicious Network Tool in Container Rule
0.33.0
July 27, 2021
Default Policy Changes
Enable the Sysdig GCP Best Practices policy by default.
0.32.0
July 25, 2021
Rule Changes
GCP events were consumed directly from the protoPayload, which removed some fields that are used and are not part of the protoPayload itself. All the rules that use jevt.value are updated now to reference protoPayload in the root path. It is a breaking change for GCP rules, and you are required to use cloud-connector versions above v0.8.0.
Updated GCP rules to use protoPayload JSON path. Affected rules:
GCP Create API Keys for a Project
GCP Delete Bucket
GCP Create Bucket
GCP List Buckets
GCP List Bucket Objects
GCP Put Bucket ACL
GCP Set Bucket IAM Policy
GCP Update Bucket
GCP Create Cloud Function Not Using Latest Runtime
GCP Create Cloud Function
CloudRun Create Service
CloudRun Replace Service
GCP Create a Default VPC Network
GCP Disable Subnet Flow Logs
GCP Enable Connecting to Serial Ports for a VM Instance
GCP Creation of a VM Instance with IP Forwarding Enabled
GCP Suspected Disable of OS Login in a VM Instance
GCP Enable Project-wide SSH keys for a VM Instance
GCP Shield Disabled for a VM Instance
GCP Create or Patch DNS Zone without DNSSEC
GCP Describe Instance
GCP Command Executed on Unused Region
GCP Create GCP-managed Service Account Key
GCP Create User-managed Service Account Key
GCP Invitation Sent to Non-corporate Account
GCP Operation by a Non-corporate Account
GCP Super Admin Executing Command
GCP Update, Disable or Delete Sink
GCP Monitoring Alert Deleted
GCP Monitoring Alert Updated
GCP Disable Automatic Backups for a Cloud SQL Instance
GCP Disable the Requirement for All Incoming Connections to Use SSL for a Cloud SQL Instance
Added a new rule: GCP Set a Public IP for a Cloud SQL Instance
0.31.0
July 22, 2021
No rule changes. No default policy changes.
Fix a defect related to installing rules for older backend versions (Sysdig 4.0.*).
0.30.0
July 20, 2021
Default Policy Changes
Sysdig AWS Best Practices severity is now set to 'medium'
Sysdig GCP Best Practices severity is now set to 'medium'
0.29.0
July 19, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false positive for rules:
DB program spawned process Rule
Change thread namespace
The docker client is executed in a container
0.28.0
July 16, 2021
Default Policy Changes
Disabled Access Cryptomining Network Policy by default
0.27.0
July 15, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false positive for rules:
Run shell untrusted
DB program spawned process
Change thread namespace
0.26.0
July 11, 2021
Default Policy Changes
Rule changes have been applied in the following default policies:
Suspicious Package Management Changes
Notable Filesystem Changes
Suspicious Filesystem Reads Policy
Suspicious Filesystem Changes
User Management Changes
Disallowed Network Activity
Inadvised Container Activity
Disallowed Container Activity
Suspicious Container Activity
New default policies created:
Suspicious Lateral Movement Activity to Cloud
Notable Network Activity
Default policies removed:
Suspicious Package Management Changes
Suspicious Filesystem Reads Policy
User Management Changes
Disallowed Network Activity
Disallowed Container Activity
Inadvised Container Activity
Existent policies status changes:
Access AcceCryptomining Network enabled by Default
0.25.0
July 01, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false proofs for rules:
Netcat Remote Code Execution in Container
Launch Sensitive Mount Container
Redirect STDOUT/STDIN to Network Connection in Container
0.24.0
June 25, 2021
Rule Changes
Add additional exceptions formats to aid in addressing false proofs for rules:
Write below root
Change thread namespace
0.23.0
June 22, 2021
Rule Changes
Add additional exceptions formats for rules:
Change thread namespace
Create Privileged Pod
Modify Shell Configuration File
Write below binary dir
Launch Privileged Container
The docker client is executed in a container
ClusterRole With Wildcard Created
Create HostNetwork Pod
Service Account Created in Kube Namespace
K8s Role/Clusterrole Created
K8s Role/Clusterrole Deleted
K8s Role/Clusterrolebinding Created
Netcat Remote Code Execution in Container
Delete Bash History
ClusterRole With Write Privileges Created
Clear Log Activities
Modify binary dirs
Unexpected outbound connection destination
Unexpected UDP Traffic
0.22.0
June 19, 2021
A new policy, Sysdig GCP Best Practices, has been added.
Rule Changes
New GCP Rules have been added for AuditLog:
GCP Create API Keys for a Project
GCP Create Bucket
GCP Delete Bucket
GCP List Buckets
GCP List Bucket Objects
GCP Put Bucket ACL
GCP Set Bucket IAM Policy
GCP Update Bucket
GCP Create Cloud Function Not Using Latest Runtime
GCP Create Cloud Function
GCP Update Cloud Function
CloudRun Create Service
CloudRun Replace Service
GCP Create a Default VPC Network
GCP Disable Subnet Flow Logs
GCP Enable Connecting to Serial Ports for a VM Instance
GCP Creation of a VM Instance with IP Forwarding Enabled
GCP Suspected Disable of OS Login in a VM Instance
GCP Enable Project-wide SSH keys for a VM InstanceGCP Shield Disabled for a VM Instance
GCP Create or Patch DNS Zone without DNSSEC
GCP Describe Instance
GCP Command Executed on Unused Region
GCP Create GCP-managed Service Account Key
GCP Create User-managed Service Account Key
GCP Invitation Sent to Non-corporate Account
GCP Operation by a Non-corporate Account
GCP Super Admin Executing Command
GCP Update, Disable or Delete SinkGCP Monitoring Alert Deleted
GCP Monitoring Alert Updated
GCP Disable Automatic Backups for a Cloud SQL Instance
GCP Disable the Requirement for All Incoming Connections to Use SSL for a Cloud SQL Instance
0.21.0
June 17, 2021
Fixed a defect in v0.20.3. The fix is for the detection of older backend versions when looking for accounts scheduled for deletion.
0.20.4
June 17, 2021
Skip accounts scheduled for deletion when verifying Falco rules compatibility.
0.20.3
June 16, 2021
Rule Changes
Add additional exceptions formats to allow addressing false positives for rules:
Launch Package Management Process in Container
Set Setuid or Setgid bit
Terminal shell in container
0.20.2
June 11, 2021
Rules Changes
Add additional exceptions formats to help address false positives for rules:
Run shell untrusted
Set Setuid or Setgid bit
0.20.1
June 03, 2021
Rule Changes
The Non sudo setuid rule: Add macmnsvc (mcafee service host) to set of programs that are allowed to setuid.
The Launch Suspicious Network Tool in Container rule: Add another zookeeper image pattern that's allowed to run network tools.
The Clear Log Activities rule: Add another fluentd image as allowed to clear log files.
Add additional exceptions formats to aid in addressing false positives for rules:
System procs network activity
K8s Serviceaccount Created
K8s Serviceaccount Deleted
K8s Role/Clusterrole Created
K8s Role/Clusterrole Deleted
0.20.0
June 01, 2021
Rule Changes
The Read Sensitive File Untrusted rule:
Allow clamscan to read sensitive files
Allow db2ckpw (IBM DB2 Credential Checker) to read sensitive files
The Launch Suspicious Network Tool in Container rule: Add another zookeeper image that is allowed to run nc inside a container.
Add additional exception patterns for the following rules:
Launch Package Management Process in Container
K8s Serviceaccount Created
K8s Serviceaccount Deleted
K8s Role/Clusterrole Created
K8s Role/Clusterrole Deleted
0.19.0
May 26, 2021
Rule Changes
Add additional Qualys binaries as exceptions for rules:
Read sensitive file untrusted
User mgmt binaries
Write below etc
The Write below etc rule:
Allow newrelic to write below /root/newrelic instead of specific files
Allow nessuscli write state file
Allow masvc to write below /etc/ma.d/
Allow grafana to write state
The Write below root rule : Add an additional cmdline writing to exec.fifo.
The DB program spawned process rule: Allow sqlplus spawn oracle.
Add additional sets of exception fields for rules:
Write below monitored dir
The docker client is executed in a container
0.18.0
May 25, 2021
The Sysdig AWS Best Practices policy no longer includes the Logged in without Using MFA rule.
Rule Changes
Add five new rules for AWS Cloudtrail events.
Disable the AWS Cloudtrail rule, Logged in without Using MFA.
The Read Sensitive File Untrusted rule: Let the TaniumEndpoint agent read additional sensitive files.
The Write below root rule, docker_writing_state macro: Allow for paths that simply specify a path below an implied / or /root of current working directory.
The DB program spawned process rule: Add additional allowed Postgres backup utilities.
The Write below root rule:
Use a more flexible string match against the /exec.fifo paths.
Allow newrelic CLI to write to CLI log file.
Allow the docker cleanup image utility to write state files below /.
The Write below rpm database rule: Allow tanium endpoint script to write to the rpm database.
The Contact K8S API Server From Container rule: Add another fluent-bit program that is allowed to contact the API Server.
0.17.0
May 20, 2021
Rule Changes
Added exception to the following to address false positives:
The Non sudo setuid rule: Let swiagent read setuid.
The Read sensitive file untrusted rule:
Let refresh-mcollec (tive-metadata), part of puppet, read sensitive files.
Let puppet directly read sensitive files.
Let Tanium endpoint read sensitive files.
Let ir_agent (rapid7 agent) read sensitive files.
The Write below root rule:
Add an additional command line pattern for Cassandra to allow writes to /root/.cassandra.
Add additional exec.fifo path below root for runc.
Let docker write to certain files below /. It is part of some docker-in-docker setups.
Let Tanium joval write to /root/.jOVAL/.
The Change thread namespace rule:
Add an additional weaveworks/kured process name.
Let avinetworks/se images run programs that can change thread namespaces.
The System procs network activity rule : Add an additional exception pattern.
The User mgmt binaries: Let refresh-mcollec (tive-metadata), part of puppet, run user management binaries.
The Contact K8S API Server From Container rule: Let fluent-bit images run programs to contact the API server.
The Launch Suspicious Network Tool in Container rule: Let certain Openshift images run dig to perform DNS lookups.
The Clear Log Activities rule: Let certain Workinggrafana-related images clear log files in the container.
0.16.0
May 19, 2021
Rule Changes
Additional exception fields are added to the following rules to aid in customization:
K8s Secret Created
K8s Secret Deleted
0.15.1
May 18, 2021
Rule Changes
The Detect outbound connections to common miner pool ports rule: Add additional known miner domains.
Add additional exception fields to the following rules to aid customization:
Modify Shell Configuration File
Write below monitored dir
Write below etc
Write below root
Write below rpm database
Launch Privileged Container
Launch Sensitive Mount Container
Terminal shell in container
System procs network activity
Launch Suspicious Network Tool in Container
Set Setuid or Setgid bit
Launch Remote File Copy Tools in Container
The docker client is executed in a container
Disallowed K8s User
Create Privileged Pod
Create Sensitive Mount Pod
Create HostNetwork Pod
Attach/Exec Pod
Pod Created in Kube Namespace
Service Account Created in Kube Namespace
ClusterRole With Wildcard Created
K8s Secret Created
K8s Secret Deleted
The Change thread namespace rule: Add an additional exception for the Sysdig agent.
The Pod created in the Kube Namespace rule: Allow users starting with "system:" to create pods in the kube-system/kube-public namespaces.
The Read sensitive file untrusted rule: Allow puppet to run scripts that might read sensitive files.
The Write below root rule: Add an additional way to detect Cassandra to allow writes to /root/.cassandra.
The Change thread namespace rule: Allow Weaveworks Kured (Kubernetes Reboot Daemon) to change thread namespaces.
0.15.0
May 17, 2021
Rule Changes
Add rpmdb_verify as an RPM Package Management program. This affects the following rules:
Update Package Repository
Write below binary dir
Write below monitored dir
Write below etc
Read sensitive file untrusted
Modify binary dirs
Mkdir binary dirs
Run shell untrusted
Package management process ran inside container
Write below etc: Add haproxy-ingress as a program that can write below /etc/haproxy.
Change thread namespace: Allow images ending with /ext-cilium-startup-script to change namespaces.
Launch Suspicious Network Tool in Container: Allow images ending with sysdig/cassandra and bitnami/zookeeper to run network tools inside containers.
Set setuid or setgid bit: Allow the images in the sysdig_commercial_images list to include applications with setuid/setgid binaries.
0.14.0
May 05, 2021
Rule Changes
Add a macro to allow backward compatibility for using older pre-exceptions rules content.
0.13.2
May 05, 2021
Rule Changes
Remove the aws_cloudtrail rule named Create Internet-facing AWS Public Facing Load Balancer without Required Tags from the previous release that uses features yet to be released.
0.13.1
May 04, 2021
Added the Launch Root User Container rule to the Notable Container Activity policy.
Rule Changes
All Rules with the source, aws_cloudtrail: Switch from using jevt.value[/path] to aws.xxx to extract information out of aws_cloudtrail events.
A new rule, Launch Root User Container , has been added. It matches when a container is started and is configured to run as root. This works for Docker and CRI-O container runtimes, but not for Openshift 4.x, which does not make the necessary information available.
Macro spawned_process: Consider only successful executables. For example, where the return value is 0. This affects the following rules:
Schedule Cron Jobs
DB program spawned process
Run shell untrusted
System user interactive
Terminal shell in container
Program run with disallowed http proxy env
User mgmt binaries
Launch Package Management Process in Container
Netcat Remote Code Execution in Container
Launch Suspicious Network Tool in Container
Launch Suspicious Network Tool on Host
Search Private Keys or Passwords
Remove Bulk Data from Disk
Delete Bash History
Launch Remote File Copy Tools in Container
Detect crypto miners using the Stratum protocol
The docker client is executed in a container
Linux Kernel Module Injection Detected
Container Run as Root User
This could affect the following rules if they are triggered based on an exec() process rather than a container-started event.
Launch Privileged Container
Launch Sensitive Mount Container
Launch Disallowed Container
Launch Root User Container
0.13.0
April 09, 2021
Rule Changes
Restore several old macros and lists that are no longer used by any of the default rules, but might be used by some users' local rules.
0.12.2
April 05, 2021
Fixed a defect that could prevent deploying rules to several older Sysdig backend versions.
0.12.1
March 31, 2021
Rule Changes
Added new versions of falco_rules.yaml/k8s_audit_rules.yaml that uses exceptions instead of collections of macros and long condition strings. The rules coverage should be identical to older versions.
0.12.0
March 19, 2021
Fixed minor problems with the rules installation script.
0.11.1
March 11, 2021
Rule Changes
Added 164 rules that detect suspicious/anomalous/notable behavior from a stream of AWS CloudTrail events. This requires a Sysdig backend that supports policy types and running the Cloud Connector for Secure for cloud..
The new policy, Sysdig AWS Best Practices, includes 41 of the above rules that Sysdig recommends using for the AWS environments.
0.11.0
February 9, 2021
Rule Changes
rule Change thread namespace: Let cilium nsenter
rule Change thread namespace: Let dynatrace setns
rule Change thread namespace: Let sysdig agent setns (the process name was changed recently)
rule Clear Log Activities: Allow fluentd to write/access log files in a container
macro exe_running_docker_save: Added support for Crio setting up containers. This affects several rules including:
Modify Shell Configuration File
Update Package Repository
Write below binary dir
Write below monitored dir
Write below etc
Write below root
Write below rpm database
Modify binary dirs
mkdir binary dirs
Set Setuid or Setgid bit
Create Hidden Files or Directories
rule Launch Package Management Process in Container: Let sysdig node-image-analyzer run rpm
0.10.5
December 14, 2020
Rule Changes
Add a new rule, Container Run as Root User ,to the Inadvised Container Activity policy.
Add crio and multus to the user_known_change_thread_namespace_binaries list
0.10.4
December 1, 2020
Rule Changes
Ensure that falco_rules_local.yaml is evaluated against all the default files.
Ensure that the logs clearly show which files are being evaluated.
0.10.3
November 16, 2020
Rule Changes
Add the new rule, Linux Kernel Module Injection Detected, to the Notable Filesystem Changes policy.
Add the multipath_writing_conf macro as an exception in the Write below etc rule.
Add the chage_list macro as exception in the User mgmt binaries rule
Update compliance tags.
0.10.2
October 14, 2020
Add CSRF token protection.
Rule Changes
Add a new rule, Outbound Connection to C2 Servers, to the Disallowed Network Activity policy.
0.10.1
September 30, 2020
Rule Changes
Write below root: Similar to the rules that rely on a process name for exceptions, events will not be triggered if the process name is missing. For example, "".
Delete or rename shell history. Ignore docker programs that would prevent modifying shell history, when the path is expressed within the container filesystem (/.bash_history) and host filesystem (/var/lib/docker/overlay/.../.bash_history).
All Rules: Changes to the tags to add NIST 800-53 and SOC2 tags:
Renamed previous NIST 800-190 tags to use the prefix NIST_800-190_.
Fixed rule names for some Kubernetes rules.
0.10.0
September 23, 2020
Rule Changes
Launch Sensitive Mount Container: Change image matching to correctly identify Sysdig images as compared to names starting with "sysdig..."
Detect shell history deletion: Ignore paths below /var/lib/docker. For example, the container filesystem overlay images that are removed when a container is removed.
The Packet socket created in container rule is now enabled by default.
0.9.1
September 10, 2020
Rule Changes
All Rules: Add user.loginuid as an output field. This uid is generally unchanging across sudo/su commands, and can more reliably identify users.
Launch Privileged Container: Add additional images that can run with privileged=true.
Launch Sensitive Mount Container: Fix a typo that allows docker.io/sysdig/agent-slim to perform sensitive mounts.
Read sensitive file untrusted: Allow linux-bench to read sensitive files containing user information.
Update Package Repository: Restrict checks to files below known package management directories.
Write below etc: Add exceptions related to calico within containers.
Write below root: Allow mysqlsh write to /root/.mysqlsh .
Change thread namespace: Trigger only when the process name is known.
Create HostNetwork Pod: Allow several images related to GKE + default metrics/routing services run with hostnetwork=true.
Disallowed Kubernetes User: Add several known Kubernetes users to allowed list.
Pod Created in Kube Namespace: Allow several images related to GKE + default metrics/routing services run in kube-system/kube-public namespaces.
System ClusterRole Modified/Deleted: Allow modifications to the role system:managed-certificate-controller.
0.9.0
September 08, 2020
Added support for updating Falco rules across multiple accounts in an on-prem setup.
0.8.3
August 17, 2020
Rule Changes
Created a new rule, EphemeralContainers Created for the Suspicious K8s Activity policy.
Replace the endswith operator when checking with an image repository.
Whitelisted sysdig/agent and sysdig/agent-slim . They are not available with the open-source Falco Rules.
Whitelisted dockerd-current and docker-current in the exe_running_docker_save macro.
0.8.2
August 03, 2020
Rule Changes
Add the k8s_image_list list to the trusted_pod macro
0.8.1
July 27, 2020
Rule Changes
Move the Write below root rule from the Suspicious Filesystem Changes policy to the Notable Filesystem Changes policy
Delete the NIST 800-190 Application Container Security Guide policy
Delete the Payment Card Industry Data Security Standard (PCI DSS) policy
Add a new macro, user_read_sensitive_file_containers for the Read sensitive file untrusted rule
Add docker.io/falcosecurity/falco to the falco_privileged_images list
Add kubernetes-admin to the allowed_k8s_users list
0.8.0
July 20, 2020
Rule Changes
Disable Disallowed K8s Activity policy
Add placeholder macros for multiple rules
Fix the root_dir macro
Add snapd to the package_mgmt_binaries list
Add zmap to the network_tool_binaries list
Whitelist protokube, dockerd, tini, and aws in the change thread namespace rule
Add sysdig/agent-slim and sysdig/node-image-analyzer images to the user_trusted_containers macro
Add kube-apiserver-healthcheck to the allowed_k8s_users list
0.7.9
July 7, 2020
Remove unnecessary logging.
Add a new flag, --saas
0.7.8
July 1, 2020
Handle an improper error.
0.7.7
June 25, 2020
Disable rule Container Drift Detected (chmod) by default
0.7.6
June 23, 2020
Update rule Container Drift Detected (open+create) to avoid warning
0.7.5
June 22, 2020
Rule Changes
Added two new rules: Container Drift Detected (chmod) and Container Drift Detected (open+create) to policy Suspicious Container Activity
The Container Drift Detected (open+create) rule is disabled until an agent is released that supports the new evt.is_open_exec filter.
Updated macros bin_dir_mkdir and bin_dir_rename using evt.arg.path instead of evt.arg
Added placeholder macro user_known_write_below_binary_dir_activities to rule Write below binary dir
Fixed rule Anonymous Request Allowed to update the auth decision with ka.auth.decision=allow instead of ka.auth.decision!=reject
0.7.4
May 28, 2020
Rule Changes
Write below etc: Added lvs as a logical volume writing program that can write below /etc/lvm.
Clear Log Activities: Allowed additional Fluentd images to write to log file directories.
Set Setuid or Setgid bit: Added macro user_known_set_setuid_or_setgid_bit_conditionsthat makes it easier to add locally provided exceptions.
Launch Remote File Copy Tools in Container: Fixed the use of the list remote_file_copy_binaries so the list items are included.
The docker client is executed in a container: Now allow hcp-tunnelfront to run kubectl in containers.
Disallowed K8s User: Added vertical pod autoscaler programs as known Kubernetes users.
0.7.3
May 5, 2020
Rule Changes
For a brief time, Falco rules/macros had fields with k8s.* in them. These fields do not work in Sysdig Secure, so the relevant macros have been rewritten to omit them:
calico_writing_state
user_known_metadata_access
k8s_containers
user_known_k8s_client_container
0.7.2
May 1, 2020
Rule Changes
Add new rule Redirect stdout/stdin to network connection in container to policy Suspicious Container Activity
Add new rules Network Connection outside Local Subnet and Outbound or Inbound Traffic not to Authorized Server Process and Port to policy Suspicious Network Activity
Add new rules K8s Secret Created and K8s Secret Deleted to policy All K8s Object Modifications
Add rules Untrusted Node Successfully Joined the Cluster and Untrusted Node Unsuccessfully Tried to Join the Cluster to policy Suspicious K8s Activit
Add rule Full K8s Administrative Access to policy Suspicious K8s User Activity
Add rule Ingress Object without TLS Certificate Created to policy Inadvised K8s Activity
Check dsc_host in macro ms_oms_writing_conf
Add macros mcafee_writing_cma_d and avinetworks_supervisor_writing_ssh as exceptions in rule Write below etc
Add macro runc_writing_exec_fifo as exception in rule Write below root
Use "pmatch" instead of "in" operator to check known files under root directory
Update rule Change thread namespace to check exit event only
Add macro known_system_procs_network_activity_binaries for rule System procs network activity
0.7.1
April 9, 2020
Rule Changes
Add PCI/NIST tags to the following rules:
Disallowed SSH Connection
Unexpected outbound connection destination
Unexpected inbound connection source
Write below binary dir
Write below monitored dir
Write below etc
Write below root
Read sensitive file untrusted
DB program spawned process
Modify binary dirs
Mkdir binary dirs
Change thread namespace
Launch Privileged Container
Launch Sensitive Mount Container
Launch Disallowed Container
Terminal shell in container
Unexpected UDP Traffic
Create files below dev
Contact K8S API Server From Container
Unexpected K8s NodePort Connection
Search Private Keys or Passwords
Clear Log Activities
Create Symlink Over Sensitive Files
Detect crypto miners using the Stratum protocol
Write below etc:
Add "dsc_host" as a MS OMS program
Let McAfee write to /etc/cma.d
Let AVI Networks supervisor write somessh cfg files
Allow writes to /etc/pki from OpenShift secrets dir
Write below root:
Let runc write to /exec.fifo
Change thread namespace
Only allow Kubernetes/Docker programs to use setns directly on the host
Let children of kubelet/hyperkube use setns
Run shell untrusted
Let Puma reactor spawn shells
Detect outbound connections to common miner pool ports
When attempting to resolve crypto mining hostnames, exclude hosts that resolve to localhost/rfc1918 ips
Default Policy Changes
Remove the default Policy Launch Privileged Container.
The rule it used is also in the existing default policy Inadvised Container Activity, so there's no change in rule coverage.
New default policies Payment Card Industry Data Security Standard (PCI DSS) and NIST 800-190 Application Container Security Guide, which are disabled by default, contain rules specifically related to PCI and NIST standards.
0.7.0
Dec 9, 2019
Expand allowed_k8s_users list with default users created by Kops
Add macro calico_writing_envvars to whitelist of rule Write below etc
Update operators with intersect
Add calico/node in the falco_privlieged_image list
Add amazon/amazon-ecs-agent in falco_sensitive_mounts_image list
Add hyperkube to the whitelist of rule
Set Setuid or Setgit bit
Add docker-runc-cur to container_entrypoint macro
Add a rule to detect Kubernetes client tool in container
Add rules Contact cloud metadata service from container and Packet socket created in container to policy Suspicious Container Activity
Update macro exe_running_docker_save
Add exe_running_docker_save as exception to rules Modify Shell Configuration File, and Update Package Repository
Create macro automount_using_mtab and add it as exception to rule Write below etc
Update macro k8s_api_server with Kubernetes headless service name
Add placeholder macro user_known_package_manager_in_container to rule Launch Package Management Process in Container
Add kubelet to list user_known_chmod_applications
Create macro user_known_k8s_client_container and add it as exception to rule The docker client is executed in a container
Add more directories to Sensitive mounts rules
0.6.0
Oct 9, 2019
Add rule Delete or rename shell history (a better version of Delete Bash History) to policy Suspicious Filesystem Changes
Add rule Detect crypto miners using the Stratum protocol to policy Suspicious Container Activity
Add a new policy, Access Cryptomining Network ,with a new rule Detect outbound connections to common miner pool ports associated (disabled by default)
Add new macros chmod and modify_repositories
Enhance rules Update Package Repository, Set Setuid or Setgid bit, and Create Hidden Files or Directories
Add imagefluent/fluentd-kubernetes-daemonset to macro trusted_logging_images
0.5.0
Aug 21, 2019
Update rule Update Package Repository with modify action
Update rule Delete Bash History with more bash history files
Update rule Set Setuid or Setgid bit using system calls instead of process name
Update rule Create Hidden Files or Directories with modify action
0.4.9
Aug 1, 2019
Add /exec.fifo to known_root_files macro (GKE)
Add macro amazon_linux_running_python_yum as exception in rule Write below rpm database (Amazon Linux 2)
Add docker.io/google/cadvisor and docker.io/prom/node-exporter to list falco_sensitive_mount_images
0.4.8
July 23, 2019
Add image k8s.gcr.io/kube-proxy to list falco_privileged_images
Add runc to macro container_entrypoint
Add macro trusted_logging_images for rule Clear Log Activities
Add image docker.io/netdata/netdata to list falco_sensitive_mount_images
0.4.7
July 1, 2019
Add placeholder for user macro
Add rfc 1918 addresses
Add image prometheus-node-exporter to macro openshift_image
Add weaveworks_scope macro used by rule Change thread namespace
0.4.6
June 20, 2019
Add whitelist to rules Change thread namespace and Non sudo setuid
0.4.5
June 17, 2019
Add trusted_container macro back
0.4.4
June 13, 2019
Extend macro mkdir with syscall mkdirat
Add placeholder for whitelist in rule Clear Log Activities
Add docker.io/ to the trusted images list
Add container.id and image in the rule output, except those rules with "not container" in condition
0.4.3
June 6, 2019
Remove image check from rancher_write_conf macro
Remove healthcheck from rancher_writing_conf
Update nginx_writing_conf macro
0.3.7
June 5, 2019
Updated macro container_started
IBM Cloud Kubernetes Service is a hosted Kubernetes from IBM
Allow Ansible to run using Python 3
Fix egrep rule and ncat rule
Add Sematext Monitoring & Logging agents to trusted Kubernetes containers
0.3.6
May 30, 2019
Add rules: remote file copy in container, create symlink over sensitive files
In macro prometheus_conf_writing_conf, use startswith instead of =
0.3.5
Apr 18, 2019
Add MITRE tags to existing rules
Add new MITRE rules mainly for persistence category
0.3.4
7 -
Enhanced Metric Store
Sysdig has launched our next generation metric store, introducing a number of new features, as well as changes to and removal of some features in Sysdig Monitor. This document covers the major enhancements and changes introduced by the metric store.
New Features and Enhancements
Prometheus-Compatible Naming Conventions for Metrics & Labels
In prior versions of Sysdig Monitor, metrics were inconsistent between PromQL and Form querying. This behavior has been changed. Metrics are now unified — all the metrics are now presented in a Prometheus compatible naming convention, as opposed to the previous statsd compatible naming convention. For example, underscore is used instead of dot notation as given below:
kubernetes.node.allocatable.cpuCores will be mapped to kube_node_status_allocatable_cpu_cores and kubernetes.namespace.name to kube_namespace_name.
Your existing dashboards, alerts and notifications will be automatically migrated to the new naming convention. Sysdig APIs support metrics and labels in both old and new naming conventions. Note that for the initial release, Labels will not be migrated to the new naming convention in the old explore, events, and team settings.
Notifications sent via alerts (webhooks, PagerDuty, etc) will use the new label/metric convention. If you are performing further processing to parse the metric or label names within these notification messages please update your scripts as appropriate.
If you have any concerns or questions regarding this mapping or you feel like you need more time to adjust you notification tools, please contact Sysdig Support.
Metrics such as cpu.used.percent previously would either be showing values from a process, container, or host depending on your query segmentation or scope. This has been improved by creating new sets of context-specific metrics and resource specific semantics of Prometheus naming convention. For example:
Network metrics previously would either be showing values from a host, container, program, or connection depending on your query segmentation or scope. This has been improved by creating also a new sets of context-explicit metrics, in this case also per connection metrics:
Your existing dashboards, alerts and notifications will be automatically migrated to the new naming convention. Sysdig APIs support metrics in both old and new naming conventions.
Queries now perform faster and handle larger volumes of data. You can expect queries executed in Sysdig Monitor to be noticeably faster.
Single Stat Panels Displays Latest Value
Number panels, tables, histograms, and toplist panels can now show the latest value for an entity. This can be done without having to aggregate multiple values over the time selection.
Overview Displays Latest Data
Overview pages now shows the latest data as opposed to an aggregated value for widgets over the time window selected. Time navigation has been removed to focus this view on the live (latest) status of your infrastructure.
Scope Variable in PromQL Dashboard
You can easily reference a dashboard scope in PromQL queries. To do so, use the reserved $__scope variable as shown below:
Under the hood $__scope will be substituted with the expression specified in the dashboard scope. This is achieved by leveraging Sysdig ServiceVision technology which allows for automatically enriching metrics with Kubernetes and application context. Learn more about ServiceVision.
Mixed-Metric Granularity
Sysdig Monitor can now display metrics scraped at different intervals, for example 10s and 1m, on the same graph.
Improved Granularity for PromQL panels
Granularity of graphs has been improved for promQL panels. For example, a 1 hour selection now shows metrics with 10 second intervals. In prior versions, 1-hour selection in Dashboards showed metrics in 1-minute interval.
Removed Re-Alignment
Previously, Sysdig Monitor would re-align time selections in graphs due to certain performance limitations. This time re-alignment has been removed to show more up-to-date metrics.
Troubleshooting Metrics
Troubleshooting metrics (program metrics, connection-level network metrics, and Kubernetes troubleshooting metrics) are being reported on a granular level at 10s and will be stored for 4 days. For the list of troubleshooting metrics and the labels that you can use to segment them, see Troubleshooting Metrics.
Discontinued Features
Discontinued Metrics and Labels
Below is the list of metrics and labels that are going to be discontinued. We made an effort to not deprecate any metrics or labels used in existing alerts, but in case you encounter any issues please contact us.
It is important to note that we have applied automatic mapping of all net.*.request.time.worst metrics to net.*.request.time, as max aggregation gives equivalent results and it was almost exclusively used in combination with these metrics.
Discontinued Metrics
The following metrics are no longer supported:
net.request.time.file
net.request.time.file.percent
net.request.time.local
net.request.time.local.percent
net.request.time.net
net.request.time.net.percent
net.request.time.nextTiers
net.request.time.nextTiers.percent
net.request.time.processing
net.request.time.processing.percent
net.request.time.worst.in
net.request.time.worst.out
net.incomplete.connection.count.total
net.http.request.time.worst
net.mongodb.request.time.worst
net.sql.request.time.worst
net.link.clientServer.bytes
net.link.delay.perRequest
net.link.serverClient.bytes
Discontinued Labels
The following labels are no longer supported:
net.connection.client
net.connection.client.pid
net.connection.direction
net.connection.endpoint.tcp
net.connection.udp.inverted
net.connection.errorCode
net.connection.l4proto
net.connection.server
net.connection.server.pid
net.connection.state
net.role
cloudProvider.resource.endPoint
host.container.mappings
host.ip.all
host.ip.private
host.ip.public
host.server.port
host.isClientServer
host.isInstrumented
host.isInternal
host.procList.main
proc.id
proc.name.client
proc.name.server
program.environment
program.usernames
mesos_cluster
mesos_node
mesos_pid
In addition to this, composite labels ending with the ‘.label’ string will no longer be supported. For example kubernetes.service.label will be deprecated, but kubernetes.service.label.* labels will continue to be supported.
Removed Featurees
Topology Maps
Topology Maps will be deprecated due to their incompatibility with the new data store and had limitations at scale for certain users.
Agent Percentiles
Agent derived percentiles will be deprecated. If you have been using these, your query will stop working and you will have to manually migrate your queries to leverage Prometheus histograms or PromQL functions such as histogram_quantile to achieve more precise results.
Change in Functionality
Usage of Labels in Table Panel
Querying labels as metrics is limited to Infrastructure labels. For example, you can use all the host level labels (for example, agent tags), aws tags (for example, region) and the Kubernetes labels (for example, workload) to build table panels.
Aggregated Data for Non Timecharts
Due to the underlying changes we made to our core metric ingestion engine, charts that are not Timecharts (e.g. Number panels) will sometimes not display aggregated data for the full requested time range. In this case, we will
aggregate a portion of data. This will in all cases be no less than across the span of 2 weeks
clearly define the time range for which we were able to aggregate in the warning message. It is important to note that this is a transient side effect, and will be less likely to happen over time.
Contact Us
If you have any questions or comments about these changes, feel free to Sysdig Support or contact your Sysdig representative.
