2022 Archive

2022 Archive of Sysdig Monitor (SaaS) release notes.

December 12, 2022

Integrate with Azure Cloud Metrics

Sysdig now supports Azure Cloud Metrics. Sysdig Monitor now can ingest metrics directly from Azure, allowing you to fully integrate all your existing Azure service metrics into Sysdig Monitor. For more information, see Azure Account

If you wish to monitor Azure Resource Quotas, you must manually enable that feature by using the Sysdig API (until this option is available in the Cloud Metrics Integrations UI. To learn about enabling pulling Azure Resource Quotas in your Sysdig Monitor account, see Monitor Azure Resource Quotas.

AWS Lambda Telemetry API Support for Sysdig Monitor

Sysdig has rolled out preview availability of the new Sysdig Monitor Lambda Extension for AWS Lambda Telemetry API. This new Lambda extension allows Sysdig Monitor users to consume metrics directly from Lambda events as functions are executed, bypassing the need to route Lambda metrics through another platform such as AWS CloudWatch.

The normal way Lambda users receive function metrics is by connecting Lambda to AWS CloudWatch. The Sysdig Monitor users can then consume the pre-configured metrics from Lambda using the CloudWatch API/Streams integration but with a certain latency CloudWatch routing entails and collecting the extra metrics which may not be necessary. With the Sysdig Monitor Lambda Extension for AWS Lambda Telemetry API, you can consume the most critical function execution metrics with an up to 85% reduction is metrics ingestion latency.

For more information, see AWS Lambda.

PromQL Query Inspector

Query Inspector helps you understand the underlying causes of a No Data message in Dashboards. For more information, see Query Inspector.

Support for New KSM Metrics

Sysdig Monitor supports the following:

KSM ingress metrics

  • kube_ingress_info
  • kube_ingress_labels
  • kube_ingress_created
  • kube_ingress_path
  • kube_ingress_tls

KSM certificate signing request metrics

  • kube_certificatesigningrequest_created
  • kube_certificatesigningrequest_condition
  • kube_certificatesigningrequest_labels
  • kube_certificatesigningrequest_cert_length

Taint metrics

  • kube_node_spec_taint

Monitoring Integrations

Integrations

  • Added the following integrations:

    • AWS Lambda
    • AWS MetricsStream CloudFront
    • Azure API Management
    • Azure Synapse Analytics
    • Azure AKS
    • Azure Cluster AutoScaler
    • Azure Blob Storage
    • Azure Files
    • Azure Queue Storage
    • Azure SQL
    • Azure Storage Accounts
    • Azure Table Storage

Dashboards and Alerts

  • Added a new alert to Redis to test lack of data alerts.
  • Add a new alert to detect exporter down in alerts templates
  • Removed deprecated storage metrics from alerts library
  • The Event Feed now displays tags associated with Custom Events

Richer Query Syntax for Events

Queries in the Event Feed and Event Overlay now support a richer query syntax.

November 08, 2022

New Advisories

The following new Advisories have been introduced:

  • Cluster pod capacity - cluster is reaching pod capacity, when this happens new pods cannot be scheduled.
  • Replicas unavailable - a workload has unavailable replicas which can affect app availability
  • Cluster CPU overcommitment - cluster is overcommitting CPU which may affect availability
  • Cluster memory overcommitment - cluster is overcommitting memory which may affect availability

Filtering AWS Cloudwatch Metric Streams

Sysdig now provides you the ability to filter (drop) metrics that are coming from AWS CloudWatch Metric Streams via Kinesis Firehose, providing our AWS users full control over what metrics are coming from Streams are ingested and stored by Sysdig Monitor. AWS currently does not offer the ability to filter CloudWatch Streams metrics that are pushed to an endpoint like Sysdig Monitor. With CloudWatch Steams Metrics Filtering, you can now choose to only ingest and store the metrics that are important for you, on a per-service basis, thereby, reducing the data storage cost. You can include or exclude specific metrics from individual AWS namespaces as they are ingested.

Dashboard Enhancements

  • Minimum Interval for PromQL Queries: You can now define a minimum interval for PromQL Queries, which is handy when working with scarce metrics. For more information, see Define Minimum Interval for PromQL Queries.
  • Bulk Delete Dashboards: Dashboard Manager now gives you the ability to bulk delete dashboards. see Dashboard Manager.

Alert Enhancements

When a metric stops reporting data, you now have the option to ignore or notify on the notification channel associated with the alert threshold.

Notification Channels

Sysdig now allows you to refine which sections are used when sending a Slack notifications. See Customize Notifications.

Monitoring Integrations

Integrations

  • Added the following integrations:

    • OpenShift 4 Scheduler
    • OpenShift 4 Controller Manager
    • OpenShift 4 API Server
    • OpenShift 4 Kubelet
    • Azure Virtual Machines
    • Azure Virtual Machine Scale Sets

  • Enable OpenShift CoreDNS job

  • Add support for OpenShift in Fluentd integration

  • Update the postgresql-exporter and elasticsearch-exporter images with critical vulnerability fixes

Dashboards and Alerts

  • Added openshift-api scopes in OpenShift v4 API Server Dashboard
  • Added the minimum interval option in AWS MetricsStream dashboard templates

September 29, 2022

Mapping IdP Groups to Roles and Teams

The IdP (Identity Provider) integration has been improved by supporting the ability to map groups to roles and teams.

  • IdP group can be mapped to a single role and one or more teams
  • Only team users can be mapped. No support for admin users at the moment.
  • SAML 2.0 is supported

For more information, see Group Mappings.

SAML Single Logout

SAML single logout, the facility to terminate multiple Sysdig user sessions simultaneously, is now available on all the regions. Furthermore, Sysdig now supports Okta for SAML single logout in federated authentication environments.

Case Sensitive Labels in PromQL Queries

To comply with PromQL specification for filtration expressions, label names in PromQL filtering expressions in Sysdig Monitor will be case sensitive. If the casing of a filtering label is incorrect, the query will return an empty response.

As majority of PromQL queries were crafted using auto-complete for existing label names, changes to the label casing will not have a major impact. However, there could be rare cases where auto-complete is ignored or PromQL queries are crafted via API, which should be reviewed to make sure casing is correct.

For example:

If the given label name is ‘host_hostname’, and you want to match the time series of the ‘sysdig_host_cpu_used_percent’ metric to the host ‘foo’, the correct query would be:

sysdig_host_cpu_used_percent{host_hostname='foo'}

Previously, both the following queries returned results.

sysdig_host_cpu_used_percent{HOST_hostname='foo'}
sysdig_host_cpu_used_percent{HoSt_HoStNamE=‘foo'}

The following are unimpacted by this change:

  • The alert and dashboard queries created by using the Form UI

  • Label values

    Label values are already case sensitive in Sysdig Monitor.

Google Chat Integration

You can now use Google Chat as a notification channel in Sysdig Monitor. See Configure a Google Chat Channel for more details.

Stacked Bars

Timechart panels support creating statcked bar charts. For more information, see Timechart.

Monitoring Integrations

Rename Dashboard Templates to Dashboard Library

In order to align with the rest of Monitor, Dashboard Templates has been renamed to Dashboard Library.

Integrations

  • Added the following integrations:

    • OpenShift API Server
    • Openshift 4 CoreDNS
    • Openshift 4 etcd
    • Calico
    • Cassandra

  • Split the k8s-control-plane integration to different integrations per application

  • Improved the Troubleshooting guide by removing scope from the promQL queries.

Dashboards and Alerts

  • Added OpenShift v4 API Server dashboard including the openshift-api scopes
  • Made Etcd and CoreDNS dashboards compatible with Kubernetes and Openshift v4 (both OKD and ROKS)
  • Changed the AWS Metrics ECS MetricStream template to include ECS in the name

Promcat.io

  • Updated Cassandra integration details with JMX exporter

August 17, 2022

New Permission for Changing Team Roles

Team management has been improved with the addition of the new permission, Team Membership Roles. This new permission will allow you to change the roles of team members separately while adding users to the teams.

For more information, see:

August 08, 2022

Advisor

Accelerate Troubleshooting by Up to 10x with Advisories

Advisories evaluate the thousands of data points being collected by the Sysdig agent, and displays a prioritized view of key problems in your infrastructure that affect the health and availability of your clusters and the workloads running on them.

See Sysdig Advisor: Making Kubernetes troubleshooting effortless on the Sysdig blog.

Entire Infrastructure Overview

Entire Infrastructure shows an aggregated view of all Advisories, active alerts, events, and a quick snapshot of the state of your Kubernetes infrastructure. This is shown before selecting a cluster or workload, and is the new default landing page of the Monitor product.

Display Improvements

Display and representation of data has been improved, including the use of new panel types. Information such as workload availability or resource limits are now displayed as a table instead of a chart.

Dashboard Manager

Sysdig introduces Dashboard Manager to organizes all the dashboards associated with your account. The page acts as the repository for all the dashboards that you have created, that your teams have shared with you, and that you have marked as favorite, as well as the dashboard templates available to you.

For more information, see Dashboard Manager.

Prometheus Alertmanager Notifications

You can now integrate Prometheus Alertmanager as a notification channel in Sysdig Monitor. See Prometheus Alertmanager Notifications for more details.

Contextual Tooltip

The Contextual Tooltip has been enhanced to display all segments. To enable this feature, toggle the Contextual Tooltip in Dashboards in the Settings > User Profile screen. The option is found under the Beta Features section.

Enhanced Label Selector

The label selector in Dashboards and Metrics Explorer has been enriched with the following sought after features:

  • Label documentation
  • Preview of label values
  • Suggested labels

New PromQL Variables

The following PromQL variables have been added:

  • $__interval_sec
  • $__range_sec

They are used for translating the rate time aggregation in a Form query into a PromQL query. For example:

avg(sum_over_time(sysdig_container_cpu_used_percent{$__scope}[$__interval])) / $__interval_sec

For more information, see Using PromQL.

Events Feed Enhancements

The Events module has been refreshed to show metrics and labels in Prometheus notation.

Monitoring Integrations

Integrations

  • Added the following integrations:

    • HAProxy Openshift integration
    • Istio integration

  • Removed metrics filtering in envoy job in Istio agent configuration. This will allow for collecting other custom metrics merged into the Envoy sidecar.

  • Enhanced the OpenShift HAProxy configuration to use ClusterRole

  • Added the following to Promcat.io:

    • HAProxy Openshift 4.7
    • Istio 1.14

Dashboards and Alerts

  • Enhanced RDS description for PostgreSQL

  • Enhanced the calculation of used vs request/limits in Kubernetes Capacity Planning Dashboard

  • Enhanced promQL in Kubernetes Dashboards to avoid operations occuring in ephemeral containers

  • Added updated Time Series Usage Dashboard Template to the repository

  • Removed the deprecated ‘OutOfDisk’ condition on Node Status and Performance Dashboard

  • Updated Kubelet metrics for Kubernetes v1.19 and above in Dashboard Templates

    • kubelet_running_container_count to kubelet_running_containers
    • kubelet_running_pod_count to kubelet_running_pods

  • Removed duplicated Dashboard Templates

Exporter

  • Upgraded exporters Jenkinsfile for scratch and ubi images
  • Fixed the error in JMX exporter image.
  • Fixed port information in Memcached exporter scratch image.
  • Added the following Security updates in UBI images of all the exporters:

    • Apache

      quay.io/sysdig/apache-exporter:v0.11.1-ubi

      quay.io/sysdig/apache-exporter:v0.11.1

    • Elasticsearch

      quay.io/sysdig/elasticsearch-exporter:v1.3.4-ubi

      quay.io/sysdig/elasticsearch-exporter:v1.3.4

    • Grok

      quay.io/sysdig/sysdig/grok-exporter:v1.0.4-ubi

      quay.io/sysdig/sysdig/grok-exporter:v1.0.4

    • JMX

      quay.io/sysdig/promcat-jmx-exporter:v0.17.3-ubi

      quay.io/sysdig/promcat-jmx-exporter:v0.17.3

    • Memcached

      quay.io/sysdig/memcached-exporter:v0.10.2-ubi

      quay.io/sysdig/memcached-exporter:v0.10.2

    • MongoDB

      quay.io/sysdig/mongodb-exporter:v0.11.9-ubi

      quay.io/sysdig/mongodb-exporter:v0.11.9

    • MySQL

      quay.io/sysdig/mysql-exporter:v0.14.1-ubi

      quay.io/sysdig/mysql-exporter:v0.14.1

    • NGINX

      quay.io/sysdig/nginx-exporter:v0.10.1-ubi

      quay.io/sysdig/nginx-exporter:v0.10.1

    • Node exporter

      quay.io/sysdig/node-exporter:v1.2.4-ubi

      quay.io/sysdig/node-exporter:v1.2.4

    • NTP

      quay.io/sysdig/ntp-exporter:v2.0.4-ubi

      quay.io/sysdig/ntp-exporter:v2.0.4

    • PHP-FPM

      quay.io/sysdig/php-fpm-exporter:v2.3.2-ubi

      quay.io/sysdig/php-fpm-exporter:v2.3.2

    • PostgreSQL

      quay.io/sysdig/postgresql-exporter:v0.10.8-ubi

      quay.io/sysdig/postgresql-exporter:v0.10.8

    • Redis

      quay.io/sysdig/redis-exporter:v1.43.1-ubi

      quay.io/sysdig/redis-exporter:v1.43.1

July 13, 2022

Integrate AWS CloudWatch Metric Streams

Sysdig has rolled out support for AWS CloudWatch Metric Streams. Based on Kinesis Firehose, AWS CloudWatch Metric Streams is a real-time metrics aggregation and delivery tool for AWS cloud services. Sysdig Monitor now can ingest metrics directly from Kinesis Firehose, allowing you to fully integrate all your existing AWS service metrics into Sysdig Monitor. Configuring AWS CloudWatch Metric Streams to send metrics to Sysdig can either be done by using the AWS CloudFormation template available directly on the Monitor UI, by manually deploying the CloudFormation template, or by manually selecting Sysdig as an HTTP receiver through the AWS Kinesis Fire configuration.


In addition, we have also released 9 out-of-the-box dashboards and alerts for the following AWS CloudWatch Metric Streams services:
  • AWS ALB
  • AWS EBS
  • AWS ELB
  • AWS Fargate
  • AWS Lamda
  • AWS RDS
  • AWS S3
  • AWS SQS

For other services, custom dashboards and alerts can be configured for all the service metrics coming in from AWS CloudWatch Metric Streams.

For more information, see Cloud Integrations.

July 06, 2022

Live Logs

Sysdig introduces Live logs support for Kubernetes in Advisor to help you debug infrastructure problems. Advisor displays live logs for a container, which is the equivalent of running kubectl logs. This strengthens Sysdig Monitor capabilities for troubleshooting, allowing you to debug problems, such as pods in a CrashLoopBackOff state and consolidates tooling, and reducing the need to switch to other tools for troubleshooting and root cause analysis.

Live logs requires Sysdig agent v12.7.0 or above. For more information, see Live Logs.

Enhanced Alerts Editor

Sysdig introduces a new Alert Editor with an improved user experience thanks to a redesigned look and feel. We’ve also added the ability to link a dashboard and a runbook to the alert definition to expedite troubleshooting.


We are deprecating the existing Anomaly Detection and Group Outlier alert types. Previously created alerts of this type can still be viewed and edited. We will be bringing new alert types in the future.

The new Alerts Editor will be available only in environments where the new metric store is enabled. For more information, see Alerts.

PromQL Panel Enhancements

The Compare To function is now supported in Timechart and Number PromQL panels.

Monitoring Integrations

Integrations

  • Added the following integrations:
    • HaProxy
    • PHP-fpm
  • Split Kubelet PVC-and-Storage integration into two different integrations, PVC and Storage.
  • Enabled Kubelet-PVC metrics by default.
  • Updated agent jobs for kube-controller-manager and kube-scheduler to support HTTPS and authentication.
  • Added Helm chart for ElasticSearch exporter with CA certificates option.

Dashboards and Alerts

  • Added dashboard and alert templates for HAProxy
  • Changed the rules to toggle showing Kubernetes dashboards to prevent hiding when encountering unstable metrics or disconnected agents
  • Fixed waiting time in Portworx alert templates with predict linear functions
  • Fixed used request in the Cluster Capacity Planning dashboard

Exporter

  • New exporter image for PHP-FPM:
    • quay.io/sysdig/php-fpm-exporter:v2.3.0
    • quay.io/sysdig/php-fpm-exporter:v2.3.0-ubi
  • Updated the JMX exporter image
    • quay.io/sysdig/promcat-jmx-exporter:v0.17.0
    • quay.io/sysdig/promcat-jmx-exporter:v0.17.0-ubi

June 7, 2022

Enhanced Metric and Label Selection

The metric and label selectors in Dashboards and Metrics Explorer have been improved to provide easier search and find what you are looking for.

Improvements include:

  • Suggested labels now show only relevant labels for a selected metric.
  • Displays 500 labels by default for a selected metric. Previously it was 50.
  • Supports inline editing of metric and label names.
  • Provides improved search relevancy.

Dashboard enhancements

Translate Form-Query to PromQL

You no longer require advanced Prometheus knowledge to build complex PromQL queries in Sysdig Monitor. With a single click, you can translate form query to PromQL, and build PromQL-based dashboards in no time. For more information, see Build PromQL Panels from Form Query.

PromQL Support for Toplist

Toplist panels support running PromQL queries.

Multi-Query Support for Stacked Area Charts

Timechart now supports visualizing multiple queries as stacked areas in the same y-axis.

With this feature, it’s easier to visualize and compare sparse metrics.

Lazy Loading of Dashboard Panels

Dashboards now supports lazy loading panels. Lazy loading greatly reduces the initial page loading time by only loading panels once they become visible on screen.

Monitoring Integrations

Integrations

  • Added the following integrations:

    • Fluentd
    • NTP

  • Improved CoreDNS Prometheus job to be detected in IKS clusters

  • Changed troubleshooting metrics in some integrations for metrics inside the filter of the Prometheus job

Dashboards and Alerts

  • Added the following templates for dashboard and alert:

    • Fluentd
    • NTP

  • Changed OOTB K8s dashboards to use “is” vs “in” scoping to improve performance.

  • Changed the following dashboards:

    • Cluster/Namespace Available Resources
    • Cluster Capacity Planning
    • Pod Rightsizing & Workload Capacity Optimization
    • Pod Scheduling Troubleshooting
    • Kubernetes HPA

  • Added the containers with limits/requests only in certain panels in the Cluster Capacity Planing dashboard

  • Limited the use of the label job to some panels in the Kubernetes CoreDNS dashboard

Exporters

  • Added support for CA files in ElasticSearch exporter Helm chart
  • Removed duplicated securityContext in ElasticSearch exporter Helm chart
  • Changed the ElasticSearch wizard and Helm chart to use secrets for URL of the ElasticSearch server
  • Bumped Helm chart repository version to include NTP exporter and fixes in Elasticsearch
  • The following Exporter images for NTP exporter have been added:
    • quay.io/repository/sysdig/ntp-exporter:v2.0.3
    • quay.io/repository/sysdig/ntp-exporter:v2.0.3-ubi
  • New version of grok exporter with security updates:
    • quay.io/sysdig/grok-exporter:v1.0.2
    • quay.io/sysdig/grok-exporter:v1.0.2-ubi

May 23, 2022

Custom Roles

A custom role is an admin-defined role that allows Sysdig administrators to bundle a set of permissions and assign those permissions to individual users or teams. Custom roles allow for finer-grained definition beyond the standard out-of-the-box Sysdig Roles. Once defined, a custom role can be assigned to any user inside a particular team, and also be configured as the default role for new users in that team. For more information, see Custom Roles.

The addition of custom roles into the platform is transparent, meaning that standard roles and assignments that already exist will not experience any changes.

May 4, 2022

Sysdig Platform Audit

We are glad to announce that Sysdig Platform now supports the capability of tracking, logging, and reporting on all changes in the system.

  • Track all activities on the API level
  • Retention period: 90 days
  • Simple API for retrieving audit information (no UI)
  • Events Forwarding support to be included in the near future (to be announced)
  • Enabled by default for all SaaS users

See Sysdig Platform Audit for more information.

Sysdig Platform Login Banner

We would like to announce that Sysdig Monitor and Secure now allow you to define a Login Message that will be presented to all users. Added to boost Sysdig compliance/enterprise readiness, requested originally by the IRS.

  • Users are not allowed to access the system until they acknowledge the message
  • One login banner per account
  • Only Admin users can enable/update the message
  • Single banner for both Monitor and Secure (for Platform customers)
  • Available on SaaS for all users

See Configure Login Message for more information.

April 13, 2022

Advisor

Advisor brings your metrics, alerts, and events into a focused and curated view to help you operate and troubleshoot Kubernetes infrastructure. To help you solve problems faster, over time, Advisor will surface your infrastructure issues that you should pay attention to. For more information, see Advisor.

Metrics Explorer

Metrics Explorer has been rebuilt from the ground up to focus on advanced metric exploration and querying.


Improvements to Metrics Explorer include:
  • Simple querying that builds PromQL queries under the hood. Metrics Explorer is the easiest way to build PromQL queries.
  • Graph multiple metrics at once for correlation. For example, CPU usage vs Kubernetes limits.
  • Queries are ungrouped by default, showing the individual time series for a metric. This allows you to spot any problems faster. For example, 1 of 50 Cassandra nodes with high pending compactions. Instead of segmenting, you now group by one or more labels, for example, workload, pod, and container.
  • When selecting a scope in the tree, only those metrics that are applicable to that entity are displayed.
  • Metrics are now more logically categorized by metric namespace (prefix).
  • Resolution has been improved. For example a 1-hour view now shows 10-seconds data. Additionally, the concept of time re-alignment has been removed.

For more information, see Explorer.

February 10, 2022

Improved Usability with New Navigation

The Sysdig Monitor UI has been enhanced to provide you with a smoother and smarter left-hand navigation experience.

Check out a video walk-through of the new feature!

  • Collapsible main menu: Allows you to toggle the visibility of menu options. The collapsible left-hand navigation prevents long lists from displaying by default and gives you a clear structure that is easy to scan and locate.

  • Hoverable sub-menu: With each module that has additional menu options, hover over the respective module to quickly navigate.

New Menu Option for Integrations

A dedicated Integrations menu option provides an easy way to access both inbound and outbound integrations with Sysdig.

  • Inbound: Access Monitoring Integrations quickly and understand which applications and services are running. You can also manage your AWS Account and review the Sysdig agent installation.
  • Outbound: Manage the Notification Channels and S3 Capture Storage.

Revamped User Menu

The User menu provides the following:

  • Option to efficiently switch between Sysdig Teams.
  • Access Management to the Administrator.
  • Sysdig API Tokens to the authenticated user.
  • Documentation and What’s new links

The Settings sub-menu link is provided to review all the available options for the current user.

January 26, 2022

Support for PVC Metrics

Contact your Sysdig representative or Sysdig Support to enable PVC metrics in your environment.

With Sysdig agent v12.2.0 or above installed in your monitoring environment, Sysdig Monitor can help you surveil your Kubernetes PV/PVCs objects. Use the PVC dashboard and alert templates to get an insight into your PV usage, such as disk usage, inodes, storage latency, errors, and so on.

For more information, see Configure PVC Metrics.

New KSM Troubleshooting Metrics

Sysdig provides the following new troubleshooting metrics:

  • kube_workload_pod_status_phase
  • kube_workload_pod_status_reason
  • kube_pod_status_unschedulable
  • kube_pod_container_status_waiting
  • kube_pod_container_status_waiting_reason
  • kube_pod_container_status_terminated
  • kube_pod_container_status_terminated_reason

These metrics give insights into why pods are stuck or crashing (CrashLoopBackOff, OOMKilled, DeadlineExceeded etc.). To support this:

  • The Kubernetes Alerts Library has been updated to provide additional alerts for errors such as CrashLoopBackOff.

  • New panels has been added to the Kubernetes Workload Status & Performance dashboard.

In environments running older versions of Sysdig agent, the Kubernetes Dashboards will display a banner prompting you to upgrade to agent v12.2.0 or above for these metrics to be automatically collected.