Alerts

Alert is the responsive component of Sysdig Monitor. Alerts notify you when an event or issue occurs that requires attention. Events and issues are identified based on changes in the metric values collected by Sysdig Monitor. The Alerts module displays out-of-the-box alerts and a wizard for creating and editing alerts as needed.

Alert Types

You use the Alert Editor to create alerts. The types of alerts available in Sysdig Monitor are:

  • Downtime: Monitor any type of entity, such as a host, a container, or a process, and alert when the entity goes down.

  • Threshold: Monitor time-series metrics, and alert if they violate user-defined thresholds.

  • Prometheus: Monitor metrics through a PromQL query.

  • Event: Monitor occurrences of specific events, and alert if the total number of occurrences violates a threshold. Useful for alerting on container, orchestration, and service events like restarts and unauthorized access.

Alert Tools

The following tools help with alert creation:

  • Alert Library: Sysdig Monitor provides a set of alerts by default. Use it as it is or as a template to create your own.

  • Sysdig API: Use Sysdig’s Python client to create, list, delete, update and restore alerts. See examples.

  • Import Prometheus Rules: Sysdig Monitor allows you to import Prometheus rules or create new rules on the fly and add them to the existing list of alerts.

Guidelines for Creating Alerts

Steps

Description

Decide What to monitor

Determine what type of problem you want to be alerted on. See Alert Types to choose a type of problem.

Define how it will be monitored

Specify exactly what behavior triggers a violation. For example, Marathon App is down on the Kubernetes Cluster named Production for ten minutes.

Decide Where to monitor

Narrow down your environment to receive fine-tuned results. Use Scope to choose an entity that you want to keep a close watch on. Specify additional segments (entities) to give context to the problem. For example, in addition to specifying a Kubernetes cluster, add a namespace and deployment to refine your scope.

Define when to notify

Define the threshold and time window for assessing the alert condition.

Setting up a Warning Threshold allows you to notify of incidents earlier.

For example, a database using 60% disk may trigger a warning to Slack but the same database using 80% disk may page the on-call team.

Decide how notifications are sent

Alert supports customizable notification channels, including email, mobile push notifications, OpsGenie, Slack, and more. To see supported services, see Set Up Notification Channels.

To create alerts, simply:

  1. Choose an alert type.

  2. Configure alert parameters.

  3. Configure the notification channels you want to use for alert notification.

Sysdig sometimes deprecates outdated metrics. Alerts that use these metrics will not be modified or disabled, but will no longer be updated. See Deprecated Metrics and Labels.

Create Alerts for CloudWatch Metrics

CloudWatch metrics queries are displayed as no data in the Alerts Editor. This is because our metric store does not currently store CloudWatch metrics and therefore, the UI displays the missing metrics as no data. However, you can successfully create alerts using these metrics.

Topics in This Section
Configure Alerts

Configure alerts based on specific metrics, conditions, and thresholds.

Manage Alerts

Alerts can be managed individually, or as a group, by using the checkboxes on the left side of the Alert UI and the customization bar.

Alert Types

Alerts Library

To help you get started quickly, Sysdig provides a set of curated alert templates called Alerts Library.

Silence Alert Notifications

Silencing Rules prevent alert notifications from being sent to notification channels.

Alert Inhibition

Alert Inhibition prevents alert notifications from being sent when another specified alert occurrence is already active. This helps to reduce alert noise by ensuring only the most relevant alert notifications are forwarded for action.

Legacy Alerts Editor

If you do not have the new Sysdig metric store enabled, you will not be able to use the latest Alert Editor features. You will continue to use the legacy Alerts Editor to create and edit alert notifications.

Service Limits

Usage thresholds help ensure that your Sysdig Monitor environment stays healthy and performs optimally.