Alerts

Alert is the responsive component of Sysdig Monitor. Alerts notify you when an event or issue occurs that requires attention. Events and issues are identified based on changes in the metric values collected by Sysdig Monitor. The Alerts module displays out-of-the-box alerts and a wizard for creating and editing alerts as needed.

Alert Types

The types of alerts available in Sysdig Monitor:

  • Downtime: Monitor any type of entity, such as a host, a container, or a process, and alert when the entity goes down.

  • Metric: Monitor time-series metrics, and alert if they violate user-defined thresholds.

  • PromQL: Monitor metrics through a PromQL query.

  • Event: Monitor occurrences of specific events, and alert if the total number of occurrences violates a threshold. Useful for alerting on container, orchestration, and service events like restarts and unauthorized access.

Alert Tools

The following tools help with alert creation:

  • Alert Library: Sysdig Monitor provides a set of alerts by default. Use it as it is or as a template to create your own.

  • Sysdig API: Use Sysdig’s Python client to create, list, delete, update and restore alerts. See examples.

  • Import Prometheus Rules: Sysdig Monitor allows you to import Prometheus rules or create new rules on the fly and add them to the existing list of alerts.

Create Alerts for CloudWatch Metrics

CloudWatch metrics queries are displayed as no data in the Alerts Editor. This is because our metric store does not currently store CloudWatch metrics and therefore, the UI displays the missing metrics as no data. However, you can successfully create alerts using these metrics.

Guidelines for Creating Alerts

Steps

Description

Decide What to monitor

Determine what type of problem you want to be alerted on. See Alert Types to choose a type of problem.

Define how it will be monitored

Specify exactly what behavior triggers a violation. For example, Marathon App is down on the Kubernetes Cluster named Production for ten minutes.

Decide Where to monitor

Narrow down your environment to receive fine-tuned results. Use Scope to choose an entity that you want to keep a close watch on. Specify additional segments (entities) to give context to the problem. For example, in addition to specifying a Kubernetes cluster, add a namespace and deployment to refine your scope.

Define when to notify

Define the threshold and time window for assessing the alert condition.

Single Alert fires an alert for your entire scope, while Multiple Alert fires if any or every segment breach the threshold at once.

Multiple Alerts include all the segments you specified to uniquely identify the location and thus provides a full qualification of where the problem occurred. The higher the number of segments the easier to uniquely identify the affected entities.

A good analogy for multiple alerts is alerting on cities. For example, creating multiple alerts on San Francisco would trigger an alert which will include information such as the country that it is part of is the USA and the continent is North America.

Trigger gives you control over how notifications are created. For example, you may want to receive a notification for every violation, or want only a single notification for a series of consecutive violations.

Decide how notifications are sent

Alert supports customizable notification channels, including email, mobile push notifications, OpsGenie, Slack, and more. To see supported services, see Set Up Notification Channels.

To create alerts, simply:

  1. Choose an alert type.

  2. Configure alert parameters.

  3. Configure the notification channels you want to use for alert notification.

Sysdig sometimes deprecates outdated metrics. Alerts that use these metrics will not be modified or disabled, but will no longer be updated. See Deprecated Metrics and Labels.


Configure Alerts

Use the Alerts Editor to create or edit alerts.

Manage Alerts

Alerts can be managed individually, or as a group, by using the checkboxes on the left side of the Alert UI and the customization bar.

Alert Types

Alerts Library

To help you get started quickly, Sysdig provides a set of curated alert templates called Alerts Library.

Silence Alert Notifications

Sysdig Monitor allows you to silence alerts for a given scope for a predefined amount of time. When silenced, alerts will still be triggered but will not send any notifications. You can schedule silencing in advance. This helps administrators to temporarily mute notifications during planned downtime or maintenance and send downtime notifications to selected channels.

Legacy Alerts Editor

If you do not have the new Sysdig metric store enabled, you will not be able to use the latest Alert Editor features. You will continue to use the legacy Alerts Editor to create and edit alert notifications.