Alert Inhibition

Glossary

• Alert Type: A category of alert rules, each with distinct evaluation logic, such as Prometheus Alert, Group Outlier Alert, Threshold Alert, or Event Alert.

• Alert Rule: The specific configuration of an alert type, which is evaluated by Sysdig. For example, Instance Down: Up == 0.

• Alert Occurrence: The event when an alert rule’s conditions are met, triggering the alert rule. For example, Up{job="nginx"} == 0.

• Alert Notification: The payload sent from a triggered alert rule to a configured notification channel, such as JIRA, Slack, or Webhooks.

Use Cases for Alert Inhibition

Alert Inhibition helps reduce alert fatigue by ensuring that only the most relevant and actionable alert notifications are sent out. For example, if a data center has gone down, it is redundant to receive notifications about individual servers being down. Similarly, if a network interface has failed, alerts for high network latency can distract from the underlying root cause. Alert Inhibition Rules allow you to suppress these secondary notifications when an upstream issue has already been identified.

Alert Inhibition lets you prioritize high severity alerts over low severity ones. If a “database full” alert is active, you can suppress “database almost full” alerts.

Alert Inhibition allow systems to stabilize to prevent false positives. If a “deployment in progress” alert is active, suppress “service endpoint unreachable” alerts as they may resolve once the deployment completes.

Unlike Alert Silencing which has specific start and end times, Alert Inhibition rules are always active and evaluate whenever an alert rule is triggered. Alert Inhibition rules can also be more targeted than Silence Rules, taking into account the current context of the system to selectively suppress less relevant alerts notifications.

Configure Alert Inhibition Rules

Inhibition Rules determine whether an alert occurrence identified by the targetMatcher should be suppressed by another alert occurrence identified by the sourceMatcher. If both triggering alert occurrences match on a label specified by the equal parameter, the alert notification for the targetMatcher is suppressed and not sent out.

In order to precisely suppress the correct alert notifications, the equal field must specify labels that are shared between the sourceMatcher and targetMatcher. We would not want an offline database in {region=us-east-1, component=cache} to suppress a notification in {region=us-east-2, component=cache}. We also wouldn’t want an offline database in {region=us-east-1, component=cache} to suppress a notification in {region=us-east-1, component=API} if the API layer and the cache layer don’t share the same database.

With this inhibition rule, application_connection_failure alert notifications are suppressed if a database_offline alert is active, provided both alerts share the same region and component label values:

sourceMatchers:
  - labelName: alertName
    operator: Equal
    value: database_offline
targetMatchers:
  - labelName: alertName
    operator: Equal
    value: application_connection_failure
equal:
  - region
  - component

Access Next Gen API Docs Using Regional Endpoints

For further detail regarding API usage, Access Next Gen API docs.

Suppress Downstream Alert Notification without an Equal Field

Without an equal field, any alert matched by the sourceMatcher will suppress every alert notification matched by the targetMatcher. This is useful when all alerts matched by the targetMatcher are downstream of the sourceMatcher. For instance, in a single cluster, if the Kubernetes Control Plane is down, it is likely the Kubernetes API server will also be down. However, it is generally best practice to include an equal value to prevent unintended inhibition:

sourceMatchers:
  - labelName: alertName
    operator: Equal
    value: KubernetesControlPlaneDown
targetMatchers:
  - labelName: alertname
    operator: Equal
    value: KubernetesAPIServerDown

Considerations for Alert Inhibition

Only Notifying Alerts Can Be Inhibited

Inhibition Rules only suppress alert notifications that are configured to forward to a notification channel. If an alert rule does not have a notification channel configured, its notifications cannot be inhibited. Inhibition Rules only affect the forwarding of alert notifications; they do not influence the evaluation of alert rules.

Alerts Cannot Inhibit Themselves

If a triggering alert occurrence matches both the sourceMatcher and targetMatcher, the inhibition rule does not apply and an alert notification will still be sent. This ensures that critical alerts notifications are always forwarded, even in scenarios where the alert occurrence may inadvertently match both conditions of the inhibition rule.

Circular Dependencies

Circular dependencies occur when the triggering alert matched by the targetMatcher of one inhibition rule also matches the sourceMatcher of another inhibition rule. For example, if AlertA inhibits AlertB, but AlertB also inhibits AlertA, no alert notifications will be sent for either alert. Careful management of Inhibition Rules is necessary to avoid such scenarios.

Alert Notifications that match the targetMatcher may be delayed

By default, alert notifications matching the targetMatcher are delayed by 2 minutes to allow time for a sourceMatcher to be identified. If a sourceMatcher is found during this delay, the alert notification is suppressed. However, to avoid delaying critical alerts, notifications matching the targetMatcher will be sent after the configured delay period, even if no sourceMatcher is identified. The default delay can be adjusted by submitting a support request.

Interaction of Silenced Alerts with Alert Inhibition

Silencing an alert occurrence does not prevent it from inhibiting an alert notification matched by the targetMatcher. Alert Silencing affects only the forwarding of alert notifications, not the evaluation of alert rules. The state of an alert occurrence remains unchanged whether it is silenced or not. If it matches a sourceMatcher, it can still inhibit an alert notification matched by the targetMatcher.

Additionally, an alert notification can be both silenced and inhibited, indicating it was triggered during an active silence window and affected by a relevant inhibition rule.