Troubleshoot Notification Channels

Message Throttling in Sysdig Secure

In Sysdig Secure, when notifications are working normally, there are situations in which not every event triggers a unique notification.

When: If multiple events occur on the same policy-and-rule combination within a 5-minute window, only the first notification event is sent. This is done to avoid spamming Slack, email, or other channels.

Why: Notifications, even via the webhook channel, should be used as an alert that something has occurred, and the investigation should continue from the Events UI page in Sysdig Monitor or Sysdig Secure. They should not be relied upon to send data from the Events platform.

If you wish to generate a call to an external service for every instance of an event from Sysdig, Event Forwarding might be more appropriate to your use case. See Event Forwarding.

Notification Failures

Notification failures occur when the system attempts to deliver a notification and receives any 4xx HTTP errors except the following: 409 CONFLICT, 408 TIMEOUT, 423 LOCKED, 429 TOO MANY REQUESTS.

Sysdig will send failure alerts to administrators. After 5 failed attempts, the notification channel is automatically disabled.

To reactivate the channel: Manually re-enable after the underlying issues have been resolved.

Failure Alert Formats

Events Feed

A Sysdig Event occurs in the Events feed titled: “Warning: Notification attempt [Attempt #] of 5 through channel [Channel Name] failed”.

Email

Administrators are sent delivery failure emails titled: “Warning: Notification attempt [Attempt #] of 5 through channel [Channel Name] failed.”