Troubleshoot Notification Channels

Use this page to troubleshoot notification channels, including message throttling and channel failure alerts.

Message Throttling in Sysdig Secure

In Sysdig Secure, when notifications are working normally, there are situations in which not every event triggers a unique notification.

When: If multiple events occur on the same policy-and-rule combination within a 5-minute window, only the first notification event is sent. This is done to avoid spamming Slack, email, or other channels.

Why: Notifications, even via the webhook channel, should be used as an alert that something has occurred, and the investigation should continue from the Events UI page in Sysdig Monitor or Sysdig Secure. They should not be relied upon to send data from the Events platform.

If you wish to generate a call to an external service for every instance of an event from Sysdig, Event Forwarding might be more appropriate to your use case. See Event Forwarding.

Notification Failures

Notification failures occur when the system attempts to deliver a notification and receives any 4XX HTTP errors except the following:

  • 409 CONFLICT
  • 408 TIMEOUT
  • 423 LOCKED

When other 4XX errors occur in a notification channel:

  1. Sysdig puts the channel under observation for 24 hours, during which subsequent errors will not trigger any further behavior. This helps to control noise.
  2. If an error occurs after this 24 hour window has expired:
    • In Sysdig Monitor, an event will appear in the event feed.
    • Sysdig Platform sends an email to relevant Account Admins, Team Admins, and Team Managers, notifying you the channel has an error.
    • Another 24 hour observation period begins for the channel, during which additional errors will not trigger any further behavior.
  3. If, after this second 24 hour window, an error occurs again, another Monitor event and Platform email are generated, and a new 24 hour observation window begins.
  4. If five such error events occur, Sysdig disables the channel and sends an email to inform you of what has occurred.

To reactivate, manually re-enable the channel after the underlying issues have been resolved.

If notifications begin to flow again during this process, the system resets to baseline, and treats subsequent incidents as new.

Failure Alert Formats

Events Feed

A Sysdig Event occurs in the Events feed titled:

“Warning: Notification attempt [Attempt #] of 5 through channel [Channel Name] failed”.

Email

Administrators are sent delivery failure emails titled:

“Warning: Notification attempt [Attempt #] of 5 through channel [Channel Name] failed.”