Configure a Custom Webhook Channel

Sysdig Monitor supports using a custom webhook notification channel to send custom alert notifications to any destination. This is often useful for having Sysdig create alerts with custom tools or other webhook-based applications for which Sysdig does not yet have a native integration.

The custom webhook notification channel lets you customize payload from within Sysdig Monitor, based on alert properties, by dynamically referencing alert variables and scope labels. To create the payload, Sysdig provides you with the Sysdig Templating Language that enables variable interpolation and basic conditional logic.

Enable Webhook

Basic Configuration

1. In Sysdig Monitor, complete steps 1-3 in Set Up a Notification Channel and select Custom Webhook.

2. Enter the webhook channel configuration options:

   • URL: The destination URL to which alert notifications will be sent.

   • Channel Name: Add a meaningful name, such as Trello, Moogsoft, and so on.

   • Enabled: Toggle enabling and disabling the webhook channel.

   • Notification options: Send notifications when alerts are resolved or acknowledged.

   • Test notification: Send a test notification upon saving to confirm that the webhook URL is reachable.

   • Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

3. Complete the steps described in Method and Headers, Payload, and Configure Test Notification.

4. Click Save.

When the channel is created, you can use it on any alerts that you create. When the alert fires, the notification will be sent as an HTTP request of the specified method in JSON format to your webhook endpoint.

For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.

Method and Headers

• HTTP Method: You can choose an HTTP method for the request to be sent.

  • POST: Used for the creating resources.
  • PUT: Used for the editing an existing resource.
  • PATCH: Used for partially modifying an existing resource.
  • DELETE: Used for deleting a resource.

• Content Type: For the custom webhook channel, application/json will always be the value of the content-type header for the request, even if this header does not appear on the headers list.

• Allow insecure connections: Check this option to skip TLS verification over HTTPS.

• Custom headers: Add custom headers to your alert notification.

  If your webhook integrations require additional headers you can specify them by using a custom header. For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default header, but you can add it using a custom header, such as Authorization: Bearer mytoken. Use this component to specify suitable header names and header values.

  Note that any Content type, Accept, and User-Agent header specified in this component will be discarded, as the default header value of Content-type:application/json cannot be overridden.

Payload

Editor: The custom webhook channel allows you to fully customize the JSON payload that you want to send. The content of the editor represents the request body of the HTTP call you are sending to the third-party webhook. The data must be a valid JSON document. Systems that receive webhook alerts can parse the data and take action based on the contents.

When triggered, the webhook channel allows dynamic referencing to properties of the alert and event which were created using the Sysdig Templating Language. Using the same system, basic conditional logic can also be written.

Configure Test Notification

As an alternative to sending a test notification upon creating the channel, you can also test the channel before creating it. Use the Configure Test Notification editor to do so.

In the editor, you must pick the severity and alert type that best suits your scenario, and optionally trigger a test notification to the destination webhook URL.

Build Payloads

Use the Sysdig Templating Language to customize your payload using variable interpolation and basic conditional logic:

• Reference alert variables with {{@alert_, for instance {{@alert_name}}
• Reference event variables with {{@event_, for instance {{@event_state}}
• Use conditional logic with {{#if_, for instance {{#if_metric_alert}}{{/if}}
• Reference labels with {{@event_labels}} for all, or specifically with for instance {{@event_labels.kube_cluster_name}}

See Reference for all the allowed syntax parameters.

Any text not matching the above keywords is accepted and dynamically rendered when the alert is triggered.

For more information, browse the on-screen documentation and examples that are provided in the form of snippets next to the code editor.

Reference

Sysdig Templating Language contains two types of constructs: statements and variables. Both of them are accessed from within {{ }} brackets, which are the only special characters in the payload that you define. Everything else is interpreted as plain text and passed along as is.

Statements

Statements let you perform basic conditional logic. The common use case would be sending a different JSON payload in case of a high severity alert, such as:

{
    {{#if_severity_high}}
        "code": "incident"
    {{#else}}
        "code": "warning"
    {{/if}}
}

Whenever an alert triggers on this notification channel, the template is evaluated. In the presence of a high severity alert, the following payload is sent:

{
    "code": "incident"
}

Whereas for all other cases, the following is sent:

{
    "code": "warning"
}

Statements thus allow for conditional rendering of different sections of the payload.

All statements

if_severity_high

Conditional statement that matches an alert that was set with high severity.

if_severity_medium

Conditional statement that matches an alert that was set with medium severity.

if_severity_low

Conditional statement that matches an alert that was set with low severity.

if_severity_info

Conditional statement that matches an alert that was set with info severity.

if_metric_alert

Conditional statement that matches a Threshold Alert. Threshold Alerts were formerly known as Metric Alerts, and if_metric_alert remains valid syntax for backwards compatibility.

if_downtime_alert

Conditional statement that matches a downtime alert.

if_prometheus_alert

Conditional statement that matches a prometheus alert.

if_event_alert

Conditional statement that matches an event alert.

if_anomaly_alert

Conditional statement that matches an anomaly detection alert.

if_outlier_alert

Conditional statement that matches an outlier alert.

if_resolved_event

Conditional statement that matches an event which has been resolved.

if_main_threshold

Conditional statement that matches an event triggered on a main alert threshold.

if_warning_threshold

Conditional statement that matches an event triggered on a warning alert threshold.

escape

An escape block can be used to insert the grammar restricted characters, such as {{, anywhere in the payload. For example:

{
    "text": "{{#escape}}a string with special {{ chars{{/escape}}"
}

An escape block can also spread between multiple lines, like so:

{
    "text": "
        {{#escape}}
            a string
            with special {{
            chars
        {{/escape}}
    "
}

Variables

Variables allow you to access alert and event properties. These are dynamically substituted upon rendering with the underlying properties of the configured alert or triggered event.

Variables of type string are automatically escaped, but you can enclose them in quotes in order to generate a valid JSON payload.

Add the variables to your Notification Channel’s payload:

Alert Variables

alert_id

Type: Number.

The auto-generated unique alert identifier.

Example: 1.

alert_name

Type: String.

The configured alert name.

Example: High CPU Usage alert.

alert_description

Type: String.

The configured alert description.

Example: Alert for when cpu load goes over 75%.

alert_scope

Type: String.

The configured alert scope.

Example: kube_pod_name in ('backend-api').

alert_severity

Type: String.

The configured alert severity label.

Possible values: HIGH, MEDIUM, LOW, NONE.

Example: HIGH.

alert_group_by

Type: JSON String Array.

A JSON String array with all the labels on which the alert has been grouped by.

This applies only to metric and event alerts.

Example: ['kube_cluster_name', 'agent_id']'.

alert_condition

Type: String.

The configured alert main condition.

This applies only to metric and event alerts.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 75.0.

alert_warning_condition

Type: String,

The configured alert warning condition.

This applies only to metric and even alerts.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 50.0.

alert_timespan

Type: Number.

The configured alert timespan.

Example: 600000000.

alert_url

Type: String

The link to Sysdig Monitor’s Alert edit page.

Example: https://app.sysdigcloud.com/#/alerts/rules/123456.

Event Variables

event_state

Type: String.

Possible values: ACTIVE for triggered conditions and renotifications. OK for resolved conditions.

Example: ACTIVE.

event_trigger_timestamp

Type: Number.

The microseconds timestamp in which the event fired.

Example: 1674140923000000.

event_resolved_timestamp

Type: Number.

The microseconds timestamp in which the event resolved.

Example: 1674140923000000.

event_trigger_metadata

Type: JSON Object.

A JSON object, with metadata related to the triggering or renotification event, including the metric value that generated the event.

Example:

{
  "entity": "host_mac = '08:00:27:70:1a:03' and container_name = 'container1_0'",
  "metricValues": [
    {
      "metric": "sysdig_container_cpu_used_percent",
      "aggregation": "avg",
      "groupAggregation": "avg",
      "value": 93.2359617776846
    }
  ]
}

event_resolved_metadata

Type: JSON Object.

A JSON object, with metadata related to the resolve event, including the metric value that cleared the alert condition.

Example:

{
  "entity": "host_mac = '08:00:27:70:1a:03' and container_name = 'container1_0'",
  "metricValues": [
    {
      "metric": "sysdig_container_cpu_used_percent",
      "aggregation": "avg",
      "groupAggregation": "avg",
      "value": 33.2359617776846
    }
  ]
}

event_labels

Type: JSON Object.

A JSON String to String object, mapping label names from the event’s context to their values.

Example:

{
    "kube_cluster_name": "my-cluster",
    "agent_id": "123"
}

event_title

Type: String.

The configured title for the event.

Example: High CPU Usage is Triggered on my-cluster.

event_body

Type: String

The body text for the event.

Example:

TEST ALERT: Testing Notification Channel dasdads

Event Generated:

Severity: Low
Metric: sysdig_container_cpu_used_percent = 93.236 %
Segment: container_name = 'container1_0' and host_mac = '08:00:27:70:1a:03'
Scope: host_hostname = 'Host-0 (08:00:27:70:1a:03)'

Time: 01/19/2023 03:10 PM UTC
State: Triggered
Notification URL: https://app-staging.sysdigcloud.com/#/events/notifications/l:2419200/2168/details

------

Triggered by Alert:
Name: TEST ALERT: Testing Notification Channel dasdads
Description: Alert description
Team: Team for standard users.
Scope:
    host_hostname = 'Host-0 (08:00:27:70:1a:03)'
Segment by: host_mac, container_name
Alert When: avg(sysdig_container_cpu_used_percent) > 85
For at least: 10 m
Alert URL: https://app-staging.sysdigcloud.com/#/alerts/rules?alertId=4322

event_condition

Type: String.

The condition that made the event trigger or resolve.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 75.0.

event_condition_value

Type: String.

The condition value that made the alert trigger or resolve formatted as a String.

Note: For No Data triggering alerts this variable will be populated with “No Data”.

Example: For an alert with condition avg(avg(sysdig_container_cpu_used_percent)) > 75.0 and an alert triggering at 80%, this variable will be populated with 80 %.

event_entity

Type: String.

The triggering alert segment (labels and values).

Example: container_name = 'container1_0' and host_mac = '08:00:27:70:1a:03'.

sysdig_event_id

Type: String.

The sysdig_event_id refers to the identifier associated with an event in the event feed. Unlike the event_id, which pertains specifically to alert occurrences, the sysdig_event_id allows you to directly reference and retrieve detailed information about an event recorded in the event feed.

Example: 1234567891234567891.

event_id

Type: Number.

The event_id is a unique identifier associated with each alert occurrence. Unlike the sysdig_event_id, which is unique for every event in the event feed, the event_id remains constant across the entire lifecycle of an alert occurrence. You can use it as an incident key in incident management systems to acknowledge and resolve an alert notification.

Example: 123456.

event_username

Type: String.

The username of the user responsible for triggering the event, if applicable, such as in the cases of test notifications.

Example: user.name@sysdig.com.

event_url

Type: String.

The url to the event details page within Sysdig Monitor.

Example: https://appsysdigcloud.com/#/event-feed?last=3600&eventId=123456.

event_threshold

Type: String.

Possible values: MAIN for main condition triggering events, WARNING for warning condition triggering events.

Learn More

• Integrate with Trello
• Integrate with Moogsoft