Configure a Custom Webhook Channel

Sysdig Monitor supports using a custom webhook notification channel to send custom alert notifications to any destination. This is often useful for having Sysdig create alerts with custom tools or other webhook-based applications for which Sysdig does not yet have a native integration.

The custom webhook notification channel allows you to customize payload from within Sysdig Monitor based on alert properties by dynamically referencing alert variables and scope labels. To create the payload, Sysdig provides you with the Sysdig Templating Language that allows for variable interpolation and basic conditional logic.

Enable Webhook

Basic Configuration

1. Complete steps 1-3 in Set Up a Notification Channel and select Custom Webhook.

   

2. Enter the webhook channel configuration options:

   • URL: The destination URL to which alert notifications will be sent.

   • Channel Name: Add a meaningful name, such as Trello, PagerDuty, OpsGenie, and so on.

   • Enabled: Toggle enabling and disabling the webhook channel.

   • Notification options: Send notifications when alerts are resolved or acknowledged.

   • Test notification: Send a test notification upon saving to confirm that the webhook URL is reachable.

   • Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

3. Complete the steps described in Method and Headers, Payload, and Configure Test Notification.

4. Click Save.

When the channel is created, you can use it on any alerts that you create.

When the alert fires, the notification will be sent as an HTTP request of the specified method in JSON format to your webhook endpoint.

For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.

Method and Headers

• HTTP Method: You can choose an HTTP method for the request to be sent.

  • POST: Used for the creating resources.
  • PUT: Used for the editing an existing resource.
  • PATCH: Used for partial modifying an existing resource.
  • DELETE: Used for deleting a resource.

• Content Type: For the custom webhook channel, application/json will always be the value of the content-type header for the request, even if this header does not appear on the headers list.

• Allow insecure connections: Check this option to skip TLS verification over HTTPS.

• Custom headers: Add custom headers to your alert notification.

  If your webhook integrations require additional headers you specify them by using a custom header.

  For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default header, but you can add it using a custom header, such as Authorization: Bearer mytoken. Use this component to specify suitable header names and header values.

  Note that any Content type, Accept, and User-Agent header specified in this component will be discarded, as the default header value of Content-type:application/json cannot be overridden.

Payload

Editor: The custom webhook channel allows you to fully customize the JSON payload that you want to send. The content of the editor represents the request body of the HTTP call you are sending to the third-party webhook. The data must be a valid JSON document. Systems that receive webhook alerts can parse the data and take action based on the contents.

When triggered, the webhook channel allows dynamic referencing to properties of the alert and event which were created using the Sysdig Templating Language. Using the same system, basic conditional logic can also be written.

Configure Test Notification

As an alternative to sending a test notification upon creating the channel, you can also test the channel on the fly before creating it. Use the Configure Test Notification editor to do so.

In the editor, you can pick a variety of scenarios, ranging from different severity alerts to alert types, and trigger a test notification to the destination webhook URL.

Build Payloads

Use the Sysdig Templating Language to customize your payload using variable interpolation and basic conditional logic:

• Reference alert variables with {{@alert_, for instance {{@alert_name}}
• Reference event variables with {{@event_, for instance {{@event_state}}
• Use conditional logic with {{#if_, for instance {{#if_metric_alert}}{{/if}}
• Reference labels with {{@event_labels}} for all, or specifically with for instance {{@event_labels.kube_cluster_name}}

See Reference for all the allowed syntax parameters.

Any text not matching the above keywords is accepted and dynamically rendered when the alert is triggered.

For more information, browse the on-screen documentation and examples that is provided in the form of snippets next to the code editor.

Reference

Sysdig Templating Language is composed of two tipes of constructs: statements and variables. Both of them are accessed from within {{ }} brackets, which are the only special characters in the payload that you define. Everything else is interpreted as plain text and passed along as is.

Statements

Statements allow you to perform basic conditional logic. The common use case would be sending a different alerting JSON payload in case of a high severity alert, such as:

{
    {{#if_severity_high}}
        "code": "incident"
    {{#else}}
        "code": "warning"
    {{/if}}        
}

Whenever an alert triggers on this notification channel, the template is evaluated. In the presence of a high severity alert, the following payload is sent:

{
    "code": "incident"      
}

Whereas for all other cases, the following is sent:

{
    "code": "warning"      
}

Statements thus allow for conditional rendering of different sections of the payload.

All statements

if_severity_high

Conditional statement that matches an alert that was set with high severity.

if_severity_medium

Conditional statement that matches an alert that was set with medium severity.

if_severity_low

Conditional statement that matches an alert that was set with low severity.

if_severity_info

Conditional statement that matches an alert that was set with info severity

if_metric_alert

Conditional statement that matches a metric alert.

if_downtime_alert

Conditional statement that matches a downtime alert.

if_prometheus_alert

Conditional statement that matches a prometheus alert.

if_event_alert

Conditional statement that matches an event alert.

if_anomaly_alert

Conditional statement that matches an anomaly detection alert.

if_outlier_alert

Conditional statement that matches an outlier alert.

if_resolved_event

Conditional statement that matches an event which has been resolved.

if_main_threshold

Conditional statement that matches an event triggered on a main alert threshold.

if_warning_threshold

Conditional statement that matches an event triggered on a warning alert threshold.

escape

An escape block can be used to insert anywhere in the payload the grammar restricted characters, such as {{, as such:

{
    "text": "{{#escape}}a string with special {{ chars{{/escape}}"
}

An escape block can also spread between multiple lines, as such:

{
    "text": "
        {{#escape}}
            a string
            with special {{
            chars
        {{/escape}}
    "
}

Variables

Variables allow you to access alert and event properties. These are dynamically substituted upon rendering with the underlying properties of the configured alert or triggered event.

Variables of type string are automatically escaped, but enclose them in quotes in order to generate a valid JSON payload.

Alert Variables

alert_id

Type: Number.

The auto-generated unique alert identifier.

Example: 1.

alert_name

Type: String.

The configured alert name.

Example: High CPU Usage alert

alert_description

Type: String.

The configured alert description.

Example: Alert for when cpu load goes over 75%.

alert_scope

Type: String.

The configured alert scope.

Example: kube_pod_name in ('backend-api')

alert_severity

Type: String.

The configured alert severity label.

Possible values: HIGH, MEDIUM, LOW, NONE.

Example: HIGH.

alert_group_by

Type: JSON String Array.

A JSON String array with all the labels on which the alert has been grouped by.

This applies only to metric and event alerts.

Example: ['kube_cluster_name', 'agent_id']'.

alert_condition

Type: String

The configured alert main condition.

This applies only to metric and event alerts.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 75.0.

alert_warning_condition

Type: String

The configured alert warning condition.

This applies only to metric and even alerts.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 50.0.

alert_timespan

Type: Number

The configured alert timespan.

Example: 600000000.

alert_url

Type: String

The link to Sysdig Monitor’s Alert edit page.

Example: https://app.sysdigcloud.com/#/alerts/rules/123456.

Event Variables

event_state

Type: String

Possible values: ACTIVE for triggered conditions and renotifications, OK for resolved conditions.

Example: ACTIVE.

event_trigger_timestamp

Type: Number

The microseconds timestamp in which the event fired.

Example: 1674140923000000.

event_resolved_timestamp

Type: Number

The microseconds timestamp in which the event resolved.

Example: 1674140923000000.

event_trigger_metadata

Type: JSON Object.

A JSON object, with metadata related to the triggering or renotification event, including the metric value that generated the event.

Example:

{
  "entity": "host_mac = '08:00:27:70:1a:03' and container_name = 'container1_0'",
  "metricValues": [
    {
      "metric": "sysdig_container_cpu_used_percent",
      "aggregation": "avg",
      "groupAggregation": "avg",
      "value": 93.2359617776846
    }
  ]
}

event_resolved_metadata

Type: JSON Object.

A JSON object, with metadata related to the resolve event, including the metric value that cleared the alert condition.

Example:

{
  "entity": "host_mac = '08:00:27:70:1a:03' and container_name = 'container1_0'",
  "metricValues": [
    {
      "metric": "sysdig_container_cpu_used_percent",
      "aggregation": "avg",
      "groupAggregation": "avg",
      "value": 33.2359617776846
    }
  ]
}

event_labels

Type: JSON Object

A JSON String to String object, mapping label names from the event’s context to their values.

Example:

{
    "kube_cluster_name": "my-cluster",
    "agent_id": "123"
}

event_title

Type: String

The configured title for the event.

Example: High CPU Usage is Triggered on my-cluster.

event_body

Type: String

The body text for the event.

Example:

TEST ALERT: Testing Notification Channel dasdads

Event Generated:

Severity: Low
Metric: sysdig_container_cpu_used_percent = 93.236 %
Segment: container_name = 'container1_0' and host_mac = '08:00:27:70:1a:03'
Scope: host_hostname = 'Host-0 (08:00:27:70:1a:03)'

Time: 01/19/2023 03:10 PM UTC
State: Triggered
Notification URL: https://app-staging.sysdigcloud.com/#/events/notifications/l:2419200/2168/details

------

Triggered by Alert:
Name: TEST ALERT: Testing Notification Channel dasdads
Description: Alert description
Team: Team for standard users.
Scope:
    host_hostname = 'Host-0 (08:00:27:70:1a:03)'
Segment by: host_mac, container_name
Alert When: avg(sysdig_container_cpu_used_percent) > 85
For at least: 10 m
Alert URL: https://app-staging.sysdigcloud.com/#/alerts/rules?alertId=4322

event_condition

Type: String

The condition that made the event trigger or resolve.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 75.0.

event_condition_value

Type: String

The condition value that made the alert trigger or resolve formatted as a String.

Note: For No Data triggering alerts this variable will be populated with “No Data”.

Example: For an alert with condition avg(avg(sysdig_container_cpu_used_percent)) > 75.0 and an alert triggering at 80%, this variable will be populated with 80 %.

event_id

Type: Number

The internal id for the event.

Example: 123456.

event_username

Type: String

The username of the user responsible for triggering the event, if applicable, such as in the cases of test notifications.

Example: user.name@sysdig.com

event_url

Type: String

The url to the event details page within Sysdig Monitor.

Example: https://appsysdigcloud.com/#/event-feed?last=3600&eventId=123456

event_threshold

Type: String

Possible values: MAIN for main condition triggering events, WARNING for warning condition triggering events.

Learn More

• Integrate with Trello
• Integrate with Moogsoft