Configure a Webhook Channel

Sysdig Monitor and Sysdig Secure support sending an alert notification to a destination, such as a website, custom application, and so on for which Sysdig does not have a native integration. Do this using a custom Webhook channel.

Prerequisites

  • Webhooks via HTTPS only work if a signed/valid certificate is in use.

  • Have your desired destination URL on hand.

Enable Webhook

  1. Complete steps 1-3 in Set Up a Notification Channel and choose Webhook.

    webhook-channel.png
  2. Enter the webhook channel configuration options:

    • URL: The destination URL to which notifications will be sent.

    • Channel Name: Add a meaningful name, such as Ansible, PagerDuty, OpsGenie, and so on.

    • Enabled: Toggle on and off notifications.

    • Notification options: Toggle for notifications when alerts are resolved or acknowledged.

    • Test notification: Toggle to be notified that the configured URL is working.

    • Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

    • Allow insecure connections: Enable if you want to skip the TLS verification.

    • Custom headers: Add custom headers to your alert notification.

      If your webhook integrations require additional headers you specify them by using a custom header.

      For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default header, but you can add it using a custom header.

      Alternatively, you can choose to add custom headers programmatically as described in Configure Custom Headers Programmatically.

  3. Click Save.

When the channel is created, you can use it on any alerts you create.

Then, when the alert fires, the notification will be sent as a POST in JSON format to your webhook endpoint. (See Alert Output, below.)

For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.

Configure Custom Headers Programmatically

By default, alert notifications follow a standard format (see Description of POST Data, below).

However, some integrations require additional headers and/or data, which you can append to the alert format using a custom header or custom data entry.

For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default alert template built into Sysdig, but you can add it using a custom header.

In addition to the Webhook UI option, you can do this from the command line, as described below.

Note

  • additionalHeaders is usually used for authentication

  • customData is used to add values to the alert

Warning

After it has been created via the API, any manipulation will mangle the notification channel. Use with care.

Sample Use Case

This example adds two custom headers and defines additional custom data, as well as the format for that data.

  1. Use the curl command to retrieve all configured notification channels:

    curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'Authorization: Bearer API-KEY'
  2. Add the custom headers and execute the request:

    curl -X PUT https://app.sysdigcloud.com/api/notificationChannels/1 -H 'Authorization: Bearer API-KEY' -H 'Content-Type: application/json' -d '{
      "notificationChannel": {
        "id": 1,
        "version": 1,
        "type": "WEBHOOK",
        "enabled": true,
        "name": "Test-Sysdig",
        "options": {
          "notifyOnOk": true,
          "url": "https://hookb.in/v95r78No",
          "notifyOnResolve": true,
          "customData": {
            "String-key": "String-value",
            "Double-key": 2.3,
            "Int-key": 23,
            "Null-key": null,
            "Boolean-key": true
          },
          "additionalHeaders": {
            "Header-1": "Header-Value-1",
            "Header-2": "Header-Value-2"
          }
        }
      }
    }'

Standard Alert Output

Alerts that use a custom webhook for notification send a JSON-format with the following data.

Description of POST Data:

{
  "timestamp": 1620222000000000, // Time when the alert triggered in microseconds
  "timespan": 60000000, // duration of the alert in microseconds (how long the value should be true before triggering)
  "alert": {
    "severity": 2, // severity from 0 to 7, use severityLabel for a human readable version
    "editUrl": "https://app-staging.sysdigcloud.com/#/alerts/21998727", // alert edit URL
    "severityLabel": "Medium", // human readable version of severity
    "subject": "CPU temp is High on homebridge:9100 is Triggered", // Alert subject
    "scope": null, // scope of the alert if set from the UI
    "name": "CPU temp is High", // name of the alert
    "description": null, // description, not used ATM
    "id": 21998727, // alert id
    "body": "CPU temp is High on homebridge:9100 is Triggered\n\n\nEvent Generated:\n\nSeverity:         Medium\n    Metric:\n    node_hwmon_temp_celsius = 65.8121\nSegment:\n    instance = 'homebridge:9100'\nScope:\n    Everywhere\n\nTime:             05/05/2021 01:40 PM UTC\nState:            Triggered\nNotification URL: https://app-staging.sysdigcloud.com/#/events/notifications/l:2419200/14918845/details\n\n------\n\nTriggered by Alert:\n\nName:         CPU temp is High\nTeam:         Monitor Operations\nScope:\n    Everywhere\nSegment by:   instance\nWhen:         avg(avg(node_hwmon_temp_celsius)) > 40\nFor at least: 1 m\nAlert URL:    https://app-staging.sysdigcloud.com/#/alerts/21998727\n\n\n"
  },
  "event": {
    "id": 14918845, // id of the generated event
    "url": "https://app-staging.sysdigcloud.com/#/events/notifications/l:604800/14918845/details" // url of the event in the feed
  },
  "state": "ACTIVE", // status of the alert, can be ACTIVE or OK
  "resolved": true, 
  "entities": [ // list of entities that triggered the alert, at the moment we send a notification per entity, so this array will always contain a single object
    {
      "entity": "instance = 'homebridge:9100'", // segment that triggered
      "metricValues": [ // value of the metric at the time of triggering
        {
          "metric": "node_hwmon_temp_celsius",
          "aggregation": "avg",
          "groupAggregation": "avg",
          "value": 65.812167
        }
      ]
    }
  ],
  "endEntities": [ // list of entities when the alert was resolved (same as "entities")
    {
      "entity": "instance = 'homebridge:9100'",
      "metricValues": [
        {
          "metric": "node_hwmon_temp_celsius",
          "aggregation": "avg",
          "groupAggregation": "avg",
          "value": 39.812167
        }
      ]
    }
  ],
  "condition": "avg(avg(node_hwmon_temp_celsius)) > 40", // alert condition in string form
  "source": "Sysdig Cloud", // source of the event
  "labels": { // list of labels associated to this event (they strongly depend on the segmentation and scope of the alert)
    "instance": "homebridge:9100"
  }
}

Example of POST Data:

{
 "timestamp": 1471457820000000,
 "timespan": 60000000,
 "alert": {
   "severity": 4,
   "editUrl": "http://app.sysdigcloud.com/#/alerting/alerts/1/edit",
   "scope": "host.mac = \"00:0c:29:04:07:c1\"",
   "name": "alertName",
   "description": "alertDescription",
   "id": 1
 },
 "event": {
   "id": 1,
   "url": "http://app.sysdigcloud.com/#/alerting/notifications/l:604800/1/details"
 },
 "state": "ACTIVE",
 "resolved": false,
 "entities": [{
   "entity": "host.mac = '00:0c:29:04:07:c1'",
   "metricValues": [{
     "metric": "cpu.used.percent",
     "aggregation": "timeAvg",
     "groupAggregation": "none",
     "value": 100.0
   }],
   "additionalInfo": [{
     "metric": "host.hostName",
     "value": "sergio-virtual-machine"
   }]
 }],
 "condition": "timeAvg(cpu.used.percent) > 10"
}

Example of Failure

$ curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'authorization: Bearer dc1a42cc-2a5a-4661-b4d9-4ba835fxxxxx’'

{"timestamp":1543419336542,"status":401,"error":"Unauthorized","message":"Bad credentials","path":"/api/notificationChannels"}

Example of Success

$ curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'Authorization: Bearer dc1a42cc-2a5a-4661-b4d9-4ba835fxxxxx'
{"notificationChannels":[{"id":18968,"version":2,"createdOn":1543418691000,"modifiedOn":1543419020000,"type":"WEBHOOK","enabled":true,"sendTestNotification":false,"name":"robin-webhook-test","options":{"notifyOnOk":true,"url":"https://postb.in/6dtwzz7l","notifyOnResolve":true}}]}
$

The webhook feature is used to integrate the following channels: