Prometheus Alertmanager Notifications

Sysdig supports integrating Prometheus Alertmanager using a custom webhook.

Prerequisites

• HTTPS webhooks only work when a signed/valid certificate is in use.

• Have your desired destination URL on hand.

Enable Prometheus Alert Manager

1. Complete steps 1-3 in Set Up a Notification Channel and select Prometheus Alert Manager.

2. Enter the Prometheus Alert Manager channel configuration options:

   • URL: The destination URL to which notifications will be sent.

   • Channel Name: Add a meaningful name, such as “Prometheus channel”.

   • Enabled: Toggle notifications on and off.

   • Notification options: Toggle for notifications when alerts are resolved or acknowledged.

   • Test notification: Toggle to be notified that the configured URL is working.

   • Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

   • Allow insecure connections: Enable if you want to skip the TLS verification.

   • Custom headers: Add custom headers to your alert notification. If your Alertmanager channel requires additional headers you can specify them by using a custom header.

     Alternatively, you can choose to add custom headers programmatically as described in Configure Custom Headers Programmatically.

3. Click Save.

When the channel is created, you can use it on any alerts you create.

When the alert fires, the notification will be sent as a POST in JSON format to your webhook endpoint.

For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.

Configure Custom Headers Programmatically

Alert notifications, by default, follow a standard format. However, some integrations require additional headers which you can append to the alert format by using a custom header entry.

For example, some applications use token-based authentication, which requires an entry for the bearer token. This entry is not included in Sysdig’s default alert template, but you can add it using a custom header.

The following example adds two custom headers:

1. Use the following curl command to retrieve all configured notification channels:

   curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'Authorization: Bearer API-KEY'
   

2. Add the custom headers and execute the request:

   curl -X PUT https://app.sysdigcloud.com/api/notificationChannels/1 -H 'Authorization: Bearer API-KEY' -H 'Content-Type: application/json' -d '{
     "notificationChannel": {
       "id": 1,
       "version": 1,
       "type": "PROMETHEUS_ALERT_MANAGER",
       "enabled": true,
       "name": "Test-Sysdig",
       "options": {
         "notifyOnOk": true,
         "url": "https://hookb.in/v95r78No",
         "notifyOnResolve": true,
         "additionalHeaders": {
           "Header-1": "Header-Value-1",
           "Header-2": "Header-Value-2"
         }
       }
     }
   }'
   

Learn More

• Alertmanager
• Alertmanager Configuration