Administration

• 1: Administration Settings
• 1.1: User Profile and Password
• 1.1.1: Retrieve the Sysdig API Token
• 1.2: User and Team Administration
• 1.2.1: Manage Users
• 1.2.2: Manage Teams, Roles, and Service Accounts
• 1.2.3: Manage Custom Roles
• 1.2.4: Detailed Role Permissions
• 1.2.5: Group Mappings
• 1.3: Notifications Management
• 1.3.1: Set Up Notification Channels
• 1.3.1.1: Amazon SNS Notifications
• 1.3.1.2: Email Notifications
• 1.3.1.3: Team Email Notifications
• 1.3.1.4: PagerDuty Notifications
• 1.3.1.5: Slack Notifications
• 1.3.1.6: VictorOps Notifications
• 1.3.1.7: OpsGenie Notifications
• 1.3.1.8: Microsoft Teams Notifications
• 1.3.1.9: Prometheus Alertmanager Notifications
• 1.3.1.10: ServiceNow Notifications
• 1.3.1.11: Configure a Custom Webhook Channel
• 1.3.1.12: Configure a Webhook Channel
• 1.3.1.13: Configure a Google Chat Channel
• 1.3.1.14: Configure IBM Cloud Functions Channel
• 1.3.2: Disable or Delete a Notification Channel
• 1.3.3: Troubleshooting Notification Channels
• 1.4: AWS: Integrate AWS Account and CloudWatch Metrics (Optional)
• 1.4.1: IAM Policy Code to Use
• 1.4.2: Integrate with AWS Role Delegation
• 1.5: Storage: Configure Options
• 1.6: Find Your Customer ID and Name
• 1.7: Agent Installation: Overview and Key
• 1.8: Subscription
• 1.8.1: Time Series Billing
• 1.8.2: About AWS Cloudwatch Licensing
• 1.9: Authentication and Authorization (SaaS)
• 1.9.1: Google OAuth (SaaS)
• 1.9.2: SAML (SaaS)
• 1.9.2.1: Okta (SAML)
• 1.9.2.2: OneLogin (SAML)
• 1.9.2.3: ADFS (SAML)
• 1.9.2.4: Azure Active Directory (SAML)
• 1.9.3: OpenID Connect (SaaS)
• 1.9.3.1: Okta (OpenID)
• 1.9.3.2: OneLogin (OpenID)
• 1.9.3.3: Keycloak (OpenID)
• 1.9.3.4: Azure (OpenID)
• 1.9.4: Disable Password Authentication (SaaS)
• 1.9.5: Configure Customized Session Expiration
• 1.10: Configure Theme Preference
• 1.11: Certificates Management
• 1.12: Ticketing Integration
• 1.12.1: JIRA Ticketing
• 1.13: Configure Login Message
• 2: On-Premises Deployments
• 2.1: Architecture & System Requirements
• 2.1.1: Architecture
• 2.1.2: System Requirements
• 2.1.3: Securing User Passwords
• 2.2: On-Premises Installation
• 2.2.1: Installer (Kubernetes | OpenShift)
• 2.2.1.1: Frequently Used Installer Configurations
• 2.2.2: Manual Install 3.0.0+ (Kubernetes)
• 2.2.3: Installer (Kubernetes | OpenShift) 2.5.0-3.2.2
• 2.2.4: Manual Install (OpenShift)
• 2.2.5: Install with Replicated
• 2.2.5.1: Airgapped Installation
• 2.2.5.2: Install Components (Replicated)
• 2.2.5.3: Post-Install Configuration
• 2.2.6: Troubleshooting On-Premises Installation
• 2.3: On-Premises Upgrades
• 2.3.1: Installer Upgrade (3.5.0-3.5.1)
• 2.3.2: Installer Upgrade (2.5.0+)
• 2.3.3: Manual Upgrade (3.0.0+)
• 2.3.4: Manual Upgrade (2.4.1- 2.5.0)
• 2.3.5: Manual Upgrade (2.3.0)
• 2.3.6: Manual Upgrade (v2435)
• 2.3.7: Basic Upgrade (Replicated)
• 2.4: Find the Super Admin Credentials and API Token
• 2.5: Configure Interactive Session Expiration
• 2.6: Upgrade an On-Premises License
• 2.7: Authentication and Authorization (On-Prem Options)
• 2.7.1: Google OAuth (On-Prem)
• 2.7.2: SAML (On-Prem)
• 2.7.2.1: Okta (SAML On-Prem)
• 2.7.2.2: OneLogin (SAML On-Prem)
• 2.7.2.3: Azure Active Directory (SAML On-Prem)
• 2.7.2.4: ADFS (SAML On-Prem)
• 2.7.3: OpenID Connect (On-Prem)
• 2.7.3.1: Okta (OpenID On-Prem)
• 2.7.3.2: OneLogin (OpenID On-Prem)
• 2.7.3.3: Keycloak (OpenID On-Prem)
• 2.7.3.4: Azure (OpenID On-Prem)
• 2.7.4: LDAP
• 2.7.4.1: LDAP Authentication Configuration (for Platform v.1586+)
• 2.7.4.2: LDAP Authentication Configuration (for Platform v.1149 - 1511)
• 2.7.4.3: Mapping LDAP Users to Sysdig Teams
• 2.7.4.4: LDAP/SAML Hybrid Authentication
• 2.7.5: Disable Password Authentication
• 2.7.6: Configure Split DNS
• 3: Data Retention
• 4: Sysdig Platform Audit
• 5: SaaS Regions and IP Ranges
• 6: Get Help | Using Sysdig Support
• 6.1: Self-Help
• 6.2: Contact Support
• 6.2.1: Add Agent Files
• 6.2.2: Add Browser Files
• 6.2.3: Add Backend Files (On-Premises Only)
• 6.3: Terms of Use

1 - Administration Settings

The Settings panel can be accessed from both Sysdig Monitor and Sysdig Secure UIs, and by both administrator and non-admin users.

Access Settings Panel

1. Log in to Sysdig Monitor or Sysdig Secure.

2. Hover the mouse over the User menu in the lower left corner of the navigation bar.

   The two ways to access the Settings links are displayed:

   • Quick links in the User menu, organized into their usage groups
   • The Settings gear symbol in the right corner of the User menu, which takes you to the Settings landing page.

1.1 - User Profile and Password

Access the User Profile page to review and perform necessary actions:

Access the User Profile Page

1. Log in to Sysdig Monitor or Sysdig Secure and select Settings from the user menu.

2. Select User Profile.

   

3. Review settings and perform actions below.

Review User Email, Role, and Current Team

The current user’s login email address, current team, and role on that team are listed in the User Profile section.

Change Admin Settings

This option is visible to admins only.

If logged on as Administrator, you can access Admin Settings on this page which apply globally.

Hide Agent Install: Toggle this slider to hide the Agent Installation link in the Settings menu from non-admin users.

See Navigate the Settings Panel: Admin vs User and Agent Installation: Overview and Key for more information.

Change Your Password

Use the Password Management fields to change this user’s password.

Required: Minimum 8 characters, not the last password used.

Recommended: We advise following NIST’s most up-to-date recommendations, with an emphasis on length and uniqueness.

Enable Beta Functions from Sysdig Labs

Toggle the feature settings listed under Sysdig Labs to enable/disable specific beta functionalities to your installation. Data that has already been stored will not be affected by beta toggles.

(If there are no beta features, Sysdig Labs will not be displayed.)

1.1.1 - Retrieve the Sysdig API Token

When using the Sysdig API with custom scripts or applications, an API security token (specific to each user+team) must be supplied.

1. Log in to Sysdig Monitor or Sysdig Secure and select Settings from the user menu.

2. Select User Profile.

   The Sysdig Monitor or Sysdig Secure API token is displayed (depending on which interface and team you logged in to).

   

3. You can Copy the token for use, or click the Reset Token button to generate a new one.

1.2 - User and Team Administration

This page describes the concepts behind Sysdig’s users, teams, and role permissions.

Understanding Sysdig Users

Users in Sysdig are identified by user name, email address, and password or third-party authentication option.

Users are either:

• Invited manually by an Administrator via the Sysdig UI

• Authenticated through a third-party system

• Entered directly in the Sysdig database through the Admin API, which can bypass the invitation process if needed.

When invited, the new user is created in the Sysdig database upon the user’s first successful login to the Sysdig UI. Before the user accepts the invitation, enters a password, and logs in, they have a “pending” status.

System-Based Privileges

From the outset, users in the Sysdig environment have one of three types of system privileges

• (Super) Admin: This is the administrator whose email address is associated with the Sysdig billing account. This user has administrator access to everything. Most relevant in on-prem installations.

• Administrator: Any administrator can grant Admin system privileges to any user. Administrators are automatically members of all teams.

  Administrators can create/delete users; create/configure/delete teams; create/delete notification channels; manage licenses; and configure Agents from links in the Settings menu that are hidden from non-admins.

• User (non-admin): By default, new users have read/write privileges to create, delete, and edit content in the Sysdig interface. They do not see options in the Settings menu that are restricted to Administrators.

  User rights are further refined based on team and team role assignments, as described below.

When a user is created, it is automatically assigned to a default team (described below).

Understanding Sysdig Teams

Teams can be thought of as service-based access control. Teams are created and assigned separately in Sysdig Monitor and Sysdig Secure.

Purpose of Teams

Organizing users into teams enables enforcing data-access security policies while improving users’ workflows. There are different team roles, each of which has read/write access to different aspects of the app.

This limits the exposure of data to those who actually need it, and also makes users more productive by focusing them on data that is relevant to them.

In addition to users, Sysdig Monitor and Secure also support Team based service accounts, which provide excellent automation capabilities. Each service account has it’s own team role, which allows defining fine grained access, as well as expiry date for added security.

The following are some potential use cases for Teams.

• “Dev” vs “Prod”: Many organizations prefer to limit access to production data. Permits isolating physical infrastructure and the applications on top.

• Microservices: Scoping data for individual dev teams to see their own dashboards and field their own alerts. Permits team creation based on logical isolation using orchestration or config management metadata in Sysdig Monitor.

• Platform as a Service: Where Ops teams need to see the entire platform. Enabling certain people to see all data for all services as well as the underlying hardware. This is perfect for managed service providers who are managing a multi-tenant environment, or devops teams using a similar model within their own organization.

• Restricted environments: Limiting data access for security and compliance. Certain services, such as authentication and billing, may have a very specific set of individuals authorized to access them.

• Organizations that need to segment monitoring for efficiency: Wide-ranging use case from very large organizations forming teams to simplify access, to smaller orgs creating ephemeral troubleshooting teams, to teams formed to optimize QA and Support access to system data.

Operations Teams and Default Teams

Out of the box, the Sysdig Platform has one immutable team for each product. Depending on licensing, an organization may use one or both:

• Monitor Operations team

• Secure Operations team

Key traits of the immutable Operations teams:

• The teams cannot be deleted

• Users in Operations teams have full visibility to all resources in that product

• Administrators must switch to the Operations team before changing configuration settings for any team

Administrators create additional teams and can designate any team to become the default team for that product. The number of teams allowed in an environment is determined by licensing.

Users entered in the Sysdig Monitor UI are auto-assigned to the Monitor default team; users entered in the Sysdig Secure UI are auto-assigned to the Secure default team.

Team-Based Roles and Privileges

Users can be assigned roles that expand or limit their basic system privileges on a per-team basis.

For a granular view of all the RBAC setting for default user and team roles, see Detailed Role Permissions.

Custom Roles

If the default roles and permissions don’t meet the specific needs of your organization, you can create your own custom roles. See Manage Custom Roles.

How Team Membership Affects Users’ Experience of the UI

Team membership affects user experience of the Sysdig Monitor or Sysdig Secure UIs in various ways.

At the highest level, the dashboards, alerts, and policy events you see are limited by the settings of the team you are switched to.

In more detail, team settings affect the following:

• Default landing page: The UI entry point is set on a per-team basis.

• Explore tab and dashboards: These are set per-team, per-user and can be shared with the team.

  On first login, all team members see the same Dashboards Assigned to Me view. If a user changes those dashboards, only that user will see the changes.

  Dashboards created while part of a team are only visible to the user when logged in to that team, and if shared, are only visible to other team members.

• Visible data: A team’s scope settings limit the data visible to team members while they are switched to that team, even if a user belongs to other teams with different settings that reveal additional data. In Sysdig Secure, for example, only the policy events that fired within your scope will be visible.

• Alert and Event: These settings are team-wide. Any member of a team can change the team’s alert settings, and any additions or edits are visible to all members of the team.

• Captures: Can only be taken on hosts/containers visible to team members, and members see only the list of captures initiated by other members who were switched to the current team.

• API Token: Note that the Sysdig Monitor API Token found under Settings > User Profile is unique per-user, per-team. See User Profile and Password. This is necessary to enable the generation of Custom Events via the API to target a specific team.

Switching Teams in the UI

Users can switch between all teams to which they’ve been assigned, and Administrators can switch between all teams that have been created.

To do so:

1. Click the user menu in the lower-left corner of the navigation bar.

   The assigned teams for this user are listed under Switch Teams.

   

2. NOTE: With version 3.6.0, you can also search for the teams in the user menu.

3. Click another team name.

   A popup window gives an overview of the new team-based view of the environment. The UI changes according to the team settings.

Onboarding Best Practices

Plan teams and roles strategically to isolate access to data, customize interfaces, and streamline workflows.

In general, administrators should:

• Create teams, invite users, and set roles in a planned manner

• Start with some dashboards and alerts for given teams to get started with

When a user logs in to a team for first time, they will see a wizard introducing dashboards, alerts, etc. specific to that team.

Restricting New User Rights by Default

By default, new users (added manually or through a third-party authenticator) are assigned Advanced User rights. If a administrator wants to limit new users’ rights further, there are several ways to do so.

• Between sending the invitation and the user’s first log in, change the user’s Role in the default Monitor team to Read User.

  Note that there could theoretically be a lag in which the user would briefly have had Edit status.

• Integrate users into Sysdig via the Admin API and define read-only permissions upon import.

• Create a default team, in either Sysdig Monitor or Sysdig Secure, with very limited scope and visibility. Manually assign users to additional teams with broader permissions as needed.

Integrating Users and Teams via API

If you are working with Sysdig Support Engineers to provision users and teams via the Sysdig API, note how the user and team role names within the UI map to the API ROLE names.

User roles

Regular (non-admin) = ROLE_USER

Admin = ROLE_CUSTOMER

Team roles

Advanced user = ROLE_TEAM_EDIT

Standard user = ROLE_TEAM_STANDARD

View-only user = ROLE_TEAM_READ

Team manager = ROLE_TEAM_MANAGER

Service manager (Sysdig Secure only) = ROLE_TEAM_SERVICE_MANAGER

1.2.1 - Manage Users

This page describes how to add, delete, and configure user information from within the Sysdig Monitor or Sysdig Secure UI.

Create a User

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu.

2. Select Users.

3. Click the Add User link.

4. Enter the user’s email address, first name and last name:

   

5. Click Save to send the user invite, or click Cancel to discard the user.

The new user will be added to the User Management table. Their status will be listed as Pending until the invitation is accepted.

Edit User Information

To edit an existing user:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu.

2. Select Users.

3. Select the user from the User Management table.

4. Optional: Edit the first name / last name.

5. Optional: Toggle the Admin switch to enable/disable administrator privileges.

6. Click Save to save the changes, or Cancel to revert the unsaved changes.

Delete a User

To delete an existing user:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu. `

2. Select Users.

3. Select the user from the User Management table.

4. Click Delete User.

5. Click Yes, delete to confirm the change.

   You can optionally delete the dashboards and artifacts that the user have created.

1.2.2 - Manage Teams, Roles, and Service Accounts

The use of teams provides a strategic way to organize groups, streamline workflows, or protect data, as needed by an organization. Administrators who design and implement teams should have in-depth knowledge of organizational infrastructure and goals.

For more information, including foundational concepts, see User and Team Administration.

Create a Team

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and

   select Settings from the user menu.

2. Select Teams.

3. Click Add Team.

4. Configure the team options and click Save.

For more information on each configuration option, refer to Team Settings.

Team Settings

Configure an Entry Page or Dashboard for a Team

Some Sysdig Monitor teams benefit from using a default entry point other than the usual Explore page, as users who don’t need in-depth monitoring information can onboard and navigate Sysdig Monitor more efficiently.

Use the Default Entry Point setting on the Team page, as shown in Create a Team.

Note: If selecting a dashboard, open the secondary Dashboard drop-down menu, or type the name of the dashboard to select it.

The dropdown is only populated with shared dashboards accessible by anyone on the team.

Add and Configure Team Members

Users can be assigned to multiple teams. Team assignment is made from the Team page (not the User page), and must be done by an Administrator or Team Manager.

Assign a User to a Team

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and

   select Settings from the user menu.

2. Select Teams.

3. Select the relevant team from the list, or search for it with the search box, and then select the relevant team.

4. In the Team Users section, click the Assign User button.

5. Select the user from the drop-down list, or search for it and then select it.

6. Click the Role drop-down menu to select the user role:

   

7. Optional: Repeat steps 3 to 5 for each additional user.

8. Click Save.

Assign a Team-Based Role to Users

Review Team-Based Roles and Privileges for an overview.

Note that the Advanced User permission can be further refined into either a View-only user or a Team Manager.

Managers can add or delete members from a team, or toggle members' rights between Edit, Read, or Manager.

Note that Admins have universal rights and are not designated as Team Managers, Advanced Users, View-Only users, or Standard users.

Manager or Advanced User permissions can be assigned even to Pending users; administrators do not have to wait for the user’s first login to set these roles.

To assign a role to a user on a team:

1. Log in to Sysdig Monitor or Sysdig Secure as Administrator and either create a team or select a team to edit.

2. Add a user or select a user from the list of team members.

3. Select the appropriate role from the drop-down menu.

   

   Reminder of the role privileges:

   Admin: Member of every team with full permissions. Can create/delete/configure all users and teams.

   Team Manager: Advanced User privileges + ability to add/delete team members or change team member permissions.

   Advanced User:

   In Sysdig Monitor: Read/write access to the components of the application available to the team. Can create/edit/delete dashboards, alerts, or other content.

   In Sysdig Secure: Read/write access to the components of the application available to the team. Can create, delete, or update runtime policies, image scanning policies or any other content.

   View-Only:

   In Sysdig Monitor: Read access to the environment within team scope, but cannot create, edit, or delete dashboards, alerts, or other content.

   In Sysdig Secure: Read access to every Secure feature in the team scope, but cannot modify runtime policies, image scanning policies or any other content.

   Standard User:

   In Sysdig Monitor: An Advanced User with no access to the Explore page (e.g. for developers who are not interested in Monitoring information).

   In Sysdig Secure: Can send container images to the scanning queue, view image scanning results, and display the runtime security events within the team scope. Standard Users cannot access Benchmarks, Activity Audit, Policy definitions, or certain write functions within other Secure features.

   Service Manager: Sysdig Secure only. Same as Standard User, plus ability to invite existing users to the team and manage the notifications channels assigned to the team.

4. Save edits.

Edit Team Configuration

To configure an existing team:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and

   select Settings from the user menu.

2. Select Teams.

3. Select the relevant team from the list, or search for it with the search box, and then select the relevant team.

4. Edit as needed, and click Save.

For more information regarding the configuration options, see Team Settings.

Delete a Team

When a team is deleted, some users may become “orphans”, as they are no longer a part of any team. These users will be moved to the default team.

The default team cannot be deleted. A new default team must be selected before the old default team can be deleted.

To delete a created team:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and

   select Settings from the user menu.

2. Select Teams.

3. Select the relevant team from the list, or search for it with the search box, and then select the relevant team.

4. Click Delete Team, then Yes, delete to confirm the change.

Service Accounts

Service Accounts are team based and are available when editing a team. Service Accounts can be used instead of users’ API keys to access Sysdig APIs by applications or scripts. Service accounts are not bound to a user, but to a team. You can generate as many team service accounts as needed. Each service account must have exactly one role.

Unlike users, service accounts have 0 permissions out of the box. They have only the permissions coming from the role you assign to them. In addition, these tokens are not retrievable after they are generated and have a pre-defined retention time.

When creating a team-based Service account you need to define:

• Name: Arbitrary token name
• Role: Any role from the list of previously defined roles
• Expiration: 14 days, 90 days, 6 months or 1 year (custom expiration is possible via API only)

1.2.3 - Manage Custom Roles

A custom role is a admin-defined role which allows Sysdig administrators to bundle a set of permissions and allocate it to one or more users or teams. This page describes how to create and use custom roles.

Custom Roles is supported only on SaaS. The feature is not currently available for on-prem environments.

Understand Custom Roles

Custom roles give you the ability to provide granular access to users according to a selected list of permissions. If the default user and team roles don’t meet the specific needs of your organization, you can create your own custom roles. Select the permissions you want them to have based on the resource they should have the access to and bundle it together. Just like built-in Sysdig roles, you can assign custom roles to users and teams. Custom roles ensures that the users have only the permission they need and help prevent unwanted access to other resources.

Custom roles operate on concepts similar to roles-based access control system (RBAC).

Benefits of Using Custom Roles

• Allow you to give access to a specific set of predefined dashboards to a group of users, who should not be able to view any additional data, nor change or share these dashboards.

• Allow you to create a service account for Sysdig Secure that is not tied to a particular user but can be used to automate your CI/CD pipeline.

  • Give custom set of permissions to the CI/CD account
  • Give permission to create these accounts to a certain set of users

• Allow you to identify the owner of a particular image so the security issue can be assigned to the actual team who owns the issue.

• Create a team role that can only invite users but not actually manage the team.

Create a Custom Role

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings.

2. Select Roles.

3. Click New Role. The New Role page is displayed.

4. Specify the following:

   • Role Name: A unique name to identify the role you create.
   • Role Description: A short explanation of the role that you have created.
   • Product: A filter that gives a fine-grained view of the product-specific features.

5. Select the features and do one of the following:

   • From the drop-down, select one of the following: No Access, Read Only, Full Access, Custom.
   • Click Customize to provide grant granular permissions to a sub-set of features. This is an alternative to clicking Custom from the drop-down.

6. Click Save New Role.

Assign a Custom Role to Teams

You can set up a custom role as the default user role for teams. To do so:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings.

2. Select Teams.

3. Do one of the following:

   • Select the relevant team from the list of teams.
   • Click Add Team.

4. From the Default User Role drop-down, select one of the custom role you have created.

   

5. Complete creating or editing the team as given in Manage Teams and Roles.

6. Click Save.

Custom Roles and Privileges

Click Customize to view and select granular permissions for each product features. Alternatively, use the drop-down to grant read access or full access to all the privileges simultaneously.

Sysdig Monitor

Sysdig Secure

1.2.4 - Detailed Role Permissions

When deciding whether to use default team roles or create a custom role, it can be helpful to review the RBAC permissions that Sysdig grants to the roles of Standard User, Advanced User, etc.

Sysdig Monitor

Standard User

View Only

Team Manager

Advanced User

Sysdig Secure Team Roles

Standard User

Service Manager

View Only

Team Manager

Advanced User

1.2.5 - Group Mappings

Group mapping is beneficial to:

• Manage permissions for and access to Sysdig resources from your organization’s IdP itself.

  For example, to allow your Analytics team to access a set of Dashboards, you can create a group named Analytics and grant group members access only to the dashboards they need access to.

• Update or completely remove user access to Sysdig resources as soon as it’s updated in the IdP.

As an admin, you can

• Enter one or more IdP groups and assign a custom role and map teams.

• Map a group to one of more teams, or all the teams.

• Select a user role for each group individually.

When group mapping is enabled:

• Group mapping will ignore the users that are manually set as administrators, allowing them to perform administrator functions without having the mapping permissions overwriting their existing permissions.

• If a user does not belong to any of the mapped groups, or the mapping is misconfigured, the user will be assigned to the default team with the default role.

• If user creation is disabled while group mapping is enabled, non-existing users will not be created. However, the team and role information associated with the existing users will be processed on each login.

Enable Group Mapping

To enable groups mapping in Sysdig Secure or Sysdig Monitor:

1. Navigate to Settings > Authentication.

2. Select SAML from Connection Settings.

3. Enable Group Mapping.

   

4. Specify the Group Attribute Name.

   It is the configurable metadata of an IdP group that is used in the SAML assertion statement. Sysdig uses this SAML attribute to identify the group and determine associated permissions. This value is processed on every login attempt to read the groups that the user belongs to.

5. Click Save Settings.

Add a Mapping

You can map a group to one role and one or more teams.

1. Navigate to Settings > Group Mappings.

2. Enter the Group ID. This is the unique name assigned to the group on the IdP side.

3. Select a role from the Role drop-down. You can select only one role for a group maaping. Ensure that the roles aren’t conflicting with each other because the mapping will not work if there are conflicting roles for a user.

4. Select one of more teams from the Teams drop-down.

5. Optionally, add additional mapping by clicking Add Group and repeating the same steps.

6. Click Save.

1.3 - Notifications Management

Alerts are used in Sysdig Monitor when Event thresholds have been crossed, and in Sysdig Secure when Policy violations have occurred. Alerts can be sent over a variety of supported notification channels.

Notification Management describes how to add, edit, or delete a variety of notification channel types, and how to disable or delete notifications when they are not needed, for example, during scheduled downtime.

1.3.1 - Set Up Notification Channels

Alerts are used in Sysdig Monitor when Event thresholds have been crossed, and in Sysdig Secure when Policy violations have occurred. Alerts can be sent over a variety of supported notification channels.

In the Settings panel of either Sysdig Monitor or Sysdig Secure, set up the notification channels to be used for alerting.

Notification channel management can be finessed by role-based access as follows:

• Notification channels can now be “global” or limited to a particular team

• Global channels can be managed by admins and can be viewed/used by other roles, while team-limited channels are available only to team members

• Team Manager , Advanced User, and Service Manager (Secure) roles can create/update/delete team-scoped notification channels, they can also read and use the global ones

• Standard and View Only roles can read team-limited and global notification channels

• Admins will be able to create global notification channels and migrate channels from “global” to “team-limited”, and also from one team to another.

Add a Notification Channel

To add a new notification channel:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu.

2. Select Notification Channels.

   The Notifications main page is displayed:

3. Click Add Notification Channel +, and select the desired notification channel.

   

4. Follow the channel-specific steps to complete the configuration process:

   • Amazon SNS Notifications

   • Email Notifications

   • Team Email Notifications

   • PagerDuty Notifications

   • Slack Notifications

   • VictorOps Notifications

   • OpsGenie Notifications

   • Configure a Webhook Channel

   • Configure a Google Chat Channel

   • Configure IBM Cloud Functions Channel

After you have set up a notification channel, it will appear as an available option to be assigned when you Add an Alert.

Edit a Notification Channel

To edit a notification channel:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings.

2. Select Notification Channels.

3. Locate the target channel and click the Edit button.

4. Make the edits and click Done Editing to save the changes.

Test a Notification Channel

To test a notification channel:

1. Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings.

2. Select Notification Channels.

3. Select the three dots next to a created Notification Channel and click Test Channel.

   

Report Unsuccessful Notification Attempts

When an unsuccessful notification has been attempted on a given notification channel, [Sysdig Events](/en/docs/Sysdig Monitor/Events/Event Types/#sysdig-events) are generated to warn you about it. At the fifth failed notification attempt, the notification channel will be disabled and a corresponding Sysdig Event will be generated. To view the list of Sysdig Events:

1. Log in to Sysdig Monitor and select Events.

2. On the Events page, select Sysdig from the All Types drop-down.

   

1.3.1.1 - Amazon SNS Notifications

Sysdig Monitor integrates easily with AWS Simple Notification Service (SNS).

On the AWS side:

To automatically push Sysdig Monitor alerts to the SNS topic of your choice:

1. From the AWS console, open the SNS management console

2. In the Create topic section, Create a new topic (if needed).

   The topic’s Name, ARN, (optional) Display name, and Topic owner’s AWS account ID are displayed in the Details section.

3. Select the topic on the list.

4. Expand Access policy - optional.

5. Select Basic (By default).

6. Under Define who can publish messages to the topic, select Only the specified AWS accounts and enter the Sysdig Monitor account ID: 273107874544 (US-East Only).

   

   For account IDs corresponding to other regions, see SaaS Regions and IP Ranges.

7. Click Create topic.

8. Ensure that you subscribe to the created topic.

   1. On the navigation panel, choose Subscriptions.

   2. On the Create subscription page, enter the Topic ARN of the topic you created earlier.

   3. Specify other details and click Create subscription.

For further information about AWS SNS, refer to the AWS documentation.

For SNS notification, you can click the ‘help’ button for tips on setting up your SNS topic.

You will need to allow publishing rights to the Sysdig Monitor account ID corresponding to your region.

This can be done by creating a new policy on your SNS topic in AWS Console as shown in the below images:

1. Select “Edit topic policy” as shown below from “Other topic actions.”

   

2. In the “Basic view” tab of the “Edit topic policy” dialog, select “Only these AWS users” from the publisher’s list and enter the Sysdig ID.

   

In the Sysdig Monitor UI:

1. Complete steps 1-3 in Set Up a Notification Channel to log in to the Sysdig UI and select Amazon SNS Topic.

   

2. Enter the Topic created on the AWS side, along with a Channel Name, Enablement, and Notification toggles as appropriate.

3. From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

4. Click Save.

1.3.1.2 - Email Notifications

To send an alert notification via email, you must first set up the email notification channel.

To do so, complete steps 1-3 in Set Up a Notification Channel, then:

1. Select Email.

2. Enter the relevant details for the email notification:

   

3. From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

4. Click Save.

   If you enabled Test notification, a test email will be sent.

You can now configure an alert to use email notifications.

1.3.1.3 - Team Email Notifications

You can notify all the users of a team when an alert is triggered. Sysdig allows you to create a new notification channel where you can select a team from a list of existing ones as the target of the channel.

To send an alert notification via email to a team, you must first set up the team email notification channel. To do so, complete steps 1-3 in Set Up a Notification Channel, then:

1. Select Team Email.

2. Enter the relevant details for the email notification:

   

   Select the Team Name and specify a Channel Name to identify the channel you are creating.

3. From Shared With drop-down, choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

4. Click Save.

   If you enabled Test notification, a test email will be sent.

You can now configure an alert to use email notifications.

1.3.1.4 - PagerDuty Notifications

To send an alert notification via PagerDuty, you must first set up the PagerDuty notification channel.

Prerequisites

• Have an account configured at PagerDuty.com.

• Have your PagerDuty credentials available (account, password, and service).

• Have the base user role of Manager. With a PagerDuty base user role of Manager, you can auto-fetch the service information during the Sysdig/PagerDuty integration process.

• Check your team and base user permissions. If your PagerDuty team permissions are Manager but base user permissions are Responder or lower, you can enter the necessary data in the Sysdig UI manually.

  

  Base user roles in the PagerDuty UI.

Configure PagerDuty

1. To launch the process from the Sysdig UI, complete steps 1-3 in Set Up Notification Channels and select PagerDuty.

2. Do one of the following:

   • Select Auto-fetch when prompted. Ensure that you have the base user role of Manager or higher in PagerDuty.

   • Select Manual and enter the necessary configuration parameter. Skip to Step 5 for details.

   Once you complete the pre-configuration, the PagerDuty Integration screen is displayed.

   

3. Do one of the following:

   • Enter the email and password associated with your PagerDuty account, and click Sign In.

   • Enter the appropriate PagerDuty subdomain for single sign-on and click Sign In Using Your Identity Provider.

4. A PagerDuty service selection screen is displayed.

   • Option 1: If you have never integrated before, you are prompted to choose a PagerDuty Servicename.

   • Option 2: If at least one service has already been integrated, you can select that one.

   

5. Click Connect.

   Once integration is authorized, the Sysdig page for a new PagerDuty notification channel is displayed, with the information auto-filled.

   

6. From Shared With, choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

7. Do one of the following:

   • Confirm the auto-populated information and click Save.

   • If you chose Manual entry in Step 2, then type the information and click Save.

You can now Add an Alert to use PagerDuty notifications.

Known Issues

There is a known issue whereby changing a notification from “Acknowledged” to “Unacknowledged” does not update correctly in PagerDuty.

What occurs:

• Event has triggered Notification, Notification is sent to PagerDuty.

• Open Event and click on “Acknowledge” button in Sysdig.

• Notification is sent to PagerDuty, and status is changed to “Acknowledged.”

• Open Event and click on “UnAcknowledge” button in Sysdig.

  

  Status is not changed in PagerDuty. It remains “Acknowledged” rather than changing to “Triggered” in PagerDuty.

Unknown issues & support

Sysdig support section.

1.3.1.5 - Slack Notifications

To send an alert notification via Slack you must first set up the Slack notification channel and have a Slack account configured at Slack.com.

Configuration

To launch the process from the Sysdig UI:

1. Complete steps 1-3 in Set Up Notification Channels and select Slack.

You will be prompted to log in to your Slack account.

2. Select a Slack channel from the drop-down list to be used for notifications and click Authorize.
3. Determine whether to apply this channel globally (All Teams) or to a specific team.
4. Complete configuration as desired and click Save.
5. Optionally, click Test to check if the slack notification works as expected.

You can now configure an alert to use Slack notifications.

Customize Notifications

You can customize the sections that will be included in the messages sent to each configured Slack channel. This gives you the ability the fine tune the contents of Slack messages according to your preferences.

1.3.1.6 - VictorOps Notifications

To integrate with your VictorOps

1. Log in to VictorOps.

2. Go to Settings > Alert Behavior > Integrations in the VictorOps interface.

3. Select REST from the list of Featured Integrations.

   

4. Complete steps 1-3 in Set Up a Notification Channel to log in to the Sysdig UI and select VictorOps.

   

5. Enter the VictorOps parameters in the Sysdig Notification Channel fields, as follows:

   API Key: everything between "/alert/" and “/$routing_key” in the REST URL

   Routing Key: A VictoOps way of routing alerts to appropriate teams. See their Routing Keys documentation for details, if needed.

   Channel Name: Choose a meaningful name like “VictorOps”.

   Enable the channel and desired notification types.

6. From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

7. Click Save.

1.3.1.7 - OpsGenie Notifications

1. Go directly to the OpsGenie Integrations Page to configure the integration on the OpsGenie side.

   OpsGenie maintains documentation on how to integrate with Sysdig products (formerly called Sysdig Cloud) here.

2. Complete steps 1-3 in Set Up a Notification Channel to log in to the Sysdig UI and select OpsGenie.

   

3. Copy/paste your OpsGenie integration API key and add a Channel Name, Enablement, and Notification toggles as appropriate.

4. From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

5. Click Save.

1.3.1.8 - Microsoft Teams Notifications

Sysdig Monitor supports sending an alert notification to Microsoft teams. Microsoft Teams has different types of integrations for third-party applications, of which Sysdig supports Incoming Webhooks.

About Incoming Webhooks

Incoming Webhooks are a type of Connector in Teams that provide a simple way for an external app to share content in team channels. They are often used as tracking and notification tools. Microsoft Teams provides a unique URL to which you can send a JSON payload with the message that you want to POST, typically in a card format. Cards are UI containers that contain content and actions related to a single topic and are a way to present message data in a consistent way.

You will need to enter the URL that you copied from the Connector. Sysdig will format a message by using a custom card template and send it to the channel. The message will show up as a new notification in the Microsoft application.

Prerequisites

• Have the destination URL handy. You can copy it from the Connectors > Incoming Webhook window on the Microsoft Teams UI. For more information, see Add an incoming webhook to a Teams channel.

  

• Webhooks via HTTPS work only if a signed or valid certificate is in use.

Enable Microsoft Teams

1. Complete steps 1-3 in Set Up a Notification Channel and choose Microsoft Teams.

   

2. Enter the configuration options:

   • URL: The destination URL you have copied from Microsoft Teams UI.

   • Channel Name: Add a meaningful name for your Microsoft Teams channel.

   • Enabled: Toggle on or off.

   • Notification options: Toggle for notifications when alerts are resolved or acknowledged.

   • Test notification: Toggle to be notified that the configured URL is working.

   • Shared With: Choose whether to apply this channel globally. All Teams or to a specific team from the drop-down.

3. Click Save.

1.3.1.9 - Prometheus Alertmanager Notifications

Sysdig supports integrating Prometheus Alertmanager using a custom webhook.

Prerequisites

• Webhooks via HTTPS only work if a signed/valid certificate is in use.

• Have your desired destination URL on hand.

Enable Prometheus Alert Manager

1. Complete steps 1-3 in Set Up a Notification Channel and select Prometheus Alert Manager.

   

2. Enter the Prometheus Alert Manager channel configuration options:

   • URL: The destination URL to which notifications will be sent.

   • Channel Name: Add a meaningful name, such as Prometheus channel.

   • Enabled: Toggle on and off notifications.

   • Notification options: Toggle for notifications when alerts are resolved or acknowledged.

   • Test notification: Toggle to be notified that the configured URL is working.

   • Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

   • Allow insecure connections: Enable if you want to skip the TLS verification.

   • Custom headers: Add custom headers to your alert notification. If your alertmanager channel requires additional headers you specify them by using a custom header.

     Alternatively, you can choose to add custom headers programmatically as described in Configure Custom Headers Programmatically.

3. Click Save.

When the channel is created, you can use it on any alerts you create.

When the alert fires, the notification will be sent as a POST in JSON format to your webhook endpoint.

For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.

Configure Custom Headers Programmatically

Alert notifications, by default, follow a standard format. However, some integrations require additional headers which you can append to the alert format by using a custom header entry.

For example, some applications uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default alert template built into Sysdig, but you can add it using a custom header.

The following example adds two custom headers:

1. Use the curl command to retrieve all configured notification channels:

   curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'Authorization: Bearer API-KEY'
   

2. Add the custom headers and execute the request:

   curl -X PUT https://app.sysdigcloud.com/api/notificationChannels/1 -H 'Authorization: Bearer API-KEY' -H 'Content-Type: application/json' -d '{
     "notificationChannel": {
       "id": 1,
       "version": 1,
       "type": "PROMETHEUS_ALERT_MANAGER",
       "enabled": true,
       "name": "Test-Sysdig",
       "options": {
         "notifyOnOk": true,
         "url": "https://hookb.in/v95r78No",
         "notifyOnResolve": true,
         "additionalHeaders": {
           "Header-1": "Header-Value-1",
           "Header-2": "Header-Value-2"
         }
       }
     }
   }'
   

Learn More

• alertmanager
• Alertmanager Configuration

1.3.1.10 - ServiceNow Notifications

Sysdig can be integrated with ServiceNow using a custom webhook.

ServiceNowSetup

Prerequisites

• Have a ServiceNow account set up and working.

• If necessary, refer to ServiceNow developer documentation here.

Create Scripted Rest API Details in ServiceNow GUI

1. Login to ServiceNow (developer entry) and create a Scripted REST API:

   

2. Click New and submit the form with the following:

   Name: SysdigAlert API ID: sysdigalert

3. Return to the Scripted REST APIs and open the resource just created.

   Scroll down to the related list area, select Resources, and click New. This will create a new Scripted REST API resource.

4. Fill in the Name field e.g. Demo.

   

5. Scroll down to Security and clear the checkbox that requires authentication.

   

6. Change the HTTP method from GET to POST.

The resource is created.

Add Code to the New Scripted API

Now give the resource the code to execute.

The default objects to work with in a Scripted REST API Resource are response and request.

For more details on request and response see Scripted_REST_Request_API and Scripted_REST_Response_API

The created resource will already have some example code:

(function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {

    // implement resource here

})(request, response);

1. Change this default code to:

   (function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
   
    gs.info(request.body.dataString);
   
   })(request, response);
   

2. Note the following resource path to this newly created resource is now visible: /api/snc/sysdigalert.

   The url to this resource would be https://yourInstance.service-now.com/<resource_Path or https://yourInstance.service-now.com/api/snc/sysdigalert ``

   

3. Click Submit/Update on this resource.

Sysdig Webhook Setup

Now that the custom API endpoint in ServiceNow is created, you can configure Sysdig alerts to use a custom webhook to trigger the ServiceNow integration.

API URL: your instance name URL

Name: ServiceNow (or whatever name you’d like for this Sysdig alert webhook)

Notify when OK: Optional

Notify when Resolved: Optional

Test Notification: Use this toggle and/or set up a test alert as described in the following section.

Test Integration

To test if this ServiceNow integration is set up and working correctly, you can set up a test alert to trigger.

For example, you could create an alert for CPU usage:

In ServiceNow, navigate to System Log > All to see a sample triggered webhook.

1.3.1.11 - Configure a Custom Webhook Channel

Sysdig Monitor supports using a custom webhook notification channel to send custom alert notifications to any destination. This is often useful for having Sysdig create alerts with custom tools or other webhook-based applications for which Sysdig does not yet have a native integration.

The custom webhook notification channel allows you to customize payload from within Sysdig Monitor based on alert properties by dynamically referencing alert variables and scope labels. To create the payload, Sysdig provides you with the Sysdig Templating Language that allows for variable interpolation and basic conditional logic.

Enable Webhook

Basic Configuration

1. Complete steps 1-3 in Set Up a Notification Channel and select Custom Webhook.

   

2. Enter the webhook channel configuration options:

   • URL: The destination URL to which alert notifications will be sent.

   • Channel Name: Add a meaningful name, such as Trello, PagerDuty, OpsGenie, and so on.

   • Enabled: Toggle enabling and disabling the webhook channel.

   • Notification options: Send notifications when alerts are resolved or acknowledged.

   • Test notification: Send a test notification upon saving to confirm that the webhook URL is reachable.

   • Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

3. Complete the steps described in Method and Headers, Payload, and Configure Test Notification.

4. Click Save.

When the channel is created, you can use it on any alerts that you create.

When the alert fires, the notification will be sent as an HTTP request of the specified method in JSON format to your webhook endpoint.

For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.

Method and Headers

• HTTP Method: You can choose an HTTP method for the request to be sent.

  • POST: Used for the creating resources.
  • PUT: Used for the editing an existing resource.
  • PATCH: Used for partial modifying an existing resource.
  • DELETE: Used for deleting a resource.

• Content Type: For the custom webhook channel, application/json will always be the value of the content-type header for the request, even if this header does not appear on the headers list.

• Allow insecure connections: Check this option to skip TLS verification over HTTPS.

• Custom headers: Add custom headers to your alert notification.

  If your webhook integrations require additional headers you specify them by using a custom header.

  For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default header, but you can add it using a custom header, such as Authorization: Bearer mytoken. Use this component to specify suitable header names and header values.

  Note that any Content type, Accept, and User-Agent header specified in this component will be discarded, as the default header value of Content-type:application/json cannot be overridden.

Payload

Editor: The custom webhook channel allows you to fully customize the JSON payload that you want to send. The content of the editor represents the request body of the HTTP call you are sending to the third-party webhook. The data must be a valid JSON document. Systems that receive webhook alerts can parse the data and take action based on the contents.

When triggered, the webhook channel allows dynamic referencing to properties of the alert and event which were created using the Sysdig Templating Language. Using the same system, basic conditional logic can also be written.

Configure Test Notification

As an alternative to sending a test notification upon creating the channel, you can also test the channel on the fly before creating it. Use the Configure Test Notification editor to do so.

In the editor, you can pick a variety of scenarios, ranging from different severity alerts to alert types, and trigger a test notification to the destination webhook URL.

Build Payloads

Use the Sysdig Templating Language to customize your payload using variable interpolation and basic conditional logic:

• Reference alert variables with {{@alert_, for instance {{@alert_name}}
• Reference event variables with {{@event_, for instance {{@event_state}}
• Use conditional logic with {{#if_, for instance {{#if_metric_alert}}{{/if}}
• Reference labels with {{@event_labels}} for all, or specifically with for instance {{@event_labels.kube_cluster_name}}

See Reference for all the allowed syntax parameters.

Any text not matching the above keywords is accepted and dynamically rendered when the alert is triggered.

For more information, browse the on-screen documentation and examples that is provided in the form of snippets next to the code editor.

Use Case: Integrate with Trello

This section helps you create a custom webhook channel and integrate with Trello. When a configured alert triggers in the channel, a new Trello card is created with information about the escalation.

Prerequisites

1. A Trello account with admin rights.

2. Obtain your trello apiKey and token.

   For more information, see the REST API Authorization guide.

3. Create a Trello board on your account and ensure that you create at least one list on it.

4. Perform the following request in order to obtain the id of one the lists existing on the board:

   $ curl --request GET --url 'https://api.trello.com/1/boards/<boardId>/lists?key=<key>&token=<token>

   Where boardId is the ID present on the URL in the browser when accessing the board’s main page, and key and token are the authentication requirements obtained in step 2.

   An example response is given below:

   [{"id":"63c68794fb31c10334ab443b","name":"Escalations","closed":false,"idBoard":"63c68794fb31c10334ab4434","pos":16384,"subscribed":false,"softLimit":null},...]
   

Where you can see that the board contains a list named Escalations. Copy and save the ID for the webhook configuration.

Configuration

With all prerequisites are met, you can now configure a webhook which uses Trello’s REST API to create a card on a list.

• URL: Type the following:

  https://api.trello.com/1/cards?idList=<idList>&key=<apiKey>&token=<token>

  This is the REST route for creating a card.

• Name: Enter Trello Escalation Card.

• Method: Make sure that you select the POST method.

• Editor: Type the following content on the editor and define the structure of the card you want to create:

  {
     "name": "Escalation: {{@alert_name}} on {{@event_labels.kube_cluster_name}}",
     "desc": "{{@event_body}}",
     "pos": "top"
  }
  

This snippet creates a card whose name suggests that an escalation for the given alert name is just triggered. It also informs on which cluster the incident occurred by accessing directly the kube_cluster_name variable from within event_labels. Note that in order to reference this variable, the original alert that was fired needs to group or filter by this variable.

For the description field, provide the standard Sysdig event body, which is similar to the default message produced by an email notification channel.

"pos": "top" indicates that you want this new card to be created on top.

Testing

Experiment with different testing configurations to see how the resulting card appears to the recipient.

Save

Save the channel and configure an alert that triggers on it.

Reference

Sysdig Templating Language is composed of two tipes of constructs: statements and variables. Both of them are accessed from within {{ }} brackets, which are the only special characters in the payload that you define. Everything else is interpreted as plain text and passed along as is.

Statements

Statements allow you to perform basic conditional logic. The common use case would be sending a different alerting JSON payload in case of a high severity alert, such as:

{
    {{#if_severity_high}}
        "code": "incident"
    {{#else}}
        "code": "warning"
    {{/if}}        
}

Whenever an alert triggers on this notification channel, the template is evaluated. In the presence of a high severity alert, the following payload is sent:

{
    "code": "incident"      
}

Whereas for all other cases, the following is sent:

{
    "code": "warning"      
}

Statements thus allow for conditional rendering of different sections of the payload.

All statements

if_severity_high

Conditional statement that matches an alert that was set with high severity.

if_severity_medium

Conditional statement that matches an alert that was set with medium severity.

if_severity_low

Conditional statement that matches an alert that was set with low severity.

if_severity_info

Conditional statement that matches an alert that was set with info severity

if_metric_alert

Conditional statement that matches a metric alert.

if_downtime_alert

Conditional statement that matches a downtime alert.

if_prometheus_alert

Conditional statement that matches a prometheus alert.

if_event_alert

Conditional statement that matches an event alert.

if_anomaly_alert

Conditional statement that matches an anomaly detection alert.

if_outlier_alert

Conditional statement that matches an outlier alert.

if_resolved_event

Conditional statement that matches an event which has been resolved.

if_main_threshold

Conditional statement that matches an event triggered on a main alert threshold.

if_warning_threshold

Conditional statement that matches an event triggered on a warning alert threshold.

Variables

Variables allow you to access alert and event properties. These are dynamically substituted upon rendering with the underlying properties of the configured alert or triggered event.

Variables of type string are automatically escaped, but enclose them in quotes in order to generate a valid JSON payload.

Alert Variables

alert_id

Type: Number.

The auto-generated unique alert identifier.

Example: 1.

alert_name

Type: String.

The configured alert name.

Example: High CPU Usage alert

alert_description

Type: String.

The configured alert description.

Example: Alert for when cpu load goes over 75%.

alert_scope

Type: String.

The configured alert scope.

Example: kube_pod_name in ('backend-api')

alert_severity

Type: String.

The configured alert severity label.

Possible values: HIGH, MEDIUM, LOW, NONE.

Example: HIGH.

alert_group_by

Type: JSON String Array.

A JSON String array with all the labels on which the alert has been grouped by.

This applies only to metric and event alerts.

Example: ['kube_cluster_name', 'agent_id']'.

alert_condition

Type: String

The configured alert main condition.

This applies only to metric and event alerts.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 75.0.

alert_warning_condition

Type: String

The configured alert warning condition.

This applies only to metric and even alerts.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 50.0.

alert_timespan

Type: Number

The configured alert timespan.

Example: 600000000.

alert_url

Type: String

The link to Sysdig Monitor’s Alert edit page.

Example: https://app.sysdigcloud.com/#/alerts/rules/123456.

Event Variables

event_state

Type: String

Possible values: ACTIVE for triggered conditions and renotifications, OK for resolved conditions.

Example: ACTIVE.

event_trigger_timestamp

Type: Number

The microseconds timestamp in which the event fired.

Example: 1674140923000000.

event_resolved_timestamp

Type: Number

The microseconds timestamp in which the event resolved.

Example: 1674140923000000.

event_trigger_metadata

Type: JSON Object.

A JSON object, with metadata related to the triggering or renotification event, including the metric value that generated the event.

Example:

{
  "entity": "host_mac = '08:00:27:70:1a:03' and container_name = 'container1_0'",
  "metricValues": [
    {
      "metric": "sysdig_container_cpu_used_percent",
      "aggregation": "avg",
      "groupAggregation": "avg",
      "value": 93.2359617776846
    }
  ]
}

event_resolved_metadata

Type: JSON Object.

A JSON object, with metadata related to the resolve event, including the metric value that cleared the alert condition.

Example:

{
  "entity": "host_mac = '08:00:27:70:1a:03' and container_name = 'container1_0'",
  "metricValues": [
    {
      "metric": "sysdig_container_cpu_used_percent",
      "aggregation": "avg",
      "groupAggregation": "avg",
      "value": 33.2359617776846
    }
  ]
}

event_labels

Type: JSON Object

A JSON String to String object, mapping label names from the event’s context to their values.

Example:

{
    "kube_cluster_name": "my-cluster",
    "agent_id: "123"
}

event_title

Type: String

The configured title for the event.

Example: High CPU Usage is Triggered on my-cluster.

event_body

Type: String

The body text for the event.

Example:

TEST ALERT: Testing Notification Channel dasdads

Event Generated:

Severity: Low
Metric: sysdig_container_cpu_used_percent = 93.236 %
Segment: container_name = 'container1_0' and host_mac = '08:00:27:70:1a:03'
Scope: host_hostname = 'Host-0 (08:00:27:70:1a:03)'

Time: 01/19/2023 03:10 PM UTC
State: Triggered
Notification URL: https://app-staging.sysdigcloud.com/#/events/notifications/l:2419200/2168/details

------

Triggered by Alert:
Name: TEST ALERT: Testing Notification Channel dasdads
Description: Alert description
Team: Team for standard users.
Scope:
    host_hostname = 'Host-0 (08:00:27:70:1a:03)'
Segment by: host_mac, container_name
Alert When: avg(sysdig_container_cpu_used_percent) > 85
For at least: 10 m
Alert URL: https://app-staging.sysdigcloud.com/#/alerts/rules?alertId=4322

event_condition

Type: String

The condition that made the event trigger or resolve.

Example: avg(avg(sysdig_container_cpu_used_percent)) > 75.0.

event_id

Type: Number

The internal id for the event.

Example: 123456.

event_username

Type: String

The username of the user responsible for triggering the event, if applicable, such as in the cases of test notifications.

Example: user.name@sysdig.com

event_url

Type: String

The url to the event details page within Sysdig Monitor.

Example: https://appsysdigcloud.com/#/event-feed?last=3600&eventId=123456

event_threshold

Type: String

Possible values: MAIN for main condition triggering events, WARNING for warning condition triggering events.

1.3.1.12 - Configure a Webhook Channel

Sysdig Secure supports sending an alert notification to a destination, such as a website, custom application, and so on for which Sysdig does not have a native integration. Do this using a Webhook channel.

Prerequisites

• Webhooks via HTTPS only work if a signed/valid certificate is in use.

• Have your desired destination URL on hand.

Enable Webhook

1. Complete steps 1-3 in Set Up a Notification Channel and choose Webhook.

   

2. Enter the webhook channel configuration options:

   • URL: The destination URL to which notifications will be sent.

   • Channel Name: Add a meaningful name, such as Ansible, PagerDuty, OpsGenie, and so on.

   • Enabled: Toggle on and off notifications.

   • Notification options: Toggle for notifications when alerts are resolved or acknowledged.

   • Test notification: Toggle to be notified that the configured URL is working.

   • Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.

   • Allow insecure connections: Enable if you want to skip the TLS verification.

   • Custom headers: Add custom headers to your alert notification.

     If your webhook integrations require additional headers you specify them by using a custom header.

     For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default header, but you can add it using a custom header.

     Alternatively, you can choose to add custom headers programmatically as described in Configure Custom Headers and Custom Data Programmatically.

   • Custom Data: Specify the custom data you want to attach to the alert notification. The data must be a valid JSON document. This information will be included in the request body of the HTTP call. Systems that receive these webhook alerts can parse the data and take action based on the contents.

3. Click Save.

When the channel is created, you can use it on any alerts you create.

Then, when the alert fires, the notification will be sent as a POST in JSON format to your webhook endpoint. (See Alert Output, below.)

For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.

Configure Custom Headers and Custom Data Programmatically

By default, alert notifications follow a standard format (see Description of POST Data, below).

However, some integrations require additional headers and/or data, which you can append to the alert format using a custom header or custom data entry.

For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default alert template built into Sysdig, but you can add it using a custom header.

In addition to the Webhook UI option, you can do this from the command line, as described below.

Sample Use Case

This example adds two custom headers and defines additional custom data, as well as the format for that data.

1. Use the curl command to retrieve all configured notification channels:

   curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'Authorization: Bearer API-KEY'
   

2. Add the custom headers and execute the request:

   curl -X PUT https://app.sysdigcloud.com/api/notificationChannels/1 -H 'Authorization: Bearer API-KEY' -H 'Content-Type: application/json' -d '{
     "notificationChannel": {
       "id": 1,
       "version": 1,
       "type": "WEBHOOK",
       "enabled": true,
       "name": "Test-Sysdig",
       "options": {
         "notifyOnOk": true,
         "url": "https://hookb.in/v95r78No",
         "notifyOnResolve": true,
         "customData": {
           "String-key": "String-value",
           "Double-key": 2.3,
           "Int-key": 23,
           "Null-key": null,
           "Boolean-key": true
         },
         "additionalHeaders": {
           "Header-1": "Header-Value-1",
           "Header-2": "Header-Value-2"
         }
       }
     }
   }'
   

Standard Alert Output

Alerts that use a custom webhook for notification send a JSON-format with the following data.

Description of POST Data

{
  "timestamp": 1620222000000000, // Time when the alert triggered in microseconds
  "timespan": 60000000, // duration of the alert in microseconds (how long the value should be true before triggering)
  "alert": {
    "severity": 2, // severity from 0 to 7, use severityLabel for a human readable version
    "editUrl": "https://app-staging.sysdigcloud.com/#/alerts/21998727", // alert edit URL
    "severityLabel": "Medium", // human readable version of severity
    "subject": "CPU temp is High on homebridge:9100 is Triggered", // Alert subject
    "scope": null, // scope of the alert if set from the UI
    "name": "CPU temp is High", // name of the alert
    "description": null, // description, not used ATM
    "id": 21998727, // alert id
    "body": "CPU temp is High on homebridge:9100 is Triggered\n\n\nEvent Generated:\n\nSeverity:         Medium\n    Metric:\n    node_hwmon_temp_celsius = 65.8121\nSegment:\n    instance = 'homebridge:9100'\nScope:\n    Everywhere\n\nTime:             05/05/2021 01:40 PM UTC\nState:            Triggered\nNotification URL: https://app-staging.sysdigcloud.com/#/events/notifications/l:2419200/14918845/details\n\n------\n\nTriggered by Alert:\n\nName:         CPU temp is High\nTeam:         Monitor Operations\nScope:\n    Everywhere\nSegment by:   instance\nWhen:         avg(avg(node_hwmon_temp_celsius)) > 40\nFor at least: 1 m\nAlert URL:    https://app-staging.sysdigcloud.com/#/alerts/21998727\n\n\n"
  },
  "event": {
    "id": 14918845, // id of the generated event
    "url": "https://app-staging.sysdigcloud.com/#/events/notifications/l:604800/14918845/details" // url of the event in the feed
  },
  "state": "ACTIVE", // status of the alert, can be ACTIVE or OK
  "resolved": true,
  "entities": [ // list of entities that triggered the alert, at the moment we send a notification per entity, so this array will always contain a single object
    {
      "entity": "instance = 'homebridge:9100'", // segment that triggered
      "metricValues": [ // value of the metric at the time of triggering
        {
          "metric": "node_hwmon_temp_celsius",
          "aggregation": "avg",
          "groupAggregation": "avg",
          "value": 65.812167
        }
      ]
    }
  ],
  "endEntities": [ // list of entities when the alert was resolved (same as "entities")
    {
      "entity": "instance = 'homebridge:9100'",
      "metricValues": [
        {
          "metric": "node_hwmon_temp_celsius",
          "aggregation": "avg",
          "groupAggregation": "avg",
          "value": 39.812167
        }
      ]
    }
  ],
  "condition": "avg(avg(node_hwmon_temp_celsius)) > 40", // alert condition in string form
  "source": "Sysdig Cloud", // source of the event
  "labels": { // list of labels associated to this event (they strongly depend on the segmentation and scope of the alert)
    "instance": "homebridge:9100"
  }
}