Events

• 1: Event Types
• 2: Collect Event Data
• 3: Event Sources
• 4: Custom Events
• 5: Severity and Status
• 6: Event Scope
• 7: Configure Event Alerts
• 8: Filtering and Searching Events
• 9: Review Events
• 10: Process Kubernetes Events

1 - Event Types

There are three primary types of events displayed in the Sysdig Monitor Events feed: alert events, infrastructure events, Sysdig events, and custom events.

Alert Events

Alert events are triggered by user-configured alerts. For more information on configuring alerts, refer to the Sysdig Monitor Alerts documentation.

Sysdig Events

The event type Sysdig identifies internal events which might contain useful information. One example is failing notiifcations. For more information, see Notification Failures.

Infrastructure Events

Events can be collected from supported services within the production environment. The Sysdig agent automatically discovers these services and is configured to collect event data for a select group of events by default. Additional events can be added to the list by configuring the dragent.yaml file.

Sysdig currently supports event monitoring for the following infrastructure services:

• Docker

• ContainerD

• Kubernetes

Events marked with * are enabled by default. For more information on configuring additional infrastructure events, refer to the Enable/Disable Event Data.

Docker Events

The following Docker events are supported.

docker:
    container:
      - attach       # Container Attached      (information)
      - commit       # Container Committed     (information)
      - copy         # Container Copied        (information)
      - create       # Container Created       (information)
      - destroy      # Container Destroyed     (warning)
      - die          # Container Died          (warning)
      - exec_create  # Container Exec Created  (information)
      - exec_start   # Container Exec Started  (information)
      - export       # Container Exported      (information)
      - kill         # Container Killed        (warning)
      - oom          # Container Out of Memory (warning)
      - pause        # Container Paused        (information)
      - rename       # Container Renamed       (information)
      - resize       # Container Resized       (information)
      - restart      # Container Restarted     (warning)
      - start        # Container Started       (information)
      - stop         # Container Stopped       (information)
      - top          # Container Top           (information)
      - unpause      # Container Unpaused      (information)
      - update       # Container Updated       (information)
    image:
      - delete # Image Deleted  (information)
      - import # Image Imported (information)
      - pull   # Image Pulled   (information)
      - push   # Image Pushed   (information)
      - tag    # Image Tagged   (information)
      - untag  # Image Untaged  (information)
    volume:
      - create  # Volume Created    (information)
      - mount   # Volume Mounted    (information)
      - unmount # Volume Unmounted  (information)
      - destroy # Volume Destroyed  (information)
    network:
      - create     # Network Created       (information)
      - connect    # Network Connected     (information)
      - disconnect # Network Disconnected  (information)
      - destroy    # Network Destroyed     (information)

ContainerD Events

The following ContainerD events are supported.

containerd:
    container:
      - create # Container created               (information)
      - exit   # Container exited                (information)
      - die    # Container exited with an error  (warning)*
      - oom    # Container out of memory         (warning)*
    image:
      - create # Image created  (information)
      - update # Image updated  (information)
      - delete # Image deleted  (information)

Kubernetes Events

The following Kubernetes events are supported.

kubernetes:
    node:
      - TerminatedAllPods       # Terminated All Pods      (information)
      - RegisteredNode          # Node Registered          (information)*
      - RemovingNode            # Removing Node            (information)*
      - DeletingNode            # Deleting Node            (information)*
      - DeletingAllPods         # Deleting All Pods        (information)
      - TerminatingEvictedPod   # Terminating Evicted Pod  (information)*
      - NodeReady               # Node Ready               (information)*
      - NodeNotReady            # Node not Ready           (information)*
      - NodeSchedulable         # Node is Schedulable      (information)*
      - NodeNotSchedulable      # Node is not Schedulable  (information)*
      - CIDRNotAvailable        # CIDR not Available       (information)*
      - CIDRAssignmentFailed    # CIDR Assignment Failed   (information)*
      - Starting                # Starting Kubelet         (information)*
      - KubeletSetupFailed      # Kubelet Setup Failed     (warning)*
      - FailedMount             # Volume Mount Failed      (warning)*
      - NodeSelectorMismatching # Node Selector Mismatch   (warning)*
      - InsufficientFreeCPU     # Insufficient Free CPU    (warning)*
      - InsufficientFreeMemory  # Insufficient Free Mem    (warning)*
      - OutOfDisk               # Out of Disk              (information)*
      - HostNetworkNotSupported # Host Ntw not Supported   (warning)*
      - NilShaper               # Undefined Shaper         (warning)*
      - Rebooted                # Node Rebooted            (warning)*
      - NodeHasSufficientDisk   # Node Has Sufficient Disk (information)*
      - NodeOutOfDisk           # Node Out of Disk Space   (information)*
      - InvalidDiskCapacity     # Invalid Disk Capacity    (warning)*
      - FreeDiskSpaceFailed     # Free Disk Space Failed   (warning)*
    pod:
      - Pulling           # Pulling Container Image          (information)
      - Pulled            # Ctr Img Pulled                   (information)
      - Failed            # Ctr Img Pull/Create/Start Fail   (warning)*
      - InspectFailed     # Ctr Img Inspect Failed           (warning)*
      - ErrImageNeverPull # Ctr Img NeverPull Policy Violate (warning)*
      - BackOff           # Back Off Ctr Start, Image Pull   (warning)
      - Created           # Container Created                (information)
      - Started           # Container Started                (information)
      - Killing           # Killing Container                (information)*
      - Unhealthy         # Container Unhealthy              (warning)
      - FailedSync        # Pod Sync Failed                  (warning)
      - FailedValidation  # Failed Pod Config Validation     (warning)
      - FailedScheduling  # FailedScheduling                 (warning)*
      - OutOfDisk         # Out of Disk                      (information)*
      - HostPortConflict  # Host/Port Conflict               (warning)*
    replicationController:
      - SuccessfulCreate    # Pod Created        (information)*
      - FailedCreate        # Pod Create Failed  (warning)*
      - SuccessfulDelete    # Pod Deleted        (information)*
      - FailedDelete        # Pod Delete Failed  (warning)*
    replicaSet:
      - SuccessfulCreate    # Pod Created       (information)*
      - FailedCreate        # Pod Create Failed (warning)*
      - SuccessfulDelete    # Pod Deleted       (information)*
      - FailedDelete        # Pod Delete Failed (warning)*
    deployment:
      - SelectingAll                        # Selecting All Pods       (warning)*
      - ScalingReplicaSet                   # Scaling Replica Set      (information)*
      - DeploymentRollbackRevisionNotFound  # No revision to roll back (warning)*
      - DeploymentRollbackTemplateUnchanged # Skipping Rollback        (warning)*
      - DeploymentRollback                  # Rollback Done            (information)*
    daemonSet:
      - SelectingAll                        # Selecting All Pods       (warning)*
    statefulSet:
      - SuccessfulCreate    # Pod Created       (information)*
      - FailedCreate        # Pod Create Failed (warning)*
      - SuccessfulDelete    # Pod Deleted       (information)*
      - FailedDelete        # Pod Delete Failed (warning)*
      - RecreatingFailedPod # Recreate fail pod (warning)*
    service:
      - CreatingLoadBalancerFailed  # Load Balancer Create Failed    (warning)*
      - CleanupLoadBalancerFailed   # Load Balancer Cleanup Failed   (warning)*
      - DeletingLoadBalancer        # Load Balancer Delete Start     (information)
      - DeletingLoadBalancerFailed  # Load Balancer Delete Failed    (warning)*
      - DeletedLoadBalancer         # Load Balancer Delete End       (information)
      - UnAvailableLoadBalancer     # Unavailable Load Balancer      (warning)*
      - UpdatedLoadBalancer         # Updated Load Balancer          (information)*
      - UpdateLoadBalancerFailed    # Load Balancer Update Failed    (warning)*
      - SyncLoadBalancerFailed      # Load Balancer Sync Failed      (warning)*
    horizontalPodAutoscalar:
      - SelectorRequired             # Selector Required          (warning)*
      - InvalidSelector              # Selector Not Valid         (warning)*
      - FailedConvertHPA             # Failed to Convert HPA      (warning)*
      - FailedGetScale               # Failed to get Scale        (warning)*
      - FailedComputeMetricsReplicas # Failed to Compute Replicas (warning)*
      - FailedRescale                # Failed to rescale          (warning)*
      - FailedUpdateStatus           # Failed to update status    (warning)*

Custom Events

Additional events can be collected by the Sysdig agent and displayed in the Events module, but require more comprehensive configuration steps. These custom events can be integrated via:

• The Sysdig Monitor Slackbot

• Python scripts (either pre-built by Sysdig or user-created)

• A CURL request

For brief sample scripts regarding configuring other custom events, refer to the Custom Events. For more information, contact Sysdig Support.

LogDNA Events

Sysdig provides the ability to view LogDNA alerts as Sysdig events.

If you are both a LogDNA and Sysdig Monitor user, you can send alerts from the LogDNA platform to Sysdig Monitor as Sysdig events. These events will provide a link redirecting you to the LogDNA for further investigation. Similar to other types of Sysdig Events, you can create alerts based on the LogDNA events.

The log data provided by LogDNA carries additional details about system health. The ability to view relevant LogDNA events in Sysdig helps you debug and monitor the health of a system efficiently.

For example, if the number of logs generated during a deployment is higher than expected, you get notified with your Sysdig Events feed.

There is no configuration required on the Sysdig Monitor side. For information on configuring LogDNA to send alerts to Sysdig Monitor, see Sysdig Alert Integration.

2 - Collect Event Data

Sysdig Monitor supports event integrations with certain applications by default. The Sysdig agent will automatically discover these services and begin collecting event data from them.

The following applications are currently supported:

• Docker

• Kubernetes

See Custom Events for more information on ingesting custom events into Sysdig Monitor.

Enable Events

By default, only a limited set of events is collected for a supported application, and are listed in the agent’s configuration file. To enable collecting other supported events, add an events entry to dragent.yaml. Events marked with * are enabled by default and are listed in the default configuration file.

You can also change log entry in dragent.yaml to filter events by severity.

Learn more about it in the following sections.

Customize Events Collection

To customize the default events collected for a specific application (by either enabling or disabling events), add an events entry todragent.yaml as described in the examples below.

An entry in a section in dragent.yaml overrides the entire section in the default configuration.

For example, the Pulling entry below will permit only kubernetes pod Pulling events to be collected and all other kubernetes pod events settings in the default configuration file will be ignored.

However, other kubernetes sections - node and replicationController - remain intact and will be used as specified in the default configuration file.

Example 1: Collect Only Certain Events

Collect only ‘Pulling’ events from Kubernetes for pods:

events:
  kubernetes:
    pod:
       - Pulling

Example 2: Disable All Events in a Section

To disable all events in a section, set the event section to none:

events:
  kubernetes: none
  docker: none

Example 3: Combine Methods

These methods can be combined. For example, disable all kubernetes node and docker image events and limit docker container events to[attach, commit, copy] . The components events in other sections will be collected as specified by default:

events:
  kubernetes:
    node: none
  docker:
    image: none
    container:
      - attach
      - commit
      - copy

Format Sequences as List or Single Line

In addition to bulleted lists, sequences can also be specified in a bracketed single line. For example:

events:
  kubernetes:
    pod: [Pulling, Pulled, Failed]

Therefore, the following two settings are equivalent, permitting only Pulling, Pulled, Failed events for pods to be emitted:

events:
  kubernetes:
    pod: [Pulling, Pulled, Failed]

events:
  kubernetes:
    pod:
      - Pulling
      - Pulled
      - Failed

Change Event Collection by Severity

Events are limited globally at the agent level based on severity, using the log settings in dragent.yaml.

The default setting for the events severity filter is information. Only warning and higher severity events are transmitted.

Valid severity levels are: none, emergency, alert, critical, error, warning, notice, information, debug.

Example 1: Block Low-Severity Messages

Block all the low-severity messages (notice, information, debug):

log:
  event_priority: warning

Example 2: Block All Event Collection

Block all the event collection:

log:
  event_priority: none

For other uses of the log settings see Optional: Change the Agent Log Level.

3 - Event Sources

Event sources indicate the origin of Sysdig Monitor Events. The sources are used to narrow down the set of events to be considered in Event Alerts.

Different source values are applicable to different Event Types.

Alert Event Sources

Since Alert Events are generated from user-configured alerts, these events are not populated with a source field.

Infrastructure Event Sources

Infrastructure events have different source values based on their origin:

• docker for Docker events
• containerd for ContainerD events
• kubernetes for Kubernetes events

Custom Event Sources

Custom events ingested through the Events API are automatically attached a value of api as their source. You can customise this value by specifying it in the ingestion payload, in two different ways:

• As a source field in the JSON event object

  For example, the following call will ingest an event with a customised source equal to jenkins:

  #!/bin/bash
  SDC_ACCESS_TOKEN='626abc7-YOUR-TOKEN-HERE-3a3ghj432'
  ENDPOINT='app.sysdigcloud.com'
  
  curl -X POST -s https://${ENDPOINT}/api/v2/events \
  -H 'Content-Type: application/json; charset=UTF-8' \
  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H "Authorization: Bearer ${SDC_ACCESS_TOKEN}" \
  --data-binary '
  {"event": {"name": "Jenkins - start wordpress deploy", "description": "deploy", "severity": "MEDIUM", "source": "jenkins", "scope": "host.hostName = \"ip-10-1-1-1\" and build = \"89\""}}
  '
  sleep 5
  

• As a tag with key source in the tags section of the event object

  For example, the following call will ingest an event with a customised source equal to jenkins:

  #!/bin/bash
  SDC_ACCESS_TOKEN='626abc7-YOUR-TOKEN-HERE-3a3ghj432'
  ENDPOINT='app.sysdigcloud.com'
  
  curl -X POST -s https://${ENDPOINT}/api/v2/events \
  -H 'Content-Type: application/json; charset=UTF-8' \
  -H 'Accept: application/json, text/javascript, */*; q=0.01' -H "Authorization: Bearer ${SDC_ACCESS_TOKEN}" \
  --data-binary '
  {"event": {"name": "Jenkins - start wordpress deploy", "description": "deploy", "severity": "MEDIUM", "tags": {"source" : "jenkins"}, "scope": "host.hostName = \"ip-10-1-1-1\" and build = \"89\""}}'
  sleep 5
  

4 - Custom Events

Sysdig Monitor can ingest any custom event created, including code deploys, auto-scaling activities, and business level actions. These events will be automatically overlayed on charts and graphs for easy correlation of all performance data. The sections below outline the different ways custom events can be sent to Sysdig Monitor.

Application Integrations

Sysdig Monitor supports event integrations with certain applications by default. The Sysdig agent will automatically discover these services and begin collecting event data from them. For more information, refer to the Events documentation.

Sysdig Monitor Slackbot

Sysdigbot, the Sysdig Monitor Slackbot, allows users to post custom events directly to the Sysdig Cloud through chats with a Slack bot.

Prebuilt Python Script

The Sysdig python script provides a way to send events to Sysdig Monitor directly from the command line, using the following command structure:

python post_event.py SYSDIG_TOKEN NAME [-d DESCRIPTION] [-s SEVERITY] [-c SCOPE] [-t TAGS] [-h]

For more information, refer to the Sysdig Github repository.

Python Sample Client

The Sysdig Monitor python client acts as a wrapper around the Sysdig Monitor REST API, exposing most of the REST API functionality to provide an easy to use and install python interface. The post_event() function can be used to send events to Sysdig Monitor from any custom script. An example script is shown below:

import os
import sys

sys.path.insert(0, os.path.join(os.path.dirname(os.path.realpath(sys.argv[0])), '..'))

from sdcclient import SdcClient

# Parse arguments
sdc_token = sys.argv[1]
name = sys.argv[2]

# Instantiate the SDC client
sdclient = SdcClient(SDC_TOKEN)

# Post the event using post_event(self, name, description=None, severity=None, event_filter=None, tags=None)
res = sdclient.post_event(NAME)

Curl Sample Client

The Sysdig Monitor REST API offers the full functionality of the Sysdig Monitor app over API, allowing custom events to be sent directly to the Sysdig Cloud over the REST API. The example below is a curl request:

#!/bin/bash
SDC_ACCESS_TOKEN='626abc7-YOUR-TOKEN-HERE-3a3ghj432'
ENDPOINT='app.sysdigcloud.com'

curl -X POST -s https://${ENDPOINT}/api/v2/events \
-H 'Content-Type: application/json; charset=UTF-8' \
-H 'Accept: application/json, text/javascript, */*; q=0.01' -H "Authorization: Bearer ${SDC_ACCESS_TOKEN}" \
--data-binary '{"event": {"name": "Jenkins - start wordpress deploy", "description": "deploy", "severity": "MEDIUM", "scope": "host.hostName = \"ip-10-1-1-1\" and build = \"89\""}}'
sleep 5

See also Enable/Disable Event Data.

5 - Severity and Status

Event Severity

Event severity is broken down into four categories in the Sysdig Monitor UI, to better visualize issue priority, and allow for easier filtering practices.

The image below outlines the severity value breakdown:

Event Status

There are two primary event states: triggered, and resolved. In addition, there are two additional statuses available to improve filtering practices.

For more information on filtering the Events feed, refer to Filtering and Searching Events.

6 - Event Scope

By default, Events feed displays events from the entire environment. However, the feed can be configured to only show events from a particular scope within that environment. The scope of the event feeds can be configured by labels.

Labels refer to a set of meaningful key-value pair (whitelist) that is defined by Sysdig Monitor. As a user, you have the ability to configure the whitelist. For example, if you are using ECS and have custom container labels you have defined, you have the ability to configure the whitelist and add the labels you need. Once done, all the infrastructure events related to containers are enriched with these labels and the event scope will display associated metadata.

For more information on scoping, refer to the Using Labels documentation.

Configure Event Scope

To configure the events feed scope:

1. From the Events page, click the Edit Events Scope.

   

2. Expand the drop-down menu.

3. Select the desired label, either by scrolling through the list, or by typing the name/partial name into the search bar, and selecting it.

   

4. Open the Operator drop-down menu, and select the relevant option.

5. Open the Value drop-down menu, and select the relevant options.

6. Optional: Open the next level drop-down menu, and repeat steps 3-5.

7. Optional: Repeat step 6 for each additional layer of scope required.

   Individual layers of the scope can be removed if necessary, by clicking the Delete (x) icon beside the relevant layer.

8. Click the Apply button to save the new scope.

Filter Events by Scope

Events are by default filtered by scope in Dashboards and Explore to show the most relevant events associated with the selected scope. This capability enables you to quickly narrow down the potential problems in the area under purview. However, you can turn the filtering off and see Events from the complete scope. To do so in Explore:

1. From the Dashboard menu, select a dashboard of your choice.

2. Click the Options (ellipsis icon) and select Events Display.

   The Events panel appears. you can do the following:

   • Determine whether to show events or not.

   • Filter events by

     • Scope: Determine whether to show events by scope. Supported scopes are Team Scope and Dashboard Scope. Select an optionto see only the relevant events.
     • Severity: The supported severity levels are all severity types, high severity, and both high and medium levels. See Severity and Status for more information.
     • Types: The types of events supported are custom events and alerts. See Event Types for more information.
     • Status: The supported statuses include both status of the event as well as the resolution of the event. See Severity and Status for more information.

3. Click Save.

Reset the Environment Scope

To reset the scope to the entire environment:

1. From the Events page, click the Edit Events Scope.

2. Click Clear All.

3. Click the Apply to save the changes.

7 - Configure Event Alerts

Event alerts can be created (for custom events) and configured (for alert events, and custom events with a previously created alert) from the Event Details panel:

1. From the Events module, select the event from the feed to open the Event Details panel.

2. Open the Configure Alert panel:

   1. For existing alerts, click Edit Alert.

   2. For new alerts, click Create Alert from Event.

      

3. Configure the alert as necessary.

   New alerts will be auto-filled with information from the custom event.

   For more information on configuring alerts, see Alerts.

4. Click Create for new alerts. For existing alerts, click Save to apply the changes.

8 - Filtering and Searching Events

Filter Events

The events feed can be filtered in multiple ways, to drill-down into the environment’s history and refine the events displayed. The feed can be filtered by severity, type, and/or status. Examples of each are shown below.

The example below shows only high and medium severity events:

The example below shows only Kubernetes events:

The example below shows only events that are Unacknowledged:

The example below shows medium severity Alert events that remain Triggered, but have been acknowledged:

Search Events

In conjunction with filters, the event feed can be searched by using the search field on the top bar:

Search Fields

The search terms are used in a fulltext search across the following event fields:

• Id
• Name
• Description
• Tag values

Additionally, for Alert Events, the following fields are included in the full text search:

• Alert Condition
• Alert State
• Alert Threshold
• Alert Type
• Alert Notification Title

Search Syntax

Event search supports the following operators:

• + signifies AND operation (all the terms have to be in the document)
• | signifies OR operation
• - negates a single term
• " wraps a number of terms to signify a phrase for searching
• * at the end of a term signifies a prefix query
• ( and ) signify precedence

The default operator binding together the search terms is OR. Implications of this are shown in the examples.

Example Searches

Container Killed: Match the events containing any search term (Container OR Killed) because the default operator is OR.

Container + Killed: Match the events containing all search terms (Container AND Killed).

"Container Killed": Match the events containing the exact phrase "Container Killed".

Cont*: Match the events containing any term starting with Cont.

"Container + (Killed | Starting)": Match the events containing either the two terms Container and Killed or the two terms Container and Starting

Container -Killed: Match the events that either contain the term Container or do not contain the term Killed. The default operator here is OR.

Container +-Killed: Match the events that contain the term Container but do not contain the term Killed. The query overrides the default OR operator by using an explicit +.

9 - Review Events

Events can be reviewed in detail by clicking on the event listing in the feed:

To review the environment at the time of the event in detail, click Explore to navigate to the Explore module. The Explore module will automatically drill-down to the impacted environment objects. To view the visualization of the impacted objects, click Dashboard.

The Event Details Panel

The Event Details panel contains detailed information about the event. This information is different, depending on whether the event is an Alert event or a Custom event.

Alert Events

The example below is of an Alert event:

Security Events

Policy

The example shows an event notifying a potentially unauthorized terminal shell in a container. For more information on Policy alerts, see Secure Events.

Scanning

The example is a high severity event alerting a change in the scan result of an elasticsearch image on Quay. For more information on Scanning, see Scanning Alerts.

Infrastructure and Custom Events

Infrastructure and custom events display the same set of information in the Event Details panel. The example below is a Docker event:

10 - Process Kubernetes Events

Process Kubernetes Events

To streamline Sysdig agent processing times and reduce CPU load, you can use an updated processing engine written in Go.

To do so, edit the following code in dragent.yaml:

go_k8s_user_events: true

Kubernetes Audit Events

The agent listens on /k8s-audit for Kubernetes audit events. Configure the path using the following configuration option:

security:{k8s_audit_server_path_uris: [path1, path2]}

For more information, see Kubernetes Audit Logging.

Working with containerd in K3S

If you have containerd using a custom socket, you can specify this parameter in the agent configuration to correctly capture the containers’ metadata:

cri:
  socket_path: /run/k3s/containerd/containerd.sock