Container Drift Policy

Overview

Drift Control helps you:

• Prevent attacks by blocking container drift in production: Drift Control automatically flags and denies deviations from the original container, blocking malicious executables before damage is done.
• Enforce immutability best practice: Drift Control ensures that container software is not modified during its lifetime, driving good practices, consistency from source to run, and preventing actions that could be part of an attack.
• Enable easy and effective security: Teams are often overwhelmed by cloud-native complexity and blind to container drift, especially at scale. Now, security teams and IT can enable Drift Control for the entire container environment and immediately start protecting it.

The Container Drift policy has the following unique attributes:

• It includes only one pre-configured rule, which cannot be edited, and no other rules can be added to Drift Control policies.
• It is a custom policy, not a managed policy.

Prerequisites

• Agent version 12.16 and above for Drift policy

• Agent version 13.0 and above for captures and container stop/pause/kill actions

• On agent version v13.1 and above:

  • ignore_container_action: Ignores kill, stop, pause container operation
  • ignore_action: Ignores all the actions including kill, prevent malware, prevent drift and container actions.

• Kernel version 5.0 and above (see Prevent action)

• Agent version 13.1.0 and above to Detect Volume Binaries

  • In v13.1.1, it is enabled by default.

  • In v13.1.0, add the following configuration to the dragent.yaml file:

    drift_deny_execution_from_volumes: true
    

See Configure Drift Control in Agent Configuration.

Create a Container Drift Policy

To configure a Container Drift Policy in the Sysdig Secure UI:

1. Select Policies > Threat Detection > Runtime Policies to display the Runtime Policies page.

2. Click + Add Policy in the top right of the page.

3. Select Container Drift policy type.

Next, you can configure the policy.

Configure a Container Drift Policy

Name: Enter a policy name.

Description: Provide a meaningful and searchable description.

Enabled: Toggle to enable or disable the policy. When the policy is enabled, it generates events.

Severity: Choose the appropriate severity level as you would like to see it in the Runtime Policies UI: High, Medium, Low, or Info.

• Policy severity is subjective and is used to group policies within a Sysdig Secure instance.
• There is no inheritance between the underlying rule priorities and the severity you assign to the policy.

Scope: Define the scope to which the policy will apply, based on the type-dependent options listed.

• Auto-tuning is not used with Drift policies. If you have too many false positives, use Scope to tune them.

Link to Runbook: (Optional) Enter the URL of a company procedure that should be followed for events resulting from this policy. For example, https://www.mycompany.com/our-runbook-link.

• If you enter a value here, a View Runbook option will be displayed in any corresponding Event.

Detect

Drifted Binaries: Toggle on to dynamically detect the execution of drifted binaries.

• A drifted binary is any binary that was not part of the original image of the container. It is typically downloaded or compiled into a running container.

• If a detected binary attempts to run when this toggle is enabled, Sysdig will create an alert. To prevent the binary from running, use Prevent.

Volume Binaries: Toggle this on to treat all binaries from mounted volumes as drifted. This will generate events and prevent the binaries from being executed.

Exceptions

Create exceptions to a policy to prevent execution based on attributes of a file or the process attempting to execute the file. If a file or process matches any condition in the Exceptions list, it will be allowed.

Define binaries with their full path, separated by commas, using string or regex (if enabled). Defining with string is very specific, while regex is pattern matching and more flexible.

Use Regex for exceptions: Toggle this on to use regex (regular expression) when you configure exceptions.

• Regex allows you to create patterns for matching file names or process paths. For example, .*\.log$ matches any file ending with “.log”. You can use regex to identify drifted files or processes that should be allowed or blocked by matching their names or paths the the regex pattern defined.
• To learn more about regex syntax, see Regular Expression Syntax.
• Agent version 13.2.0 or above is required for this feature.

File-based Exceptions

Exceptions: Specify which drifted files can execute within a container. If a file matches any condition in the Exceptions list, it will be allowed. Even if a file is prohibited, has been modified or added post-deployment, it can still be executed if it is listed as an exception. This is useful for scenarios where certain files need to be updated or added and executed as part of legitimate operations, such as configuration scripts or updates.

Prohibited Binaries: Specify which drifted files are not allowed to execute. Prohibited binaries will be prevented from executing unless they are listed as an Exception.

Process-based Exceptions

Exceptions: Specify which processes can execute drifted files. If a process matches any condition in the Exceptions list, it will be allowed. Specified processes can interact with modified or newly added files without triggering security alerts, facilitating legitimate operations that require such interactions. Use Exceptions sparingly to ensure that only necessary drifted processes are allowed.

Prohibited Binaries: Specify binaries of processes whose execution is blocked even if they are built with the image. Prohibited binaries will be prevented from executing unless they are listed as an Exception.

Actions

Determine what should be done if a Policy is violated.

Prevent

Prevent Execution: Toggle this to stop detected executables from running.

• Depending on the kernel version, the agent will either:

  • Kill the process once it is detected as drifted (kernel version <5.0)

  • Prevent drifted processes from running (kernel version 5.0+)

  The latter method is preferable because it stops the drifted process from running for any period. The agent determines which mode to use at agent startup. One of the above-mentioned types of prevention is always available with 12.15+ agents. For example, Bottlerocket prevents by using the kill action.

Containers

Select what should happen to affected containers if the policy rules are breached. The appropriate action depends on your use case:

• No container action: Do not change the container behavior; send a notification according to Notification Channel settings.
• Kill: Kills one or more running containers immediately. This is recommended to make an attack harder for the attacker.
• Stop: Allows a graceful shutdown (10-seconds) before killing the container. This is recommended if stopping an attacker is your priority. However, this might affect services that rely on the container.
• Pause: Suspends all processes in the specified containers.

For a detailed breakdown and use cases, see Container Actions.

Capture

Toggle Capture ON if you want to create a capture in case of an event, and define the number of seconds before and after the event that should be in the snapshot.

See also: Captures.

Notify

Select a notification channel from the drop-down to send notifications of events to appropriate personnel.

This will ensure you know there’s a serious security incident. You can then save the notification as a record of what happened for later analysis.

See also: Set Up Notification Channels.

Container Actions

In Threat Detection Policies, Kill, Stop and Pause are different response actions, each with distinct impacts on workloads and use cases:

Kill: Killing the container immediately terminates it, removing it from the cluster. Kubernetes will typically attempt to restart the container based on its configuration. This can stop a malicious activity quickly, but can result in data loss (in stateful applications) and service disruption. It is the best choice for severe and urgent threats.

Stop: Stopping a container shuts down its processes and removes it from the scheduling pool, but unlike killing, it allows for a more graceful shutdown of services. Stopping a container might require manual intervention to bring it back online. It is best for stateful applications.

Pause: Pausing a container suspends its execution, freezing its processes and preserving its runtime state. The container remains in the cluster but is temporarily halted. It’s ideal for forensic analysis and limiting disruption, but may result in higher resource consumption and prolonged risk due to lack of resolution.

Summary

Check Events

When the policy is enabled, you can check for any detected events:

1. Log in to Sysdig Secure and select Events.

2. Type Drift in the filter bar to find where the Drift policy was triggered and drill down to examine the event details.

   

Caveats

There are certain conditions that the Drift policy will not catch.

Script Execution using Binaries from the Base Image

Drift control tracks the execution of binaries that were not part of the original image. It does not cover cases where a script is executed leveraging binaries from the original image. For example, if an attacker downloads a Python script and leverages an existing Python binary from the base image, drift detection will not detect it at runtime.

Persistent Volumes

As the Drift policy uses overlays for detection, execution from persistent volumes is not considered “drifted.” Therefore, if a malicious binary is downloaded to a persistent volume and executed, it will not be caught.

Suppose you are using persistent volumes exclusively for data. In that case, you can set the following option in the agent config file to treat the execution of all binaries from persistent volumes as drifted:

drift_deny_execution_from_volumes: true

Use with caution. When this option is used with the Prevent action, the execution of any binary from any mounted volumes will be stopped.

Container Limits

• For kernel versions below v5.13, Drift Control can monitor up to 128 containers per node.

• For kernel versions v5.13 or above, you can modify the container limit as described in Configure Container Limits.