Runtime Threat Detection Policy Tuning

When auto-tune is enabled, exceptions are automatically added to Falco policy rules to remove particularly noisy sets of events and leave lower-volume events for later analysis.

Additionally, event details (accessed either in Events feed or the Insights panels) may include Tunable Exception suggestions which can be ignored, applied, or edited and applied.

The tuner can be especially helpful when deploying Sysdig Secure runtime policies in a new environment. Your environment may include applications that legitimately perform actions such as running Docker clients in containers, changing namespaces, or writing below binary directories, but which trigger unwanted floods of related policy events in the default policies and rules provided by Sysdig.

Use the Runtime Policy Tuner

There are two ways to use the policy tuner:

• Auto-Tune: enable/disable
• Tunable Exceptions: Accessed in the Secure Events feed and Insights, these features do not require the auto-tuning feature to be enabled.

Enable/Disable Auto Tuning

The tuner is enabled and disabled as needed to tame false positives and optimize the use of the Events feed.

Default: Enabled

1. Log in to Sysdig Secure as Admin and choose Policies > Threat Detection | Runtime Policy Tuning.

2. Enable/disable the feature with the Tuning Engine toggle.

   It may take up to 24 hours to see the initial Applied Tuning Exceptions listed in the left panel.

   

   In the background, the tuner will evaluate policy events as they are received by the Sysdig backend, find applicable exceptions values, and add them. The Applied Tuning Exceptions file is passed along to all Sysdig agents, along with the rules and policies.

3. Toggle the Tuning Engine off when you feel the feature has addressed the most commonly occurring (unnecessary) policy events.

   NOTE: Turning the auto tuner off does not disable the exceptions that were added by the auto tuner, nor any exceptions manually added from the tuner suggestions.

   To start over from scratch, clear the Applied Tuning Exceptions text and re-enable the Tuning Engine toggle.

Edit & Review Exceptions

Edit in Runtime Policy Tuning Page: After an exception is added by either the auto tuning feature or tunable exceptions, admins can directly edit or review the exceptions in the editor file in the Runtime Policy Tuning page.

Review in the Rules Library: View rules that have been tuned by selecting those that have “Published By: Tuner 0.0.0”

Understand How the Tuning Engine Works

When Does the Tuner Add Exceptions?

The Policy Tuning feature is conservative, only adding exceptions for commonly occurring events for a single rule with similar attributes.

All the conditions must be met:

• The rule has generated at least 25 policy events in the past hour

• A candidate set of exception values is applicable to at least 25% of the events in the past hour

This ensures the tuning feature only adds exceptions for high-volume sets of events that can be easily addressed with a single set of exception values.

Exceptions Behind the Scenes

To understand the process of exception insertion by the tuner, consider a sample rule:

- rule: Write below root
  desc: an attempt to write to any file
   directly below / or /root
  condition: root_dir and evt.dir = < and
   open_write
  exceptions:  - name: proc_writer
  fields: [proc.name, fd.filename]

And a stream of policy events with outputs such as:

File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest
File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest
File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest
File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest

Then the tuner would add the following exception values to address the false positives:

- rule: Write below root
  exceptions:
  - name: proc_writer
    values:
       - [my-app-server, /state.txt]
   append: true

See the Falco proposal for more background information on using exceptions.

Tuning Best Practices

Auto Tuning

Learning/Onboarding to Sysdig Secure

The auto-tuner is best used when you are onboarding to Sysdig Secure with environments that have a lot of tooling that may look like some nefarious actions, but are benign. For example, a database backup tool can trigger certain rules that are monitoring for data exfiltration.

Note: In many instances, we have already added exceptions out of the box, and auto tuning is not needed.

Alternative Actions

In most cases, you may only get a handful of alerts from the same rule for the same workloads. See the best practices for tunable exceptions

• Use Insights and review a grouped set of rules, using the Tunable Exceptions to add exceptions for your workloads or applications
• Use the Events feed and review an event within a grouped policy, using the Tunable Exceptions to add exceptions for your workloads or applications

Review the auto-tuned exceptions after a learning/onboarding period

After or during the auto-tuning period, you can review rules that have been modified in the Rules Library by viewing the rules that have “Published By: Tuner 0.0.0”

How long should I leave auto-tuning on to learn about my environment?

In many cases, a week should suffice, as most workloads would have likely taken any actions that were required

Tunable Exceptions

The tunable exceptions feature is available in the Secure Events feed and Insights. When you select tunable exceptions a modal will open.

Should I apply all suggestions?

No, in most cases the first suggestion will create an exception that will encompass all of the other suggestions. If the number of events shown on the top right of each suggestion are the same, it’s likely that you are just duplicating exceptions, which is mostly okay.

Use the first suggestion

Do not blindly accept all exceptions. The top suggestion targets a combination of the most events with the most targeted set of fields and values. Take, for example, the following suggestions for the rule Read sensitive file trusted after startup.

The first suggestion successfully stops alerting for all fifteen events by specifying an exception, known_sensitive_reader with a specified process, httpd, when reading a specific type of password file, /etc/shadow.

The second suggestion targets the same 15 events, but specifies only the process httpd. If implemented, this would prevent Read sensitive file trusted after startup rule from triggering for any httpd process, including if the process opened other sensitive files such as /etc/password.