This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

  • 1:

    Runtime Policy Tuning

    The Runtime Policy Tuning feature assists in reducing noisy false positives in the Sysdig Secure Events feed. Built on top of the Falco Rules Tuner, it automatically adds Exceptions to rules, thereby removing particularly noisy sets of policy events and leaving the lower-volume events for later analysis.

    The tuner may be especially helpful when deploying Sysdig Secure runtime policies in a new environment. Your environment may include applications that legitimately perform actions such as running Docker clients in containers, changing namespaces, or writing below binary directories, but which trigger unwanted floods of related policy events in the default policies and rules provided by Sysdig.

    Earlier versions of Sysdig used the The Falco Rules Tuner (Legacy) .

    Using Runtime Policy Tuner

    Prerequisites

    • Sysdig agent 11.0.0+

    • Sysdig SaaS or Sysdig On-Prem 5.0+

    Please contact Sysdig Support to make this feature available in your environment.

    Enable, View, Edit Exceptions, Disable

    The tuner is enabled and disabled as needed to tame false positives and optimize the use of the Events feed. By default, it is disabled.

    1. Log in to Sysdig Secure as Admin and choose Policies > Runtime Policy Tuning.

    2. Enable the feature with the Tuning Engine toggle.

      It may take up to 24 hours to see the initial Applied Tuning Exceptions listed in the left panel.

      In the background, the tuner will evaluate policy events as they are received by the Sysdig backend, find applicable exceptions values, and add them. The AppliedTuning Exceptions file is passed along to all Sysdig agents, along with the rules and policies.

    3. If needed, you can edit the Exceptions created directly in the left-hand panel.

      Any changes will be retained as the tuner evaluates additional events.

    4. Toggle the Tuning Engine off when you feel the feature has addressed the most commonly occurring (unwanted) policy events.

      NOTE: Any exceptions in the Applied Tuning Exceptions panel will still be passed along to agents.

      To start over from scratch, clear the Applied Tuning Exceptions text and re-enable with the Tuning Engine toggle.

    Understanding How the Tuning Engine Works

    When Does the Tuner Add Exceptions?

    The Policy Tuning feature is conservative, only adding exceptions for commonly occurring events for a single rule with similar attributes.

    All three of the following conditions must be met:

    • The rule has generated at least 50 policy events

    • The average rate of events over the measurement period is greater than 10 events per minute

    • A candidate set of exception values is applicable to at least 25% of the events in that period

    This ensures the tuning feature only adds exceptions for high-volume sets of events that can be easily addressed with a single set of exception values.

    Exceptions Behind the Scenes

    If you want to understand the process of exception insertion by the tuner, consider a sample rule:

    - rule: Write below root
      desc: an attempt to write to any file
       directly below / or /root
      condition: root_dir and evt.dir = < and
       open_write
      exceptions:  - name: proc_writer
      fields: [proc.name, fd.filename]
    

    And a stream of policy events with outputs such as:

    File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest
    File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest
    File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest
    File below / or /root opened for writing (user=root user_loginuid=-1 command=/usr/local/bin/my-app-server parent=java file=/state.txt program=my-app-server container_id=a97d44bbe437 image=my-registry/app-server:latest
    

    Then the tuner would add the following exception values to address the false positives:

    - rule: Write below root
      exceptions:
      - name: proc_writer
        values:
           - [my-app-server, /state.txt]
       append: true
    

    See the Falco proposal for more background information on using exceptions.

    1 -

    The Falco Rules Tuner (Legacy)

    This version of the tuner has been updated for Sysdig SaaS; this content is preserved for older on-prem Sysdig environments.

    Sysdig policies are built on rules, including Falco rules and macros. (For review: Understanding Sysdig Secure Rules and Using Falco within Sysdig Secure.) Sysdig is always working to improve its out-of-the-box policies based on activity captured about well-known containers and OSS applications. Nevertheless, proprietary software running in unique user environments can require a customized approach.

    The Falco Rule Tuner was created to simplify the process of updating the existing ruleset to reduce false positives.

    The tool fetches policy events generated during a configurable time window (EVENT_LOOKBACK_MINUTES), and based on occurrence threshold (EVENT_COUNT_THRESHOLD), it suggests updates to rules. It’s up to the user to evaluate the suggestions and selectively apply the changes.

    To use the Rule Tuner, you will provide some environment variables, run as a Docker container, review the output in a Slack channel or the terminal window, and then apply the recommended tuning adjustments as desired, in the Sysdig Secure Rules Editor.

    Requirements

    • Sysdig Secure SaaS or On-Prem version 3.5.0+

    • An available Slack channel (optional, for receiving output information)

    • Environment variable values listed in the table below

    Set Variables and Run the Container

    Gather the values needed for the following environment variables.

    Required Environment Variables for Falco Rule Tuner

    Variable

    Description

    SECURE_CUSTOMER

    Optional: Name of the business entity. Default: test

    SECURE_ENDPOINT

    The endpoint for the tuning engine to query.

    For SaaS, see SaaS Regions and IP Ranges.

    For On-Prem, the endpoint has been user-defined.

    SECURE_TOKEN

    The Sysdig Secure API token used to access the Secure backend. See Find Sysdig API Token.

    SLACK_WEBHOOK

    Optional: The Slack webhook URL to receive the events summary and rule tuning recommendations.

    For example: https://hooks.slack.com/services/...

    EVENT_LOOKBACK_MINUTES

    The number of minutes the Falco Rule Tuner should look back to gather the events. Default: 60

    EVENT_COUNT_THRESHOLD

    The threshold number of events over which a tuning is recommended. Default: 5.

    Setting the threshold to 1 would mean that every policy event should be considered a false positive.

    Required Environment Variables for Falco Rule Tuner

    Run as a Docker container:

    docker run -e SECURE_ENDPOINT=${SECURE_ENDPOINT} -e SECURE_TOKEN=${SECURE_TOKEN} quay.io/sysdig/falco_rules_tuner
    

    The output in the terminal window will show the recommended rules to be adjusted and the recommended/generated macros and their conditions, e.g.:

    ... <etc.>
    
    # Change for rule: Write below root
    - macro: elasticsearch-scripts_python_access_fileshost_exe_access_files
      condition: (container, image, repository endswith locationservices/elasticsearch-scripts and proc.name=python and (fd.name startswith=/root/app/))
    

    Check Output in Slack Channel (Optional)

    The output provided in the terminal window includes only the recommended rule changes. If you provide a Slack channel URL in the environment variables, the Tuner gives both an event summary and the recommended rule changes.

    For review: How to Use the Rules Editor.

    The Tuner detects rules that may be triggering excess alert “noise” and proposes content relevant macros and macro conditions that would reduce the noise.

    To implement the suggestions, you can 1) copy the rule contents directly into the left panel of the Rules Editor and edit them, or 2) find the existing placeholder macro that was created for that rule (usual format: user_known_<rule_name> ) and add the suggested macros and conditions there.

    Note that editing the definition of a rule directly could cause overwrite issues when upgrading Sysdig versions. Creating custom rules or using the user_known placeholders is a safer procedure.

    For example, suppose you decide to implement the Tuner prompt 4 in the image above, which suggests changing the configuration of the rule Write below root. One way to proceed:

    1. Search [CTRL F] the falco_rules.yaml for Write below root.

      You will find both the Rule itself

      and placeholder macros, user_known_write_below_root_activities and user_known_write_below_root_conditions. Either one can be used.

    2. Copy one placeholder to the left-hand Custom Rules panel of the Editor: user_known_write_below_root_activities .

    3. Copy the tuner-generated macro (elasticsearch-scripts_python_access_files in this case), and conditions into the Custom Rules panel, overwriting the never_true default condition. The result is something like:

      # generated by tuner and copied to here (custom panel in the rules editor)
      - macro: elasticsearch-xxx
        condition: (...)
      - macro: user_known_write_below_root_acitivies
        condition: (elasticsearch-xxx) # updated from "never_true" with the generated macro name
      
    4. Click Save.

      The tuning adjustment will apply when the Write below root rule is invoked in a policy.

      These changes will apply anywhere that the edited macro ( user_known_write_below_root) is used. Some macros have been embedded in multiple rules and/or other macros. Edit at your discretion.