Observations

Prerequisites

Observations require:

• Agent >= 13.6.0

Stateful Falco Rules Using Observations

Without the Observation extension, Falco rules evaluate conditions against information from a single event and its associated context, such as reconstructed state from past events. For example, an open() syscall event provides details like the file being opened, the process name, and the container environment.

Although powerful, this evaluation is stateless — it cannot “remember” past events to evaluate them together with present events to create a runtime policy event. For example, when Falco detects an open() syscall, it cannot remember which other files a process has opened, or whether the process has performed other actions, such as opening a network connection or spawning a subprocess.

This limitation makes it difficult to detect multi-step attack patterns and generate a runtime policy event. For example, a spawning shell might be innocuous. However, when this occurs along with actions such as accepting an inbound connection, opening a temporary file, and executing a shell from that file, it is highly suspicious and a strong indicator of a remote code execution attack.

Sysdig Secure addresses this gap with Observations, which extend Falco’s rules file format and detection. Key features include:

• A new top level object, observation, to denote events that are notable but not necessarily worthy of a runtime policy event.
• A new top level object, obs_link_fields, specifies how two observations must be related to each other.
• Falco rules have a new event source, source: observation, with fields obs.link/obs.occurs that can be used to relate a group a set of observations into a runtime policy event.

Any standard Falco rule can also be used in place of an observation. For example, rules with an event source of observation can group together a mix of observations and falco rules to generate a runtime policy event.

This example assumes there is an existing workload rule “Terminal Shell in container”:

- macro: package_management_program
  condition: (proc.name in (tar, rpm, dpkg))

- observation: Spawn Package Management Program
  condition: spawned_process and package_management_program

- obs_link_fields: same_session
  fields:
    - [proc.sid, proc.sid]

- rule: Spawn Package Management Program Below Container Exec
  desc: Detect an attempt to run a package management program from a container exec
  condition: >-
    obs.occurs["Terminal shell in container"]=true and
    obs.link["Terminal shell in container-->Spawn Package Management Program, same_session"]=true
  output: >
    An attempt was made to run a package management program from a container exec
    (user=%user.name cmdline=%proc.cmdline container_id=%container.id
    image=%container.image.repository)
  priority: WARNING
  source: observation

This rule generates a policy event when someone execs into a container and then runs a package management program.

Limitations

Only Managed Policies can contain these extensions. Rules using these extensions can not be customized or tuned.

Observations are only supported for workload events, for example source: syscall, and not cloud or Kubernetes Audit events.

The top level observation object

In Sysdig Secure, a new top level object called observation allows tracking notable activities without necessarily resulting in a runtime policy event. Here’s an example:

- macro: open_write
  condition: >-
    ((evt.type=open or evt.type=openat)
     and evt.is_open_write=true
     and fd.typechar='f' and fd.num>=0)

- macro: spawned_process
  condition: (evt.type in (execve) and evt.dir=< and evt.arg.res=0)

- observation: Write File
  condition: open_write

- observation: Launch Process
  condition: spawned_process
  source: syscall

An observation object has a name (for example, the value for the observation key), a condition, and an optional source key. The condition behaves identically to rule conditions, and can contain macro/list references. The source names the event source on which the observation runs. If not specified, the source is syscall.

Note that there is no output, no priority, and no tags. This is because events that match observations are only held in memory. They are not sent to any output channel and do not result in a policy event.

Observation Link Fields objects: relate observations to each other

In Sysdig Secure, a new top level rules object called obs_link_fields specifies how one observation can relate to another. Here’s an example:

- obs_link_fields: fd_matches_exepath
  fields:
    - [fd.name, proc.exepath]

An obs_link_fields object has a name (for example, the value for the obs_link_fields key) and a list of field tuples under the fields property. For two observations to be related, the value of the field for the left hand side of the tuple, from the event matching the first observation, must be equal to the value of the field for the right hand side of the tuple, from the second observation.

For example, if the first observation were Write File and the second observation were Launch Process, the two events would be related if the value of the fd.name field (such as the written file) was equal to the value of the proc.exepath (the process being executed).

Observation rules: generating policy events from a group of related observations

Sysdig Secure defines a new Falco Rule event source called observation. Conditions for observation rules can contain the following fields:

• obs.occurs[<Observation/Rule Name>]: This field evaluates to true if an observation has occurred with the provided name. This field helps limit the number of events that need to be held in memory, by specifying an initial restriction.
• obs.link[<Observation/Rule 1>--><Observation/Rule 2>, <obs_link_fields name>]: This field evaluates to true if there are existing observations 1 and 2 that are related by the fields named in the provided obs_link_fields object.

Like other rules, Falco Rules with event source observation contain a description, condition, output, priority, tags, and source key.

When generating the output string for a policy event, the last matching event (such as the final syscall) is used to fill in the templated output string.

Using the observations and obs_link_fields objects presented earlier, we could write the following rule:

- obs_link_fields: same_session
  fields:
    - [proc.sid, proc.sid]
    
- rule: Spawned Written File via Container Exec
  desc: Detect an attempt to spawn a program from an opened file below a container exec
  condition: >-
    obs.occurs["Terminal shell in container"]=true and
    obs.link["Terminal shell in container-->Write File, same_session"]=true and
    obs.link["Write File-->Launch Process, fd_matches_exepath"]=true
  output: >
    A process was launched from a file written during a container exec (user=%user.name cmdline=%proc.cmdline container_id=%container.id image=%container.image.repository)
  priority: WARNING
  source: observation

Breaking down the condition expression, the rule evaluates to true when:

• Someone execs into a container using docker exec/kubectl exec/etc.
• Below that container exec, someone writes a file. The session id (value of proc.sid) of the process that denotes the exec into the container must be the same as the session id of the process writing the file.
• Someone launches a process using the written file as an executable. The value of fd.name for the syscall that writes the file must be the same as proc.exepath for the syscall that spawns the new process.