Understand Agent Drivers

The Sysdig Agent can receive system call events from the Linux kernel via one of three different drivers:

• Kernel module (kmod)
• eBPF
• Universal eBPF

Each driver has its own prerequisites, advantages, and trade-offs. All three drivers share common features, but differ in the way they implement these common features.

Driver Comparison Table

Kernel Module

The kernel module driver is what its name suggests: an additional driver that is inserted into the Linux kernel. It implements kernel event handlers and a custom channel for sending event data to the agent’s user space program. It must be built against the headers for the specific kernel it will run on. The kernel module also allocates its event buffers from kernel memory, giving it the lowest user-visible memory footprint.

eBPF

The eBPF driver uses the same runtime and kernel instrumentation techniques as Universal eBPF. However, it relies on older kernel features which limits its efficiency and capabilities. Like the kernel module driver, the eBPF driver must be built against headers for the specific kernel it will run on. It differs, however, in that the eBPF driver must allocate event buffers from memory in user space. This increases the eBPF driver’s user-visible memory footprint by 8MiB per CPU, relative to the kernel module.

Helm Installation

agent:
  ebpf:
    enabled: true
    kind: legacy_ebpf

For more information, see Helm charts.

Host-Based Installation

./install_agent.sh  --bpf ... 

Universal eBPF

The Universal eBPF driver is a suite of programs that run in a special virtual machine within the Linux kernel and attaches to the same kernel functions as the kmod event handlers do. eBPF programs exchange data with user space and each other through a collection of eBPF specific data structures including maps and ring buffers. The Universal eBPF driver is pre-built and embedded in the agent binary. It does not require kernel headers from the target system to function.

Prerequisites

Kernel version 5.8 and later While kernel versions 5.8 and later generally support the Universal eBPF probe, the probe does not strictly require kernel version 5.8. You can backport the necessary features to earlier kernel versions. However, even kernel versions 5.8 and later may occasionally lack certain required features. The key requirements are:

• BPF ring buffer support
• A kernel that exposes BTF (BPF Type Format)

The Sysdig agent can automatically detect if the features are available on the running machine and notify you if something is missing. Alternatively, you can use bpftool to manually check for the required features by running the following commands:

sudo bpftool feature probe kernel | grep -q "map_type ringbuf is available" && echo "true" || echo "false"
sudo bpftool feature probe kernel | grep -q "program_type tracing is available" && echo "true" || echo "false"

Helm Installation

agent:
  ebpf:
    enabled: true
    kind: universal_ebpf

For more information, see Helm charts.

Host-Based Installation

./install_agent.sh --universal_ebpf ...

During Agent installation process on Linux nodes, you can define the --universal_ebpf setting to install the agent with eBPF support. For more information, see onboarding workflow.

Memory Accounting in the Linux Kernel

• When inspecting memory usage using tools like pmap or by checking /proc/<PID>/smaps, you may notice that the Resident Set Size (RSS) appears to double-account for the eBPF ring buffer maps. This can lead to an inflated memory footprint. The visible memory footprint is 16MiB per ring buffer map, not 8MiB as expected.

• The memory statistics can give a false impression that the agent using eBPF drivers consumes more memory than an agent using the kernel module due to how the Linux kernel accounts for memory.

• In Linux containerized environments, a kernel feature called cgroups is used to measure, limit and isolate resources consumed by the group of processes forming the container.

• Starting with version 5.11, the Linux kernel switched from rlimits to memory cgroup (memcg) accounting for managing memory limits on processes handling eBPF objects in the kernel. For this reason, when using Universal eBPF driver, the memory consumed by the eBPF ring buffer maps should be considered when setting the memory limit for the agent container.

Learn More

• What is eBPF? An Introduction and Deep Dive into eBPF the Technology
• What is the Difference Between User Space and Kernel Space in Linux
• Prefixes for binary multiples