Understand Agent Drivers

The Sysdig Agent can receive system call events from the Linux kernel via one of three different drivers:

  • Kernel module (kmod)
  • eBPF
  • Universal eBPF (Technical Preview)

Each driver has its own prerequisites, advantages, and trade-offs.

All three drivers share common features. They attach event handlers to functions within the Linux kernel, such as a system call entry point or exit point. They also allocate buffers from the instrumented system’s RAM to send kernel events to the agent program running in the user space. However, they differ in the way they implement these common features.

Kernel Module

The kernel module driver is what its name suggests: an additional driver that is inserted into the Linux kernel. It implements kernel event handlers and a custom channel for sending event data to the agent’s user space program. It must be built against the headers for the specific kernel it will run on. The kernel module also allocates its event buffers from kernel memory, giving it the lowest user-visible memory footprint.

eBPF

The current eBPF driver uses the same runtime and kernel instrumentation techniques as Universal eBPF. However, it relies on older kernel features which limits its efficiency and capabilities. Like the kernel module driver, the current eBPF driver must be built against headers for the specific kernel it will run on. It differs, however, in that the current eBPF driver must allocate event buffers from memory in user space. This increases the current eBPF driver’s user-visible memory footprint by 8MiB per CPU, relative to the kernel module.

Universal eBPF (Technical Preview)

The Universal eBPF driver is a suite of programs that run in a special virtual machine within the Linux kernel and attaches to the same kernel functions as the kmod event handlers do. eBPF programs exchange data with user space and each other through a collection of eBPF specific data structures including maps and ring buffers. The Universal eBPF driver is pre-built and embedded in the agent binary. It does not require kernel headers from the target system to function.

However, it does rely on kernel features that are only available in upstream kernel versions 5.8 and newer. The kernel must also be built with embedded BPF Type Format, (BTF), information at /sys/kernel/btf/vmlinux. Unlike the kernel module, the Universal eBPF driver must allocate event buffers from memory in user space. This increases the Universal eBPF driver’s user-visible memory footprint by 8MiB per CPU, relative to the kmod driver.

Driver Comparison Table

Driver NameMinimum Compatible Kernel VersionRequires Kernel HeadersEvent Buffer User Space Memory FootprintIdeal Use Case
Kernel Module2.6YesN/ASystems with pre-5.8 Linux kernels
eBPF4.14Yes8MiB per CPUSystems with Linux kernel version 4.14 to 5.7 where third-party kernel modules are not allowed.
Universal eBPF (Technical Preview)5.8No8MiB per CPUNew agent installations on systems with Linux kernel version 5.8 or newer.

Agent driver compatibility is described in terms of upstream or “vanilla” kernel versions because some popular Linux distributions back-port features from newer kernels into the older kernel versions they ship with their distributions. RHEL is especially well-known for this practice.

For information on configuration, see Configuration Library.

Learn More