Tuning Sysdig Agent

The resource requirements for the Sysdig agent are subjective to the size and load of the host. Increased activity equates to higher resource requirements.

You might see 5 to 20 KiB/s of bandwidth consumed. Different variables can increase the throughput required. For example:

• The number of metrics

• The number of events

• Kubernetes objects

• Products and features enabled

When a Sysdig Capture is being collected, you can expect to see a spike in the bandwidth while the capture file is being ingested.

Sysdig does not recommend placing bandwidth shaping or caps on the agent to ensure that data is sent to the Sysdig Collection service.

In general, in larger clusters, the agent requires more memory, and in servers with a high number of cores, the agent requires more CPU cores to monitor all the system calls. You will use CPU cores on the host and the Kubernetes nodes visible to the agent as proxies for the rate of events processed in the agent.

Similarly, there are different factors that are at play, and considering all the factors, we recommend the following:

Small: CPU core count <= 8. Kubernetes nodes <=10

Medium: 8 < CPU core count <= 32. 10 < Kubernetes nodes <= 100

Large: CPU core count > 32. Kubernetes nodes > 100

While you can expect the behavior with the given numbers to be better than simply using the default values, Sysdig cannot guarantee that resource allocation will be correct for all the cases.

Note that the agent has its own memory watchdog to prevent runaway memory consumption on the host in case of memory leaks. The default values of the watchdog are specified in the following agent configuration file.

watchdog:
  max_memory_usage_mb: 1024
  max_memory_usage_subprocesses:
    sdchecks: 128
    sdjagent: 256
    mountedfs_reader: 32
    statsite_forwarder: 32
    cointerface: 512
    promscrape: 640

The recommended value for promscrape depends on the amount of timeseries and label data that required to be scraped on a particular node. The cluster size does not have an effect on promscrape memory usage.

max_memory_usage_mb corresponds to the dragent process in the agent. All the values are given in MiB.

For example, use the following agent configuration to match the agent watchdog settings with large values:

watchdog:
  max_memory_usage_mb: 2048
  max_memory_usage_subprocesses:
    sdchecks: 128
    sdjagent: 256
    mountedfs_reader: 32
    statsite_forwarder: 32
    cointerface: 4096
    promscrape: 640