- 1:
- 1.1:
- 1.1.1:
- 1.1.2:
- 1.1.3:
- 1.1.3.1:
- 1.1.3.2:
- 1.1.3.3:
- 1.1.3.4:
- 1.1.3.5:
- 1.1.3.6:
- 1.1.4:
- 1.1.5:
- 1.1.6:
- 1.1.7:
- 1.1.8:
- 1.1.9:
- 1.1.10:
- 1.2:
- 1.2.1:
- 1.2.2:
- 1.2.2.1:
- 1.2.2.2:
- 1.2.2.3:
- 1.2.3:
- 1.2.4:
- 1.2.4.1:
- 1.2.4.2:
- 1.2.4.3:
- 1.2.4.4:
- 1.2.4.5:
- 1.2.4.6:
- 1.2.4.7:
- 1.2.5:
- 1.2.6:
- 1.2.7:
- 1.2.8:
- 1.2.8.1:
- 1.2.8.2:
- 1.2.8.3:
- 1.2.9:
- 1.2.10:
- 1.3:
- 1.4:
- 1.5:
- 2:
- 3:
- 4:
- 5:
- 6:
- 7:
Installation
This guide describes deployment options for various Sysdig components:
1 -
Sysdig Agent
Sysdig agents are simple to deploy and upgrade, and out of the box, they
will gather and report on a wide variety of pre-defined metrics. Agent
installation can be as simple as copying and pasting a few lines of code
from a Wizard prompt, or you can follow step-by-step instructions to
check supported environments and distributions, review installation
options, and customize the agent configuration file post-install.
About Sysdig Agent
Sysdig agent is a lightweight host component that processes syscall,
creates capture files, and
performs auditing and compliance. It is a platform that supports
monitoring, securing, and troubleshooting cloud-native environments
In addition, the agent performs the following:
Metrics processing, analysis, aggregation, and transmission
Policy monitoring and alert notification through bidirectional
communication with the Sysdig collector
Integration with third-party software for consolidating customer
ecosystem data
Full assimilation into containerized and orchestrated environment
Matches runtime policies against the syscall events and generate policy events
Learn More
1.1 -
Agent Installation
Sysdig agents are delivered as either a container or a service and can be deployed with or without an orchestrator such as Kubernetes or Mesos.
A quick install involves just a few lines of code from the Getting Started wizard copied into a shell. The complete install instructions address checking for and installing kernel headers if needed, any prerequisite permissions settings for particular environments, and tips about updating the configuration files after initial installation.
Plan the Installation
Host Requirements | Review the platforms, runtimes, Linux distributions, orchestration, browsers, etc. that are supported. |
Access key | An agent access key is provided with a Sysidg trial |
Installation Options | Different ways in which you can install Sysdig agent. |
Troubleshooting Agents | Troubleshooting tips for agent installation, tuning agents, and compiling kernel modules. |
Installation Options
In the default mode of agent installation, you install the agent package as two containers, each container responsible for different functions as given below. The agent-slim
reduces the surface area of attack for potential vulnerabilities and is, therefore, more secure.
agent-kmodule
: Responsible for downloading and building the kernel
module. The image is short-lived. The container exits after the
kernel module is loaded. The transient nature of the container
reduces the time and opportunities for exploiting any potential
vulnerabilities present in the container image.
Prerequisites: The package depends on Dynamic Kernel Module
Support (DKMS) and requires the compiler and kernel headers
installed if you are using the agent-kmodule
to build the kernel
probe. Alternatively, you can use it without the kernel headers. In
such cases, the agent-kmodule
will attempt to download a pre-built
kernel probe if it is present in the Sysdig probe repository.
The module contains:
agent-slim
: Responsible for running the agent module once the
kernel module has been loaded. Slim agent functions the same way as the regular agent and retains the feature parity.
Use the instruction below to install agent on your chosen environment:
Legacy Agent: The legacy agent can be run as a single container or a service. It includes the components for downloading and building the kernel module, as well as for gathering and reporting on a wide variety of pre-defined metrics. For more information, see Installing Agent as a Single Container.
Helm
Helm is the preferred way of installing Sysdig agent. It is used in most cloud environments, for example, Amazon EKS or EC2 on AWS Cloud or AWS Outpost, EC2, and Azure AKS.
Manual
With the Getting Started wizard, you can copy a simple line of code to deploy agents in a variety of environments.
Behind the scenes, the wizard auto-detects and completes configuration items such as the required access key and port information. The wizard can also be launched from the Start a Free Trial button at sysdig.com.
After the first install, Sysdig Secure and Monitor users can access the wizard at any time from the Rocket icon on the navigation bar.
1.1.1 -
Agent Installation Requirements
Sysdig agents can be installed on a wide array of Linux hosts. Check
your environment to ensure it meets the minimum supported platform,
operating system, runtime, and orchestration requirements and uses the
appropriate installation instructions.
Versioning Scheme
We recommend that you use the latest version of the agent. Sysdig supports n-3 versions back based on the minor number. For example, if the latest release is v12.0.0
, we will support n-3 versions back, up to v11.2.0
.
End of Support
Sysdig agents that are older than version 0.85.1, released October 1, 2018, will no longer connect to the Sysdig US-East SaaS platform with default agent values.
Going forward all the agent releases will have a 3-year deprecation policy. This implies:
Sysdig Support might not be able to help you troubleshoot or address the problems with agents past the deprecation date.
Sysdig will no longer provide prebuilt kernel probe binaries for these agent releases. You need to build the kernel probe binaries on the fly by using the hosts kernel headers.
These changes is effective starting Sysdig agent v12.1.0.
Agent Installation Requirements
Support Matrix for Kubernetes
Sysdig agent versions 12.4.0 and above has been tested on the following list of latest Kubernetes versions. The matrix provides a single view into the supported operating systems, architecture, and runtime versions for different flavors of Kubernetes orchestrators.
Cluster | Operating System | Kubernetes Version | Architecture | Runtime |
---|
RedHat OpenShift Kubernetes Service (ROKS) | Ubuntu 18.04.6 LTS | v1.22.7+IKS | x86_64 | containerd |
Rancher | SUSE Linux Enterprise Server 15 SP2 | v1.20.4 | x86_64 | docker |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 46.82.202110110956-0 (Ootpa) | v1.19.14+fcff70a | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 47.84.202202070903-0 (Ootpa) | v1.20.11+e880017 | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 48.84.202202142303-0 (Ootpa) | v1.21.6+4b61f94 | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 49.84.202202230006-0 (Ootpa) | v1.22.3+b93fd35 | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 49.84.202203081945-0 (Ootpa) | v1.22.5+5c84e52 | x86_64 | cri-o |
OpenShift (okd3) | CentOS Linux 7 (Core) | v1.11.0+d4cacc0 | x86_64 | docker |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.20.0 | x86_64, arm64 | containerd |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.21.0 | x86_64, arm64 | containerd |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.22.0 | x86_64, arm64 | containerd |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.21.9 | x86_64 | containerd |
Kubernetes | Ubuntu 20.04.2 LTS | v1.23.0 | x86_64 | docker |
Kubernetes | Ubuntu 20.04.2 LTS | v1.21.0 | x86_64 | docker |
IBM Cloud Kubernetes Service (IKS) | Ubuntu 18.04.6 LTS | v1.22.7+IKS | x86_64 | containerd |
Google Kubernetes Engine (GKE) | Container-Optimized OS from Google | v1.21.6-gke.1503 | x86_64 | containerd |
Google Kubernetes Engine (GKE) | Container-Optimized OS from Google | v1.21.9-gke.1002 | x86_64 | containerd |
Amazon Elastic Kubernetes Service (EKS) | Amazon Linux 2 | v1.21.5-eks-9017834 | x86_64 | docker |
If you are not using an orchestrator in your environment, follow the instructions for Agent Install Non-Orchestrated .
Linux Distributions and Kernels
Support Matrix for Linux Distributions
Sysdig agent has been tested on the following linux destros:
Operating System | Architecture |
---|
Amazon Linux 2 | x86_64 |
CentOS Linux 7 (Core) | x86_64 |
Fedora 33 (Cloud Edition) | x86_64 |
Fedora 34 (Cloud Edition) | x86_64 |
Red Hat Enterprise Linux 8.5 (Ootpa) | x86_64 |
Red Hat Enterprise Linux Server 7.9 (Maipo) | x86_64 |
Ubuntu 16.04.7 LTS (Xenial Xerus) | x86_64 |
Ubuntu 18.04.6 LTS (Bionic Beaver) | x86_64 |
Ubuntu 20.04.4 LTS | x86_64 |
(Beta) Linux Distributions
Sysdig agent is supported on the following Linux distributions:
Core Set | Debian Ubuntu Ubuntu (Amazon) CentOS Red Hat Enterprise Linux (RHEL) SuSE Linux Enterprise Server RHEL CoreOS (RHCOS) Fedora Fedora CoreOS Linux Mint Amazon Linux Amazon Linux v2 Amazon Bottlerocket Google Container Optimized OS (COS) Oracle Linux (UEH) Oracle Linux (RHCK)
|
AWS EC2 | Amazon Linux 2 Amazon Bottlerocket Core set (see above)
|
GCP | |
Azure | |
Container Runtimes
Sysdig agent supports the detection of the following:
- Docker
- LXC
- CRI-O
- containerd
- Podman
- Mesos
Support Matrix for Docker
Operating System | Architecture |
---|
Amazon Linux 2 | x86_64, arm64 |
CentOS Linux 7 (Core) | x86_64 |
Debian GNU/Linux 9 (stretch) | x86_64 |
Debian GNU/Linux 9.13 (stretch) | x86_64 |
Fedora 34 (Cloud Edition) | x86_64 |
Fedora Linux 35 (Cloud Edition) | x86_64 |
Red Hat Enterprise Linux 8.5 (Ootpa) | x86_64, arm64 |
Red Hat Enterprise Linux Server 7.9 (Maipo) | x86_64 |
Ubuntu 16.04.7 LTS (Xenial Xerus) | x86_64, arm64 |
Ubuntu 18.04.6 LTS (Bionic Beaver) | x86_64, arm64 |
Ubuntu 20.04.4 LTS (Focal Fossa) | x86_64, arm64 |
Prerequisites for Podman Environments
Sysdig agent supports running as a
Podman container.
Enable Podman API Service for all the users.
The agent will not able to collect Podman-managed container
metadata, such as the container name, if the API service is not
enabled.
Secure rules and policies that depend on container metadata other
than the container ID will not work.
Pausing and terminating containers will not work because Policy
actions for Podman are not supported.
The containers started as a non-root user will have the
podman_owner_uid
label associated with it if the API service is
enabled for that user. The value of podman_owner_uid
will be the
numeric user ID corresponding to the user that started the
container.
Container Registries
Quay.io
For example, to pull the latest agent container from Quay.io:
docker pull quay.io/sysdig/agent
CPU Architectures
x86
Supported Agent Containers
agent
agent-slim
agent-kmodule
ARM (aarch64)
Unsupported Features
- Pre-built probes
- Activity Audit
- Sysdig agent installation using the
agent
container
s390x (zLinux)
Unsupported Features
Probes
No support for pre-built probes on zLinux.
For kernel instrumentation, use the kernel module. eBPF probes are not supported on zLinux.
Captures
Capture is not supported on zLinux.
Legacy Agent Installation
Sysdig agent installation using agent
container is not supported.
Java Versions and Vendors
Sysdig agent supports the following:
- Java versions: v7 and above
- Vendors: Oracle, OpenJDK
For Java-based applications (Cassandra, Elasticsearch, Kafka, Tomcat,
Zookeeper and etc.), the Sysdig agent requires the Java runtime
environment (JRE) to be installed to poll for metrics (beans).
If the Docker-container-based Sysdig agent is installed, the JRE is
installed alongside the agent binaries and no further dependencies
exist. However, if you are installing the service-based agent
(non-container) and you do not see the JVM/JMX metrics reporting, your
host may not have the JRE installed or it may not be installed in the
expected location: usr/bin/java
Minimum Resource Requirements
The resource requirements of the agent are subjective to the size and
load of the host— more activity equates to more resources required.
It is typical to see between 5-20KiB/s of bandwidth consumed—different
variables can increase the throughput required such as the number of
metrics, events, Kubernetes objects, and which products and features are
enabled. When a Sysdig Capture is being collected, you can expect to see
a spike in bandwidth while the capture file is being ingested.
We do not recommend placing bandwidth shaping or caps on the agent to
ensure data can be sent to our collection service. For more information, see Tuning Sysdig Agent.
Additional Requirements
Access key
The installation of the Sysdig agent requires an access key.
This key and the agent installation instructions are presented to you after
activating your account and using a web-based wizard upon initial login.
The same information can also be found in the
Settings > Agent Installation
menu of the web interface after logging
in. See Agent Installation: Overview and
Key for details.
Network connection
A Sysdig agent (containerized or native) is installed into each host
being monitored and will need to be able to connect to the Sysdig
Monitor backend servers to report host metrics. The agent must be able
to reach the Sysdig Collector addresses. For example, for US East, it is
‘collector.sysdigcloud.com’ (via
multiple IPs) over port tcp/6443
. See Sysdig Collector
Ports for supported ports
for other regions.
The agent supports the HTTP proxy for communicating with Sysdig backend
components. For more information, see Enable HTTP Proxy for
Agents.
1.1.2 -
Quick Install Sysdig Agent
Sysdig provides you with quick-install commands pre-filled with some of your environment variables to get started with Sysdig agent. You choose the deployment type and Sysdig gives you auto-generated command to ease your installation experience.
Access from Get Started Pages
Log in as admin to Sysdig Monitor or Sysdig Secure.
Select the Get Started page.

Click Install the Agent, select the appropriate deployment type, and copy the auto-generated code, filling in remaining variable values as required.
Sample Usage
Helm
Helm is the recommended option for installing agents on Kubernetes.
helm install sysdig-agent --namespace <value> --set sysdig.accessKey=<value> \
--set sysdig.settings.collector=<value> --set sysdig.settings.collector_port=<value> \
--set nodeAnalyzer.apiEndpoint=<value> --set secure.vulnerabilityManagement.newEngineOnly=true sysdig/sysdig
Example
kubectl create ns sysdig-agent
helm repo add sysdig https://charts.sysdig.com
helm repo update
helm install
--namespace=`dev`
--sysdig.accessKey=`1234-your-key-here-1234`
--sysdig.settings.collector='mycollector.elb.us-west-1.amazonaws.com'
--sysdig.settings.collector_port=`6443`
--set sysdig.settings.tags='linux:ubuntu,dept:dev,local:nyc'
--set sysdig.settings.k8s_cluster_name='my_cluster'
sysdig/sysdig
Options
Option | Description |
---|
namespace | If a value is provided, the agent will be deployed to the specified Kubernetes namespace. The default is sysdig-agent. |
accessKey | The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure. |
collector | The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. See SaaS Regions and IP Ranges for more information. It is a custom value in on-prem installations. |
collector_port | The default is 6443. |
nodeAnalyzer.apiEndpoint | Node analyzer Endpoint. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges for more information. |
secure.vulnerabilityManagement.newEngineOnly | The default is false . Installs the vulnerabilitiy management engine and the runtime scanner. If you are still using the legacy engine, set secure.vulnerabilityManagement.newEngineOnly to false ; otherwise the Get Started code snippet will prompt you to use true . It is still possible to deploy the Runtime scanner by setting nodeAnalyzer.runtimeScanner.deploy to true . See Helm Charts for more information. |
nodeAnalyzer.runtimeScanner.settings.eveEnabled | The default is false . Enables Sysdig EVE, a requirement for the runtime feature Risk Spotlight. |
nodeAnalyzer.runtimeScanner.deploy | The default is false . Deploys the Runtime Scanner. This option works only if the secure.vulnerabilityManagement.newEngineOnly flag is set to false . |
Kubernetes
install-agent-kubernetes \
[-a | --access_key <value>] [-t | --tags <value>] \
[-c | --collector <value>] [-cp | --collector_port <value>] [-s | --secure <value>] \
[-cc | --check_certificate <value>] [-ns | --namespace | --project <value>] \
[-ac | --additional_conf <value>] [-op | --openshift] [-as | --agent_slim] \
[-av | --agent_version <value>] [-ae | --api_endpoint <value> ] [-na | --nodeanalyzer ] \
[-ia | --imageanalyzer ] [-am | --analysismanager <value>] [-ds | --dockersocket <value>] \
[-cs | --crisocket <value>] [-cv | --customvolume <value>] \
[-cn | --cluster_name <value>] [-r | --remove ] [-h | --help]
Example
curl -s https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s -- \
--access_key 1234-your-key-here-1234 \
--collector collector-staging.sysdigcloud.com --collector_port 6443 \
--nodeanalyzer --api_endpoint secure-staging.sysdig.com
Options
-a
| The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure. |
-t
| The list of tags to identify the host where the agent is installed. For example: role:webserver , location:europe , role:webserver . |
-c or collector_url
| The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. |
-cp
| The collector port. The default is 6443. |
-s
| Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default. |
-cc
| Enable strong SSL certificate check. The default is true. |
-ns
| If a value is provided, the agent will be deployed to the specified namespace/project. The default is sysdig-agent . |
-op
| If provided, perform the agent installation using the OpenShift command line. |
-ac
| If a value is provided, the additional configuration will be appended to the agent configuration file. |
-av
| If a version is provided, use the specified agent version. The default is the latest version. |
-r
| If a value is provided, the daemonset, configmap, cluster role binding, service acccount and secret associated with the Sysdig Agent will be removed from the specified namespace. |
-ae
| The api_endpoint is the region-dependent domain for the Sysdig product, without the protocol. E.g. secure.sysdig.com , us2.app.sysdig.com , eu1.app.sysdig.com |
-h
| Print this usage and exit. |
Sysdig Secure Only | |
-na
| If provided, will install the Node Analyzer tools. It is an error to set both -ia and -na. |
-ds
| The docker socket for Image Analyzer. |
-cs
| The CRI socket for Image Analyzer. |
-cv
| The custom volume for Image Analyzer. |
-h
| Print this usage and exit. |
Sysdig Secure Only (Legacy) These values apply to the Node Image Analyzer (v1) in Sysdig Secure. | |
-am
| The Analysis Manager endpoint for Sysdig Secure. |
-ia
| If provided, will install the Node Image Analyzer (v1). It is an error to set both -ia and -na. The v1 Node Image Analyzer will be deprecated and replaced by the NA tools. |
Docker
Install agent-kmodule
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
Install agent-slim
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=<ACCESS_KEY> \
-e COLLECTOR=<COLLECTOR_URL> \
-e SECURE=true \
[-e TAGS=<LIST_OF_TAGS>] \
-e ADDITIONAL_CONF= <LIST_OF_CONFIG> \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
Example
Install agent-kmodule
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
Install agent-slim
docker run \
--name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=1234-your-key-here-1234 \
-e COLLECTOR=mycollector.elb.us-west-1.amazonaws.com \
-e COLLECTOR_PORT=6443 \
-e CHECK_CERTIFICATE=false \
-e TAGS=my_tag:some_value \
-e ADDITIONAL_CONF="log:\n file_priority: debug\n console_priority: error" \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=350m \
quay.io/sysdig/agent-slim
Options
Option | Description |
---|
ACCESS_KEY | The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure. |
tags | Enter meaningful tags you want applied to your instances. |
COLLECTOR | The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges. |
collector_port | The default is 6443. |
SECURE | Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default. |
CHECK_CERTIFICATE | (On-prem) Determines strong SSL certificate check for Sysdig Monitor on-premises installation. Set to true when using SSL/TLS to connect to the collector service to ensure that a valid SSL/TLS certificate is installed. |
ADDITIONAL_CONF | Optional. Use this option to provide custom configuration values to the agent as environment variables. If provided, will be appended to agent configuration file. For example, For example, file log configuration. |
bpf | Enables eBPF probe. |
Linux
$ curl -s https://download.sysdig.com/stable/install-agent -a | \
--access_key <value> [-t | --tags <value>] [-c | --collector <value>] \
[-cp | --collector_port <value>] [-s | --secure <value>] \
[-cc | --check_certificate] [-ac | --additional_conf <value>] \
[-b | --bpf] [-h | --help]
Example
curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- \
--access_key <ACCESS_KEY> --collector collector-staging.sysdigcloud.com \
--secure true
Options
Option | Description |
---|
access-key | The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure. |
tags | Enter meaningful tags you want applied to your instances. |
collector | The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges. |
collector_port | The default is 6443. |
secure | Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default. |
check_certificate | Disables strong SSL certificate check for Sysdig Monitor on-premises installation. |
additional_conf | Optional. Use this option to provide custom configuration values to the agent as environment variables. If provided, the value will be appended to agent configuration file. For example, file log configuration. |
bpf | Enables eBPF probe. |
1.1.3 -
Agent Install: Kubernetes
This document describes how to install Sysdig agent
in a Kubernetes environment. This document
assumes you will run the agent container as a Kubernetes pod, which then
enables the Sysdig agent automatically to detect and monitor your
Kubernetes environment.
It is relevant for any platform where Kubernetes is deployed, including
Amazon environments (EKS,
EC2, ECS),
Azure Container Service (AKS), Google Kubernetes Engine
(GKE), Red Hat
OpenShift, Oracle Kubernetes Engine (OKE), and IBM Cloud Kubernetes
Service (IKS).
You can also use DaemonSets
to deploy agents on every node in your Kubernetes environment. Once
deployed, Sysdig Monitor automatically begins monitoring all of your
hosts, apps, pods, and services and automatically connects to the
Kubernetes API server to pull relevant metadata about the environment.
If licensed, Sysdig Secure launches with default policies that you can
view and configure to suit your needs. You can access the front-end web
interfaces for Sysdig Monitor and Sysdig Secure immediately.
Prerequisites
A supported distribution. See Agent Installation Requirements for details.
Kubernetes v 1.9+: The agent installation on Kubernetes requires
using v1.9 or higher because the APIs used to fetch kubernetes
metadata are only present in v1.9+.
Sysdig account and access key: Request a trial or full account
at Sysdig.com and click the Activate Account button. You create
a Sysdig user name and password.
The Getting Started Wizard provides an access key.
Runtime Support: CRI-O and Containerd
By default, Sysdig agents deployed in Kubernetes automatically detect
metadata from containerd and CRI-O (in
addition to Docker), as long as the prerequisites are fulfilled.
After reviewing the information on this page, continue with the Sysdig
agent installation steps: Kubernetes Agent Installation Steps.
Containerd Support
As of agent version 0.88.1, the Sysdig agent will automatically detect
containerd metadata (as well as any Docker
metadata) in your environment, as long as the prerequisites are
fulfilled.
Prerequisites
Agent version: Sysdig agent version 0.88.1 or higher
NOTE: If you are upgrading from an earlier version of the agent,
you must also download the latest
sysdig-agent-daemonset-v2.yaml
from GitHub.
Configuration parameter: In the agent config file,
new_k8s: true
must be set.
As of agent 9.6.0, new_k8s
is enabled by default.
See Enable Kube State Metrics and Cluster Name below for details
on editing the config file.
Kubernetes-only: The containerd API must support
CRI
(a Kubernetes runtime interface).
Results in the Sysdig Monitor UI
If the Sysdig agent detects containerd metadata, it will be reported in
the front end as follows:
Explore/Dashboard views: The icon next to container-specific
items (container.name,
container.id, etc.) shows whether it’s a
Docker or containerd object.

Spotlight: Updated for containerd display.
Events: Containerd events die
and oom
are enabled by
default.
Events create
and exit
are also supported.

CRI-O Support
The Sysdig agent will automatically detect CRI-O
metadata (as well as any Docker and/or containerd metadata) in your
environment, as long as the Prerequisites are fulfilled.
Prerequisites
Platform version: Sysdig SaaS March 2019or higher
Agent version: Sysdig agent v 0.89.4 March 27, 2019 or higher.
NOTE: If you are upgrading from an earlier version of the agent,
you must also download the latest
sysdig-agent-daemonset-v2.yaml
from GitHub.
Configuration parameter: In the agent config file,
new_k8s: true
must be set.
See Enable Kube State Metrics and Cluster Name below for details
on editing the config file.
Kubernetes-only: The API must support
CRI
(a Kubernetes runtime interface).
Results in the Sysdig Monitor UI
Events: There are no CRI-O events, so the Events pane remains
unchanged.
Explore/Dashboard views: The icon next to container-specific
items (container.name,
container.id, etc.) shows CRI-O type.
Supported Metrics: By default, the same metrics are supported
for CRI-O as for Docker and containerd, except for image id
(container.image.id
).
As of agent version 0.92.1, this setting is enabled by default.
To enable image id metrics, edit the agent configuration file
dragent.yaml
to contain the following:
cri:
extra_queries: true
See Understanding the Agent Config
Files for more information
on editing dragent.yaml
.
Complete the Installation
Choose the appropriate link to complete the installation steps:
1.1.3.1 -
Steps for Kubernetes (Vanilla)
Preparation
The Sysdig agent requires kernel header files to install successfully on
a host.
This setup step is required for some environments and not others, as
noted.
If the hosts in your environment match the pre-compiled kernel modules
available from Sysdig, no special action is required.
In some cases, the host(s) in your environment may use Unix versions
that do not match the provided headers, and the agent may fail to
install correctly. In those cases, you must install the kernel headers
manually.
To do so:
For Debian-style distributions, run the command:
apt-get -y install linux-headers-$(uname -r)
For RHEL-style distributions, run the command:
yum -y install kernel-devel-$(uname -r)
Background info: see also About Kernel Headers and the Kernel Module.
Prerequisites
You can review Agent Install: Kubernetes | GKE | OpenShift |IBM and the Agent Installation Requirements for additional context, if desired.
Installation Steps
Deploy Using Helm Charts
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export CLUSTER_NAME=my-cluster # Kubernetes cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information, see charts.
Deploy Using Daemonsets
To deploy agents using Kubernetes daemonsets, you will
download
the following configuration files, edit them as required, and deploy
them.
sysdig-agent-clusterrole.yaml
sysdig-agent-service.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
Deploy the Agents
Download
the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever naming you prefer. In this document, we used
sysdig-agent
for both the namespace and the service account.
The default service account name was automatically defined in
sysdig-agent-daemonset-v2.yaml, at
the line:
serviceAccount: sysdig-agent.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
Create a cluster role and service account, and define the cluster
role bindingthat grants the Sysdig agent rules in the cluster role,
using the commands:
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information:
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
(All installs) Apply the sysdig-agent-configmap.yaml
file:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
(All installs) Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the
Kubernetes API server. See Kubernetes Audit
Logging for information
on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file :
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed. See Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
Enable Kube State Metrics and Cluster Name
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected,
monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State
Metrics
entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment
metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
There are two ways to update the agent
configuration
Option 1: Edit the files locally and apply the changes with
kubectl apply -f
:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Option 2: Use kubectl edit
to edit files on the fly:
kubectl edit configmap sysdig-agent
-n sysdig-agent
Running agents will automatically pick the new configuration after
Kubernetes pushes the changes across all the nodes in the cluster.
Additional Options
Connect to the Sysdig Backend via Static IPs (SaaS only)
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
Set the collector: value to
collector-static.sysdigcloud.com
See SaaS Regions and IP
Ranges and identify the
correct URL associated with your Sysdig collector and region.
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
Verify Metrics in Sysdig Monitor UI
Log in to Sysdig Monitor to verify that the agent deployed and the
metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP
Ranges and identify the
correct domain URL associated with your Sysdig application and
region. For example, for US East, the URL is
https://app.sysdigcloud.com.
For other regions, the format is
https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig
application is hosted. For example, for Sysdig Monitor in the EU,
you use
https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.

Select an individual container or pod to see details.

Kubernetes metadata (pods, deployments etc.) appear a minute or two
later than the nodes/containers themselves; if pod names do not appear
immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC
addresses. See Troubleshooting Agent Installation for tips.
1.1.3.2.1 -
Installing Sysdig Agent on GKE Autopilot
Autopilot is an operation mode for creating and managing clusters in GKE. In brief, with Autopilot, Google configures and manages the underlying node infrastructure for you. This topic helps you use helm to install Sysdig agent on a GKE cluster installed in Autopilot mode.
NodeAnalyzer is not supported on Autopilot environments.
Prerequisites
Install a GKE cluster in Autopilot mode.
Connect the GKE cluster.
Install your workload.
Deploy Sysdig Agent
After connecting to the GKE cluster, you can install the pre-configured Sysdig agent using helm.
To customize the configuration of the agent, see the Sysdig Agent Helm Chart.
Add the Sysdig Agent Helm Chart repository:
$ helm repo add sysdig https://charts.sysdig.com/ `
Create a namespace for the Sysdig agent:
$ kubectl create ns sysdig
Install the Sysdig agent chart:
$ helm install agent --namespace sysdig --set sysdig.accessKey=$SYSDIG_AGENT_KEY --set sysdig.settings.collector=$COLLECTOR_URL sysdig/sysdig --set gke.autopilot=true
Replace the values as follows:
$SYSDIG_AGENT_KEY
: Use the Sysdig agent key for your Sysdig Platform.
$COLLECTOR_URL
: The URL is region-dependent in Sysdig SaaS. See Regions and IP Ranges. The Collector URL is custom for on-prem installations.
Once the installation is complete, you can get started with Sysdig Secure and Sysdig Monitor.
Verify Metrics on the Sysdig Monitor UI
Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.
Given below is one way to do so.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP Ranges and identify the
correct domain URL associated with your Sysdig application and
region. For example, for US East, the URL is
https://app.sysdigcloud.com.
For other regions, the format is
https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysdig
application is hosted. For example, for Sysdig Monitor in the EU,
you use
https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
Verify that kube state metrics and cluster name are working correctly: select the Explore
tab and create a grouping by kube_cluster_name
and
kube_pod_name
.
Select an individual container or pod to see the details.
1.1.3.2.2 -
Installing Sysdig Agent on GKE Standard
Google Kubernetes Engine
(GKE) is a managed
environment for running Kubernetes in Google Cloud, in order to deploy
containerized applications. As of Sysdig agent version 0.88, Sysdig
supports all flavors of GKE, including Ubuntu and GKE’s default
Container-Optimized OS
(COS).
GKE COS environments require eBPF probe to support agent installation.
Preparation
Open Port 6443 for Agent Egress
Because GKE uses stateful firewalls, you must actively open port 6443
for the Sysdig agent outbound traffic.
In earlier versions, the Sysdig Agent connected to port 6666. This
behavior has been deprecated, as the Sysdig agent now connects to port
6443.
GKE COS/eBPF-Specific Requirements
Linux kernel version >= 4.14.
When performing the installation steps, you will add one additional
parameter to install the eBPF probe. See Step 7, below. Note that
the eBPF probe is supported only in GKE COS environments.
Prerequisites
You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.
Installation Steps
Deploy Using Helm Charts
To deploy agent using Helm charts, run the following:
Create environment variables for each value you will need during the installation phases:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (Integrations > Agent Installation > Your access key).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export GKE_CLUSTER_NAME=my-cluster # GKE cluster name
See SaaS Regions and IP Ranges for configuring collector URL and node analyzer URL for your region if you are using the SaaS version. This is a custom valyue in on-prem installations.
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$GKE_CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL --set ebpf.enabled=true
For more information,charts.
Deploy Using Daemonsets
To deploy agents using Kubernetes daemonsets, you will
download
the following configuration files, edit them as required, and deploy
them.
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
Deploy the Agents
Download
the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever name you want. In this document, we used
sysdig-agent
for both the namespace and the service account.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
If you are running Kubernetes 1.6 or higher, you must
Grant your user the ability to create roles in Kubernetes by
running the following command (see Google
documentation
for more):
kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org
Create a service account for the Sysdig agent using the
clusterrole.yaml
file.
The Sysdig agent must be granted read-only access to certain
Kubernetes APIs, which the agent uses to populate metadata and
provide component metrics.
Sysdig provides a config file in GitHub. Deploying this file creates
a cluster role and service account in Kubernetes, and defines
cluster role binding that grants the Sysdig agent rules in the
cluster role.
Run the following commands (using whatever namespace you defined in
Step 2):
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information :
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
(All installs) Apply the sysdig-agent-configmap.yaml
file using
the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
FOR GKE COS ONLY: To enable the eBPF probe required for COS,
uncomment the following parameters in
sysdig-agent-daemonset-v2.yaml
under the env section:
env:
- name: SYSDIG_BPF_PROBE
value: ""
Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the
Kubernetes API server. See Kubernetes Audit
Logging for information
on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file using the command:
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed and you can see some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.
Enable Kube State Metrics and Cluster Name
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected,
monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State
Metrics
entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment
metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
There are two ways to update the agent
configuration
Option 1: Edit the files locally and apply the changes with
kubectl apply -f
:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Option 2: Use kubectl edit
to edit files on the fly:
kubectl edit configmap sysdig-agent
-n sysdig-agent
Running agents will automatically pick the new configuration after
Kubernetes pushes the changes across all the nodes in the cluster.
Verify Metrics in Sysdig Monitor UI
Log in to Sysdig Monitor to verify that the agent deployed and the
metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP
Ranges and identify the
correct domain URL associated with your Sysdig application and
region. For example, for US East, the URL is
https://app.sysdigcloud.com.
For other regions, the format is
https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig
application is hosted. For example, for Sysdig Monitor in the EU,
you use
https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.

Select an individual container or pod to see details.

Kubernetes metadata (pods, deployments etc.) appear a minute or two
later than the nodes/containers themselves; if pod names do not appear
immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC
addresses. See Troubleshooting Agent
Installation for tips.
Additional Options
Connect to the Sysdig Backend via Static IPs (SaaS only)
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
Set the collector: value to
collector-static.sysdigcloud.com
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
1.1.3.3 -
Steps for OKE
Oracle Kubernetes Engine (OKE) is a managed environment for running Kubernetes in Oracle Cloud, in order to deploy containerized applications. As of Sysdig agent version 12.0.1, Sysdig supports all flavors of OKE.
OKE environments require eBPF probe to support agent installation.
The instructions below describe a standard OKE agent install and call out the special steps needed to install the eBPF probe.
Preparation
Open Port 6443 for Agent Egress
Because OKE uses stateful firewalls, you must actively open port 6443
for the Sysdig agent outbound traffic.
OKE by default allows network access to the sysdig Agent on 6443, but ensure that firewall rules are open and the agent can connect to the Sysdig backends.
eBPF-Specific Requirements
Linux kernel version >= 4.14.
When performing the installation steps, you will add one additional
parameter to install the eBPF probe. See Step 7, below.
Installation Steps
Identify the appropriate endpoint depending on your Sysdig account region. For more information, see SaaS Regions and IP Ranges. More info here https://docs.sysdig.com/en/docs/administration/saas-regions-and-ip-ranges/
After making clear which region your account belongs to, please choose one of the following methods:
Deploy Using Helm Charts
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export OKE_CLUSTER_NAME=my-cluster # OKE cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$OKE_CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information,charts.
Deploy Using Daemonsets
Download
the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever name you want. In this document, we used
sysdig-agent
for both the namespace and the service account.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
If you are running Kubernetes 1.6 or higher, you must Create a service account for the Sysdig agent by using the clusterrole.yaml
file.
The Sysdig agent must be granted read-only access to certain
Kubernetes APIs, which the agent uses to populate metadata and
provide component metrics.
Sysdig provides a config file in GitHub. Deploying this file creates
a cluster role and service account in Kubernetes, and defines
cluster role binding that grants the Sysdig agent rules in the
cluster role.
Run the following commands by using the namespace you defined in Step 2:
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information :
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
(All installs) Apply the sysdig-agent-configmap.yaml
file using
the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
To enable the eBPF probe uncomment the following parameters in
sysdig-agent-daemonset-v2.yaml
under the env section:
env:
- name: SYSDIG_BPF_PROBE
value: ""
Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the
Kubernetes API server. See Kubernetes Audit
Logging for information
on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file using the command:
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed and you can see Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
Verify Metrics in Sysdig Monitor UI
Log in to Sysdig Monitor to verify that the agent deployed and the
metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP
Ranges and identify the
correct domain URL associated with your Sysdig application and
region. For example, for US East, the URL is
https://app.sysdigcloud.com.
For other regions, the format is
https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig
application is hosted. For example, for Sysdig Monitor in the EU,
you use
https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.

Select an individual container or pod to see details.

Kubernetes metadata (pods, deployments etc.) appear a minute or two
later than the nodes/containers themselves; if pod names do not appear
immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC
addresses. See Troubleshooting Agent
Installation for tips.
Additional Options
Connect to the Sysdig Backend via Static IPs (SaaS only)
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
Set the collector: value to
collector-static.sysdigcloud.com
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
1.1.3.4 -
Steps for OpenShift
You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.
RHCOS environments require eBPF probe to support agent installation.
Preparation
RHCOS/eBPF-Specific Requirements
- Linux kernel version 4.14 or above.
- When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below.
The Sysdig agent requires kernel header files to install successfully on
a host.
This setup step is required for some environments and not others, as
noted.
If the hosts in your environment match the pre-compiled kernel modules
available from Sysdig, no special action is required.
In some cases, the host(s) in your environment may use Unix versions
that do not match the provided headers, and the agent may fail to
install correctly. In those cases, you must install the kernel headers
manually.
To do so:
For Debian-style distributions, run the command:
apt-get -y install linux-headers-$(uname -r)
For RHEL-style distributions, run the command:
yum -y install kernel-devel-$(uname -r)
Background info: see also About Kernel Headers and the Kernel
Module.
If you are using Red Hat OpenShift, these steps are required. They
describe how to create a project, assign and label the node selector,
create a privileged service account, and add it to a cluster role.
Copy/Paste Sample Code Block
In the example code, this document uses sysdig-agent
for the
PROJECT NAME (-n
), the SERVICE ACCOUNT (-z
), and the NODE SELECTOR.
You can copy-paste the code as is, or follow the steps below to
customize your naming conventions.
oc adm new-project sysdig-agent --node-selector=''
oc project sysdig-agent
oc create serviceaccount sysdig-agent
oc adm policy add-scc-to-user privileged -n sysdig-agent -z sysdig-agent -z node-analyzer
oc adm policy add-cluster-role-to-user cluster-reader -n sysdig-agent -z sysdig-agent -z node-analyzer
Customize the Code
You can use your own Project name and Service Account name if desired.
Note that if you use a different Service Account name, you will
need to edit the default service account in the Sysdig Installation
Steps, below.
Create a new OpenShift project for the Sysdig agent deployment and
use an empty string for the node selector:
oc adm new-project PROJECT-NAME --node-selector=""
Change to the new OpenShift Project for the Sysdig agent deployment:
oc project PROJECT-NAME
Create a service account for the project:
oc create serviceaccount SERVICE-ACCOUNT
Add the service account to privileged Security Context
Constraints:
oc adm policy add-scc-to-user privileged -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
Add the service account to the cluster-reader
Cluster Role:
oc adm policy add-cluster-role-to-user cluster-reader -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
Sysdig Installation Steps
Deploy Using Helm Charts
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export CLUSTER_NAME=my-cluster # OpenShift cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information,charts.
Deploy Using Daemonsets
To deploy agents using Kubernetes daemonsets, you
download
the configuration files, edit them as required, and deploy them.
sysdig-agent-daemonset-v2.yaml
sysdig-agent-clusterrole.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Deploy the Agents
Download
the sample files:
sysdig-agent-daemonset-v2.yaml
sysdig-agent-clusterrole.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create the sysdig-agent
cluster role and assign it to the service account:
oc apply -f sysdig-agent-clusterrole.yaml
oc adm policy add-cluster-role-to-user sysdig-agent -n PROJECT-NAME -z SERVICE-ACCOUNT
Create a secret key using the command:
oc create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
If you created a service account name other than sysdig-agent
:
Edit sysdig-agent-daemonset-v2.yaml
to provide your custom value:``
serviceAccount: sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information:
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
(All installs) Apply the sysdig-agent-configmap.yaml
file using
the command:
oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
FOR RHCOS ONLY: To enable the eBPF probe required for COS, uncomment the following parameters in sysdig-agent-daemonset-v2.yaml
under the env section:`
env:
- name: SYSDIG_BPF_PROBE
value: ""
Apply the sysdig-agent-service.yaml
file:
oc apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the
Kubernetes API server. See Kubernetes Audit
Logging for information
on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file:
oc apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed and you can see Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
Enable Kube State Metrics and Cluster Name
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected,
monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State
Metrics
entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment
metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
There are two ways to update the agent
configuration
Option 1: Edit the files locally and apply the changes with
oc apply -f
:
oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Option 2: Use oc edit
to edit files on the fly:
oc edit configmap sysdig-agent
-n sysdig-agent
Running agents will automatically pick the new configuration after
Kubernetes pushes the changes across all the nodes in the cluster.
Verify Metrics in Sysdig Monitor UI
Log in to Sysdig Monitor to verify that the agent deployed and the
metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP
Ranges and identify the
correct domain URL associated with your Sysdig application and
region. For example, for US East, the URL is
https://app.sysdigcloud.com.
For other regions, the format is
https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig
application is hosted. For example, for Sysdig Monitor in the EU,
you use
https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.

Select an individual container or pod to see details.

Kubernetes metadata (pods, deployments etc.) appear a minute or two
later than the nodes/containers themselves; if pod names do not appear
immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC
addresses. See Troubleshooting Agent
Installation for tips.
Additional Options
Connect to the Sysdig Backend via Static IPs (SaaS only)
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
Set the collector: value to
collector-static.sysdigcloud.com
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
1.1.3.5 -
Steps for Rancher
Preparation
General Requirements
You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.
The Sysdig agent requires a kernel module in order to be installed
successfully on a host. On RancherOS distributions, the Unix version
does not match the provided headers, and the agent might fail to install
correctly. Therefore, you must install the kernel headers manually.
For RancherOS distributions, the kernel headers are available in the
form of a system service and therefore are enabled using the ros
service command:
$ sudo ros service enable kernel-headers-system-docker
$ sudo ros service up -d kernel-headers-system-docker
Some cloud hosting service providers supply pre-configured Linux
instances with customized kernels. You may need to contact your
provider’s support desk for instructions on obtaining appropriate header
files, or for installing the distribution’s default kernel.
Installation steps
Deploy Using Helm Charts
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export CLUSTER_NAME=my-cluster # Rancher cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information,charts.
Deploy Using Daemonsets
Install Agent
To deploy agents using Kubernetes daemonsets,
download
the following configuration files, edit them as required, and deploy
them.
sysdig-agent-clusterrole.yaml
sysdig-agent-service.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
HELM CHART OPTION Kubernetes also offers a package manager,
Helm, which uses charts to simplify this process.
If you are using Helm charts in your K8s environment, we recommend using
them to deploy Sysdig agents, as described
here.
Deploy Agent
Download
the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever naming you prefer. In this document, we used
sysdig-agent
for both the namespace and the service account.
The default service account name was automatically defined in
sysdig-agent-daemonset-v2.yaml, at
the line:
serviceAccount: sysdig-agent.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
Create a cluster role and service account, and define the cluster
role bindingthat grants the Sysdig agent rules in the cluster role,
using the commands:
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information:
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
(All installs) Apply the sysdig-agent-configmap.yaml
file:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
(All installs) Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the
Kubernetes API server. See Kubernetes Audit
Logging for information
on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file :
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed. See Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
Enable Kube State Metrics and Cluster Name
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected,
monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State
Metrics
entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment
metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
Connect to the Sysdig Backend via Static IPs (SaaS only)
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
Set the collector: value to
collector-static.sysdigcloud.com
See SaaS Regions and IP
Ranges and identify the
correct URL associated with your Sysdig collector and region.
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
Verify Metrics in Sysdig Monitor UI
Log in to Sysdig Monitor to verify that the agent deployed and the
metrics are detected and collected appropriately.
Kubernetes metadata (pods, deployments etc.) appear a minute or two
later than the nodes/containers themselves; if pod names do not appear
immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC
addresses. See Troubleshooting Agent
Installation for tips.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP
Ranges and identify the
correct domain URL associated with your Sysdig application and
region. For example, for US East, the URL is
https://app.sysdigcloud.com.
For other regions, the format is
https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig
application is hosted. For example, for Sysdig Monitor in the EU,
you use
https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.

Select an individual container or pod to see details.

1.1.3.6 -
Using Node Leases
The Sysdig agent uses Kubernetes
Lease
to control how and when connections are made to the Kubernetes API
Server. This mechanism prevents overloading the Kubernetes API server
with connection requests during agent bootup.
Kubernetes node leases are automatically created for agent version
12.0.0 and above. On versions prior to 12.0.0, you must configure node
leases as given in the KB article.
Prerequisites
Types of Leases
The agent creates the following leases:
Cold Start
During boot up, the Sysdig agent connects to the Kubernetes API server
to retrieve Kubernetes metadata and build a cache. The cold-start
leases control the number of agents that build up this cache at any
given time. An agent will grab a lease, build its cache, and then
release the lease so that another agent can build its cache. This
mechanism prevents agents from creating a “boot storm” which can
overwhelm the API server in large clusters.
Delegation
In Kubernetes environments, two agents are marked as delegated
in each
cluster. The delegated
agents are the designated agents to request
more data from the API server and produce KubeState metrics. The
delegation
leases will not be released until the agent is terminated.
View Leases
To view the leases, run the following:
$ kubectl get leases -n sysdig-agent
You will see an output similar to the following:
NAME HOLDER AGE
cold-start-0 20m
cold-start-1 20m
cold-start-2 21m
cold-start-3 ip-10-20-51-167 21m
cold-start-4 21m
cold-start-5 21m
cold-start-6 20m
cold-start-7 21m
cold-start-8 20m
cold-start-9 ip-10-20-51-166 21m
delegation-0 ip-10-20-52-53 21m
delegation-1 ip-10-20-51-98 21m
Troubleshoot Leases
Verify Configuration
When lease-based delegation is working as expected, the agent logs show
one of the following:
Getting pods only for node <node>
Getting pods for all nodes.
Both (occasionally on the delegated nodes)
Run the following to confirm that it is working:
$ kubectl logs sysdig-agent-9l2gf -n sysdig-agent | grep -i "getting pods"
The configuration is working as expected if the output on a pod is
similar to the following:
2021-05-05 02:48:32.877, 15732.15765, Information, cointerface[15738]: Only getting pods for node ip-10-20-51-166.ec2.internal
Unable to Create Leases
The latest Sysdig ClusterRole is required for the agent to create
leases. If you do not have the latest ClusterRole or if you have not
configured the ClusterRole correctly, the logs show the following error:
Error, lease_pool_manager[2989554]: Cannot access leases objects: leases.coordination.k8s.io is forbidden: User "system:serviceaccount:sysdig-agent:sysdig-agent" cannot list resource "leases" in API group "coordination.k8s.io" in the namespace "sysdig-agent"
Contact Sysdig Support for help.
Optional Agent Configuration
Several configuration options exist for leases. It is recommended to not
change the default settings unless prompted by Sysdig Customer Support.
k8s_coldstart:
enabled: <true/false>
| true above agent versions 12.0.0
| When true, the agent will attempt to create cold-start leases to control the number of agents which are allowed to build their cache at one time. |
k8s_coldstart:
max_parallel_cold_start: <int>
| 10 | The number of cold-start leases to be created. This is the number of agents that can connect to the API Server simultaneously during agent initialization. |
k8s_coldstart:
namespace: <string>
| sysdig-agent
| The namespace to be created. This shouldn't be needed in agent version 12.0.0 because the DownwardAPI in the ClusterRole will provide the appropriate namespace. |
k8s_coldstart:
enforce_leader_election: <true/false>
| false
| When true , the agent will not fall back to the previous method if it cannot create leases.This can be useful if the previous method caused API Server problems. |
k8s_delegation_election: <true/false>
| true above agent versions 12.0.0
| When true , the agent will create delegation leases to control which set of agents generate global cluster metrics. |
1.1.4 -
Agent Install: Non-Orchestrated
This section describes how to install the Sysdig agent directly on a
Linux host, without using an orchestrator, such as Kubernetes or Mesos.
The agent can be installed in two ways:
The steps for each flavor differ slightly depending on whether you are
using the SaaS or on-premises version of the Sysdig platform.
If you are installing the Sysdig agent in an environment that has Kubernetes, use the Agent Install: Kubernetes instructions instead.
Prerequisites
On kernel headers: The Sysdig agent requires kernel header files in
order to install successfully on a host, and the agent is delivered with
precompiled headers. If the hosts in your environment match the kernel
versions included with the agent, no special action is needed.
In some cases, the host(s) in your environment may use Unix versions
that do not match the provided headers, and the agent may fail to
install correctly. In those cases, you must install the kernel headers
manually. See About Kernel Headers and the Kernel Module for details.
Run any commands as root or with the sudo
command.
Have your Sysdig access key on hand.
If you launch an agent install from
www.sysdig.com, the welcome wizard will
present an access key.
Installing Agent Using Containers
The Sysdig agent can be deployed as a Docker container.
The commands below can also be copied from the Get Started page. In that case, your access key will already be included in the command automatically.
SaaS
Installing As Two Containers
The agent is installed by running sysdig/agent-kmodule
, followed by running sysdig/agent-slim
.
Every host restart requires subsequent running of agent-kmodule
and
agent-slim
containers.
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the address for your region.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that is either built or downloaded.
Build and load the kernel module:
If you are not using eBPF, use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules \
quay.io/sysdig/agent-kmodule
If you are using eBPF use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-e SYSDIG_BPF_PROBE="" \
-v /etc/os-release:/host/etc/os-release:ro \
-v /root/.sysdig:/root/.sysdig \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
Configure kernel module to load during system boot.
If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.
$ sudo mkdir -p /etc/modules-load.d
$ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
Run the agent module providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
[-e TAGS=[TAGS]]
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host\
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
[-e TAGS=[TAGS]]
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /root/.sysdig:/root/.sysdig \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
Installing As Single Container (Legacy)
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the address for your region.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that was either built or downloaded.
Run the agent container providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host\
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host\
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e TAGS=[TAGS] \
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
On-Premises
Installing As Two Containers
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the URL of your environment.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that was either built or downloaded.CHECK_CERTIFICATE
: Set to false
if a self-signed certificate or private CA-signed certificate is used. For more information, see Set Up SSL Connectivity to the Backend.
Build and load the kernel module:
If you are not using eBPF, use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules \
quay.io/sysdig/agent-kmodule
If you are using eBPF use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-e SYSDIG_BPF_PROBE="" \
-v /etc/os-release:/host/etc/os-release:ro \
-v /root/.sysdig:/root/.sysdig \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
Configure kernel module to load during system boot.
If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.
$ sudo mkdir -p /etc/modules-load.d
$ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
Run the agent module providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
Installing As Single Container (Legacy)
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the URL of your environment.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that was either built or downloaded.CHECK_CERTIFICATE
: Set to false
if a self-signed certificate or private CA-signed certificate is used. For more information, see Set Up SSL Connectivity to the Backend.
Run the agent module providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
Installing Agent as a Service on Linux Host
Use these instructions to install the agent on the host itself, not in a
container. Install on each host in the environment.
The command lines below can also be copy/pasted from the Welcome wizard
or the Settings>Agent Installation
page in the Sysdig Monitor
interface.
In that case, your access key will already be included in the command
automatically.
The Sysdig agent depends on several python modules, some of which might
not be installed on the hosts where the agent is running as a service.
When the required dependencies are not available, the sdchecks
component in the agent will report errors in the log files, such as:
>> Error, sdchecks[0] ModuleNotFoundError: No module named 'posix_ipc'
To address these errors, install the missing modules using the
pip install
command.
SaaS
Run the following command:
curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] [--tags [TAGS]]
Where
[ACCESS_KEY]
is your unique agent access key string. For example, 1234-your-key-here-1234.
TAGS
is an optional list of user-defined agent tags. For example, role:webserver,location:europe
.
See SaaS Regions and IP Ranges to find the collector endpoint for your region.
Restart the agent and start the service:
sudo systemctl enable dragent
On-Premises
Run the following command:
curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] --secure true --check_certificate true [--tags [TAGS]]
Where
[ACCESS_KEY]
is your unique agent access key string. For example, 1234-your-key-here-1234.
TAGS
is an optional list of user-defined agent tags. For example, role:webserver,location:europe
.
check_certificate
: Set it to false
if a self-signed certificate, a private, or a CA-signed certificate is used. See Set Up SSL Connectivity to the Backend for more information.
Restart the agent and start the service:
sudo systemctl enable dragent
Connect to the Sysdig Backend via Static IPs (SaaS only)
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com:
user@host:~$ docker run --name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
-e COLLECTOR=collector-static.sysdigcloud.com \
-e COLLECTOR_PORT=6443 \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
--shm-size=512m \
quay.io/sysdig/agent-slim
Guidelines for Manual Agent Installation
In the following cases, we recommend that you manually install the agent.
See Agent Install: Manual Linux Installation for more information.
1.1.5 -
Agent Install: Manual Linux Installation
Manual installation of the native Linux agent is recommended in the
following cases:
Full control over the deployment process
Integration with configuration management tools
Custom kernel
Unsupported distribution (within Debian/Fedora flavors)
Otherwise, you may want to just follow the standard Installation Guide:
NOTE: If you are installing the Sysdig agent in an orchestrated
infrastructure such as Kubernetes, Mesos/Marathon, use the respective
Installation Guides:
Run the commands as root or with sudo.
Installation Options
Review the Host Requirements for Agent
Installation. Then follow
the steps for the appropriate Linux distribution, below.
Debian, Ubuntu
Trust the Sysdig Monitor GPG key, configure the apt repository, and
update the package list:
curl -s https://download.sysdig.com/DRAIOS-GPG-KEY.public | apt-key add -
curl -s -o /etc/apt/sources.list.d/draios.list http://download.sysdig.com/stable/deb/draios.list
apt-get update
Install kernel development files.
The following command might not work with every kernel. Make sure to
customize the name of the package properly.
apt-get -y install linux-headers-$(uname -r)
Install, configure, and restart the Sysdig agent.
apt-get -y install draios-agent
echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
service dragent restart
Replace ACCESS_KEY
with your unique access key
string.
Inability to retrieve the key indicates that the administrator of
your instance might have it turned off for non-admin users. Please
contact your Sysdig administrator to receive the key. If you still
have issues please contact Sysdig
support.
[TAGS]
is an optional parameter you can use to list one or more
tags for this host (see below).
CentOS, RHEL, Fedora, Amazon AMI, Amazon Linux 2
Trust the Sysdig Monitor GPG key, configure the yum repository.
$ rpm --import https://download.sysdig.com/DRAIOS-GPG-KEY.public
$ curl -s -o /etc/yum.repos.d/draios.repo http://download.sysdig.com/stable/rpm/draios.repo
Install the EPEL repository
The following command is required only if DKMS is not available in
the distribution. You can verify if DKMS is available with
yum list dkms
The command below contains a sample release number; be sure to
update with the correct release.
$ rpm -i http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm
Install kernel development files.
The following command might not work with every kernel. Make sure to
customize the name of the package properly.
$ yum -y install kernel-devel-$(uname -r)
Install, configure, and start the Sysdig agent.
$ yum -y install draios-agent
$ echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
$ echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
$ sudo systemctl enable dragent
$ sudo systemctl start dragent
Replace ACCESS_KEY
with your unique access key
string.
Inability to retrieve the key indicates that the administrator of
your instance might have it turned off for non-admin users. Please
contact your Sysdig administrator to receive the key. If you still
have issues please contact Sysdig
support.
[TAGS]
is an optional parameter you can use to list one or more
tags for this host (see below).
If you using a non-systemd Linux distribution, use the service
command to start dragent
.
$ service dragent restart
Other Linux Distributions
The Sysdig Agent is unsupported outside of the Debian, Fedora, and
Amazon distributions.
Tagging your hosts is highly recommended. Agent Tags allow you to sort
nodes of your infrastructure into custom groups in Sysdig Monitor.
Replace the [TAGS] parameter above with a comma-separated list of
TAG_NAME:TAG_VALUE
.
For example: role:webserver,location:europe
1.1.6 -
Agent Install: ECS
Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that helps to easily deploy, manage, and scale containerized applications.
This section describes how to install the Sysdig agent container on each underlying host in your ECS cluster. Once installed, the agent will automatically begin monitoring all of your hosts, service and tasks.
These instructions are valid only for ECS clusters using EC2 instances. For information on ECS Fargate clusters, see AWS Fargate Serverless Agents.
Installation Instructions
To install Sysdig agent on ECS, do the following:
Create an ECS task definition for the Sysdig agent.
Register the task definition in your AWS account.
Create a service with the previous task definition to run the Sysdig agent in each of the nodes of your ECS cluster.
Create an ECS Task Definition
- First make sure to collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the collector address for your region. For more information, see SaaS Regions and IP Ranges.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
Use the above values to customize the JSON snippet below and save it as a file named sysdig-agent-ecs.json
. Note that memory and cpu have both been set to 1024, depending of the size of your cluster you might want to tune those values, see Tuning Sysdig Agent for more information.
{
"family": "sysdig-agent-ecs",
"containerDefinitions": [
{
"name": "sysdig-agent",
"image": "quay.io/sysdig/agent-slim",
"cpu": 1024,
"memory": 1024,
"privileged": true,
"environment": [
{
"name": "ACCESS_KEY",
"value": "$ACCESS_KEY"
},
{
"name": "COLLECTOR",
"value": "$COLLECTOR"
},
{
"name": "TAGS",
"value": "$TAG1,TAG2"
}
],
"mountPoints": [
{
"readOnly": true,
"containerPath": "/host/boot",
"sourceVolume": "boot"
},
{
"containerPath": "/host/dev",
"sourceVolume": "dev"
},
{
"readOnly": true,
"containerPath": "/host/lib/modules",
"sourceVolume": "modules"
},
{
"readOnly": true,
"containerPath": "/host/proc",
"sourceVolume": "proc"
},
{
"containerPath": "/host/var/run/docker.sock",
"sourceVolume": "sock"
},
{
"readOnly": true,
"containerPath": "/host/usr",
"sourceVolume": "usr"
}
],
"dependsOn": [
{
"containerName": "sysdig-agent-kmodule",
"condition": "SUCCESS"
}
]
},
{
"name": "sysdig-agent-kmodule",
"image": "quay.io/sysdig/agent-kmodule",
"memory": 512,
"privileged": true,
"essential": false,
"mountPoints": [
{
"readOnly": true,
"containerPath": "/host/boot",
"sourceVolume": "boot"
},
{
"containerPath": "/host/dev",
"sourceVolume": "dev"
},
{
"readOnly": true,
"containerPath": "/host/lib/modules",
"sourceVolume": "modules"
},
{
"readOnly": true,
"containerPath": "/host/proc",
"sourceVolume": "proc"
},
{
"containerPath": "/host/var/run/docker.sock",
"sourceVolume": "sock"
},
{
"readOnly": true,
"containerPath": "/host/usr",
"sourceVolume": "usr"
}
]
}
],
"pidMode": "host",
"networkMode": "host",
"volumes": [
{
"name": "sock",
"host": {
"sourcePath": "/var/run/docker.sock"
}
},
{
"name": "dev",
"host": {
"sourcePath": "/dev/"
}
},
{
"name": "proc",
"host": {
"sourcePath": "/proc/"
}
},
{
"name": "boot",
"host": {
"sourcePath": "/boot/"
}
},
{
"name": "modules",
"host": {
"sourcePath": "/lib/modules/"
}
},
{
"name": "usr",
"host": {
"sourcePath": "/usr/"
}
}
],
"requiresCompatibilities": [
"EC2"
]
}
Register a Task Definition
Once your task definition is ready, ensure that you register it in your AWS account:
aws ecs register-task-definition \
--cli-input-json file://sysdig-agent-ecs.json
Run the Agent as an ECS Service
Using the ECS task definition you have created, create a service in the cluster that you want to monitor with Sysdig.
aws ecs create-service \
--cluster $CLUSTER_NAME \
--service-name sysdig-agent-svc \
--launch-type EC2 \
--task-definition sysdig-agent-ecs \
--scheduling-strategy DAEMON
With the agent installed, Sysdig will begin auto-discovering your containers and other resources of your ECS environment.
Using ECS Anywhere
If you’re using ECS Anywhere, change the launch type to EXTERNAL
when the service is created.
aws ecs create-service \
--cluster $CLUSTER_NAME \
--service-name sysdig-agent-svc \
--launch-type EXTERNAL \
--task-definition sysdig-agent-ecs \
--scheduling-strategy DAEMON
Enable the awslogs
Log Driver for the Sysdig Agent Containers
You can send the logs from the containers running in ECS tasks to log groups in CloudWatch Logs. To send Sysdig container logs, do the following:
Add the following section to each of the container definitions described above:
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "$YOUR_LOG_GROUP",
"awslogs-region": "$AWS_REGION",
"awslogs-stream-prefix": "sysdig"
}
Update your task definition and the service to enable the logs.
1.1.7 -
Agent Install: IBM Cloud Kubernetes Service (IKS)
IBM Cloud maintains the documentation for Sysdig agent installation on IBM Cloud Kubernetes Service (IKS).
For more information, see the IBM Cloud Monitoring
documentation:
1.1.8 -
Agent Install: Mesos | Marathon | DCOS
Marathon is the container orchestration platform for Mesosphere’s
Datacenter Operating System (DC/OS) and Apache Mesos.
This guide describes how to install the Sysdig agent container on each
underlying host in your Mesos cluster. Once installed, the agent will
automatically connect to the Mesos and Marathon APIs to pull relevant
metadata about the environment and will begin monitoring all of your
hosts, apps, containers, and frameworks.
Standard Installation Instructions
Review the Host Requirements for Agent Installation.
In this three-part installation, you:
Deploy the Sysdig agent on all Mesos Agent (aka “Slave”) nodes,
either automatically or by creating and posting a .json
file to
the leader Marathon API server.
Deploy the Sysdig agent on the Mesos Master nodes.
Special configuration steps: modify the Sysdig agent config file to
monitor Marathon instances.
Deploy the Sysdig agent on your Mesos Agent nodes
Preferred Option: Automatic install (DC/OS 1.11+)
If you’re using DC/OS 1.8 or higher, then you can find Sysdig in the
Mesosphere Universe marketplace and install it from there.
It will automatically deploy the Sysdig agent container on each of your
Mesos Agent nodes as a Marathon app.
Proceed to Deploy the Sysdig
Agent.
Alternate Option: Post a .json file
If you are using a version of DC/OS earlier than 1.8 then:
Create a JSON file for Marathon, in the following format.
The COLLECTOR
address comes from your own environment in on-prem
installations. For SaaS installations, find the collector endpoint
for your region listed
here.
COLLECTOR_PORT, SECURE,
and CHECK_CERT
are used in environments
with Sysdig’s on-premises backend installed.
{
"backoffFactor": 1.15,
"backoffSeconds": 1,
"constraints": [
[
"hostname",
"UNIQUE"
]
],
"container": {
"docker": {
"forcePullImage": true,
"image": "sysdig/agent",
"parameters": [],
"privileged": true
},
"type": "DOCKER",
"volumes": [
{
"containerPath": "/host/var/run/docker.sock",
"hostPath": "/var/run/docker.sock",
"mode": "RW"
},
{
"containerPath": "/host/dev",
"hostPath": "/dev",
"mode": "RW"
},
{
"containerPath": "/host/proc",
"hostPath": "/proc",
"mode": "RO"
},
{
"containerPath": "/host/boot",
"hostPath": "/boot",
"mode": "RO"
},
{
"containerPath": "/host/lib/modules",
"hostPath": "/lib/modules",
"mode": "RO"
},
{
"containerPath": "/host/usr",
"hostPath": "/usr",
"mode": "RO"
}
]
},
"cpus": 1,
"deployments": [],
"disk": 0,
"env": {
"ACCESS_KEY": "ACCESS_KEY=YOUR-ACCESS-KEY-HERE",
"CHECK_CERT": "false",
"SECURE": "true",
"TAGS": "example_tag:example_value",
"name": "sdc-agent",
"pid": "host",
"role": "monitoring",
"shm-size": "350m"
},
"executor": "",
"gpus": 0,
"id": "/sysdig-agent",
"instances": 1,
"killSelection": "YOUNGEST_FIRST",
"labels": {},
"lastTaskFailure": {
"appId": "/sysdig-agent",
"host": "YOUR-HOST",
"message": "Container exited with status 70",
"slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
"state": "TASK_FAILED",
"taskId": "sysdig-agent.3bb0759d-3fa3-11e9-b446-c60a7a2ee871",
"timestamp": "2019-03-06T00:03:16.234Z",
"version": "2019-03-06T00:01:57.182Z"
},
"maxLaunchDelaySeconds": 3600,
"mem": 850,
"networks": [
{
"mode": "host"
}
],
"portDefinitions": [
{
"name": "default",
"port": 10101,
"protocol": "tcp"
}
],
"requirePorts": false,
"tasks": [
{
"appId": "/sysdig-agent",
"healthCheckResults": [],
"host": "YOUR-HOST-IP",
"id": "sysdig-agent.0d5436f4-3fa4-11e9-b446-c60a7a2ee871",
"ipAddresses": [
{
"ipAddress": "YOUR-HOST-IP",
"protocol": "IPv4"
}
],
"localVolumes": [],
"ports": [
4764
],
"servicePorts": [],
"slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
"stagedAt": "2019-03-06T00:09:04.232Z",
"startedAt": "2019-03-06T00:09:06.912Z",
"state": "TASK_RUNNING",
"version": "2019-03-06T00:09:04.182Z"
}
],
"tasksHealthy": 0,
"tasksRunning": 1,
"tasksStaged": 0,
"tasksUnhealthy": 0,
"unreachableStrategy": {
"expungeAfterSeconds": 0,
"inactiveAfterSeconds": 0
},
"upgradeStrategy": {
"maximumOverCapacity": 1,
"minimumHealthCapacity": 1
},
"version": "2019-03-06T00:09:04.182Z",
"versionInfo": {
"lastConfigChangeAt": "2019-03-06T00:09:04.182Z",
"lastScalingAt": "2019-03-06T00:09:04.182Z"
}
}
See Table 1: Environment Variables for Agent Config
Filef
or the Sysdig name:value
definitions.
Complete the “cpus
”, “mem
” and “labels
” (i.e. Marathon labels)
entries to fit the capacity and requirements of the cluster
environment.
Update the created.json
file to the leader Marathon API server:
$ $curl -X POST http://$(hostname -i):8080/v2/apps -d @sysdig.json -H "Content-type: application/json"
Deploy the Sysdig Agent
After deploying the agent to the Mesos Agent nodes, you will install
agents on each of the Mesos Master nodes as well.
If any cluster node has both Mesos Master and Mesos Agent roles, do not
perform this installation step on that node. It already will have a
Sysdig agent installed from the procedure in step A. Running duplicate
Sysdig agents on a node will cause errors.
Use the Agent Install:
Non-Orchestrated
instructions to install the agent directly on each of your Mesos Master
nodes.
When the Sysdig agent is successfully installed on the master nodes, it
will automatically connect to the local Mesos and Marathon (if
available) API servers via http://localhost:5050
and
http://localhost:8080
respectively, to collect cluster configuration
and current state metadata in addition to host metrics.
Special Configuration Steps
In certains situations, you may need to add additional configurations to
the dragent.yaml
file:
Descriptions and examples are shown below.
If the Sysdig Agent Cannot Run On the Mesos API Server
Mesos allows multiple masters. If the API server can not be instrumented
with a Sysdig agent, simply delegate ONE other node with an agent
installed to remotely receive infrastructure information from the API
server.
NOTE: If you manually configure the agent to point to a master with a
static configuration file entry, then automatic detection/following of
leader changes will no longer be enabled.
Add the following Mesos parameter to the delegated agent’s
dragent.yaml
file to allow it to connect to the remote API server and
authenticate, either by:
a. Directly editing dragent.yaml
on the host, or
b. Converting the YAML code to a single-line format and adding it as an
ADDITIONAL_CONF
argument in a Docker command.
See Understanding the Agent Config
Files for details.
Specify the API server’s connection method, address, and port. Also
specify credentials if necessary.
YAML example:
mesos_state_uri: http://[acct:passwd@][hostname][:port]
marathon_uris:
- http://[acct:passwd@][hostname][:port]
Although marathon_uris:
is an array, currently only a single “root”
Marathon framework per cluster is supported. Multiple side-by-side
Marathon frameworks should not be configured in order for our agent to
function properly. Multiple side-by-side “root” Marathon frameworks on
the same cluster are currently not supported. The only supported
multiple-Marathon configuration is with one “root” Marathon and other
Marathon frameworks as its apps.
If the Mesos API server requires authentication
If the agent is installed on the API server but the API server uses a
different port or requires authentication, those parameters must be
explicitly specified.
Add the following Mesos parameters to the API server’s dragent.yaml
to
make it connect to the API server and authenticate with any unique
account and password, either by:
a. Directly editing dragent.yaml
on the host, or
b. Converting the YAML code to a single-line format and adding it as an
ADDITIONAL_CONF
argument in a Docker command.
See Understanding the Agent Config
Files for details.
Specify the API server’s protocol, user credentials, and port:
mesos_state_uri: http://[username:password@][hostname][:port]
marathon_uris:
- http://[acct:passwd@][hostname][:port]
*HTTPS protocol is also supported.
In troubleshooting cases where auto-detection and reporting of your
Mesos infrastructure needs to be temporarily turned off in a designated
agent:
Comment out the Mesos parameter entries in the agent’s dragent.yaml
file.
Example parameters to disable: mesos_state_uri, marathon_uris
If the agent is running on the API server (Master node) and
auto-detecting a default configuration, you can add the line:
mesos_autodetect: false
either directly in the dragent.yaml file or as an ADDITIONAL_CONF
parameter in a Docker command.
Restart the agent.
1.1.9 -
Airgapped Agent Installation
Airgapped environments are those that do not have the network access to
pull images from the container repository. Agent installation requires
sysdigcloud-probe
and you cannot download a pre-compiled module in an
airgapped environment. Therefore, ensure that you compile your own
sysdigcloud-probe
before installing the agent.
Prepare the Sysdig Probe Builder Images
On a machine with internet connectivity, build the Sysdig probe
container and create a tar file of the image.
Get the probe builder artifacts from the repository:
$ git clone https://github.com/draios/sysdig
$ git checkout probe-builder
$ cd sysdig
Build the container image:
$ docker build -t airgap/sysdig-probe-builder probe-builder/
Create the container and run:
$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock airgap/sysdig-probe-builder:latest -P -b airgap/
Save the images to a tar archive:
$ docker save airgap/sysdig-probe-builder | gzip > builders.tar.gz
Ensure that you make this tar available to the airgapped machines
where you intend to install the Sysdig agent.
Set Up Kernel Module
Set up a local repository to host the pre-compiled kernel module:
$ kubectl run my-nginx --image=nginx --port=80
$ kubectl expose deployment my-nginx --port=80 --type=NodePort
Copy sysdigcloud-probe
to the repository you have created:
$ kubectl cp sysdigcloud-probe-<version> my-nginx-xxxxxxxx-xxxx:/usr/share/nginx
Install Agent in Docker Environment
Install Sysdig agent by pointing SYSDIG_PROBE_URL
to the local
repository:
For docker-based installations:
$ docker run -d --name sysdig-agent --restart always --privileged --net host --pid host -e ACCESS_KEY=WWWWW-YYYY-XXXX-ZZZZ-123456789 -e SECURE=true -e SYSDIG_PROBE_URL=http://www.mywebserver.net:80/ -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro --shm-size=512m sysdig/agent
Where -e SYSDIG_PROBE_URL=http://www.mywebserver:80/
is the local
nginx
pod with the loaded module.
To use secure communication with a self-signed or untrusted
certificate, apply the -e SYSDIG_PROBE_INSECURE_DOWNLOAD=true
environment variable.
Check the agent log. You will see a similar message:
Found custom module URL http://mywebserver:80/, will use it * Trying to download precompiled module from http://mywebserver:80/sysdigcloud-probe-<version>
Continue with the instructions in Agent Install:
Non-Orchestrated.
Install Agent in Kubernetes Environment
Open your agent daemonset and update the SYSDIG_PROBE_URL
to point
to the local repository:
- name: SYSDIG_PROBE_URL
value: http://www.mywebserver:80/
If you would like to use secure communication with a self-signed or
untrusted certificate, apply the SYSDIG_PROBE_INSECURE_DOWNLOAD
environment variable.
- name: SYSDIG_PROBE_INSECURE_DOWNLOAD
value: true
Continue with the instructions in Agent Install:
Kubernetes.
1.1.10 -
Identify Agent Version
Use one of the following methods to determine the version of the agents
installed in your environment:
Explore
Segmenting metrics by using agent.version
shows the installed versions
of agents in your environment. For example, segment the uptime
metric
across your environment by using agent.version
. Hover over the graph
to see the list of agent versions.

The image shows the list of agent versions in
n/a.
Dashboard
Use the Sysdig Agent Health Dashboard to determine the agent
versions:
Log in to the Sysdig Monitor.
Select Dashboards and expand Host Infrastructure Dashboards.
Open the Sysdig Agent Health & Status template or create your
own from the template.

The Sysdig Agent and Health & Status Dashboard shows the agent
version corresponding to each host in your environment.
1.2 -
Agent Configuration
Out of the box, the Sysdig agent will gather and report on a wide
variety of pre-defined metrics. It can also accommodate any number of
custom parameters for additional metrics collection.
Use this section when you need to change the default or pre-defined
settings by editing the agent configuration files, or for other
special circumstances.
Monitoring Integrations also require
editing the agent config files.
By default, the Sysdig agent is configured to collect metric data from a
range of platforms and applications. You can edit the agent config files
to extend the default behavior, including additional metrics for JMX,
StatsD, Prometheus, or a wide range of other applications. You can also
monitor log files for
targeted text strings.
1.2.1 -
Understand the Agent Configuration
Out of the box, the Sysdig agent will gather and report on a wide
variety of pre-defined metrics. It can also accommodate any number of
custom parameters for additional metrics collection.
The agent relies on a pair of configuration files to define metrics
collection parameters:
dragent.default.yaml
| The core configuration file. You can look at it to understand more about the default configurations provided. Location: /opt/draios/etc/dragent.default.yaml . CAUTION. This file should never be edited. |
dragent.yaml or configmap.yaml (Kubernetes)
| The configuration file where parameters can be added, either directly in YAML as name/value pairs, or using environment variables such as ADDITIONAL_CONF Location: /opt/draios/etc/dragent.yaml . |
The dragent.yaml
file can be accessed and edited in several ways,
depending on how the agent was installed. This document describes how to
modify dragent.yaml
.
One additional file, dragent.auto.yaml
is also created and used in
special circumstances. See Optional: Agent
Auto-Config for more
detail.
Access and Edit the Configuration File
There are various ways to add or edit parameters indragent.yaml
.
Option 1: With dragent.yaml (for testing)
It is possible to edit the container’s file directly on the host.
Add parameters directly in YAML.
Access dragent.yaml
directly
at"/opt/draios/etc/dragent.yaml
."
Edit the file. Use proper YAML
syntax.
See the examples at the bottom of the page.
Restart the agent for changes to take effect
Option 2: With configmap.yaml(Kubernetes)
Configmap.yaml
is the configuration file where parameters can be added,
either directly in YAML as name/value pairs, or using environment
variables such as ‘ADDTIONAL_CONF."
If you install agents as DaemonSets on a system running Kubernetes, you
use configmap.yaml
to connect with and manipulate the
underlyingdragent.yaml
file.
See also: Agent Install: Kubernetes | GKE | OpenShift |
IBM
Add parameters directly in YAML.
Edit the files locally and apply with the changes withkubectl -f.
Access theconfigmap.yaml
.
Edit the file as needed.
Apply the changes:
kubectl apply -f sysdig-agent-configmap.yaml
Running agents will automatically pick the new configuration after
Kubernetes pushes the changes across all the nodes in the cluster.
Option 3: With Docker Run (Docker)
Add -e ADDITIONAL_CONF="<VARIABLES>"
to a Docker run command, where
<VARIABLES>
contains all the customized parameters you want to
include, in a single-line format.
To insert ADDITIONAL_CONF
parameters in a Docker run command or a
daemonset
file, you must convert the YAML code into a single-line
format.
You can do the conversion manually for short snippets. To convert longer
portions of YAML, use echo|sed
commands.
In earlier versions, the Sysdig Agent connected to port 6666. This
behavior has been deprecated, as the Sysdig agent now connects to port
6443.
The basic procedure:
Write your configuration in YAML, as it would be entered directly in
dragent.yaml
.
In a bash shell, use echo
and sed
to convert to a single
line.
sed
script: echo "" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'
Insert the resulting line into a Docker run command or add it to the
daemonset
file as an ADDITIONAL_CONF
.
Example: Simple
Insert parameters to turn off StatsD collection and blacklist port 6443.
Sysdig agent uses port 6443 for both inbound and outbound communication with the Sysdig backend. The agent initiates a request and keeps a connection open with the Sysdig backend for the backend to push configurations, Falco rules, policies, and so on.
Ensure that you allow the agents’ inbound and outbound communication on TCP 6443 from the respective IPs associated with your SaaS Regions. Note that you are allowing the agent to send communication outbound on TCP 6443 to the inbound IP ranges listed in the SaaS Regions.
YAML format
statsd:
enabled: false
blacklisted_ports:
- 6443
Single-line format (manual)
Use spaces, hyphens, and \n
correctly when manually converting to a
single line:
ADDITIONAL_CONF="statsd:\n enabled: false\n blacklisted_ports:\n - 6443"
Here the single line is incorporated into a full agent startup Docker
command.
docker run
--name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=1234-your-key-here-1234 \
-e TAGS=dept:sales,local:NYC \
-e ADDITIONAL_CONF="statsd:\n enabled: false\n blacklisted_ports:\n - 6443" \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
quay.io/sysdig/agent
Example: Complex
Insert parameters to override the default configuration for a RabbitMQ
app check.
YAML format
app_checks:
- name: rabbitmq
pattern:
port: 15672
conf:
rabbitmq_api_url: "http://localhost:15672/api/"
rabbitmq_user: myuser
rabbitmq_pass: mypassword
queues:
- MyQueue1
- MyQueue2
Single-line format (echo |sed)
From a bash shell, issue the echo command and sed script.
echo "app_checks:
- name: rabbitmq
pattern:
port: 15672
conf:
rabbitmq_api_url: "http://localhost:15672/api/"
rabbitmq_user: myuser
rabbitmq_pass: mypassword
queues:
- MyQueue1
- MyQueue2
" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'
This results in the single-line format to be used with ADDITIONAL_CONF
in a Docker command or daemonset file.
"app_checks:\n - name: rabbitmq\n pattern:\n port: 15672\n conf:\n rabbitmq_api_url: http://localhost:15672/api/\n rabbitmq_user: myuser\n rabbitmq_pass: mypassword\n queues:\n - MyQueue1\n - MyQueue2\n"
If you installed the Sysdig agent in Kubernetes using a Helm
chart, then no
configmap.yaml
file was downloaded. You edit dragent.yaml
using Helm
syntax:
Example
$ helm install
--name sysdig-agent
--set sysdig.settings.tags='linux:ubuntu\,dept:dev\,local:nyc'
--set clusterName='my_cluster'
sysdig/sysdig
Will be transformed into
data:
dragent.yaml: |
tags: linux:ubuntu,dept:dev,local:nyc
k8s_cluster_name: my_cluster
Table 1: Environment Variables for Agent Config File
ACCESS_KEY
| <your Sysdig access key>
| Required |
TAGS
| <meaningful tags you want applied to your instances>
| Optional. These are displayed in Sysdig Monitor for ease of use. For example: tags: linux:ubuntu,dept:dev,local:nyc
See sysdig-agent-configmap.yaml. |
COLLECTOR
| <collector-hostname.com> or 111.222.333.400
| Enter the host name or IP address of the Sysdig collector service. Note that when used within dragent.yaml , must be lowercase collector . For SaaS regions, see: SaaS Regions and IP Ranges. |
COLLECTOR_PORT
| 6443
| On-prem only. The port used by the Sysdig collector service; default 6443. |
SECURE
| "true"
| On-prem only. If using SSL/TLS to connect to collector service value = "true" otherwise "false." |
CHECK_CERTIFICATE
| "false"
| On-prem only. Set to "true" when using SSL/TLS to connect to the collector service and should check for valid SSL/TLS certificate. |
ADDITIONAL_CONF
| | Optional. A place to provide custom configuration values to the agent as environment variables . |
SYSDIG_PROBE_URL
| | Optional. An alternative URL to download precompiled kernel module. |
Sample Docker Command Using Variables
docker run \
--name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=3e762f9a-3936-4c60-9cf4-c67e7ce5793b \
-e COLLECTOR=mycollector.elb.us-west-1.amazonaws.com \
-e COLLECTOR_PORT=6443 \
-e CHECK_CERTIFICATE=false \
-e TAGS=my_tag:some_value \
-e ADDITIONAL_CONF="log:\n file_priority: debug\n console_priority: error" \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=350m \
quay.io/sysdig/agent
1.2.2 -
Agent modes provide the ability to control metric collection to fit your
scale and specific requirement. You can choose one of the following
modes to do so:
Monitor
Monitor Light
Troubleshooting
Secure
Custom Metrics Only
Using a stripped-down mode limits collection of unneeded metrics, which
in turn prevents the consumption of excess resources and helps reduce
expenses.
Monitor
The Monitor mode offers an extensive collection of metrics. We recommend
this mode to monitor enterprise environments.
monitor
is the default mode if you are running the Enterprise
tier. To switch back to the
Monitor mode from a different mode, do one of the following:
Add the following to the dragent.yaml
file and restart the agent:
feature:
mode: monitor
Remove the parameter related to the existing mode from the
dragent.yaml
file and restart the agent. For example, to switch
from troubleshooting
mode to monitor
, delete the following
lines:
feature:
mode: troubleshooting
Monitor Light
Monitor Light caters to the users that run agents in a
resource-restrictive environment, or to those who are interested only in
a limited set of metrics.
Monitor Light provides CPU, Memory, File, File system, and Network
metrics. For more information, see Metrics Available in Monitor
Light.
Enable Monitor Light Mode
To switch to the Monitor Light mode, edit the dragent.yaml
file:
Open the dragent.yaml
file.
Add the following configuration parameter:
feature:
mode: monitor_light
Restart the agent.
Troubleshooting
Troubleshooting mode offers sophisticated metrics with detailed
diagnostic capabilities. Some of these metrics are heuristic in nature.
In addition to the extensive metrics available in the Monitor mode,
Troubleshooting mode provides additional metrics such as net.sql
and
additional segmentation for file and network metrics. For more
information, see Additional Metrics Values Available in
Troubleshooting.
Enable Troubleshooting Mode
To switch to the Troubleshooting mode, edit the dragent.yaml
file:
Open the dragent.yaml
file.
Add the following configuration parameter:
feature:
mode: troubleshooting
Restart the agent.
Secure Mode
The secure mode supports only Sysdig
Secure features.
Sysdig agent collects no metrics in the secure mode, which, in turn,
minimizes network consumption and storage requirement in the Sysdig
backend. Lower resource usage can help reduce costs and improve
performance.
In the Secure mode, the Monitor UI shows no data because no metrics are
sent to the collector.
This feature requires agent v10.5.0 or above.
Enabling Secure Mode
Open the dragent.yaml
file.
Add the following:
feature:
mode: secure
Restart the agent.
Custom Metrics Only Mode
Custom Metrics Only mode collects the same metrics as the Monitor
Light mode, but also adds the ability to collect the following:
- Custom Metrics: StatsD, JMX, App Checks, and Prometheus
- Kubernetes State Metrics
As such, Custom Metrics Only mode is suitable if would like to use most of the
features of Monitor mode but are limited in resources.
This mode is not compatible with Secure. If your account is
configured for Secure, you must explicitly disable Secure in
the agent configuration if you wish to use this mode.
This mode requires agent v12.4.0 or above.
Enabling Custom Metrics Only Mode
Open the dragent.yaml
file.
Add the following configuration parameter:
feature:
mode: custom-metrics-only
If your account is enabled for Secure, add the following:
security:
enabled: false
secure_audit_streams:
enabled: false
falcobaseline:
enabled: false
This configuration explicitly disables the Secure features in the agent. If you do not disable Secure, the agent will not start due to incompatiblity issues.
Restart the agent.
1.2.2.1 -
Metrics Available in Monitor Light
Monitor Light provides cpu, memory, file, file system, and network
metrics.
Metrics | Description |
---|
cpu.cores.used | See System. |
cpu.cores.used.percent | |
cpu.idle.percent | |
cpu.iowait.percent | |
cpu.nice.percent | |
cpu.stolen.percent | |
cpu.system.percent | |
cpu.used.percent | |
cpu.user.percent | |
load.average.percpu.1m | |
load.average.percpu.5m | |
load.average.percpu.15m | |
memory.bytes.available | |
memory.bytes.total | |
memory.bytes.used | |
memory.bytes.used | |
memory.bytes.virtual | |
memory.pageFault.major | |
memory.pageFault.minor | |
memory.swap.bytes.available | |
memory.swap.bytes.total | |
memory.swap.bytes.used | |
memory.swap.used.percent | |
memory.used.percent | |
file.bytes.in | |
file.bytes.out | |
file.bytes.total | |
file.iops.in | |
file.iops.out | |
file.iops.total | |
file.open.count | |
file.time.in | |
file.time.out | |
file.time.total | |
fs.bytes.free | |
fs.bytes.total | |
fs.bytes.used | |
fs.free.percent | |
fs.inodes.total.count | |
fs.inodes.used.count | |
fs.inodes.used.percent | |
fs.largest.used.percent | |
fs.root.used.percent | |
fs.used.percent | |
net.bytes.in | |
net.bytes.out | |
net.bytes.total | |
proc.count | |
thread.count | |
container.count | |
system.uptime | |
uptime | |
1.2.2.2 -
Additional Metrics Values Available in Troubleshooting
In addition to the extensive set of metrics available in the monitor
mode, additional metrics, such as net.sql
and net.mongodb
, as well
as additional segmentations for file and network metrics are available.
Metrics | Additional Metrics Values Available When Segmented by | Supported Agent Versions |
---|
file.error.total.count | file.name and file.mount labels | Version 10.1.0 or above |
file.bytes.total | | |
file.bytes.in | | |
file.bytes.out | | |
file.open.count | | |
file.time.total | | |
host.count | | |
host.error.count | | |
proc.count | | |
proc.start.count | | |
net.mongodb.collection | all | Version 10.2.0 or above |
net.mongodb.error.count | | |
net.mongodb.operation | | |
net.mongodb.request.count | | |
net.mongodb.request.time | | |
net.sql.query | all | |
net.sql.error.count | | |
net.sql.query.type | | |
net.sql.request.count | | |
net.sql.request.time | | |
net.sql.table | | |
net.http.error.count | net.http.url | Version 10.3.0 or above |
net.http.method | | |
net.http.request.count | | |
net.http.request.time | | |
net.bytes.in | | |
net.bytes.out | | |
net.request.time.worst.out | | |
net.request.count | | |
net.request.time | | |
net.bytes.total | | |
net.http.request.time.worst | all | |
1.2.2.3 -
Metrics Not Available in Essentials Mode
The following metrics will not be reported in the essentials
mode when
compared with monitor
mode:
Metrics | Segmented By |
---|
net.bytes.in | net.connection.server , net.connection.direction , net.connection.l4proto , and net.connection.client labels |
net.bytes.out | |
net.connection.count.total | |
net.connection.count.in | |
net.connection.count.out | |
net.request.count | |
net.request.count.in | |
net.request.count.out | |
net.request.time | |
net.request.time.in | |
net.request.time.out | |
net.bytes.total | |
net.mongodb.collection | all |
net.mongodb.error.count | |
net.mongodb.operation | |
net.mongodb.request.count | |
net.mongodb.request.time | |
net.sql.query | all |
net.sql.error.count | |
net.sql.query.type | |
net.sql.request.count | |
net.sql.request.time | |
net.sql.table | |
net.sql.query | all |
net.sql.error.count | |
net.sql.query.type | |
net.sql.request.count | |
net.sql.request.time | |
net.sql.table | |
net.http.method | |
net.http.request.count | |
net.http.request.time | |
net.http.statusCode | |
net.http.url | |
1.2.3 -
Enable HTTP Proxy for Agents
You can configure the agent to allow it to communicate with the Sysdig
collector through an HTTP proxy. HTTP proxy is usually configured to
offer greater visibility and better management of the network.
Agent Behaviour
The agent can connect to the collector through an HTTP proxy by sending
an HTTP CONNECT message and receiving a response. The proxy then
initiates a TCP connection to the collector. These two connections form
a tunnel that acts like one logical connection.
By default, the agent will encrypt all messages sent through this
tunnel. This means that after the initial CONNECT message and response,
all the communication on that tunnel is encrypted by SSL end-to-end.
This encryption is controlled by the top-level ssl
parameter in the
agent configuration.
Optionally, the agent can add a second layer of encryption, securing the
CONNECT message and response. This second layer of encryption may be
desired in the case of HTTP authentication if there is a concern that
network packet sniffing could be used to determine the user’s
credentials. This second layer of encryption is enabled by setting the
ssl
parameter to true in the http_proxy
section of the agent
configuration. See
Examples
for details.
Configuration
You specify the following parameters at the same level as http_proxy
in the dragent.yaml
file. These existing configuration options affect
the communication between the agent and collector (both with and without
a proxy).
ssl
: If set to false
, the metrics sent from the agent to the
collector are unencrypted (default is true
).
ssl_verify_certificate
: Determines whether the agent verifies the
SSL certificate sent from the collector (default is true
).
The following configuration options affect the behavior of the HTTP
Proxy setting. You specify them under the http_proxy
heading in the
dragent.yaml
file.
proxy_host
: Indicates the hostname of the proxy server. The
default is an empty string, which implies communication through an
HTTP proxy is disabled.
proxy_port
: Specifies the port on the proxy server the agent
should connect to. The default is 0, which indicates that the HTTP
proxy is disabled.
proxy_user
: Required if HTTP authentication is configured. This
option specifies the username for the HTTP authentication. The
default is an empty string, which indicates that authentication is
not configured.
proxy_password
: Required if HTTP authentication is configured.
This option specifies the password for the HTTP authentication. The
default is an empty string. Specifying proxy_user
with no
proxy_password
is allowed.
ssl
: If set to true, the connection between the agent and the
proxy server is encrypted.
Note that this parameter requires the top-level ssl
parameter to
be enabled, as the agent does not support SSL to the proxy but
unencrypted traffic to the collector. This additional security
prevents you from misconfiguring the agent assuming the metrics are
as well encrypted end-to-end when they are not.
ssl_verify_certificate
: Determines whether the agent will verify
the certificate presented by the proxy.
This option is configured independently of the top-level
ssl_verify_certificate
parameter. This option is enabled by
default. If the provided certificate is not correct, this option can
cause the connection to the proxy server to fail.
ca_certificate
: The path to the CA certificate for the proxy
server. If ssl_verify_certificate
is enabled, the CA certificate
must be signed appropriately.
Examples
No SSL
The following example shows no SSL connection between the agent and the
proxy server as well as between the proxy server and the collector.
collector_port: 6667
ssl: false
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3128
ssl: false
SSL Between Proxy and Collector
In this example, SSL is enabled only between the proxy server and the
collector.
collector_port: 6443
ssl: true
ssl_verify_certificate: true
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3128
SSL
The following example shows SSL is enabled between the agent and the
proxy server as well as between the proxy server and the collector.
collector_port: 6443
ssl: true
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3129
ssl: true
ssl_verify_certificate: true
ca_certificate: /usr/proxy/proxy.crt
SSL with Username and Password
The following configuration instructs the agent to connect to a proxy
server located at squid.yourdomain.com
on port 3128
. The agent will
request the proxy server to establish an HTTP tunnel to the Sysdig
collector at collector-your.sysdigcloud.com
on port 6443. The agent
will authenticate with the proxy server using the given user and
password combination.
collector: collector-your.sysdigcloud.com
collector_port: 6443
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3128
proxy_user: sysdig_customer
proxy_password: 12345
ssl: true
ssl_verify_certificate: true
ca_certificate: /usr/proxy/proxy_cert.crt
1.2.4 -
Filter Data
The dragent.yaml
file elements are wide-reaching. This section
describes the parameters to edit in dragent.yaml
to perform a range of
activities:
1.2.4.1 -
Blacklist Ports
Use the blacklisted_ports
parameter in the agent configuration file to
block network traffic and metrics from unnecessary network ports.
Note: Port 53 (DNS) is always blacklisted.
Access the agent configuration file, using one of the options
listed.
Add blacklisted_ports
with desired port numbers.
Example (YAML):
blacklisted_ports:
- 6443
- 6379
Restart the agent (if editing dragent.yaml
file directly), using
either the service dragent restart
or
docker restart sysdig-agent
command as appropriate.
1.2.4.2 -
Enable/Disable Event Data
Sysdig Monitor supports event integrations with certain applications by
default. The Sysdig agent will automatically discover these services and
begin collecting event data from them.
The following applications are currently supported:
Other methods of ingesting custom events into Sysdig Monitor are touched
upon in Custom Events.
By default, only a limited set of events is collected for a supported
application, and are listed in the agent’s default settings
configuration file (/opt/draios/etc/dragent.default.yaml
).
To enable collecting other supported events, add an events
entry to
dragent.yaml
.
You can also change log
entry in dragent.yaml
to filter events by
severity.
Learn more about it in the following sections.
Supported Application Events
Events marked with *
are enabled by default; see the
dragent.default.yaml
file.
Docker Events
The following Docker events are supported.
docker:
container:
- attach # Container Attached (information)
- commit # Container Committed (information)
- copy # Container Copied (information)
- create # Container Created (information)
- destroy # Container Destroyed (warning)
- die # Container Died (warning)
- exec_create # Container Exec Created (information)
- exec_start # Container Exec Started (information)
- export # Container Exported (information)
- kill # Container Killed (warning)*
- oom # Container Out of Memory (warning)*
- pause # Container Paused (information)
- rename # Container Renamed (information)
- resize # Container Resized (information)
- restart # Container Restarted (warning)
- start # Container Started (information)
- stop # Container Stopped (information)
- top # Container Top (information)
- unpause # Container Unpaused (information)
- update # Container Updated (information)
image:
- delete # Image Deleted (information)
- import # Image Imported (information)
- pull # Image Pulled (information)
- push # Image Pushed (information)
- tag # Image Tagged (information)
- untag # Image Untaged (information)
volume:
- create # Volume Created (information)
- mount # Volume Mounted (information)
- unmount # Volume Unmounted (information)
- destroy # Volume Destroyed (information)
network:
- create # Network Created (information)
- connect # Network Connected (information)
- disconnect # Network Disconnected (information)
- destroy # Network Destroyed (information)
Kubernetes Events
The following Kubernetes events are supported.
kubernetes:
node:
- TerminatedAllPods # Terminated All Pods (information)
- RegisteredNode # Node Registered (information)*
- RemovingNode # Removing Node (information)*
- DeletingNode # Deleting Node (information)*
- DeletingAllPods # Deleting All Pods (information)
- TerminatingEvictedPod # Terminating Evicted Pod (information)*
- NodeReady # Node Ready (information)*
- NodeNotReady # Node not Ready (information)*
- NodeSchedulable # Node is Schedulable (information)*
- NodeNotSchedulable # Node is not Schedulable (information)*
- CIDRNotAvailable # CIDR not Available (information)*
- CIDRAssignmentFailed # CIDR Assignment Failed (information)*
- Starting # Starting Kubelet (information)*
- KubeletSetupFailed # Kubelet Setup Failed (warning)*
- FailedMount # Volume Mount Failed (warning)*
- NodeSelectorMismatching # Node Selector Mismatch (warning)*
- InsufficientFreeCPU # Insufficient Free CPU (warning)*
- InsufficientFreeMemory # Insufficient Free Mem (warning)*
- OutOfDisk # Out of Disk (information)*
- HostNetworkNotSupported # Host Ntw not Supported (warning)*
- NilShaper # Undefined Shaper (warning)*
- Rebooted # Node Rebooted (warning)*
- NodeHasSufficientDisk # Node Has Sufficient Disk (information)*
- NodeOutOfDisk # Node Out of Disk Space (information)*
- InvalidDiskCapacity # Invalid Disk Capacity (warning)*
- FreeDiskSpaceFailed # Free Disk Space Failed (warning)*
pod:
- Pulling # Pulling Container Image (information)
- Pulled # Ctr Img Pulled (information)
- Failed # Ctr Img Pull/Create/Start Fail (warning)*
- InspectFailed # Ctr Img Inspect Failed (warning)*
- ErrImageNeverPull # Ctr Img NeverPull Policy Violate (warning)*
- BackOff # Back Off Ctr Start, Image Pull (warning)
- Created # Container Created (information)
- Started # Container Started (information)
- Killing # Killing Container (information)*
- Unhealthy # Container Unhealthy (warning)
- FailedSync # Pod Sync Failed (warning)
- FailedValidation # Failed Pod Config Validation (warning)
- OutOfDisk # Out of Disk (information)*
- HostPortConflict # Host/Port Conflict (warning)*
replicationController:
- SuccessfulCreate # Pod Created (information)*
- FailedCreate # Pod Create Failed (warning)*
- SuccessfulDelete # Pod Deleted (information)*
- FailedDelete # Pod Delete Failed (warning)*
Enable/Disable Events Collection with events Parameter
To customize the default events collected for a specific application (by
either enabling or disabling events), add an events
entry to
dragent.yaml
as described in the examples below.
An entry in a section in dragent.yaml
overrides the entire section
in the default configuration.
For example, the Pulling
entry below will permit only kubernetes pod
Pulling
events to be collected and all other kubernetes pod events
settings in dragent.default.yaml
will be ignored.
However, other kubernetes sections - node
and replicationController
-
remain intact and will be used as specified in dragent.default.yaml.
Example 1: Collect Only Certain Events
Collect only ‘Pulling’ events from Kubernetes for pods:
events:
kubernetes:
pod:
- Pulling
Example 2: Disable All Events in a Section
To disable all events in a section, set the event section to none
:
events:
kubernetes: none
docker: none
Example 3: Combine Methods
These methods can be combined. For example, disable all kubernetes node
and docker image events and limit docker container events to
[attach, commit, copy]
(components events in other sections will be
collected as specified by default):
events:
kubernetes:
node: none
docker:
image: none
container:
- attach
- commit
- copy
In addition to bulleted lists, sequences can also be specified in a
bracketed single line, eg.:
events:
kubernetes:
pod: [Pulling, Pulled, Failed]
So, the following two settings are equivalent, permitting only
Pulling, Pulled, Failed
events for pods to be emitted:
events:
kubernetes:
pod: [Pulling, Pulled, Failed]
events:
kubernetes:
pod:
- Pulling
- Pulled
- Failed
Change Event Collection by Severity with log Parameter
Events are limited globally at the agent level based on severity, using
the log
settings in dragent.yaml
.
The default setting for the events severity filter is information
(only warning and higher severity events are transmitted).
Valid severity levels are:
none, emergency, alert, critical, error, warning, notice, information, debug
.
Example 1: Block Low-Severity Messages
Block all low-severity messages (notice, information, debug
):
log:
event_priority: warning
Example 2: Block All Event Collection
Block all event collection:
log:
event_priority: none
For other uses of the log
settings see Optional: Change the Agent Log
Level.
1.2.4.3 -
Include/Exclude Custom Metrics
It is possible to filter custom metrics in the following ways:
Ability to include/exclude custom metrics using configurable
patterns,
Ability to log which custom metrics are exceeding limits
After you identify those key custom metrics that must be received, use
the new ‘include’ and ’exclude’ filtering parameters to make sure you
receive them before the metrics limit is hit.
Filter Metrics Example
Here is an example configuration entry that would be put into the agent
config file: (/opt/draios/etc/dragent.yaml)
metrics_filter:
- include: test.*
- exclude: test.*
- include: haproxy.backend.*
- exclude: haproxy.*
- exclude: redis.*
Given the config entry above, this is the action for these metrics:
test.* → send
haproxy.backend.request → send
haproxy.frontend.bytes → drop
redis.keys → drop
The semantic is: whenever the agent is reading metrics, they are
filtered according to configured filters and the filtering rule order -
the first rule that matches will be applied. Thus since the inclusion
item for test.* was listed first it will be followed and that second
’exclude’ rule for the same exact metric entry will be ignored.
Logging Accepted/Dropped Metrics
Logging is disabled by default. You can enable logging to see which
metrics are accepted or dropped by adding the following configuration
entry into the dragent.yaml config file:
metrics_excess_log: true
When logging of excess metrics is enabled, logging occurs at INFO-level,
every 30 seconds and lasts for 10 seconds. The entries that can be seen
in /opt/draios/logs/draios.log
will be formatted like this:
+/-[type] [metric included/excluded]: metric.name (filter: +/-[metric.filter])
The first ‘+’ or ‘-’, followed by ’type’ provides an easy way to quickly
scan the list of metrics and spot which are included or excluded (’+’
means “included”, ‘-’ means “excluded”).
The second entry specifies metric type (“statsd”, “app_check”,
“service_check”, or “jmx”).
A third entry spells out whether “included” or “excluded”, followed by
the metric name. Finally, inside the last entry (in parentheses), there
is information about filter applied and its effect (’+’ or ‘-’, meaning
“include” or “exclude”).
With this example filter rule set:
metrics_filter:
- include: mongo.statsd.net*
- exclude: mongo.statsd.*
We might see the following INFO-level log entries (timestamps stripped):
-[statsd] metric excluded: mongo.statsd.vsize (filter: -[mongo.statsd.*])
+[statsd] metric included: mongo.statsd.netIn (filter: +[mongo.statsd.net*])
1.2.4.4 -
Prioritize Designated Containers
To get the most out of Sysdig Monitor, you may want to customize the way
in which container data is prioritized and reported. Use this page to
understand the default behavior and sorting rules, and to implement
custom behavior when and where you need it. This can help reduce agent
and backend load by not monitoring unnecessary containers, or– if
encountering backend limits for containers– you can filter to ensure
that the important containers are always reported.
Overview
By default, a Sysdig agent will collect metrics from all containers it
detects in an environment. When reporting to the Monitor interface, it
uses default sorting behavior to prioritize what container information
to display first.
Understand Default Behavior
Out of the box, it chooses the containers with the highest
and allocates approximately 1/4 of the total limit to each stat type.
Understand Simple Container Filtering
As of agent version 0.86,
it is possible set a use_container_filter
parameter in the agent
config
file, tag/label specific containers, and set include/exclude
rules to push those containers to the top of the reporting hierarchy.
This is an effective sorting tool when:
You can manually mark each container with an include
or exclude
tag, AND
The number of includes is small (say, less than 100)
In this case, the containers that explicitly match the include
rules
will take top priority.
Understand Smart Container Reporting
In some enterprises, the number of containers is too high to tag with
simple filtering rules, and/or the include_all
group is too large to
ensure that the most-desired containers are consistently reported. As of
Sysdig agent version 0.91,
you can append another parameter to the agent config file,
smart_container_reporting
.
This is an effective sorting tool when:
The number of containers is large and you can’t or won’t mark each
one with include/exclude tags, AND
There are certain containers you would like to always prioritize
This helps ensure that even when there are thousands of containers in an
environment, the most-desired containers are consistently reported.
Container filtering and smart container reporting affect the monitoring
of all the processes/metrics within a container, including StatsD, JMX,
app-checks, and built-in metrics.
Prometheus metrics are attached to processes, rather than containers,
and are therefore handled differently.
The container limit is set in dragent.yaml under containers:limit:
Understand Sysdig Aggregated Container
The sydig_aggregated
parameter is automatically activated when smart
container reporting is enabled, to capture the most-desired metrics from
the containers that were excluded by smart filtering and report them
under a single entity. It appears like any other container in the Sysdig
Monitor UI, with the name “sysdig_aggregated.
”
Sysdig_aggregated
can report on a wide array of metrics; see
Sysdig_aggregated Container
Metrics. However, because
this is not a regular container, certain limitations apply:
container_id and container_image do not exist.
The aggregated container cannot be segmented by certain metrics that
are excluded, such as process.
Some default dashboards associated with the aggregated container may
have some empty graphs.
Use Simple Container Filtering
By default, the filtering feature is turned off. It can be enabled by
adding the following line to the agent configuration:
use_container_filter: true
When enabled, the agent will follow include/exclude filtering rules
based on:
The default behavior in default.dragent.yaml
excludes based on a
container label (com.sysdig.report
) and/or a Kubernetes pod
annotation (.sysdig.com/report
).
Container Condition Parameters and Rules
Parameters
The condition parameters are described in the following table:
container.image
| Matches if the process is running inside a container running the specified image | - include:
container.image: luca3m/prometheus-java-app
|
container.name
| Matches if the process is running inside a container with the specified name | - include:
container.name: my-java-app
|
container.label.*
| Matches if the process is running in a container that has a Label matching the given value | - include:
container.label.class: exporter
|
kubernetes.<object>.annotation.* kubernetes.<object>.label.*
| Matches if the process is attached to a Kubernetes object (Pod, Namespace, etc.) that is marked with the Annotation/Label matching the given value. | - include:
kubernetes.pod.annotation.prometheus.io/scrape: true
|
all | Matches all. Use as last rule to determine default behavior. | - include:
all
|
Rules
Once enabled (when use_container_filter: true
is set), the agent will
follow filtering rules from the container_filter
section.
Each rule is an include
or exclude
rule which can contain one or
more conditions.
The first matching rule in the list will determine if the container
is included or excluded.
The conditions consist of a key name and a value. If the given key
for a container matches the value, the rule will be matched.
If a rule contains multiple conditions they all need to match for
the rule to be considered a match.
Default Configuraton
The dragent.default.yaml
contains the following default configuration
for container filters:
use_container_filter: false
container_filter:
- include:
container.label.com.sysdig.report: true
- exclude:
container.label.com.sysdig.report: false
- include:
kubernetes.pod.annotation.sysdig.com/report: true
- exclude:
kubernetes.pod.annotation.sysdig.com/report: false
- include:
all
Note that it excludes via a container.label
and by a
kubernetes.pod.annotation.
The examples on this page show how to edit in the dragent.yaml
file
directly. Convert the examples to Docker or Helm commands, if applicable
for your situation.
Enable Container Filtering in the Agent Config File
Option 1: Use the Default Configuration
To enable container filtering using the default configuration in
default.dragent.yaml
(above), follow the steps below.
1. Apply Labels and/or Annotations to Designated Containers
To set up, decide which containers should be excluded from automatic
monitoring.
Apply the container label .com.sysdig.report
and/or the Kubernetes
pod annotation sysdig.com/report
to the designated containers.
2. Edit the Agent Configuration
Add the following line to dragent.yaml
to turn on the default
functionality:
use_container_filter: true
Option 2: Define Your Own Rules
You can also edit dragent.yaml
to apply your own container filtering
rules.
1. Designate Containers
To set up, decide which containers should be excluded from automatic
monitoring.
Note the image, name, label, or Kubernetes pod information as
appropriate, and build your rule set accordingly.
2. Edit the Agent Configuration
For example:
use_container_filter: true
container_filter:
- include:
container.name: my-app
- include:
container.label.com.sysdig.report: true
- exclude:
kubernetes.namespace.name: kube-system
container.image: "gcr.io*"
- include:
all
The above example shows a container_filter
with 3 include rules and 1
exclude rule.
If the container name is “my-app
” it will be included.
Likewise, if the container has a label with the key
“com.sysdig.report
” and with the value “true
”.
If neither of those rules is true, and the container is part of a
Kubernetes hierarchy within the “kube-system
” namespace and the
container image starts with “gcr.io
”, it will be excluded.
The last rule includes all, so any containers not matching an
earlier rule will be monitored and metrics for them will be sent to
the backend.
Use Smart Container Reporting
As of Sysdig agent version
0.91, you can add another
parameter to the config file: smart_container_reporting = true
This enables several new prioritization checks:
The sort is modified with the following rules in priority order:
User-specified containers come before others
Containers reported previously should be reported before those which
have never been reported
Containers with higher usage by each of the 4 default stats should
come before those with lower usage
Enable Smart Container Reporting and sysdig_aggregated
Set up any simple container filtering rules you need, following
either Option 1 or Option 2, above.
Edit the agent configuration:
smart_container_reporting: true
This turns on both smart_container_reporting
and
sysdig_aggregated
. The changes will be visible in the Sysdig
Monitor UI.
See also Sysdig_aggregated Container
Metrics..
Logging
When the log level is set to DEBUG, the following messages may be found
in the logs:
message | meaning |
---|
container <id>, no filter configured | container filtering is not enabled |
container <id>, include in report | container is included |
container <id>, exclude in report | container is excluded |
Not reporting thread <thread-id> in container <id> | Process thread is excluded |
See also: Optional: Change the Agent Log
Level.
1.2.4.4.1 -
Sysdig Aggregated Container Metrics
Sysdig_aggregated containers can report on the following metrics:
tcounters
other
time_ns
time_percentage
count
io_file
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
io_net
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
processing
time_ns
time_percentage
count
reqcounters
other
time_ns
time_percentage
count
io_file
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
io_net
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
processing
time_ns
time_percentage
count
max_transaction_counters
time_ns_in
time_ns_out
count_in
count_out
resource_counters
syscall_errors
count
count_file
count_file_opened
count_net
protos
http
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
mysql
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
postgres
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
mongodb
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
names
transaction_counters
time_ns_in
time_ns_out
count_in
count_out
1.2.4.5 -
Include/Exclude Processes
In addition to filtering data by container, it is also possible to
filter independently by process. Broadly speaking, this refinement helps
ensure that relevant data is reported while noise is reduced. More
specifically, use cases for process filtering may include:
Wanting to alert reliably whenever a given process goes down. The
total number of processes can exceed the reporting limit; when that
happens, some processes are not reported. In this case, an
unreported process could be misinterpreted as being “down.” Specify
a filter for 30-40 processes to guarantee that they will always be
reported.
Wanting to limit the number of noisy but inessential processes being
reported, for example: sed, awk, grep, and similar tools that may be
used infrequently.
Wanting to prioritize workload-specific processes, perhaps from
integrated applications such as NGINX, Supervisord or PHP-FPM.
Note that you can report on processes and containers independently;
the including/excluding of one does not affect the including/excluding
of the other.
Prerequisites_Processes
This feature requires the following Sysdig component versions:
Understand Process Filtering Behavior
By default, processes are reported according to internal criteria such
as resource usage (CPU/memory/file and net IO) and container count.
If you choose to enable process filtering, processes in the include list
will be given preference over other internal criteria.
Processes are filtered based on a standard priority filter description
already used in Sysdig yaml files. It is comprised of -include and
-exclude statements which are matched in order, with evaluation ceasing
with the first matched statement. Statements are considered matched if
EACH of the conditions in the statement is met.
Use Process Filtering
Edit dragent.yaml per the following patterns to implement the filtering
you need.
Process Condition Parameters and Rules
The process:
condition parameters and rules are described below.
Name | Value | Description |
---|
app_checks_always_send: | true/false | Legacy config that causes the agent to emit any process with app check. With process filtering, this translates to an extra “include” clause at the head of the process filter which matches a process with any app check, thereby overriding any exclusions. Still subject to limit. |
flush_filter: | | Definition of process filter to be used if flush_filter_enabled == true. Defaults to -include all |
flush_filter_enabled: | true/false | Defaults to false (default process reporting behavior). Set to true to use the rest of the process filtering options. |
limit: | N (chosen number) | Defines the approximate limit of processes to emit to the backend, within 10 processes or so. Default is 250 processes. |
top_n_per_container: | N (chosen number) | Defines how many of the top processes per resource category per emitted container to report after included processes. Still subject to limit. Defaults to 1. |
top_n_per_host: | N (chosen number) | Defines how many of the top processes per resource category per host are reported before included processes. Still subject to limit. Defaults to 1. |
The process: Condition Parameters
Rules
container.image: my_container_image
Validates whether the
container image associated with the process is a wild card match of
the provided image name
container.name: my_container_name Validates whether the container
name associated with the process is a wild card match of the
provided image name
container.label.XYZ: value
Validates whether the label XYZ of the
container associated with the process is a wildcard match of the
provided value
process.name: my_process_name
Validates whether the name of the
process is a wild card match of the provided value
process.cmdline: value
Checks whether the executable name of a
process contains the specified value, or any argument to the process
is a wildcard match of the provided value
appcheck.match: value
Checks whether the process has any appcheck
which is a wildcard match of the given value
all
Matches all processes, but does not whitelist them, nor does
it blacklist them. If no filter is provided, the default
is -include all
. However, if a filter is provided and no match is
made otherwise, then all unmatched processes will be blacklisted. In
most cases, the definition of a process filter should end
with -include: all
.
Examples
Block All Processes from a Container
Block all processes from a given container. No processes
from some_container_name will be reported.
process:
flush_filter_enabled: true
flush_filter:
- exclude:
container.name: some_container_name
- include:
allprocess: flush_filter: - exclude: container.name: some_container_name - include: all
Prioritize Processes from a Container
Send all processes from a given container at high priority.
process:
flush_filter_enabled: true
flush_filter:
- include:
container.name: some_container_name
- include:
all
Prioritize “java” Processes
Send all processes that contain ‘java" in the name at high priority.
process:
flush_filter_enabled: true
flush_filter:
- include:
process.name: java
- include:
all
Prioritize “java” Processes from a Particular Container
Send processes containing “java” from a given container at high
priority.
process:
flush_filter_enabled: true
flush_filter:
- include:
container.name: some_container_name
process.name: java
- include:
all
Prioritize “java” Processes not in a Particular Container
Send all processes that contain “java” in the name that are not in
container some_container_nane
.
process:
flush_filter_enabled: true
flush_filter:
- exclude:
container.name: some_container_name
- include:
process.name: java
- include:
all
Prioritize “java” Processes even from an Excluded Container
Send all processes containing “java” in the name. If a process does not
contain “java” in the name and if the container within which the process
runs is named some_container_name, then exclude it.
Note that each include/exclude rule is handled sequentially and
hierarchically so that even if the container is excluded, it can still
report “java” processes.
flush_filter:
- flush_filter_enabled: true
- include:
process.name: java
- exclude:
container.name: some_container_name
- include:
all
Prioritize “java” Processes and “sql” Processes from Different Containers
Send Java processes from one container and SQL processes from another at
high priority.
process:
flush_filter:
- flush_filter_enabled: true
- include:
container.name: java_container_name
process.name: java
- include
container.name: sql_container_name
process.name: sql
- include
all
Report ONLY Processes in a Particular Container
Only send processes running in a container with a given label.
process:
flush_filter:
- flush_filter_enabled: true
- include:
=container.label.report_processes_from_this_container_example_label: true
- exclude:
all
1.2.4.6 -
Collect Metrics from Remote File Systems
Sysdig agent does not automatically discover and collect metrics from
external file systems, such as NFS, by default. To enable collecting
these metrics, add the following entry to the dragent.yaml
file:
remotefs = true
In addition to the remote file systems, the following mount types are
also excluded because they cause high load.
mounts_filter:
- exclude: "*|autofs|*"
- exclude: "*|proc|*"
- exclude: "*|cgroup|*"
- exclude: "*|subfs|*"
- exclude: "*|debugfs|*"
- exclude: "*|devpts|*"
- exclude: "*|fusectl|*"
- exclude: "*|mqueue|*"
- exclude: "*|rpc_pipefs|*"
- exclude: "*|sysfs|*"
- exclude: "*|devfs|*"
- exclude: "*|devtmpfs|*"
- exclude: "*|kernfs|*"
- exclude: "*|ignore|*"
- exclude: "*|rootfs|*"
- exclude: "*|none|*"
- exclude: "*|tmpfs|*"
- exclude: "*|pstore|*"
- exclude: "*|hugetlbfs|*"
- exclude: "*|*|/etc/resolv.conf"
- exclude: "*|*|/etc/hostname"
- exclude: "*|*|/etc/hosts"
- exclude: "*|*|/var/lib/rkt/pods/*"
- exclude: "overlay|*|/opt/stage2/*"
- exclude: "/dev/mapper/cl-root*|*|/opt/stage2/*"
- exclude: "*|*|/dev/termination-log*"
- include: "*|*|/var/lib/docker"
- exclude: "*|*|/var/lib/docker/*"
- exclude: "*|*|/var/lib/kubelet/pods/*"
- exclude: "*|*|/run/secrets"
- exclude: "*|*|/run/containerd/*"
- include: "*|*|*"
To include a mount type:
Open the dragent.yaml
file.
Remove the corresponding line from the exclude list in the
mount_filter
.
Add the file mount to the include list under mount_filter
.
The format is:
# format of a mount filter is:
# ```
# mounts_filter:
# - exclude: "device|filesystem|mount_directory"
# - include: "pattern1|pattern2|pattern3"
For example:
mounts_filter:
- include: "*|autofs|*"mounts_filter:
- include: "overlay|*|/opt/stage2/*"
- include: "/dev/mapper/cl-root*|*|/opt/stage2/*"
Save the configuration changes and restart the agent.
1.2.4.7 -
Disable Captures
Sometimes, security requirements dictate that capture functionality
should NOT be triggered at all (for example, PCI compliance for payment
information).
To disable Captures altogether:
Access using one of the options
listed.
This example accesses dragent.yaml
directly. ``
Set the parameter:
sysdig_capture_enabled: false
Restart the agent, using the command: ``
service dragent restart
See Captures for more
information on the feature
1.2.5 -
Reduce Memory Consumption in Agent
Sysdig provides a configuration option called thin cointerface to reduce
the memory footprint in the agent. When the agent is installed as a
Kubernetes daemonset, you can optionally enable the thin cointerface in
the sysdig-agent configmap
.
Pros
- Reduces memory consumption
- Particularly useful on very large Kubernetes clusters (>10,000 pods)
Cons
- Less frequently used option which is therefore less battle-tested
- If a watch is dropped and re-list is required (e.g. in case of a network issue, and apiserver update, etc.), there is no cache to maintain the resources. In this case, the agent must process many additional events.
How It Works
In a typical Kubernetes cluster, two instances of agent daemonset are
installed to retrieve the data. They are automatically connected to the
Kubernetes API server to retrieve the metadata associated with the
entities running on the cluster and sends the global Kubernetes state to
the Sysdig backend. Sysdig uses this data to generate kube state
metrics.
A delegated agent will not have a higher CPU or memory footprint than a
non-delegated agent.
On very large Kubernetes clusters (in the range of 10,000 pods) or
clusters with several replication controllers, the agent’s data
ingestion can have a significant memory footprint on itself and on the
Kubernetes API server. Thin cointerface is provided to reduce this
impact.
Enabling this option changes the way the agent communicates with the API
server and reduces the need to cache data, which in turn reduces the
overall memory usage. Thin cointerface does this by moving some
processing from the agent’s cointerface process to the dragent process.
This change does not alter the data which is ultimately sent to the
backend nor will it impact any Sysdig feature.
The thin cointerface feature is disabled by default.
To Enable:
Add the following in either the sysdig-agent’s configmap
or via
the dragent.yaml
file:
thin_cointerface_enabled: true
1.2.6 -
Enable Kube State Metrics
Agent Versions 12.5.0 and Onward
HPA kube state metrics are no longer collected by default. To enable the agent to collect HPA kube state metrics, you must edit the agent configuration file, dragent.yaml
, and include it along with the other resources you would like to collect.
For example, to collect all supported resources including HPAs, add the following to dragent.yaml
:
k8s_extra_resources:
include:
- services
- resourcequotas
- persistentvolumes
- persistentvolumeclaims
- horizontalpodautoscalers
Agent Versions 12.3.x and 12.4.x
The Sysdig agent collects HPA, PVS, PV, Resourcequota, and Services kube state metrics by default.
To disable some of them, you must edit the agent config file, dragent.yaml
, as follows:
k8s_extra_resources:
include:
- services
- resourcequotas
- persistentvolumes
- persistentvolumeclaims
- horizontalpodautoscalers
The above list includes all the supported resources so you must remove the resources you are not interested in.
For example, if you wanted to disable Services, it should look like the following:
k8s_extra_resources:
include:
- resourcequotas
- persistentvolumes
- persistentvolumeclaims
- horizontalpodautoscalers
For more information, see Understanding the Agent Configuration Files.
1.2.7 -
Process Kubernetes Events
Use Go to Process Kubernetes Events
Required: Sysdig agent version 92.1 or higher.
As of agent version 9.5.0, go_k8s_user_events:true
is the default
setting. Set to false
to use the older, C++-based version.
To streamline Sysdig agent processing times and reduce CPU load, you can
use an updated processing engine written in Go.
To do so, edit the following code in dragent.yaml
:
go_k8s_user_events: true
Kubernetes Audit Events
The agent listens on /k8s-audit
for Kubernetes audit events. Configure
the path using the following configuration option:
security:{k8s_audit_server_path_uris: [path1, path2]}
For more information, see Kubernetes Audit
Logging.
Working with containerd in K3S
If you have containerd using a custom socket, you can specify this parameter in the agent configuration to correctly capture the containers’ metadata:
cri:
socket_path: /run/k3s/containerd/containerd.sock
1.2.8 -
Manage Agent Log Levels
Sysdig allows you to configure file log levels for agents globally and
granularly.
1.2.8.1 -
Change Agent Log Level Globally
The Sysdig agent generates log entries in /opt/draios/logs/draios.log
.
The agent will rotate the log file when it reaches 10MB in size, keeping
the 10 most recent log files archived with a date-stamp appended to the
filename.
In order of increasing detail, the log levels available are: [ none
| critical| error | warning |notice | info | debug | trace ].
The default level (info) creates an entry for each aggregated
metrics transmission to the backend servers, once per second, in
addition to entries for any warnings and errors.
Setting the value lower than info
may prohibit troubleshooting
agent-related issues.
The type and amount of logging can be changed by adding parameters and
log level arguments shown below to the agent’s user settings
configuration file here:
/opt/draios/etc/dragent.yaml
After editing the dragent.yaml
file, restart the agent at the shell
with: service dragent restart
to affect changes.
Note that dragent.yaml
code can be written in both YAML and JSON. The
examples below use YAML.
File Log Level
When troubleshooting agent behavior, increase the logging to debug for
full detail:
log:
file_priority: debug
If you wish to reduce log messages going to the
/opt/draios/logs/draios.log
file, add the log:
parameter with one of
the following arguments under it and indented two spaces: [ none |
error | warning | info | debug | trace ]
log:
file_priority: error
Container Console Logging
If you are running the containerized agent, you can also reduce
container console output by adding the additional parameter
console_priority:
with the same arguments [ none | error | warning
| info | debug | trace ]
log:
console_priority: warning
Note that troubleshooting a host with less than the default ‘info’ level
will be more difficult or not possible. You should revert to ‘info’ when
you are done troubleshooting the agent.
A level of ’error’ will generate the fewest log entries, a level of
’trace’ will give the most, ‘info’ is the default if no entry exists.
Examples
Using HELM
helm install ... \
--set sysdig.settings.log.file_priority=debug \
--set sysdig.settings.log.console_priority=debug
Using values.yaml
sysdig:
settings:
log:
file_priority: debug
console_priority: debug
Using dragent.yaml
customerid: 831f3-Your-Access-Key-9401
tags: local:sf,acct:eng,svc:websvr
log:
file_priority: warning
console_priority: info
OR
customerid: 831f3-Your-Access-Key-9401
tags: local:sf,acct:eng,svc:websvr
log: { file_priority: debug, console_priority: debug }
Using Docker Run Command
If you are using the “ADDITIONAL_CONF” parameter to start a Docker
containerized agent, you would specify this entry in the Docker run
command:
-e ADDITIONAL_CONF="log: { file_priority: error, console_priority: none }"
-e ADDITIONAL_CONF="log:\n file_priority: error\n console_priority: none"
Using deamonset.yaml in Kubernetes Infrastructure
When running in a Kubernetes infrastructure (installed using the v1
method, comment in the “ADDITIONAL_CONF” line in the agent
sysdig-daemonset.yaml
manifest file, and modify as needed:
- name: ADDITIONAL_CONF #OPTIONAL pass additional parameters to the agent
value: "log:\n file_priority: debug\n console_priority: error"
1.2.8.2 -
Manage File Logging for Agent Components
Sysdig Agent provides the ability to set component-wise log levels that
override the global file logging level controlled by the file_priority
configuration option. The components represent internal software modules
and can be found in /opt/draios/logs/draios.log
.
By controlling logging at the fine-grained component level, you can
avoid excessive logging from certain components in draios.log
or
enable extra logging from specific components for troubleshooting.
Components can also have an optional feature level logging that
can provide a way to control the logging for a particular feature
in Sysdig Agent.
To set feature-level or component-level logging:
Determine the agent feature or component you want to set the log level:
To do so,
Open the /opt/draios/logs/draios.log
file.
Copy the component name.
The format of the log entry is:
<timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
For example, the given snippet from a sample log file shows log
messages from promscrape
featture, sdjagent
, mountedfs_reader
,
watchdog_runnable
, protobuf_file_emitter
,
connection_manager
, and dragent
.
2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
2020-09-07 18:52:40.065, 27979.27980, Debug, promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
To set feature-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info.
log:
file_priority: notice
file_priority_by_component:
- "promscrape: info"
The log levels specified for feature override global settings.
To set component-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info, sdjagent
, mountedfs_reader
component log level to debug, watchdog_runnable
component log level
to warning and promscrape:prom_emitter
component log level to debug.
log:
file_priority: notice
file_priority_by_component:
- "promscrape: info"
- "promscrape:prom_emitter: debug"
- "watchdog_runnable: warning"
- "sdjagent: debug"
- "mountedfs_reader: debug"
The log levels specified for feature override global settings.
The log levels specified for component overide feature and global settings.
Restart the agent.
For example, if you have installed the agent as a service, then run:
$ service dragent restart
1.2.8.3 -
Manage Console Logging for Agent Components
Sysdig Agent provides the ability to set component-wise log levels that
override the global console logging level controlled by the
console_priority
configuration option. The components represent
internal software modules and can be found in
/opt/draios/logs/draios.log
.
By controlling logging at the fine-grained component level, you can
avoid excessive logging from certain components in draios.log
or
enable extra logging from specific components for troubleshooting.
Components can also have an optional feature level logging that
can provide a way to control the logging for a particular feature
in Sysdig Agent.
To set feature-level or component-level logging:
Determine the agent component you want to set the log level:
To do so,
Look at the console output.
If you’re using an orchestrator like Kubernetes, the log viewer
facility, such as the kubectl
log command, shows the console
log output.
Copy the component name.
The format of the log entry is:
<timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
For example, the given snippet from a sample log file shows log
messages from promscrape
featture, sdjagent
, mountedfs_reader
,
watchdog_runnable
, protobuf_file_emitter
,
connection_manager
, and dragent
.
2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
2020-09-07 18:52:40.065, 27979.27980, Debug, promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
To set feature-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info.
log:
console_priority: notice
console_priority_by_component:
- "promscrape: info"
The log levels specified for feature override global settings.
To set component-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info, sdjagent
, mountedfs_reader
component log level to debug, watchdog_runnable
component log level
to warning and promscrape:prom_emitter
component log level to debug.
log:
console_priority: notice
console_priority_by_component:
- "promscrape: info"
- "promscrape:prom_emitter: debug"
- "watchdog_runnable: warning"
- "sdjagent: debug"
- "mountedfs_reader: debug"
The log levels specified for feature override global settings.
The log levels specified for component overide feature and global settings.
Restart the agent.
For example, if you have installed the agent as a service, then run:
$ service dragent restart
1.2.9 -
Agent Auto-Config
Introduction
If you want to maintain centralized control over the configuration of
your Sysdig agents, one of the following approaches is typically ideal:
Via an orchestration system, such as using
Kubernetes or
Mesos/Marathon.
Using a configuration management system, such as
Chef or
Ansible.
However, if these approaches are not viable for your environment, or to
further augment your Agent configurations via central control, Sysdig
Monitor provides an Auto-Config option for agents. The feature allows
you to upload fragments of YAML configuration to Sysdig Monitor that
will be automatically pushed and applied to some/all of your Agents
based on your requirements.
Enable Agent Auto-Config
Independent of the Auto-Config feature, typical Agent configuration
lives in /opt/draios/etc and is derived from a combination of base
config in the dragent.default.yaml file and any overrides that may
be present in dragent.yaml. See also Understanding the Agent Config
Files.
Agent Auto-Config adds a middle layer of possible overrides in an
additional file dragent.auto.yaml.When present, the the order of
config application from highest precedence to lowest now becomes:
dragent.yaml
dragent.auto.yaml
dragent.default.yaml
While all Agents are by default prepared to receive and make use of
Auto-Config data, the file dragent.auto.yaml will not be present on
an Agent until you’ve pushed central Auto-Config data to be applied to
that Agent.
Auto-Config settings are performed via Sysdig Monitor’s REST API.
Simplified examples are available that use the Python client
library to
get
or
set
current Auto-Config settings. Detailed examples using the REST API are
shown below.
The REST endpoint for Auto-Config is /api/agents/config. Use the
GET method to review the current configuration. The following
example shows the initial empty settings that result in no
dragent.auto.yaml files being present on your Agents.
curl -X GET \
--header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
https://app.sysdigcloud.com/api/agents/config
Output:
{
"files": []
}
Use the PUT method to centrally push YAML that will be distributed
and applied to your Agents as dragent.auto.yaml files. The
content parameter must contain syntactically-correct YAML. The
filter option is used to specify if the config should be sent to one
agent or all of them, such as in this example to globally enable Debug
logging on all Agents:
curl -X PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
https://app.sysdigcloud.com/api/agents/config -d '
{
"files": [
{
"filter": "*",
"content": "log:\n console_priority: debug"
}
]
}'
Alternatively, the filter can specify a hardware MAC address for a
single Agent that should receive a certain YAML config. All MAC-specific
configs should appear at the top of the JSON object and are not
additive to any global Auto-Config specified with “filter”: “*” at
the bottom. For example, when the following config is applied, the one
Agent that has the MySQL
app check configured would not have Debug logging enabled, but all
others would.
curl -X PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
https://app.sysdigcloud.com/api/agents/config -d '
{
"files": [
{
"filter": "host.mac = \"08:00:27:de:5b:b9\"",
"content": "app_checks:\n - name: mysql\n pattern:\n comm: mysqld\n conf:\n server: 127.0.0.1\n user: sysdig-cloud\n pass: sysdig-cloud-password"
},
{
"filter": "*",
"content": "log:\n console_priority: debug"
}
]
}'
To update the active central Auto-Config settings, simply PUT a
complete replacement JSON object.
All connected Agents will receive centrally-pushed Auto-Config updates
that apply to them based on the filter settings. Any Agent whose
Auto-Config is enabled/disabled/changed based on the centrally-pushed
settings will immediately restart, putting the new configuration into
effect. Any central Auto-Config settings that would result in a
particular Agent’s Auto-Config remaining the same will not trigger a
restart.
Disable Agent Auto-Config
To clear all Agent Auto-Configs, use the PUT method to upload the
original blank config setting of ’{ “files”: [] }’.
It is also possible to override active Auto-Config on an individual
Agent. To do so, follow these steps for your Agent:
Add the following config directly to the dragent.yaml file:
auto_config: false.
Delete the file /opt/draios/etc/dragent.auto.yaml.
Restart the Agent.
For such an Agent to opt-in to Auto-Config again, remove auto_config:
false from the dragent.yaml and restart the Agent.
Restrictions
To prevent the possibility of pushing Auto-Config that would damage an
Agent’s ability to connect, the following keys will not be accepted in
the centrally-pushed YAML.
auto_config
customerid
collector
collector_port
ssl
ssl_verify_certificate
ca_certificate
compression
1.2.10 -
Using the Agent Console
Sysdig provides an Agent Console to interact with the Sysdig agent. This
is a troubleshooting tool to help you view configuration files and
investigate agent configuration problems quickly.
Access Agent Console
From Explore click the Groupings drop-down.
Select Hosts & Container or Nodes.
Click the desired host to investigate the corresponding agent
configuration.
Click Options (three dots) on the right upper corner of the
Explore tab.
Click Agent Console.
Agent Console Commands
View Help
The ?
command displays the commands to manage Prometheus configuration
and targets monitored by the Sysdig agent.
$ prometheus ?
$ prometheus config ?
$ prometheus config show ?
Command Syntax
The syntax of the Agent Console commands is as follows:
directory command
directory sub-directory command
directory sub-directory sub-sub-directory command
View Version
Run the following to find the version of the agent running in your
environment:
$ version
An example output:
12.0.0
Troubleshoot Prometheus Metrics Collection
These commands help troubleshoot Prometheus targets configured in your
environment.
For example, the following commands display and scrape the Prometheus
endpoints respectively.
$ prometheus target show
$ prometheus target scrape
Sub-Directory Commands
The Promscrape CLI consists of the following sections.
config
: Manages Sysdig agent-specific Prometheus configuration.
metadata
: Manages metadata associated with the Prometheus targets
monitored by the Sysdig agent.
stats
: Helps view the global- and job-specific Prometheus
statistics.
target
: Manages Prometheus endpoints monitored by Sysdig agent.
Prometheus Commands
Show
The show
command displays the information about the subsection. For
example, the following example displays the configuration of the
Prometheus server.
$ prometheus config show
5 Configuration Value
6 Enabled True
7 Target discovery Prometheus service discovery
8 Scraper Promscrape v2
9 Ingest raw True
10 Ingest calculated True
11 Metric limit 2000
Scrape
The scrape
command scrapes a Prometheus target and displays the
information. The syntax is:
$ prometheus target scrape -url <URL>
For example:
$ prometheus target scrape -url http://99.99.99.3:10055/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
7 # TYPE go_gc_duration_seconds summary
8 go_gc_duration_seconds{quantile="0"} 7.5018e-05
9 go_gc_duration_seconds{quantile="0.25"} 0.000118155
10 go_gc_duration_seconds{quantile="0.5"} 0.000141586
11 go_gc_duration_seconds{quantile="0.75"} 0.000171626
12 go_gc_duration_seconds{quantile="1"} 0.00945638
13 go_gc_duration_seconds_sum 0.114420898
14 go_gc_duration_seconds_count 607
View Agent Configuration
The Agent configuration commands have a different syntax.
Run the following to view the configuration of the agent running in your
environment:
$ configuration show-dragent-yaml
$ configuration show-configmap-yaml
$ configuration show-default-yaml
$ configuration show-backend-yaml
The output displays the configuration file. Sensitive data, such as the
credentials, are obfuscated.
customerid: "********"
watchdog:
max_memory_usage_mb: 2048
Security Considerations
User-sensitive configuration is obfuscated and not visible through
the CLI.
All the information is read-only. You cannot currently change any
configuration by using the Agent console.
Runs completely inside the agent. It does not use bash or any other
Linux terminals to prevent the risk of command injection.
Runs only via a TLS connection with the Sysdig backend.
Disable Agent Console
This is currently turned on by default. To turn off Agent Console for a
particular team:
Navigate to Settings > Teams.
Select the team that you want to disable Agent Console for.
From Additional Permissions, Deselect Agent CLI.
Click Save.
To turn it off in your environment, edit the following in the
dragent.yaml
file:
command_line:
enabled: false
1.3 -
Agent Upgrade
The steps to upgrade an agent differ depending on whether the agent was originally installed as a container or as a service.
Follow the upgrade best practices for a smooth upgrade and to maximize the value of Sysdig applications:
This section describes how to check the current version of the installed agents, and then how to upgrade them.
Agent Version Check
Kubernetes installation
If the agent is installed in a Kubernetes environment, run:
kubectl get pods -n sysdig-agent -o=jsonpath='{.items[0].spec.containers[:1].image}'
Container/Docker Installation
If the agent is installed as container, run:
docker exec sysdig-agent /opt/draios/bin/dragent --version
Service Installation
If the agent is installed as a service, run:
/opt/draios/bin/dragent --version
The agent version can also be found in the agent log file:
/opt/draios/logs/draios.log.
Look for the “Agent starting” message, which is logged whenever the
agent restarts.
Update Agent
Update the containerized agent version as you normally update any container; the basic steps are below.
Use the full run command as shown in the Settings > Agent Installation tab of your account.
Containerized Agent
To see which agent versions are available see tags.
Kubernetes
Helm
Update the chart:
helm repo update
Do one of the following:
For more information on using Helm, see Helm Charts.
Manual
Check whether .yaml
files must be updated:
Updating the agent image does not overwrite the daemonset.yaml
and
sysdig-agent-configmap.yaml
on your local system. Check the Sysdig
Agent Release Notes to see
if you need to download the latest .yaml
files from
GitHub.
Perform update:
kubectl set image ds/sysdig-agent sysdig-agent=sysdig/agent:<tag>
Watch update status:
kubectl rollout status ds/sysdig-agent
Docker
Basic Steps: stop the agent, remove it, pull the new agent, and install
it.
The exact Docker command can also be found in the Sysdig Settings >
Agent Installation menu.
docker stop sysdig-agent
docker rm sysdig-agent
docker pull sysdig/agent
docker run . . .
Service Agent
For service (non-containerized) agent installations, updates are
installed as part of the normal system upgrade available with
apt-get
or yum
.
Debian, Ubuntu
apt-get update
apt-get -y install draios-agent
CentOS, RHEL, Fedora, Amazon AMI, Amazon Linux 2
yum clean expire-cache
yum -y install draios-agent
1.4 -
Uninstall the Agent
This section describes uninstalling the Sysdig agent when it was
installed as a service.
If the agent was installed as a container, remove it using standard
container commands.
If the agent was installed by an orchestrator, such as Kubernetes,
remove it by using the standard orchestrator commands.
Debian/Ubuntu Distributions
To uninstall the agent from Debian Linux distributions, including
Ubuntu:
As the sudo
user, run the following command in a terminal on each
host:
sudo apt-get remove draios-agent
Fedora/CentOS/RHEL/Amazon AMI/ Amazon Linux 2 Distributions
To uninstall the agent from Fedora Linux distributions, including
CentOS, Red Hat Enterprise Linux, as well as Amazon AMI and Amazon Linux
2:
As the sudo
user, run the following command in a terminal on each
host:
sudo yum erase draios-agent
1.5 -
Troubleshooting Agent Installation
This section describes methods for troubleshooting two types of issue:
Disconnecting Agents
If agents are disconnecting, there could be problems with addresses that
need to be resolved in the agent configuration files. See also
Understanding the Agent Config
Files.
Check for Duplicate MAC addresses
The Sysdig agent will use the eth0
MAC address to identify the
different hosts within an infrastructure. In a virtualized environment,
you should confirm each of your VM’s eth0
MAC addresses are unique.
If a unique address cannot be configured, you can supply an additional
parameter in the Sysdig agent’s dragent.yaml
configuration file:
machine_id_prefix: prefix
The prefix text can be any string and will be prepended to the MAC
address as reported in the Sysdig Monitor web interface’s Explore
tables.
Example: (using ADDITIONAL_CONF
rather than Kubernetes
Configmap
)
Here is an example Docker run command installing the parameter via the
ADDITIONAL_CONF
parameter
docker run --name sysdig-agent --privileged --net host --pid host -e ACCESS_KEY=abc123-1234-abcd-4321-abc123def456 -e TAGS=tag1:value1 -e ADDITIONAL_CONF="machine_id_prefix: MyPrefix123-" -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/agent
The resulting /opt/draios/etc/dragent.yaml
config file would look like
this:
customerid:abc123-1234-abcd-4321-abc123def456
tags: tag1:value1
machine_id_prefix: MyPrefix123-
You will then see all of your hosts, provided that all the prefixes are
unique. The prefix will be visible whenever the MAC address is displayed
in any view.
See also: Agent
Configuration.
Check for Conflicting MAC addresses in GKE environments
In Google Container Engine (GKE) environments, MAC addresses could be
repeated across multiple hosts. This would cause some hosts running
Sysdig agents not to appear in your web interface.
To address this, add a unique machine ID prefix to each config you use
to deploy the agent to a given cluster (i.e. each
sysdig-daemonset.yaml
file).
Note: This example uses the
(v1) ADDITIONAL_CONF
,
rather than (v2)
Configmap
method.
- name: ADDITIONAL_CONF value: "machine_id_prefix: mycluster1-prefix-"
Can’t See Metrics After Agent Install
If agents were successfully installed, you could log in to the Sysdig
Monitor UI, but no metrics are displayed in the Explore panel, first
confirm that the agent license count has not been exceeded. Then check
for any proxy, firewall, or host security policies preventing proper
agent communication to the Sysdig Monitor backend infrastructure.
Check License Count
If network connectivity is good, the agent will connect to the backend
but will be disconnected after a few seconds if the license count has
been exceeded.
To check whether you are over-subscribed, go to
Settings > Subscription
.
See Subscription for
details.
Check Network Policy
Agent Connection Port
Check your service provider VPC security groups to verify that network
ACLs are set to allow the agent’s outbound traffic over TCP ports. See
Sysdig Collector Ports for
the supported TCP ports for each region.
Outbound IP Addresses
Due to the distributed nature of the Sysdig Monitor infrastructure, the
agent must be open for outbound connections to
collector.sysdigcloud.com on all
outbound IP addresses.
Check Amazon’s public IP ranges
file to see all the
potential IP addresses the Sysdig agent can use to communicate with the
Sysdig backend databases.
AWS metadata is used for gathering information about the instance
itself, such as instance id, public IP address, etc.
When running on an AWS instance, access to the following AWS metadata
endpoint is also needed: 169.254.169.254
Check Local Host Policy
The agent requires access to the following local system resources in
order to gather metrics:
Read/Write access to /dev/sysdig
devices.
Read access to all the files under /proc
file system.
For container support, the Docker API endpoint
/var/run/docker.sock
If any settings or firewall modifications are made, you may need to
restart the agent service. In a shell on the affected instances issue
the following command:
sudo service dragent restart
Learn More
1.5.1 -
In addition to the information on Host Requirements for Agent
Installation, this page
describes how the agent uses kernel headers and tips on troubleshooting,
if needed.
About Kernel Headers and the Kernel Module
The Sysdig agent requires a kernel module in order to install
successfully on a host. This can be obtained in three ways:
Agent compiles the module using kernel headers.
If the hosts in your environment already have kernel header files
pre-installed, no special action is needed. Or you can install the
kernel headers manually; see below.
Agent auto-downloads precompiled modules from Sysdig’s AWS storage
location.
If the headers are not already installed but the agent is able to
auto-download, no special action is needed. If there is no internet
connectivity, you can use method 3 (download from an internal URL).
Agent downloads precompiled modules from an internal URL.
Use the environment variable SYSDIG_PROBE_URL
. See also
Understanding the Agent Config
Files. Contact Sysdig
support for assistance.
Agent Installation on SELinux-Enabled Linux Distributions
On Fedora 35 or similar SELinux-enabled distributions with default restrictive policies, the agent init container, agent-kmodule
, will not install the downloaded kernel module raising an error similar to the following:
insmod: ERROR: could not insert module /root/.sysdig/sysdigcloud-probe-12.3.1-x86_64-5.16.11-200.fc35.x86_64-67098c7fdcc97105d4b9fd0bb2341888.ko: Permission denied
In such cases, we recommend that you use eBPF option while running agent-kmodule
instead.
TracePoints
All supported distribution released kernels have this support but if
creating a custom kernel, it must support the following options:
CONFIG_TRACEPOINTS
CONFIG_HAVE_SYSCALL_TRACEPOINTS
In some cases, the host(s) in your environment may use Unix versions
that do not match the provided headers, and the agent may fail to
install correctly. In those cases, you must install the kernal headers
manually.
Debian-Style
For Debian-syle distributions, run the command:
apt-get -y install linux-headers-$(uname -r)
RHEL-Style
For RHEL-style distributions, run the command:
yum -y install kernel-devel-$(uname -r)
RancherOS
For RancherOS distributions, the kernel headers are available in the
form of a system service and therefore are enabled using the
ros
service command:
sudo ros service enable kernel-headers-system-docker
sudo ros service up -d kernel-headers-system-docker
NOTE: Some cloud hosting service providers supply pre-configured Linux
instances with customized kernels. You may need to contact your
provider’s support desk for instructions on obtaining appropriate header
files, or for installing the distribution’s default kernel.
During an agent installation in an Amazon machine image (AMI) you may
encounter the following errors while the installer is trying to compile
the Sysdig kernel module:
Errors
This indicates your machine is running a kernel in an older AMI for
which the kernel headers are no longer available in the configured
repositories. The issue has to do with Amazon purging packages in the
yum repository when new Amazon Linux machine images are released.
The solution is either to update your kernel to a version for which
header files are readily available (recommended), or perform a one-time
installation of the kernel headers for your older AMI.
Option 1: Upgrade Your Host’s Kernel
First install a new kernel and reboot your instance:
sudo yum -y install kernel
sudo reboot
After rebooting, check to see if the host is reporting metrics to your
Sysdig account. If not, you may need to issue three more commands to
install the required header files:
sudo yum -y install kernel-devel-$(uname -r)
sudo /usr/lib/dkms/dkms_autoinstaller start
sudo service dragent restart
Although it is recommended to upgrade to the latest kernel for security
and performance reasons, you can alternatively install the older headers
for your AMI.
Find the the AMI version string and install the appropriate headers with
the commands:
releasever=$(cat /etc/os-release | grep 'VERSION_ID' | grep -Eo "[0-9]{4}\.[0-9]{2}")
sudo yum -y --releasever=${releasever} install kernel-devel-$(uname -r)
Issue the remaining commands to allow the Sydig Agent to start
successfully:
sudo /usr/lib/dkms/dkms_autoinstaller start
sudo service dragent restart
Reference: Find Your AWS Instance Image Version
The file /etc/image-id
shows information about the original machine
image with which your instance was set up:
[ec2-user ~]$ cat /etc/image-id
image_name="amzn-ami-hvm"
image_version="2017.03"
image_arch="x86_64"
image_file="amzn-ami-hvm-2017.03.0.20170401-x86_64.ext4.gpt"
image_stamp="26a3-ed31"
image_date="20170402053945"
recipe_name="amzn ami"
recipe_id="47cfa924-413c-d460-f4f2-2af7-feb6-9e37-7c9f1d2b"
This file will not change as you install updates from the yum
repository.
The file /etc/system-release
will tell what version of the AWS image
is currently installed:
[ec2-user ~]$ cat /etc/system-release
Amazon Linux AMI release 2017.03
1.5.2 -
Tuning Sysdig Agent
The resource requirements for the Sysdig agent are subjective to the
size and load of the host. Increased activity equates to higher resource
requirements.
You might see 5 to 20 KiB/s of bandwidth consumed. Different variables
can increase the throughput required. For example:
When a Sysdig Capture is being collected, you can expect to see a spike
in the bandwidth while the capture file is being ingested.
Sysdig does not recommend placing bandwidth shaping or caps on the agent
to ensure that data is sent to the Sysdig Collection service.
In general, in larger clusters, the agent requires more memory, and in
servers with a high number of cores, the agent requires more CPU cores
to monitor all the system calls. You will use CPU cores on the host and
the Kubernetes nodes visible to the agent as proxies for the rate of
events processed in the agent.
Similarly, there are different factors that are at play, and considering
all the factors, we recommend the following:
Small: CPU core count <= 8. Kubernetes nodes <=10
Medium: 8 < CPU core count <= 32. 10 < Kubernetes nodes
<= 100
Large: CPU core count > 32. Kubernetes nodes > 100
While you can expect the behavior with the given numbers to be better
than simply using the default values, Sysdig cannot guarantee that
resource allocation will be correct for all the cases.
Cluster Size | Small | Medium | Large |
---|
Kubernetes CPU Request | 1 | 3 | 5 |
Kubernetes CPU Limit | 1 | 3 | 5 |
Kubernetes Memory Request | 1024 MB | 3072 MB | 6144 MB |
Kubernetes Memory Limit | 1024 MB | 3072 MB | 6144 MB |
Dragent Memory Watchdog | 512 MB | 1024 MB | 2048 MB |
Cointerface Memory Watchdog | 512 MB | 2048 MB | 4096 MB |
Note that the agent has its own memory watchdog to prevent runaway
memory consumption on the host in case of memory leaks. The default
values of the watchdog are specified in the following agent
configuration file.
watchdog:
max_memory_usage_mb: 1024
max_memory_usage_subprocesses:
sdchecks: 128
sdjagent: 256
mountedfs_reader: 32
statsite_forwarder: 32
cointerface: 512
promscrape: 640
The recommended value for promscrape
depends on the amount of timeseries and label data that required to be scraped on a particular node. The cluster size does not have an effect on promscrape
memory usage.
max_memory_usage_mb
corresponds to the dragent process in the agent.
All the values are given in MiB.
For example, use the following agent configuration to match the agent watchdog settings with large values:
watchdog:
max_memory_usage_mb: 2048
max_memory_usage_subprocesses:
sdchecks: 128
sdjagent: 256
mountedfs_reader: 32
statsite_forwarder: 32
cointerface: 4096
promscrape: 640
2 -
Serverless Agents
Overview
The serverless environment: As cloud platforms have evolved, both
the convenience and the abstraction levels have increased simultaneously
and new agent models are required.
For example, with Amazon’s ECS and EKS, users remain in charge of
managing the underlying virtual host machines. In environments like
Fargate,
however, the hosts are implicitly allocated by the cloud provider and
users simply run their containers without allocating, configuring, or
having any knowledge of the underlying compute infrastructure.
While this “container as a service” model is convenient, it can
introduce risk, as many users leave the containers unattended and don’t
monitor for security events inside them that can exfiltrate secrets,
compromise business data, and increase their AWS/cloud provider costs.
In addition, it is not possible to install a standard agent in an
environment where you do not have access to a host.
For these reasons, Sysdig has introduced a new “serverless agent” that
can be deployed in such container-based cloud environments.
2.1 -
AWS Fargate Serverless Agents
Introduction
Check the Overview for an
explanation of when and why to use serverless agents in
“container-as-a-service” cloud environments.
Architecture
The Sysdig serverless agent provides runtime detection through policy
enforcement with Falco. At this time, the serverless agent is available
for AWS Fargate on ECS. It is comprised of an orchestrator agent and
(potentially multiple) workload agents.
The Sysdig serverless orchestrator agent is a collection point
installed on each VPC to collect data from the serverless workload
agent(s) and to forward them to the Sysdig backend. It also syncs
the Falco runtime policies and rules to the workload agent(s) from
the Sysdig backend.
The Sysdig serverless workload agent is installed in each task
and requires network access to communicate with the orchestrator
agent.

Understanding Install Options
- Install Option 1: As of Serverless Agent 3.0, a simplified installer lets you deploy the orchestrator agent and instrument the Fargate task definitions for each workload agent using a dowloaded YAML file from Sysdig and then editing your own CloudFormation Template and deploying.
- Install Option 2 (legacy): Here you deploy the orchestrator agent and the workload agents in two separate processes. A range of tools are supported for the workload agent deployment: you can use Kilt (automated), Terraform (automated), a manual Task Definition, or a manually instrumented container image.
The prerequisites, upgrade instructions, and downloaded YAML files differ between the two versions.
Option 1 Installation for Fargate ECS
This option presumes your services use an existing CFT and you will install the Workload Agent using an automated process which will instrument all the Fargate Task Definitions in your CFT.
- For the Orchestrator Agent, Sysdig provides a YAML to use as a CloudFormation Template which you can deploy through the AWS Console. You need one Orchestrator deployment per VPC in your environment which your organization wants to secure.
- For the Workload Agents, you need one Workload Agent per Fargate Task Definition. For example, if you have ten services and ten Task Definitions, each needs to be instrumented.
Prequisites
Before starting, ensure you have the following.
On AWS Side
- A custom CFT containing Fargate Task Definitions you want to instrument through the Sysdig Serverless Agent
- Two VPC subnets that can connect with the Internet via a NAT Gateway or an Internet Gateway
On Sysdig Side
- Sysdig Secure
- Sysdig Agent Key to install Sysdig Agents
- The endpoint of the Sysdig Collector for your region
- The Sysdig CFT serverless-instrumentation.yaml to instrument the Fargate Task Definitions in your CFT
Deployment Steps
Deploy the Sysdig Instrumentation and Orchestration Stack
Deploy the serverless-instrumentation.yaml
for each desired VPC using CloudFormation:
Log in to the AWS Console, select CloudFormation, Create Stack with new resources, and specify the serverless-instrumentation.yaml
as the Template source.
Specify the stack details to deploy the Orchestrator Agent on the same VPC where your service is running. For standard deployments, provide the parameters highlighted in the figure below:
Click Next, complete the stack creation, and wait for the deployment to complete (usually a few minutes).

Edit your CFT
Once the stack is deployed, the Output tab provides the Transformation Instruction as shown in the figure below. Copy and paste the Transformation Instruction at the root level of your CFT.

Deploy your CFT
All new deployments of your CFT will be instrumented.
When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.
Upgrade from a Prior Version
Up through version 2.3.0, the installation process deployed two stacks (using Installation Option 2):
- Orchestration stack, deployed via YAML
- Instrumentation stack, deployed via command-line installer.
From version 3.0.0, you will deploy the “Instrumentation & Orchestration” stack only, using option 1.
To upgrade to version 3.0.0:
- Deploy the new “Instrumentation and Orchestration” stack and the Workload Agents, as described in Option1
When deploying the “Instrumentation and Orchestration” stack, assign a unique name to your macro (
Transform: MyV2SysdigMacro
).
You now have two versions of the serverless agent components. When you are ready to switch from the earlier version, proceed with the next step. - Stop all running tasks and use CloudFormation to delete the earlier stacks.
- Clean up the earlier Macro using the installer:
./installer-linux-amd64 cfn-macro delete MyEarlierSysdigMacro
- Redeploy the workload stack with the updated CFT (
Transform: MyV2SysdigMacro
).
Installation Option 2 for Fargate ECS
In this scenario, the two components of the serverless agent are
installed separately.
For the orchestrator agent, Sysdig provides a
yaml
to use as a CloudFormation Template which you can deploy through the
AWS Console. You need one orchestrator deployment per VPC in your
environment which your organization wants to secure.
For the workload agents, you need one workload agent per Fargate
task definition. (If you have ten services and ten task definitions,
each needs to be instrumented.)
We assume your services use an existing CFT and you will install the
workload agent using an automated process which will instrument all
the task definitions in your CFT.
Prerequisites
On the AWS side:
AWS CLI configured and permissions to create and use an S3 bucket
Permissions to upload images to repos, deploy CloudFormation
Templates (CFTs), and create task definitions for Fargate
The Fargate tasks you want to instrument with the Sysdig serverless
agent
Two subnets that can connect with the internet. (Your service on
Fargate must reach the orchestrator agent, and the orchestrator
agent must reach the internet to communicate with Sysdig’s back
end.)
A NAT gateway or an AWS Internet Gateway so the orchestrator agent can reach to Sysdig.
On the Sysdig side:
Install the Orchestrator Agent
Obtain the Sysdig Orchestrator Agent
yaml
to be used as the CloudFormation Template source.
For more information on CloudFormation (CFN), see AWS
documentation.
Deploy the orchestrator agent for each desired VPC, using
CloudFormation.
The steps below are an outline of the important Sysdig-related
parts.
Log in to the AWS Console. Select CloudFormation and
Create Stack with new resources and specify the
orchestrator-agent.yaml
as the Template source.
Specify the stack details to deploy the orchestrator agent
on the same VPC where your service is running.

Stack name: self-defined
Sysdig Settings
Sysdig Access Key: Use the agent
key
for your Sysdig platform.
Sysdig Collector Host: collector.sysdigcloud.com
(default);
region-dependent in Sysdig SaaS; custom in Sysdig on-prem.
Sysdig Collector Port: 6443
(default), or could be
custom for on-prem installations.
Network Settings
Advanced Settings
Sysdig Agent Tags: Enter a comma-separated list of tags
(eg. role:webserver,location:europe
) Note: tags will also
be created automatically from your infrastructure’s
metadata, including AWS, Docker, etc.
Sysdig Orchestrator Agent Image:
quay.io/orchestrator-agent:latest
(default)
Check Collector SSL Certificate: Default: true
.
False
means no validation will be done on the SSL
certificate received from the collector, used for dev
purposes only.
Sysdig Orchestrator Agent Port: 6667
(default).
Port where the orchestrator service will be listening for instrumented task connections.
Click Next, complete the stack creation, and wait for the
deployment to complete (usually less than 10 minutes.)
In Output, take note of the
OrchestratorHost
and OrchestratorPort
values.

As of Serverless Agent Version 2.2.0, the orchestrator port can be configured via either SYSDIG_ORCHESTRATOR_PORT
(default 6667, which is hardcoded into other parts of the CFT) or the SysdigOrchestratorAgentPort
(alternate) parameter in the CloudFormation template.
If AWS Internet Gateway is used (as opposed to a NAT Gateway), uncomment
the line AssignPublicIp: ENABLED
in the orchestrator.yaml.
Install the Workload Agents
Choose one of the four options below (A, B, C1, C2).
A: Automated Process (Kilt)
Prerequisite: Have the orchestrator agent deployed in the
appropriate VPC and have the Orchestrator Host and Port information
handy.
Download the appropriate installer for your OS.
These set up Kilt, an open-source library mechanism for injection
into Fargate containers.
Create a macro for the serverless worker agents, using the
installer. Any service tagged with this macro will have the
serverless worker agent(s) added and Fargate data will be collected.
Log in to AWS CLI.
Create a CFN macro that applies instrumentation. You will
need the outputs from previous task. Example:
./installer-linux-amd64 cfn-macro install -r us-east-1 MySysdigMacro $OrchestratorHost $OrchestratorPort
Add the macro you created to the CFT that you use for your own
service at the root.
Use: Transform: MySysdigMacro
.
All new deployments of that template will be instrumented.
Defining Entrypoint and Command in CFT:
In this step, the macro will go over the original CloudFormation
Template looking for ContainerDefinitions
to instrument. This
includes replacing the original entry point for the Sysdig entry
point, so ideally the CFT should explicitly describe Entrypoint
and Command.
Otherwise, the macro will try to pull the image to
retrieve a default Entrypoint
and Command
, which only works if
the image is publicly available. Otherwise the pull will fail and
the instrumentation will not be completed.
Complete!
When instrumentation is complete, Fargate events should be visible
in the Sysdig Secure Events feed.
Alternatively, you can use the Sysdig Provider on the Terraform registry to deploy the workload agents, as described below.
Prerequisite: Have the orchestrator agent deployed in the
appropriate VPC and have the Orchestrator Host and Port information
handy.
Set up the Sysdig Terraform provider:
terraform {
required_providers {
sysdig = {
source = "sysdiglabs/sysdig"
version = ">= 0.4.0"
}
}
}
provider "sysdig" {
sysdig_secure_api_token = var.sysdig_access_key
}
Pass the container-definition, orchestrator host, and orchestrator port to the sysdig_fargate_workload_agent
data source.
Notes:
The input container definitions must be in JSON format, following CloudFormation naming conventions.
When deploying the orchestrator agent with CFT, the port is always 6440.
When deploying the orchestrator agent with CFT, the host is the collector endpoint for your Sysdig region; e.g.
"collector-static.sysdigcloud.com"
for US East. For other regions, see SaaS Regions and IP Ranges.
data "sysdig_fargate_workload_agent" "instrumented" {
container_definitions = file("container-definitions/hello.json")
sysdig_access_key = var.sysdig_access_key
workload_agent_image = "quay.io/sysdig/workload-agent:latest"
orchestrator_host = module.sysdig_orchestrator_agent.orchestrator_host
orchestrator_port = module.sysdig_orchestrator_agent.orchestrator_port
}
Include the instrumented JSON in your Fargate task definition:
resource "aws_ecs_task_definition" "fargate_task" {
...
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
container_definitions = "${data.sysdig_fargate_workload_agent.instrumented.output_container_definitions}"
}
Complete!
When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.
/en/docs/installation/serverless-agents/aws-fargate-serverless-agents/manage-serverless-agent-logs)
Environment Variables for Workload Agents
The most commonly used variables include:
SYSDIG_ORCHESTRATOR
: (required) use the output OrchestratorHost
valueSYSDIG_ORCHESTRATOR_PORT
: (required) use the output from OrchestratorPort
SYSDIG_EXTRA_ARGS
: use to pass any extra arguments needed to the instrumentationSYSDIG_EXTRA_CONF
: use to add any extra configuration needed to the instrumentationSYSDIG_LOGGING
: add this variable and set to debug
if needed. Any other value will print minimal logs at startup. See also: Manage Serverless Agent Logs
C. Manual Instrumentation for Workload Agents
In some cases, you may prefer not to use the Kilt or Terraform installers and instead to either:
- Instrument a Task Definition, or
- Instrument a Container Image (rare)
C1: Instrument a Task Definition
Review the installation overview and prerequisites.
Install the Orchestrator Agent, as described above. Take note of the
OrchestratorHost
and OrchestratorPort
values.
Instrument the CloudFormation task definition to deploy the workload agent manually.
Add a new container to your existing CloudFormation task definition.
- For the example, we will use
SysdigInstrumentation
for the image name. - The entrypoint/command fields can be left empty
- Obtain the image from
quay.io/sysdig/workload-agent:latest
Edit the containers you want to instrument.
Mount volumes from SysdigInstrumentation
Add SYS_PTRACE
capability to your container. See AWS Documentation for detail if needed.
Set /opt/draios/bin/instrument
as the entry point of your container.
Note: if you are using the entry point in your image, then prepend it: /opt/draios/bin/instrument,your,original,entry,point
Configure the Sysdig instrumentation through environment variables.
Example Task Definitions
TestApp:
Type: AWS::ECS::TaskDefinition
Properties:
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 2048
Memory: 8GB
ExecutionRoleArn: !Ref TestAppExecutionRole
TaskRoleArn: !Ref TestAppTaskRole
ContainerDefinitions:
- Name: App
Image: !Ref TestAppImage
EntryPoint:
- /bin/ping
- google.com
Tags:
- Key: application
Value: TestApp
Task After Manual Instrumentation
TestApp:
Type: AWS::ECS::TaskDefinition
Properties:
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 2048
Memory: 8GB
ExecutionRoleArn: !Ref TestAppExecutionRole
TaskRoleArn: !Ref TestAppTaskRole
ContainerDefinitions:
- Name: App
Image: !Ref TestAppImage
EntryPoint:
### Added for manual instrumentation
- /opt/draios/bin/instrument
### End added propertis
- /bin/ping
- google.com
### Added for manual instrumentation
LinuxParameters:
Capabilities:
Add:
- SYS_PTRACE
VolumesFrom:
- SourceContainer: SysdigInstrumentation
ReadOnly: true
Environment:
- Name: SYSDIG_ORCHESTRATOR # Required
Value: <host orchestrator output, region dependent>
- Name: SYSDIG_ORCHESTRATOR_PORT # Required
Value: "6667"
- Name: SYSDIG_ACCESS_KEY
Value: ""
- Name: SYSDIG_COLLECTOR
Value: ""
- Name: SYSDIG_COLLECTOR_PORT
Value: ""
- Name: SYSDIG_LOGGING
Value: ""
- Name: SysdigInstrumentation
Image: !Ref WorkloadAgentImage
### End of added properties
Tags:
- Key: application
Value: TestApp
C2: Instrument in a Container Image
Alternatively, you can include the workload agent in your container at build time. To do so, update your Dockerfile to copy the required files:
ARG sysdig_agent_version=latest
FROM quay.io/sysdig/workload-agent:$sysdig_agent_version AS workload-agent
FROM my-original-base
COPY --from=workload-agent /opt/draios /opt/draios
Prepend the /opt/draios/bin/instrument
command to the entrypoint of your container:
ENTRYPOINT ["/opt/draios/bin/instrument", "my", "original", "entry", "point"]
Note about Logs
To separate instrumentation logs from your workload logs, the instrument
entrypoint will attempt to send logs to localhost:32000
, where they will be mixed with the workload logs.
- If you want to separate instrumentation logs in a manual install, we advise using Option A.
- To avoid log spam, you can set
SYSDIG_LOGGING
to silent
.
See also: Manage Serverless Agent Logs
Upgrade (Up to v2.3.0)
Note: In most cases, it is advised to upgrade directly to v3.0.0+, using the Option 1 upgrade path. These instructions are kept for special cases.
To upgrade the serverless agents, you install a second version of both
components then kill all the running tasks and restart with the new
version. You then delete and clean up the old CloudFormation and Kilt
residuals.
Obtain the Sysdig Orchestrator Agent
yaml
to be used as the CloudFormation Template source.
At this time, the yaml metadata does not specify the agent version,
but this link always downloads the latest file.
Perform the steps to Install the Orchestrator
Agent.
Note that in step 2.4, the OrchestratorHost
and OrchestratorPort
values will be unique.
Perform the steps to Install the Workload
Agents (if using Kilt. Workload agents installed with Terraform are automatically updated to _latest
.)Manual instrumentation requires redoing the manual process.)
In step 4, assign a unique name to your macro
(Transform: MyV2SysdigMacro
).
You now have two versions of the serverless agent components. When
you are ready to switch from the earlier version, proceed with the
next step.
Stop all running tasks and use CloudFormataion to delete the
earlier stack. Redeploy the new stack with the updated CFT.
(Transform: MyV2SysdigMacro
)
Clean up macros: Delete the previous kilt macro using the
installer:
./installer-linux-amd64 cfn-macro delete MySysdigMacro
2.1.1 -
Manage Serverless Agent Logs
As of serverless agent version 2.2.0, task logs and instrumentation logs are handled separately.
Task logs continue to be with whatever log setup you have on your task container.
Instrumentation logs go to a separate log group created by the macro installer:
<macro_name>-SysdigServerlessLogGroup-<uid>
.
At this time, the log group name cannot be edited. By default, the logging level is set to info
.
Logging Environment Variables
Set Logging Level
SYSDIG_LOGGING
is used for instrumentation log level. Default = info
.
The full list of options are: none | critical| error | warning |notice | info | debug | trace
Log Forwarding
SYSDIG_ENABLE_LOG_FORWARD
Default = true
SYSDIG_LOG_LISTEN_PORT
Default = 32000
SYSDIG_LOG_FORWARD_ADDR
Default = localhost:32000
. Where the logwriter of the workload agent listens. Can be set to any other TCP endpoint.
Advanced Configurations
SYSDIG_BUFFER_SIZE
: Receiving buffer size in bytes for the TCP listener, which can be increased. Default 1024
.SYSDIG_DEADLINE_SECONDS
: Connection deadline, which can be increased if clients are taking longer to connect. Default 3
.
See other serverless agent environment variables here.
3 -
Prometheus Remote Write
You can collect Prometheus metrics from environments where the Sysdig
agent is not available. Sysdig uses the remote_write
capabilities to
help you do so.
In Sysdig terminology, the remote endpoints that can read Prometheus
metrics are known as Prometheus Remote Write. Prometheus Remote Write
does not require the Sysdig agent to be installed in the Prometheus
environment. This facility expands your monitoring capabilities beyond
Kubernetes and regular Linux kernels to environments where the Sysdig
agent cannot be installed.

Prometheus Remote Write can collect metrics from:
Use Sysdig agent in environments where an agent can be installed.
However, use the Prometheus Remote Write to collect metrics from
ephemeral or batch jobs that may not exist long enough to be scraped by
the agent.
With the Prometheus Remote Write, you can either monitor metrics through
the Monitor UI or you can use PromQL to query the data by using the
standard Prometheus query language.
Endpoints and Regions
Prometheus Remote Write resides in the ingest endpoints for each region
under /prometheus/remote/write
. The public Prometheus Remote Write
endpoints for each region are listed below:
You need to configure remote_write
in your Prometheus server in order
to send metrics to Sysdig Prometheus Remote Write.
The configuration of your Prometheus server depends on your
installation. In general, you configure the remote_write
section in
the prometheus.yml
configuration file:
global:
external_labels:
[ <labelname>: <labelvalue> ... ]
remote_write:
- url: "https://<region-url>/prometheus/remote/write"
bearer_token: "<your API Token>"
The communication between your Prometheus server and Prometheus Remote
Write should use the authorization header with the Sysdig API key (not
the agent access key) as the bearer token.
Alternatively, you can also use the bearer_token_file
entry to refer
to a file instead of directly including the API token.
Prometheus does not reveal the bearer_token
value on the UI.
Customize Metrics
To enable customization, Sysdig provides additional options to control
which metrics you want to send to Prometheus Remote Write.
Manage Metrics
Prometheus Remote Write by default sends all the metrics to Sysdig
Prometheus Remote Write. These metrics are sent with a
remote_write: true
label appended to it so you can easily identify
them.
Label Metrics
You can specify custom label-value pairs and send them with each time
series to the Prometheus Remote Write. Use the external_labels
block
in the global
section in the Prometheus configuration file. This is
similar to setting an agent tag and allowing you to filter or scope the
metrics in Sysdig Monitor.
For example, if you have two Prometheus servers configured to remote
write to Prometheus Remote Write, you can include an external label to
identify them easily:
Prometheus 1
global:
external_labels:
provider: prometheus1
remote_write:
- url: ...
Prometheus 2
global:
external_labels:
provider: prometheus2
remote_write:
- url: ...
Filter Metrics
With the general configuration, all the metrics are by default remotely
written to Prometheus Remote Write. You can control the metrics that you
collect and send to Sysdig. To select which series and labels to
collect, drop, or replace, and reduce the number of active series that
are sent to Sysdig, you can set up relabel configurations by using
the write_relabel_configs
block within your remote_write
section.
For example, you can send metrics from one specific namespace called
myapp-ns
as give below:
remote_write:
- url: https://<region-url>/prometheus/remote/write
bearer_token_file: /etc/secrets/sysdig-api-token
write_relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
regex: 'myapp-ns'
action: keep
Rate Limit
The default limits are configured set for each user and can be raised as
required. The defaults are good for most users, and the limits help
protect against any misconfigurations.
Parallel writes | 100 concurrent requests. This doesn't necessarily mean 100 Prometheus servers because the time at which the data is written is distributed. |
Data points per minute | One million. The number of data points sent depends on how often metrics are submitted to Sysdig. A scrape interval of 10s will submit more DPM than an interval of 60s. |
Number of writes per minute | 10,000 |
Team Scoping
It is possible to scope a Sysdig Team to only access metrics matching certain labels sent via Prometheus remote write. See Manage Teams and Roles
Limitations
Metrics sent to Prometheus Remote Write can be accessed in
Explore, but they are not compatible with the scope tree.
Label enrichment is unavailable at this point. Only labels collected
at the source can be used. You should add additional labels to
perform further scoping or pivoting in Sysdig.
Currently, Sysdig Dashboards do not support mixing metrics with
different sampling. For example, 10 seconds and 1-minute samples.
For optimal experience, configure the scrape interval to be 10s to
combine remote write metrics with agent metrics.
Remote write functionality does not support sending metric metadata.
Upstream Prometheus recently added support for propagation of
metadata (metric type, unit, description, info) and this
functionality will be supported in a future update to Prometheus
Remote Write.
Suffix the metric name with _total
, _sum
, or _count
to
store them as a counter. Otherwise, the metrics will be handled
as a gauge.
Units can be set in Dashboards manually.
Learn More
4 -
Sysdig Secure for cloud
Sysdig Secure for cloud is the software that connects Sysdig Secure
features to your cloud environments to provide unified threat detection,
compliance, forensics, and analysis.
Because modern cloud applications are no longer just virtualized compute
resources, but a superset of cloud services on which businesses depend,
controlling the security of your cloud accounts is essential. Errors can
expose an organization to risks that could bring resources down,
infiltrate workloads, exfiltrate secrets, create unseen assets, or
otherwise compromise the business or reputation. As the number of cloud
services and configurations available grows exponentially, using a cloud
security platform protects against having an unseen misconfiguration
turn into a serious security issue.
Check Sysdig Secure - Secure for cloud to review the features provided per cloud.
Multiple Installation Options
Sysdig Secure for cloud is available on a variety of cloud providers, with simplified installation instructions available from the Get Started screen.
Supported cloud providers at this time are:
Cloud-Native templates
Native template deployment for
Helm Chart based (feature limited)
Core component for Sysdig Secure for Cloud, is called cloud-connector, which is also available through following methods:
Note: Installing this component will only allow you threat detection and image scaning features, although together with SysdigCompliance IAM role, it can handle Compliance and Identity and Access Posture too.
See Also
4.1 -
Deploy Sysdig Secure for cloud on AWS
If needed, review the offering description on Sysdig Secure for cloud - AWS
Deployment Options
All following options provide all four cloud features: threat detection, CSPM benchmarks, and image and
container registry scanning
- Terraform-based for two types of AWS
account
- Organizational/management account: This is the account that you use to create the organization in AWS.
Organizational accounts create and contain member accounts.
- Single/member account: Each of these is a stand-alone account. A member account that is part of an organization is supported too.
- CloudFormation Template (CFT)-based: This option requires explicit creation of an AWS role, which is prompted by the onboarding wizard.
Terraform-based install instructions differ depending on what type of
AWS account you are using.
At this time, the options include:
- Install for a single AWS account
- Install for an organizational/management account
For Single/Member Account
The default code provided in the Get Started page of Sysdig Secure is
pre-populated with your Secure API token and will automatically install
threat detection with CloudTrail, AWS benchmarks, and container registry
and image scanning.
Prerequisites and Permissions
- A Sysdig Secure SaaS account, with administrator permissions
- An AWS account, for Secure for Cloud compute workload deployment
- You must have Administrator permissions, or permissions to create each of the resources specified in the resources list.
- Enable AWS STS in each region you would like to secure.
- Have Terraform installed on the
machine from which you will deploy the installation code.
Steps
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
AWS
tab.

Copy the code snippet under Single Account and paste it in the
terminal of your local machine. It should be pre-configured with
your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them,
then make the changes.
Confirm the Services are
Working
Check
Troubleshooting
in case of permissions or account conflict errors.
For Organizational/Management Account
For organizational accounts, the default code provided in the Get
Started page of Sysdig Secure is pre-populated with your Secure API
token and will automatically install threat detection with CloudTrail
(only).
Prerequisites and Permissions
- A Sysdig Secure SaaS account, with administrator permissions
- An AWS account on your organization, for Secure for Cloud compute workload deployment (we recommend creating an isolated member account)
- You must have Administrator permissions, or permissions to create each of the resources specified in the resources list.
Sysdig provides an IAM policy containing the required permissions.
- Enable AWS STS in each region you would like to secure.
- An existing AWS account as the organization master account with the Organizational CloudTrail service enabled.
- AWS profile credentials configuration of the master account of the organization; You must also have sufficient permissions for the IAM user or role in the management account to successfully create an organization trail.
- Have Terraform installed on the machine from which you will deploy the installation code.
Steps
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
AWS
tab.

Copy the code snippet under Organizational Account and paste it
in the terminal of your local machine. It should be pre-configured
with your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them,
then make the changes.
Confirm the Services are
Working
Check
Troubleshooting
in case of permissions or account conflict errors.
Soon, this option will be expanded to include all the features currently
in the single account option, as well as the ability to easily add
multiple member accounts.
Customizing the Install
Both the Single Account and Organizational Account code examples are
configured with sensible defaults for the underlying inputs. But if
desired, you can edit the region, module enablement, and other Inputs.
See details for:
Enabling Image Scanner
Image Scanner feature is disabled by default. If you want to enable it, just use the deploy_scanning
input variable on your snippet such as:
module "secure-for-cloud_example"{
...
deploy_image_scanning_ecs = true
deploy_image_scanning_ecr = true
}
Resources Created by Each Module
Check full list of created resources
If cloud-connector or cloud-scanning is installed, these additional
modules will be installed:
resource-group
cloudtrail
ssm
ecs-fargate-cluster
If cloud-scanning is installed, these additional modules will be
installed:
codebuild
aws_cloudwatch_log_group
aws_codebuild_project
aws_iam_role
aws_iam_role_policy
Troubleshooting
Find more troubleshooting options on the module source repository
1. Resolve 409 Conflict Error
This error may occur if the specified cloud account has already been
onboarded to Sysdig.

Solution:
The cloud account can be imported into Terraform by running:
terraform import module.cloud_bench.sysdig_secure_cloud_account.cloud_account CLOUD_ACCOUNT_ID
2. Resolve Permissions Error/Access Denied
This error may occur if your current AWS authentication session does not
have the required permissions to create certain resources.

Solution:
Ensure you are authenticated to AWS using a user or role with the
required permissions.Onboarding a Single Account using a CFT
Onboarding a Single Account using a CFT
Each of the features can be enabled from a single CloudFormation
Template (CFT) from the AWS Console.
Two options are available:
Secure For Cloud stack, deployed on ECS compute workload.
Available in all regions
Secure For Cloud stack, deployed on AppRunner compute workload.
A less resource-demanding deployment but not available in all regions; accepting ‘us-east-1’, ‘us-east-2’, ‘us-west-2’, ‘ap-northeast-1’ and ’eu-west-1’
Prerequisites
A Sysdig Secure SaaS account
An AWS account and AWS services you would like to connect to Sysdig,
with appropriate permissions to deploy a CFT.
Steps
Log in to your AWS Console and confirm that you are in the account
and AWS region that you want to secure using Sysdig Secure for
cloud.
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
AWS
tab.

Select between Install Secure For Cloud stack, deployed on ECS compute workload
or
Install Secure For Cloud stack, deployed on AppRunner compute workload
link.
The Connect Account dialog is displayed.

Enter:
The AWS account number with which you want to connect
An IAM Role name to be created for Sysdig Secure for cloud in AWS. This role name must not yet exist in your account.
The role provides read-only access to your resources to allow Sysdig to monitor and secure your cloud account. Access is scoped to the managed SecurityAudit policy.
Click Launch Stack
.
The AWS Console opens, at the
CloudFormation > Stacks > Quick Create
page. The Sysdig
CloudFormation template is pre-loaded.
Confirm that you are logged in the AWS account and region where you
want to deploy the Sysdig Template.

Provide a Stack name
or accept the default.
Fill in the Parameters:

Sysdig Settings
Sysdig Secure Endpoint
: Default (US-East): https://secure.sysdig.com
.
If your Sysdig Secure platform is installed in another region, use that endpoint.
- US West:
https://us2.app.sysdig.com
- European Union:
https://eu1.app.sysdig.com
Sysdig Secure API Token
: See Retrieve the Sysdig API Token to find yours.
Sysdig Role Name
: As specified in Step 3; IAM role name to be created for Sysdig to access your AWS account
Sysdig External ID
: Not to be modified. It’s the ExternalID to identify Sysdig on AWS, for the Trusted Identity
Sysdig Trusted Identity
: Not to be modified. It’s the ARN of the Trusted Identity of Sysdig on AWS, to be able to run CSPM benchmarks.
Modules to Deploy: Choose any or all.
CSPM/Compliance
and Threat detection using CloudTrail
capabilities will be always deployed.
ECR Image Registry Scanning:
Integrates container registry
scanning for AWS ECR.
Fargate Image Scanning:
Integrates image scanning on any
container image deployed on a serverless Fargate task (in ECS).
Existing Infrastructure: Leave all this fields blank for resources to be created.
If you want to use existing components of your infrastructure, you can provide:
Network: Only available if stack is deployed on ECS. If provided, MUST specify ALL field values:
- ECS Cluster Name where the Sysdig workload is to be deployed
- VPC ID for the ECS Cluster
- Private subnet ID(s) for the VPC. At least two subnets are required
Cloudstrail SNS Topic: Specify the URL of the SNS Topic
Confirm the Capabilities required to deploy:

Click Create Stack.
In the AWS Console, the main stack and associated substacks will
show “CREATE_IN_PROGRESS”. Refresh the status to see
“CREATE_COMPLETE” for all. There is a delay of 5-10 minutes for
events to be sent from CloudTrail, but no event is lost.

A success message also appears in the Sysdig Secure Get Started
page.
Confirm the Services are Working
Log in to Sysdig Secure and check that each module you deployed is
functioning. It may take 10 minutes or so for events to be collected and
displayed.
Check Overall Connection Status
Data Sources:
Select Select Integrations > Inbound | Cloud Accounts
to see all connected
cloud accounts.
Subscription: Select Settings > Subscription
to see an
overview of your account activity, including cloud accounts.

Insights: Check that
Insights have been
added to your navigation bar. View activity on the Cloud Account,
Cloud User, or Composite insight views.

Check Threat Detection
Policies and Rules: Check Policies > Runtime Policies
and confirm that
the AWS Best Practices
policy is enabled. This consists of the
most-frequently-recommended rules for AWS and CloudTrail. You can
customize it by creating a new policy of the AWS
CloudTrail
type.

Events: In the Events
feed, search ‘cloud’ to show events from
AWS CloudTrail.

Force an event: In case you want to manually create an event, choose one of the rules contained in the AWS Best Practices
policy and
execute it in your AWS account.
ex.: Create a S3 Bucket with Public Access Blocked. Make it public to prompt the event.
Remember that in case you add new rules to the policy you need to give it time to propagate the changes.
Check CSPM/AWS Benchmarks
Compliance: Select Compliance
and see that
AWS Foundations Benchmark
is installed.
Review the benchmark results and confirm the account, region and
date added.

Check Scanning for ECR and Fargate
Scan Results: CheckImage Scanning > Scan Results
and choose
the Origins
drop-down.
Confirm that AWS Registry
and/or AWS Fargate
are listed.

Filter by the desired origin and review scan results.
Force an event: Upload an image to a repository on your Elastic Container Registry (ECR). Follow View push commands
button guide provided by AWS.
Check Permissions and Entitlements (AWS)
Select Posture > Permissions and Entitlements
and check if the following are showing up in the Unused Permissions graphs:
- Unused permissions
- Inactive users
- Policies with the greatest number of granted vs used permissions
Follow the instructions to remediate overly permissive entitlements and reduce security risks.
) to remediate overly permissive entitlements and reduce security risks.
See Also
4.2 -
Deploy Sysdig Secure for cloud on GCP
If needed, review the offering description on Sysdig Secure for cloud - GCP.
Deployments on GCP use a Terraform file.
Terraform-based install instructions differ depending on what type of account you are using.
At this time, the options include:
The default code provided in the Get Started page of Sysdig Secure is
pre-populated with your Secure API token and will automatically install
threat detection, benchmarks, and container registry and image scanning.
Prerequisites and Permissions
- A Sysdig Secure SaaS account, with administrator permissions
- A Google Cloud Platform (GCP) account, for Secure for Cloud compute workload deployment
- Owner role, in order to create each of the resources specified in the resources list below
- For organizational Organization Admin role is required too.
- Enable the required GCP APIs
- Have Terraform installed on the
machine from which you will deploy the installation code.
Steps
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
GCP
tab.

Copy the code snippet under Single Account or Organizational Account and paste it in the
terminal of your local machine. It should be pre-configured with
your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them,
then make the changes.
Confirm the Services are
Working
Check
Troubleshooting
in case of permissions or account conflict errors.
Customizing the Install
Both the Single Account and Organizational Account code examples are
configured with sensible defaults for the underlying inputs. But if
desired, you can edit the region, module enablement, and other Inputs.
See details for:
Enabling Image Scanner
Image Scanner feature is disabled by default. If you want to enable it, just use the deploy_scanning
input variable on your snippet such as:
module "secure-for-cloud_example"{
...
deploy_scanning = true
}
Resources Created by Each Module
Check full list of created resources
- Cloud-bench
google_iam_workload_identity_pool
google_iam_workload_identity_pool_provider
google_project_iam_custom_role
google_project_iam_member
google_service_account
google_service_account_iam_binding
sysdig_secure_benchmark_task
sysdig_secure_cloud_account
- Cloud-connector
google_cloud_run_service
google_eventarc_trigger
google_project_iam_member
google_storage_bucket
google_storage_bucket_iam_member
google_storage_bucket_object
- Cloud-scanning
google_cloud_run_service
google_cloud_run_service_iam_member
google_eventarc_trigger
google_project_iam_member
google_pubsub_topic
If Cloud-connector is installed in organizational mode, this additional module will be installed:
- Organization-sink
google_logging_organization_sink
google_pubsub_topic
google_pubsub_topic_iam_member
If Cloud-connector is installed in single-project mode, this additional module will be installed:
- Project-sink
google_logging_project_sink
google_pubsub_topic
google_pubsub_topic_iam_member
If Cloud-scanning is installed, this additional module will be installed:
- Secrets
google_secret_manager_secret
google_secret_manager_secret_iam_member
- google_secret_manager_secret_version`
Troubleshooting
Find more troubleshooting options on the module source repository
1. Insufficient Permissions on Project
This error may occur if your current GCP authentication session does not have the required permissions to access the specified project.
Solution:
Ensure you are authenticated to GCP using a user or service account with the required permissions.

2. Insufficient Permissions to Create Resource
This error may occur if your current GCP authentication session does not have the required permissions to create certain resources.
Solution:
Ensure you are authenticated to GCP using a user or service account with the required permissions.

If you have sufficient permissions but still get this kind of error, try to authenticate gcloud
using:
$ gcloud auth application-default login
$ gcloud auth application-default login set-quota-project your_project_id
3. Conflicting Resources
This error may occur if the specified GCP project has already been onboarded to Sysdig.
Solution:
The cloud account can be imported into terraform by running
terraform import module.single account.module.cloud_bench.sysdig_secure_cloud_account.cloud_account PROJECT_ID
, where PROJECT_ID
is the numerical ID of the project (not the project name).

4. Workload Identity Federation pool already exists
This error may occur if a Workload Identity Federation Pool or Pool Provider has previously been created, and then deleted, either via the GCP console or with terraform destroy
. When a delete request for these resources is sent to GCP, they are not completely deleted, but marked as “deleted”, and remain for 30 days. These “deleted” resources will block creation of a new resource with the same name.
Solution:
The “deleted” pools must be restored using the GCP console, and then imported into terraform, using terraform import module.single-account.module.cloud_bench.google_iam_workload_identity_pool.pool POOL_ID
and module.single-account.module.cloud_bench.google_iam_workload_identity_pool_provider.pool_provider POOL_ID/PROVIDER_ID

Email contains error codes such as:
Error Code: topic_not_found
or
Error Code: topic_permission_denied
Cause: The resources Sysdig deployed with Terraform will eventually be consistent, but it could happen that some pre-required resources are created but not ready yet.
Solution: This is a known issue that will only take place within first minutes of the deployment. Eventually, resource health checks will pass and modules will work as expected.

Confirm the Services are Working
Log in to Sysdig Secure and check that each module you deployed is
functioning. It may take 10 minutes or so for events to be collected and
displayed.
Check Overall Connection Status
Data Sources:
Select Integrations > Inbound | Cloud Accounts
to see all connected
cloud accounts.
Subscription: Select Settings > Subscription
to see an
overview of your account activity, including cloud accounts.

Insights: Check that
Insights have been
added to your navigation bar. View activity on the Cloud Account,
Cloud User, or Composite insight views.

Check Threat Detection
Policies and Rules: Check Policies > Runtime Policies
and confirm that
the GCP Best Practices
policy is enabled. This consists of the
most-frequently-recommended rules for GCP.


Events: In the Events
feed, search ‘cloud’ to show events from
GCP.

Check GCP Benchmarks
- Tasks: Select
Compliance > Benchmarks > Tasks
and confirm a task with the name Sysdig Secure for Cloud (GCP)
exists. - Results: After a few minutes, check results of the benchmark are available by clicking on the
Sysdig Secure for Cloud (GCP)
task. Note that results may take up to 15 minutes to appear.
Check GCP Scanning
Scan Results: CheckImage Scanning > Scan Results
and choose
the Origins
drop-down.
Confirm that GCP
is listed.
Filter by the desired origin and review scan results.
Force an event: Upload an image to a new Repository in a Container Registry. Follow repository Setup Instructions
provided by GCP.
See Also
4.3 -
Deploy Sysdig Secure for cloud on Azure
If needed, review the offering description on Sysdig Secure for cloud - Azure.
Deployments on Azure use a Terraform file.
Terraform-based install instructions differ depending on what type of account you are using.
At this time, the options include:
- Install for a single subscription
- Install for tenant subscriptions
The default code provided in the Get Started page of Sysdig Secure is
pre-populated with your Secure API token and will automatically install
threat detection, benchmarks, and container registry and image scanning.
Prerequisites and Permissions
- A Sysdig Secure SaaS account, with administrator permissions
- An Azure subscription/tenant you would like to connect to Sysdig
- The installing user must have (Organizational level) Security Administrator and (Subscription level) Owner role
- More permissions detail
- Terraform installed on the
machine from which you will deploy the installation code.
Steps
Log in to Sysdig Secure as Admin
and select
Get Started > Connect your Cloud account
. Choose the
Azure
tab.

Copy the code snippet under Single Subscription or Tenant Subscriptions and paste it in the
terminal of your local machine. It should be pre-configured with
your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them,
then make the changes.
Confirm the Services are
Working
Check
Troubleshooting
in case of permissions or account conflict errors.
Customizing the Install
Both the Single Account and Organizational Account code examples are
configured with sensible defaults for the underlying inputs. But if
desired, you can edit the region, module enablement, and other Inputs.
See details for:
Enabling Image Scanner
Image Scanner feature is disabled by default. If you want to enable it, just use the deploy_scanning
input variable on your snippet such as:
module "secure-for-cloud_example"{
...
deploy_scanning = true
}
Resources Created by Each Module
Check full list of created resources
- Cloud-bench
azurerm_lighthouse_assignment
azurerm_lighthouse_definition
azurerm_role_definition
azurerm_subscription
sysdig_secure_cloud_account
sysdig_secure_benchmark_task
- Cloud-connector
azurerm_container_group
azurerm_network_profile
azurerm_storage_account
azurerm_storage_blob
azurerm_storage_container
azurerm_subnet
azurerm_virtual_network
If Cloud-connector is installed, these additional modules will also be installed:
- Container-registry
azurerm_container_registry
azurerm_eventgrid_event_subscription
- Enterprise-application
azuread_application
azuread_application_password
azuread_service_principal
azuread_service_principal_password
azurerm_role_assignment
azurerm_role_definition
- Eventhub
azurerm_eventhub
azurerm_eventhub_authorization_rule
azurerm_eventhub_namespace
azurerm_eventhub_namespace_authorization_rule
azurerm_monitor_diagnostic_setting
azurerm_resource_group
Troubleshooting
Find more troubleshooting options on the module source repository
1. Insufficient Permissions on Subscription
This error may occur if your current Azure authentication session does not have the required permissions to create resources in the specified subscription.
Solution:
Ensure you are authenticated to Azure using a Non-Guest user with the Contributor or Owner role on the target subscription.
Error: Error Creating/Updating Lighthouse Definition "dd9be15b-0ee9-7daf-b942-5e173dae13fb" (Scope "/subscriptions/***"): managedservices.RegistrationDefinitionsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="InsufficientPrivilegesForManagedServiceResource" Message="The requested user doesn't have sufficient privileges to perform the operation."
with module.cloudvision_example_existing_resource_group.module.cloud_bench.module.trust_relationship["***"].azurerm_lighthouse_definition.lighthouse_definition,
on ../../../modules/services/cloud-bench/trust_relationship/main.tf line 28, in resource "azurerm_lighthouse_definition" "lighthouse_definition":
28: resource "azurerm_lighthouse_definition" "lighthouse_definition" {
2. Conflicting Resources
This error may occur if the specified Azure Subscription has already been onboarded to Sysdig
Solution:
The resource can be imported into Terraform by using the terraform import
command. This will bring the resource under management in the current Terraform workspace.
Error: A resource with the ID "/subscriptions/***/resourceGroups/sfc-resourcegroup" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_group" for more information.
with module.cloudvision_example_existing_resource_group.module.infrastructure_eventhub.azurerm_resource_group.rg[0],
on ../../../modules/infrastructure/eventhub/main.tf line 6, in resource "azurerm_resource_group" "rg":
6: resource "azurerm_resource_group" "rg" {
Confirm the Services are Working
Log in to Sysdig Secure and check that each module you deployed is
functioning. It may take 10 minutes or so for events to be collected and
displayed.
Check Overall Connection Status
Data Sources:
Select Integrations > Inbound | Cloud Accounts
to see all connected
cloud accounts.
Subscription: Select Settings > Subscription
to see an
overview of your account activity, including cloud accounts.

Insights: Check that
Insights have been
added to your navigation bar. View activity on the Cloud Account,
Cloud User, or Composite insight views.
Check Threat Detection
Policies: Check Policies > Runtime Policies
and confirm that
the Azure Best Practices
policy is enabled. This consists of the
most-frequently-recommended rules for Azure DevOps.

Events: In the Events
feed, search ‘cloud’ to show events from
Azure.
Check Benchmarks
- Tasks: Select
Compliance > Benchmarks > Tasks
and confirm a task with the name Sysdig Secure for Cloud (Azure)
exists. - Results: After a few minutes, check results of the benchmark are available by clicking on the
Sysdig Secure for Cloud (Azure)
task. Note that results may take up to 15 minutes to appear.
Check Scanning
Scan Results: CheckImage Scanning > Scan Results
and choose
the Origins
drop-down.
Confirm that Azure
is listed.
Filter by the desired origin and review scan results.
See Also
5 -
Rapid Response: Installation
With Rapid Response, Sysdig has introduced a way to grant designated
Advanced Users in Sysdig Secure the ability to remote connect into a
host directly from the Event stream and execute desired commands there.
Rapid Response team members have access to a full shell from within the
Sysdig Secure UI. Responsibility for the security of this powerful
feature rests with you: your enterprise and your designated employees.
See also: Rapid Response.
Prerequisites
Sysdig Secure On-Premises 4.0+ or Sysdig Secure SaaS
Have on hand:
Your Sysdig agent access
key
Your Sysdig API endpoint
On-prem: custom, depending on your on-prem
installation
SaaS: Region-dependent (use the “endpoint” entries, e.g. https://us2.app.sysdig.com
for US West)
A passphrase used to encrypt all traffic between the user and
host.
NOTE: Sysdig cannot recover this passphrase. If lost, a user
will not be able to start a session, nor will any session logs
be recoverable.
Optionally, these can be added to the environment variables:
export API_ENDPOINT=https://secure-staging.mycompany.com
export ACCESS_KEY=$YOUR_SYSDIG_AGENT_INSTALLATION_KEY
export PASSPHRASE=$ENCRYPTION_PASSPHRASE
export API_TLS_SKIP_CHECK=false
Install Host Component
The Rapid Response agent can be installed as a Docker container or as a
Kubernetes DaemonSet.
As Docker Container
Mount the host directories and binaries to gain access to the host.
docker run --hostname $HOST_NAME -d quay.io/sysdig/rapid-response-host-component:latest --endpoint $API_ENDPOINT --access-key $ACCESS_KEY --password $PASSPHRASE
Customize the Docker image.
The container is simply bash shell. To add custom scripts without
needing to mount the underlying host filesystem, you can bake this
into the Docker container, e.g. by installing kubectl, gcloud,
netstat, or another command-line utility.
FROM quay.io/sysdig/rapid-response-host-component:latest AS base-image
FROM alpine:3.13
COPY --from=base-image /usr/bin/host /usr/bin/host
# add custom scripts and other directives
ENTRYPOINT ["host"]
As Kubernetes DaemonSet
Create a namespace and secrets for the Rapid Response agent:
kubectl create ns rapid-response
kubectl create secret generic sysdig-rapid-response-host-component-access-key --from-literal=access-key=$ACCESS_KEY -n rapid-response
kubectl create secret generic sysdig-rapid-response-host-component-passphrase --from-literal=passphrase=$PASSPHRASE -n rapid-response
Create the configmap and change the API-ENDPOINT parameter:
echo "apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-rapid-response-host-component
data:
api-endpoint: ${API_ENDPOINT}
api-tls-skip-check: 'false'" | kubectl apply -n rapid-response -f -
Deploy the DaemonSet.
Note: The agent does not automatically have access to the host
filesystem; there are several mounts commented-out in the manifest
that must be uncommented to investigate the host.
echo "# apiVersion: extensions/v1beta1 # If you are in Kubernetes version 1.8 or less please use this line instead of the following one
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: sysdig-rapid-response-host-component
labels:
app: sysdig-rapid-response-host-component
spec:
selector:
matchLabels:
app: sysdig-rapid-response-host-component
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: sysdig-rapid-response-host-component
spec:
hostNetwork: true
volumes:
# Add custom volume here
# Uncomment these lines if you'd like to map /root/ from the
# host into the container.
#- hostPath:
# path: /
# name: host-root-vol
- name: sysdig-rapid-response-host-component-config
configMap:
name: sysdig-rapid-response-host-component
optional: true
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
containers:
- name: sysdig-rapid-response-host-component
image: quay.io/sysdig/rapid-response-host-component
#securityContext:
# The privileged flag is necessary for OCP 4.x and other Kubernetes setups that deny host filesystem access to
# running containers by default regardless of volume mounts. In those cases, access to the CRI socket would fail.
# privileged: true
imagePullPolicy: Always
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 250m
memory: 250Mi
# Add custom volume mount here
# Uncomment these lines if you'd like to map /root/ from the
# host into the container.
#volumeMounts:
#- mountPath: /host
# name: host-root-vol
env:
- name: API_ENDPOINT
valueFrom:
configMapKeyRef:
name: sysdig-rapid-response-host-component
key: api-endpoint
- name: API_TLS_SKIP_CHECK
valueFrom:
configMapKeyRef:
name: sysdig-rapid-response-host-component
key: api-tls-skip-check
- name: ACCESS_KEY
valueFrom:
secretKeyRef:
name: sysdig-rapid-response-host-component-access-key
key: access-key
- name: PASSWORD
valueFrom:
secretKeyRef:
name: sysdig-rapid-response-host-component-passphrase
key: passphrase" | kubectl apply -n rapid-response -f -
Complete the Configuration
After installation/upgrade, complete the following steps:
Request enablement of the feature from Sysdig Support.
Configure an S3 bucket for Rapid Response
logs: If you are
using the default Cassandra storage for Capture files, you will need
to configure an AWS or custom S3 bucket to store Rapid Response log
files after a session. If you have already configured an S3 bucket
for Captures, then Rapid Response logs will be routed there
automatically, into their own folder.
Manage the following port/firewall considerations:
Ensure the host component is able to reach the endpoint defined
in API_ENDPOINT
Ensure there are no intermediate proxies that could enforce
maximum time to live (since sessions could potentially have long
durations)
Ensure that the host component can reach the object storage (S3
bucket) when configured.
Configure and use Rapid Response in the Sysdig Secure UI: See
Rapid Response.
6 -
Node Analyzer: Multi-Feature Installation
What Is the Node Analyzer?
The Node Analyzer (NA) provides a method for deploying the
components for three different Sysdig Secure features:
(Node) Image
Analyzer:
an existing tool that can now be installed and/or upgraded in a new
way, alongside the other two components.
Benchmarks:
Installs a new component (called a benchmark runner) which is
required to use Benchmarks, including an updated interface and
new improved features. The legacy Benchmark tool can still be
accessed.
Host Scanning: a
new tool for scanning not just the images/containers on a host, but
the host itself.
Installation Options
All the Node Analyzer components, along with the Sysdig agent, are
deployed per node or host. You can deploy them using various methods:
Fresh Install: Agent + Node Analyzer
If you are installing Sysdig Secure for the first time and have not yet
deployed any agents, you can use a single-line install to deploy both
the Sysdig agent and the Node Analyzer (NA) tools. The script will make
changes to each node or host within a cluster.
curl -s
https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s
-- --access_key ACCESS_KEY --collector COLLECTOR_URL --collector_port 6443 --nodeanalyzer --api_endpoint API_ENDPOINT
For SaaS, see also the Get
Started
page in Sysdig Secure. Under “Connect Your Data Sources,” the script is
generated with your endpoints automatically inserted.
On-Premises with Self-Signed Cert:
If you want the Node Analyzer to report to an On-Prem Sysdig backend
that uses a self-signed certificate, then: Add -cc false
to the
command line so the node analyzer will accept it.
To find the values yourself:
access_key:
This is the agent access key. You can
retrieve
this from Settings > Agent Installation
in the Sysdig Secure UI.
collector_url:
This value is
region-dependent in
SaaS and is auto-completed on the Get Started page in the UI. (It is
a custom value in on-prem installations.)
api_endpoint:
This is the base URL (
region-dependent) for
Sysdig Secure and is auto-completed on the Get Started page. E.g.
secure.sysdig.com
, us2.app.sysdig.com
, eu1.app.sysdig.com
.
When finished, you can Access the Node Analyzer
Features.
Use this script in the following conditions:
Agent is already installed, you just want the NA tools
Node Image Analyzer already installed; you want to upgrade it to v2
You want to add Benchmarks v2 and Host Scanning features to your
existing Sysdig Secure environment, as well as upgrade or install
the Image Analyzer.
Note that if you already have the Node Image Analyzer (v1) installed,
this script will upgrade that component automatically. An agent MUST
already be installed. The script will make changes to every node in the
cluster.
curl -s https://download.sysdig.com/stable/install-node-analyzer | sudo bash -s -- --api_endpoint API_ENDPOINT
When finished, you can Access the Node Analyzer
Features.
Daemonset Install
To deploy the Node Analyzer using Kubernetes daemonsets,
download
the following configuration files, edit them as annotated within the
files, and deploy them.
To deploy the Node Analyzer concurrently with the Sysdig agent, you
would also download the sysdig-agent-clusterrole.yaml
,
sysdig-agent-daemonset-v2.yaml
, and sysdig-agent-configmap.yaml
and
deploy them as described in Agent Install:
Kubernetes.
You need to deploy these YAMLs after installing the Sysdig agent in the
same nodes, and also in the same namespace (sysdig-agent
by default).
When finished, you can Access the Node Analyzer
Features.
Install with Helm
Use the “Sysdig” Helm chart,
which installs the Sysdig agent and the Node Analyzer, with the
following commands:
helm repo add sysdig https://charts.sysdig.com
helm repo update
helm install sysdig-agent --set sysdig.accessKey=ACCESS_KEY --set sysdig.settings.collector=COLLECTOR_URL --set sysdig.settings.collector_port=6443 sysdig/sysdig --set nodeAnalyzer.apiEndpoint=API_ENDPOINT
To find the values:
access_key:
This is the agent access key. You can
retrieve
this from Settings > Agent Installation
in the Sysdig Secure UI.
collector_url:
This value is
region-dependent in
SaaS and is auto-completed on the Get Started page in the UI. (It is
a custom value in on-prem installations.)
api_endpoint:
This is the base URL (
region-dependent) for
Sysdig Secure and is auto-completed on the Get Started page. E.g.
secure.sysdig.com
, us2.app.sysdig.com
, eu1.app.sysdig.com
.
Access the Node Analyzer Features
Log in to Sysdig Secure and check that the features are working as
expected.
Select Scanning > Image Results
.

Check for scanned container image results that originate with the
Sysdig Node Image Analyzer.

Check vulnerabilities in hosts or nodes, both for operation system
packages (e.g. rpm
, dpkg
) and non-operating system packages (e.g.
Java packages, Ruby gems).
Select Scanning > Hosts
.
Review the Host vulnerabilities listed.

Your active team scope is applied when loading host scanning
results. Log in with the broadest team and user credentials to see
the full report.
Use Benchmarks (Legacy Feature)
Select Benchmarks |Tasks
.
Either configure a new
task or review your
upgraded tasks. Click
a line item to see the associated benchmark report.
Your active team scope is applied when loading benchmarks results.
Log in with the broadest team and user credentials to see the full
report.
Alternate Install Cases
The installation options above should be sufficient for the majority of
users; the options below allow for customizations and special cases.
Running Node Analyzer Behind a Proxy
Depending on your organization’s network design, you may require the
HTTP requests from Node Analyzer features to pass through a proxy in
order to reach the Sysdig Secure backend. To do so, you must edit all
three configmaps:
These are in the sysdig-agent
namespace by default.
Configure the following variables:
http_proxy/https_proxy
Use with the relevant proxy URL, e.g.
http://my_proxy_address:8080
.
In most cases, it is enough to specify http_proxy
. as it applies
to HTTPS connections as well.
no_proxy
Use this parameter to exclude certain subnets from using
the proxy, adding a comma-separated exclusion list, e.g.
127.0.0.1,localhost,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8
If the proxy server requires authentication it is possible to specify
credentials in the URL, e.g. http://username:password@my_proxy:8080
.
Running in a Non-Kubernetes Environment
This is handled per-component.
Benchmarks (Non-Kubernetes)
It is possible to deploy the benchmark runner as a single Docker
container:
docker run -d \
-v /:/host:ro \
-v /tmp:/host/tmp \
--privileged \
--network host \
--pid host \
-e BACKEND_ENDPOINT=https://<sysdig_backend_endpoint> \
-e ACCESS_KEY=<Sysdig agent access key> \
-e BACKEND_VERIFY_TLS=false \
-e TAGS=<custom_tags> \
quay.io/sysdig/compliance-benchmark-runner:latest
Note: If you don’t want to pass the access key directly via the
command line, consider using an alternative method of passing
environment variables, such as docker-compose.
The BACKEND_ENDPOINT
is only required if for Sysdig on-prem or
when using a Sysdig SaaS region other than US-EAST.
For example, for the EU SaaS endpoint would be:
https://eu1.app.sysdig.com
.
See also: SaaS Regions and IP
Ranges.
BACKEND_VERIFY_TLS=false
is only needed if you are using an
on-prem backend with a self-signed certificate.
TAGS:
The list of tags for the host where the agent is installed.
For example: “role:webserver, location:europe
”, “role:webserver
”
or “webserver
”.
Image Analyzer (Non-Kubernetes)
It is also possible to run the image analyzer as a single Docker
container:
docker run -d \
-v /var/run:/var/run \
--privileged \
--network host \
-e AM_COLLECTOR_ENDPOINT=https://<sysdig_backend_endpoint>/internal/scanning/scanning-analysis-collector \
-e ACCESS_KEY=<Sysdig agent access key> \
-e VERIFY_CERTIFICATE=false \
quay.io/sysdig/node-image-analyzer:latest
Note: If you don’t want to pass the access key directly via the
command line, consider using an alternative method of passing
environment variables, such as docker-compose.
The AM_COLLECTOR_ENDPOINT
is only required if for Sysdig on-prem
or when using a Sysdig SaaS region other than US-EAST.
For example, for the EU SaaS endpoint would be:
https://eu1.app.sysdig.com/internal/scanning/scanning-analysis-collector
.
See also: SaaS Regions and IP
Ranges.
VERIFY_CERTIFICATE=false
is only needed if you are using an
on-prem backend with a self-signed certificate.
Host Scanning (Non-Kubernetes)
To install the Host Scanning component in a non-Kubernetes environment,
you can use:
docker run -d \
-v /:/host:ro \
--privileged \
-e HOST_BASE=/host
-e AM_COLLECTOR_ENDPOINT=https://<sysdig_backend_endpoint>/internal/scanning/scanning-analysis-collector \
-e ACCESS_KEY=<Sysdig agent access key> \
-e VERIFY_CERTIFICATE=false \
-e SCHEDULE=@dailydefault \
quay.io/sysdig/host-analyzer:latest
Note: If you don’t want to pass the access key directly via the
command line, consider using an alternative method of passing
environment variables, such as docker-compose.
The BACKEND_ENDPOINT
is only required if for Sysdig on-prem or
when using a Sysdig SaaS region other than US-EAST.
For example, for the EU SaaS endpoint would be:
https://eu1.app.sysdig.com
.
See also: SaaS Regions and IP
Ranges.
BACKEND_VERIFY_TLS=false
is only needed if you are using an
on-prem backend with a self-signed certificate.
TAGS:
The list of tags for the host where the agent is installed.
For example: “role:webserver, location:europe
”, “role:webserver
”
or “webserver
”.
For Image Analyzer Component Only
These cases affect only the Image Analyzer component of the Node
Analyzer installation.
Installing Image Analyzer Component Alone
It is still possible to install the image analyzer component without
benchmarks or host scanning. This option normally would apply only to
previous users of the former node image analyzer who want to upgrade
just that component, for whatever reason.
This can be done by downloading the
sysdig-image-analyzer-daemonset.yaml
and
sysdig-image-analyzer-configmap.yaml
and deploying.
You need to deploy these YAMLs after installing the Sysdig agent in the
same nodes, and also in the same namespace (sysdig-agent
by default).
Kubernetes Requiring Custom Socket Path
By default, the image analyzer will automatically detect the socket to
mount from:
Docker socket from /var/run/docker/docker.sock
CRI-O socket from/var/run/crio/crio.sock
CRI-containerd socket from/var/run/containerd/containerd.sock
Some setups require the analyzer to use custom socket paths.
If the socket is located outside /var/run
, the corresponding volume
must be mounted as well. You can configure it via the single line
installer script or by manually editing the daemonset and configmap
variables.
When using the installer, use the-cv
option to mount an additional
volume and add -ds -cs
or -cd
to specify a Docker, CRI, or
CRI-containerd socket respectively.
See the script -help
command for additional information.
Examples:
For K3S, which uses containerd, add:
-cd unix:///run/k3s/containerd/containerd.sock -cv /run/k3s/containerd
For Pivotal, which uses a custom path for the Docker socket, use:
-ds unix:///var/vcap/data/sys/run/docker/docker.sock -cv /var/vcap/data/sys/run/docker
Daemonset Resource Limit Considerations
During its regular operation, the Image Analyzer uses much less memory
than the limit specified in the daemonset configuration. However, in
some cases, processing an image may require more memory, depending, for
example, on image size, content or package types.
This issue can be detected by looking for abnormal spikes in the memory
usage of the Image Analyzer pods which are also showing analysis errors.
In such cases we recommend trying to increase the analyzer memory usage
up to three times the size of the unprocessed images, if the cluster
available memory allows.
Component Configurations
Image Analyzer Configmap Options
For special cases, the image analyzer can be configured by editing the
sysdig-image-analyzer
configmap in the sysdig-agent
namespace with
the following options:
docker_socket_path
| The Docker socket path, defaulting to unix:///var/run/docker/docker.sock If a custom path is specified, ensure it is correctly mounted from the host inside the container. |
cri_socket_path
| The socket path to a CRI compatible runtime, such as CRI-O, defaulting to unix:///var/run/crio/crio.sock. If a custom path is specified, ensure it is correctly mounted from the host inside the container. |
containerd_socket_path
| The socket path to a CRI-Containerd daemon, defaulting to unix:///var/run/containerd/containerd.sock If a custom path is specified, ensure it is correctly mounted from the host inside the container. |
collector_endpoint
| The endpoint to the Scanning Analysis collector, specified in the following format: https://<API_ENDPOINT>/internal/scanning/scanning-analysis-collector |
ssl_verify_certificate
| Can be set to "false" to allow insecure connections to the Sysdig backend, such as for on-premise installs that use self-signed certificates. By default, certificates are always verified. |
debug
| Can be set to "true" to show debug logging, useful for troubleshooting. |
http_proxy
| Proxy configuration variables. |
https_proxy
| |
no_proxy
| |
Host Scanning Configuration Options
The analyzer component of the Host
Scanning feature can be
configured by editing the sysdig-host-analyzer
configmap in
thesysdig-agent
namespace with the following options:
Option | Description |
---|
schedule | The scanning schedule specification for the host analyzer expressed as a crontab string such as “5 4 * * *” (more examples). The default value of @dailydefault instructs the analyzer to automatically pick a schedule that will start shortly after it is deployed and will perform a scan every 24 hours. |
dirs_to_scan | The list of directories to inspect during the scan, expressed as a comma separated list such as /etc,/var/lib/dpkg,/usr/local,/usr/lib/sysimage/rpm,/var/lib/rpm,/lib/apk/db |
collector_endpoint | The endpoint to the Scanning Analysis collector, specified in the following format: https://<API_ENDPOINT>/internal/scanning/scanning-analysis-collector |
max_send_attempts | The number of times the analysis collector is allowed to retry sending results if backend communication fails |
ssl_verify_certificate | Can be set to "false" to allow insecure connections to the Sysdig backend, such as for on-premise installs that use self-signed certificates. By default, certificates are always verified. |
debug | Can be set to "true" to show debug logging, useful for troubleshooting. |
http_proxy | Proxy configuration variables. |
https_proxy | |
no_proxy | |
Benchmark Runner Configuration Options
The benchmark runner component can be
configured by editing the sysdig-benchmark-runner
configmap in
the sysdig-agent
namespace with the following options:
Option | Description |
---|
collector_endpoint | The Secure API endpoint, specified in the following format: https://<API_ENDPOINT> |
ssl_verify_certificate | Can be set to "false" to allow insecure connections to the Sysdig backend, such as for on-premise installs that use self-signed certificates. By default, certificates are always verified. |
debug | Can be set to "true" to show debug logging, useful for troubleshooting. |
http_proxy | Proxy configuration variables. |
https_proxy | |
no_proxy | |
7 -
Install Admission Controller
If you have installed the CLI-based
version of the Admission
Controller, the UI-based version is not backwards-compatible. You will
need to uninstall the old version and install the UI-based version
instead.
To understand and use the Admission Controller after installing it, see Admission
Controller.
For a more technical documentation see Chart Documentation.
Prerequisites
- Helm 3
- Kubernetes 1.16 or higher
Install the Admission Controller
The component must be installed on each cluster where you want to use it.
Make sure kubectl
is pointing to the target cluster where the
Admission Controller will be installed.
Add and synchronize the Helm repository:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the Admission Controller on the target cluster with full capabilities , e.g.:
helm install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller \
--set sysdig.secureAPIToken=$SYSDIG_API_TOKEN \
--set clusterName=$CLUSTER_NAME \
--set sysdig.url=https://$SYSDIG_SECURE_ENDPOINT \
--set features.k8sAuditDetections=true
Check that installation was successful in the Sysdig UI.
Log in to Sysdig Secure and select Image Scanning>Admission Controller|Policy Assignments
.

Admission Controller will be disabled by default in your cluster, to avoid accidentally blocking deployment.
Cluster will be displayed in the Connected list, as healthy, but Disabled (gray colored dot).
You have to manually enable it by toggling the Enabled flag and status should change to accordingly (green colored dot):

Installation Parameters
Following parameters are the most common ones, but find the full list of available parmeters or specific use-cases
--create-namespace
: If supplied, will create a namespace--namespace
: Desired namespace where the Admission Controller will be installed--set sysdig.secureAPIToken
: Sysdig Secure API token as found in the
Sysdig UI under Settings/User Profile. Note that this
user must have administrator rights--set clusterName
: User-defined name for this cluster that will appear
in the admission controller interface in Sysdig’s backend. The
cluster name needs to match the agent cluster name.--set sysdig.url
: Sysdig endpoint. Default
https://secure.sysdig.com
is for the us-east
region.- For
us-west
use https://us2.app.sysdig.com
- For European Union, use
https://eu1.app.sysdig.com
- For APAC, use
https://app.au1.sysdig.com
- For US4 (our west Google cloud region) use
https://app.us4.sysdig.com/
- For on-prem, your own enpoints.
- See also SaaS Regions and IP Ranges.
--set features.k8sAuditDetections
: (true/false) Set true
to enable
Kubernetes audit logging via the Admission Controller. See also:
Kubernetes Audit Logging
(legacy installation) and Select the Policy Type
(Kubernetes Audit Policies)--set verifySSL
: (true/false) Sets the verification of the Sysdig Secure API; default: true (we recommend only changing this to false when doing initial testing / evaluation of an on-premises installation)--set scanner.verifyRegistryTLS
: (true/false) Verify TLS from registries on image pull; default: true (we recommend only changing this to false when doing initial testing / evaluation)--set scanner.psp.create
: (true/false) Whether to create a psp policy and role / role-binding; default: false
Enable in Sysdig Labs (for Image Scanning)
Log in to Sysdig Secure as administrator and select
Settings|User Profile
.
Under Sysdig Labs, enable the Admission Controller feature and click
Save
.

The links to the Admission Controller pages will appear under Image
Scanning in the left-hand navigation.
Upgrades
Upgrading from Scanning-Only Admission Controller
If you already have the Sysdig Admission Controller installed and want
to upgrade:
helm upgrade sysdig-admission-controller sysdig/admission-controller \
-n sysdig-admission-controller \
--set features.k8sAuditDetections=true \
--reuse-values
For those customers who already have the Admission Controller AND
already enabled Kubernetes audit logging via the legacy
method, you can still
install/upgrade to the new Admission Controller. Just be sure to set
features.k8sAuditDetections=false
to avoid collecting and displaying
duplicate events.
Uninstall the CLI-based Version
If you have installed the CLI-based
version of the Admission
Controller, the UI-based version is not backwards-compatible. You will
need to uninstall the old version and install the UI-based version
instead.
Deploy the following:
helm uninstall -n sysdig-admission-controller sysdig-admission-controller
Troubleshooting
Refer to Chart Documentation - Troubleshooting.