This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Agent Installation

Sysdig agents are delivered as either a container or a service and can be deployed with or without an orchestrator such as Kubernetes or Mesos.

A quick install involves just a few lines of code from the Getting Started wizard copied into a shell. The complete install instructions address checking for and installing kernel headers if needed, any prerequisite permissions settings for particular environments, and tips about updating the configuration files after initial installation.

Plan the Installation

Topic

Description

Host Requirements

Review the platforms, runtimes, Linux distributions, orchestration, browsers, etc. that are supported.

Access key

An agent access key is provided with a Sysidg trial

Installation Options

Different ways in which you can install Sysdig agent.

Troubleshooting Agents

Troubleshooting tips for agent installation, tuning agents, and compiling kernel modules.

Installation Options

In the default mode of agent installation, you install the agent package as two containers, each container responsible for different functions as given below. The agent-slim reduces the surface area of attack for potential vulnerabilities and is, therefore, more secure.

  • agent-kmodule: Responsible for downloading and building the kernel module. The image is short-lived. The container exits after the kernel module is loaded. The transient nature of the container reduces the time and opportunities for exploiting any potential vulnerabilities present in the container image.

    Prerequisites: The package depends on Dynamic Kernel Module Support (DKMS) and requires the compiler and kernel headers installed if you are using the agent-kmodule to build the kernel probe. Alternatively, you can use it without the kernel headers. In such cases, the agent-kmodule will attempt to download a pre-built kernel probe if it is present in the Sysdig probe repository.

    The module contains:

    • The driver sources

    • A post-install script that builds the module upon installation

  • agent-slim: Responsible for running the agent module once the kernel module has been loaded. Slim agent functions the same way as the regular agent and retains the feature parity.

Use the instruction below to install agent on your chosen environment:

EnvironmentFlavorInstallationInstall Instructions
KubernetesOpen SourceHelm and manualInstall Agent on Kubernetes
OpenShiftHelm and manualInstall Agent on OpenShift
GKEHelm and manualInstall Agent on GKE
MKEHelmInstall Agent on MKE
RancherHelm and manualInstall Agent on Rancher
OKEHelm and manualInstall Agent on OKE
Non-OrchestratedManualInstall Agent on Non-Orchestrated Environment

Legacy Agent: The legacy agent can be run as a single container or a service. It includes the components for downloading and building the kernel module, as well as for gathering and reporting on a wide variety of pre-defined metrics. For more information, see Installing Agent as a Single Container.

Helm

Helm is the preferred way of installing Sysdig agent. It is used in most cloud environments, for example, Amazon EKS or EC2 on AWS Cloud or AWS Outpost, EC2, and Azure AKS.

Manual

With the Getting Started wizard, you can copy a simple line of code to deploy agents in a variety of environments.

Behind the scenes, the wizard auto-detects and completes configuration items such as the required access key and port information. The wizard can also be launched from the Start a Free Trial button at sysdig.com.

After the first install, Sysdig Secure and Monitor users can access the wizard at any time from the Rocket icon on the navigation bar.

EnvironmentFlavorInstall Instructions
KubernetesOpen SourceHelm is the preferred way of installing Sysdig agent. It is used in most cloud environments, for example, Amazon EKS or EC2 on AWS Cloud or AWS Outpost, EC2, and Azure AKS.
OpenShift
GKEUsed for Google Kubernetes Service environment.
MKEeBPF parameter required
RanchereBPF parameter required
OKE
IKSIBM manages and documents Sysdig installs as part of IKS.
IBM Cloud Monitoring
Non-OrchestratedUsed when there is no orchestrator such as Kubernetes. Install Agent on Non-Orchestrated Environment.
LinuxRare, used with custom kernel headers, unique use cases
Agent Install: Manual Linux Installation.
Mesos |Marathon|DCOSAgent Install: Mesos |Marathon |DCOS.

1 - Agent Installation Requirements

Sysdig agents can be installed on a wide array of Linux hosts. Check your environment to ensure it meets the minimum supported platform, operating system, runtime, and orchestration requirements and uses the appropriate installation instructions.

Versioning Scheme

We recommend that you use the latest version of the agent. Sysdig supports n-3 versions back based on the minor number. For example, if the latest release is v12.0.0, we will support n-3 versions back, up to v11.2.0.

End of Support

Sysdig agents that are older than version 0.85.1, released October 1, 2018, will no longer connect to the Sysdig US-East SaaS platform with default agent values.

Going forward all the agent releases will have a 3-year deprecation policy. This implies:

  • Sysdig Support might not be able to help you troubleshoot or address the problems with agents past the deprecation date.

  • Sysdig will no longer provide prebuilt kernel probe binaries for these agent releases. You need to build the kernel probe binaries on the fly by using the hosts kernel headers.

    These changes is effective starting Sysdig agent v12.1.0.

Agent Installation Requirements

Orchestration Platforms

Support Matrix for Kubernetes

Sysdig agent version 12.8.1 has been tested on the following list of latest Kubernetes versions. The matrix provides a single view into the supported operating systems, architecture, and runtime versions for different flavors of Kubernetes orchestrators.

ClusterOperating SystemKubernetes VersionArchitectureRuntime
RedHat OpenShift Kubernetes Service (ROKS)Red Hatv1.22x86_64cri-o
RancherSUSE Linux Enterprise Server 15 SP2v1.20x86_64docker
OpenShift (okd4) 4.8Red Hat Enterprise Linux CoreOS 48v1.21zlinuxcri-o
OpenShift (okd4) 4.10
NOTE: OpenShift versions 4.10+ cannot be used with the new Vulnerability Management component.
This means that if installing the agent on OCP4.10+, the following option should not be used or be set to false
(Helm):
set nodeAnalyzer.secure.vulnerabilityManagement.newEngineOnly=false \
Red Hat Enterprise Linux CoreOS 410v1.23x86_64cri-o
OpenShift (okd3)CentOS Linux 7 (Core)v1.11.0+d4cacc0x86_64docker
Kubernetes Operations (kops)Ubuntu 20.04.4 LTSv1.21x86_64, arm64containerd
Kubernetes Operations (kops)Ubuntu 20.04.4 LTSv1.24x86_64, arm64containerd
KubernetesUbuntu 20.04.2 LTSv1.23x86_64docker
IBM Cloud Kubernetes Service (IKS)Ubuntu 18.04.6 LTSv1.23x86_64containerd
Google Kubernetes Engine (GKE)Container-Optimized OS from Googlev1.22x86_64containerd
Amazon Elastic Kubernetes Service (EKS)Bottlerocket OS 1.9v1.22x86_64, arm64containerd

(Beta) Additional Orchestration Platforms

Orchestration PlatformsDocumentation
Oracle Kubernetes Engine (OKE)Steps for OKE
Microsoft Azure Cloud ServicesAgent Install: Non-Orchestrated
Microsoft Azure Kubernetes Service (AKS)Agent Install: Kubernetes
Amazon Elastic Container Service (Amazon ECS)Agent Install: Non-Orchestrated
+ AWS Integration Instructions
RancherOSAgent Install: Non-Orchestrated
Mesos/MarathonAgent Install: Mesos/Marathon
Docker Datacenter (DDC)Agent Install: Non-Orchestrated

If you are not using an orchestrator in your environment, follow the instructions for Agent Install Non-Orchestrated.

Note: Installing the Sysdig agent into a namespace managed by Istio and configured for sidecar auto-injection is not supported. For example, setting kubectl label namespace sysdig-agent istio-injection=enabled. Because the agent behaves more like a host component, it is required to be part of the host PID and network namespace to function correctly. Due to this requirement, deploying the Sysdig agent in Istio with an Envoy sidecar is not supported. However, running the Sysdig agent in a non-injected namespace where Istio is installed and managing other namespaces is fully supported. See Istio integration for more details on using the Sysdig agent to monitor Istio control plane and sidecar metrics.

Linux Distributions and Kernels

Support Matrix for Linux Distributions

Sysdig agent version 12.8.1 (installed as a service) has been tested on the following list of latest linux distros:

Operating SystemArchitecture
Amazon Linux 2x86_64
Fedora Linux 36 (Cloud Edition)x86_64
Red Hat Enterprise Linux 8.6 (Ootpa)x86_64
Ubuntu 18.04.6 LTS (Bionic Beaver)x86_64
Ubuntu 20.04.4 LTS (Focal Fossa)x86_64
Ubuntu 22.04 LTS (Jammy Jellyfish)x86_64

(Beta) Linux Distributions

Sysdig agent is supported on the following Linux distributions:

Platforms

Linux Distribution

Core Set

  • Debian

  • Ubuntu

  • Ubuntu (Amazon)

  • CentOS

  • Red Hat Enterprise Linux (RHEL)

  • SuSE Linux Enterprise Server

  • RHEL CoreOS (RHCOS)

  • Fedora

  • Fedora CoreOS

  • Linux Mint

  • Amazon Linux

  • Amazon Linux v2

  • Amazon Bottlerocket

  • Google Container Optimized OS (COS)

  • Oracle Linux (UEH)

  • Oracle Linux (RHCK)

AWS EC2

  • Amazon Linux 2

  • Amazon Bottlerocket

  • Core set (see above)

GCP

  • Core set (see above)

  • COS

Azure

  • Core set (see above)

Container Runtimes

Sysdig agent supports the detection of the following:

  • Docker
  • LXC
  • CRI-O
  • containerd
  • Podman
  • Mesos

Support Matrix for Docker

Sysdig agent version 12.8.1 has been tested on the following list of latest linux distros:

Operating SystemArchitecture
Amazon Linux 2x86_64, arm64
Amazon Linux 2022x86_64, arm64
Debian GNU/Linux 10 (buster)x86_64, arm64
Debian GNU/Linux 11 (bullseye)x86_64, arm64
Fedora Linux 35 (Cloud Edition)x86_64, arm64
Fedora Linux 36 (Cloud Edition)x86_64, arm64
Red Hat Enterprise Linux 8.6 (Ootpa)x86_64, arm64
Red Hat Enterprise Linux 9.0 (Plow)x86_64, arm64
SLES_15 SP4x86_64
Red Hat Enterprise Linux 9.0 (Plow)x86_64, arm64
Ubuntu 18.04.6 LTS (Bionic Beaver)x86_64, arm64
Ubuntu 20.04.4 LTS (Focal Fossa)x86_64, arm64
Ubuntu 22.04.4 LTS (Jammy Jellyfish)x86_64, arm64

Prerequisites for Podman Environments

Sysdig agent supports running as a Podman container.

  • Enable Podman API Service for all the users.

    The agent will not able to collect Podman-managed container metadata, such as the container name, if the API service is not enabled.

  • Secure rules and policies that depend on container metadata other than the container ID will not work.

  • Pausing and terminating containers will not work because Policy actions for Podman are not supported.

  • The containers started as a non-root user will have the podman_owner_uid label associated with it if the API service is enabled for that user. The value of podman_owner_uid will be the numeric user ID corresponding to the user that started the container.

Container Registries

Quay.io

For example, to pull the latest agent container from Quay.io:

docker pull quay.io/sysdig/agent

CPU Architectures

x86

Supported Agent Containers
  • agent
  • agent-slim
  • agent-kmodule

ARM (aarch64)

Supported kernel versions are v4.17 and above

Unsupported Features
  • Pre-built probes
  • Activity Audit
  • Sysdig agent installation using the agent container

s390x (zLinux)

Unsupported Features
Probes

No support for pre-built probes on zLinux. For kernel instrumentation, use the kernel module. eBPF probes are not supported on zLinux.

Captures

Capture is not supported on zLinux.

Legacy Agent Installation

Sysdig agent installation using agent container is not supported.

Java Versions and Vendors

Sysdig agent supports the following:

  • Java versions: v7 and above
  • Vendors: Oracle, OpenJDK

For Java-based applications (Cassandra, Elasticsearch, Kafka, Tomcat, Zookeeper and etc.), the Sysdig agent requires the Java runtime environment (JRE) to be installed to poll for metrics (beans).

If the Docker-container-based Sysdig agent is installed, the JRE is installed alongside the agent binaries and no further dependencies exist. However, if you are installing the service-based agent (non-container) and you do not see the JVM/JMX metrics reporting, your host may not have the JRE installed or it may not be installed in the expected location: usr/bin/java

Minimum Resource Requirements

The resource requirements of the agent are subjective to the size and load of the host— more activity equates to more resources required.

It is typical to see between 5-20KiB/s of bandwidth consumed—different variables can increase the throughput required such as the number of metrics, events, Kubernetes objects, and which products and features are enabled. When a Sysdig Capture is being collected, you can expect to see a spike in bandwidth while the capture file is being ingested.

We do not recommend placing bandwidth shaping or caps on the agent to ensure data can be sent to our collection service. For more information, see Tuning Sysdig Agent.

Additional Requirements

Access key

The installation of the Sysdig agent requires an access key.

This key and the agent installation instructions are presented to you after activating your account and using a web-based wizard upon initial login.

The same information can also be found in the Settings > Agent Installation menu of the web interface after logging in. See Agent Installation: Overview and Key for details.

Network connection

A Sysdig agent (containerized or native) is installed into each host being monitored and will need to be able to connect to the Sysdig Monitor backend servers to report host metrics. The agent must be able to reach the Sysdig Collector addresses. For example, for US East, it is ‘collector.sysdigcloud.com’ (via multiple IPs) over port tcp/6443 . See Sysdig Collector Ports for supported ports for other regions.

The agent supports the HTTP proxy for communicating with Sysdig backend components. For more information, see Enable HTTP Proxy for Agents.

2 - Quick Install Sysdig Agent

Sysdig provides you with quick-install commands pre-filled with some of your environment variables to get started with Sysdig agent. You choose the deployment type and Sysdig gives you auto-generated command to ease your installation experience.

Access from Get Started Pages

  1. Log in as the administrator to Sysdig Monitor or Sysdig Secure.

  2. Select the Get Started page.

  3. Click Install the Agent, select the appropriate deployment type, and copy the auto-generated code, filling in remaining variable values as required.

Sample Usage

Kubernetes

Helm

Helm is the recommended option for installing agents on Kubernetes.

Example

The shell commands below will create a new Kubernetes namespace called sysdig-agent and deploy the agent with a Helm release name of sysdig. Be sure to replace the configuration options with the values specific to your setup.

SaaS
kubectl create ns sysdig-agent
helm repo add sysdig https://charts.sysdig.com
helm repo update
helm install sysdig-agent \
    --namespace=sysdig-agent \
    --set global.sysdig.accessKey=`1234-your-key-here-1234` \
    --set global.sysdig.region='us1' \
    --set global.clusterConfig.name='my_cluster' \
    --set agent.sysdig.settings.tags='linux:ubuntu,dept:dev,local:nyc' \
    sysdig/sysdig-deploy
On-Prem
kubectl create ns sysdig-agent
helm repo add sysdig https://charts.sysdig.com
helm repo update
helm install sysdig-agent \
    --namespace=`dev` \
    --set global.sysdig.accessKey=`1234-your-key-here-1234` \
    --set agent.collectorSettings.collectorHost='mycollector.elb.us-west-1.amazonaws.com' \
    --set agent.collectorSettings.collectorPort=`6443` \
    --set agent.sysdig.settings.tags='linux:ubuntu,dept:dev,local:nyc' \
    --set agent.sysdig.settings.k8s_cluster_name='my_cluster' \
    sysdig/sysdig-deploy

Options

For the latest helm-based installation instructions and configuration options, see sysdig-deploy.

Script

If you cannot utilize helm, we also provide a script which will download and apply Kubernetes manifests to deploy the agent as a DaemonSet. The script requires curl and kubectl installed in the $PATH on the host in which it is run.

install-agent-kubernetes \
[-a | --access_key <value>] [-t | --tags <value>] \
[-c | --collector <value>] [-cp | --collector_port <value>] [-s | --secure <value>] \
[-cc | --check_certificate <value>] [-ns | --namespace | --project <value>] \
[-ac | --additional_conf <value>] [-op | --openshift] [-as | --agent_slim] \
[-av | --agent_version <value>] [-ae | --api_endpoint <value> ] [-na | --nodeanalyzer ] \
[-ia | --imageanalyzer ] [-am | --analysismanager <value>] [-ds | --dockersocket <value>] \
[-cs | --crisocket <value>] [-cv | --customvolume <value>] \
[-cn | --cluster_name <value>] [-r | --remove ] [-h | --help]
Example
curl -s https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s -- \
--access_key <ACCESS_KEY>  \
--collector <COLLECTOR_ADDRESS> --collector_port <COLLECTOR_PORT> \
--nodeanalyzer --api_endpoint <SECURE_ENDPOINT_ADDRESS>

Options

For the complete configuration options, see Agent Install:Kubernetes.

Docker

Install agent-kmodule
docker run -it --privileged --rm --name sysdig-agent-kmodule \
  -v /usr:/host/usr:ro \
  -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro \
  quay.io/sysdig/agent-kmodule
Install agent-slim
docker run -d --name sysdig-agent \
  --restart always \
  --privileged \
  --net host \
  --pid host \
  -e ACCESS_KEY=<ACCESS_KEY> \
  -e COLLECTOR=<COLLECTOR_URL> \
  -e SECURE=true \
  [-e TAGS=<LIST_OF_TAGS>] \
  -e ADDITIONAL_CONF= <LIST_OF_CONFIG> \
  -v /var/run/docker.sock:/host/var/run/docker.sock \
  -v /dev:/host/dev \
  -v /proc:/host/proc:ro \
  -v /boot:/host/boot:ro \
  --shm-size=512m \
  quay.io/sysdig/agent-slim

Example

Install agent-kmodule
docker run -it --privileged --rm --name sysdig-agent-kmodule \
  -v /usr:/host/usr:ro \
  -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro \
  quay.io/sysdig/agent-kmodule
Install agent-slim
docker run \
  --name sysdig-agent \
  --privileged \
  --net host \
  --pid host \
  -e ACCESS_KEY=1234-your-key-here-1234  \
  -e COLLECTOR=mycollector.elb.us-west-1.amazonaws.com \
  -e COLLECTOR_PORT=6443 \
  -e CHECK_CERTIFICATE=false \
  -e TAGS=my_tag:some_value \
  -e ADDITIONAL_CONF="log:\n file_priority: debug\n console_priority: error" \
  -v /var/run/docker.sock:/host/var/run/docker.sock \
  -v /dev:/host/dev \
  -v /proc:/host/proc:ro \
  -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro \
  -v /usr:/host/usr:ro \
  --shm-size=350m \
quay.io/sysdig/agent-slim

Options

OptionDescription
ACCESS_KEYThe agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.
tagsEnter meaningful tags you want applied to your instances.
COLLECTORThe collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges.
collector_portThe default is 6443.
SECUREUse a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default.
CHECK_CERTIFICATE(On-prem) Determines strong SSL certificate check for Sysdig Monitor on-premises installation. Set to true when using SSL/TLS to connect to the collector service to ensure that a valid SSL/TLS certificate is installed.
ADDITIONAL_CONFOptional. Use this option to provide custom configuration values to the agent as environment variables. If provided, will be appended to agent configuration file. For example, For example, file log configuration.
bpfEnables eBPF probe.

Linux

$ curl -s https://download.sysdig.com/stable/install-agent | sudo bash -c -- \
--access_key [-t | --tags <value>] [-c | --collector <value>] \
[-cp | --collector_port <value>] [-s | --secure <value>] \
[-cc | --check_certificate]  [-ac | --additional_conf <value>] \
[-b | --bpf] [-h | --help]

Example

curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- \
--access_key <ACCESS_KEY> --collector collector-staging.sysdigcloud.com \
--secure true

Options

OptionDescription
access-keyThe agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.
tagsEnter meaningful tags you want applied to your instances.
collectorThe collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges.
collector_portThe default is 6443.
secureUse a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default.
check_certificateDisables strong SSL certificate check for Sysdig Monitor on-premises installation.
additional_confOptional. Use this option to provide custom configuration values to the agent as environment variables. If provided, the value will be appended to agent configuration file. For example, file log configuration.
bpfEnables eBPF probe.

3 - Agent Install: Kubernetes

The recommended method to monitor Kubernetes environments is to deploy the Sysdig agent using the helm chart. Alternatively, you can install the agent container using DaemonSet. This section helps you install the agent in both the methods.

Installing the agent using helm or as a daemonSet will deploy agent containers on every node in your Kubernetes environment. Once the agent is installed, Sysdig Monitor automatically begins monitoring all of your hosts, apps, pods, and services and automatically connects to the Kubernetes API server to pull relevant metadata about the environment. If licensed, Sysdig Secure launches with default policies that you can view and configure to suit your needs. You can access the front-end web interfaces for Sysdig Monitor and Sysdig Secure immediately.

Sysdig supports monitoring numerous Kubernetes platforms, including the following:

Prerequisites

  • A supported distribution: See Agent Installation Requirements for details.

  • Kubernetes v1.9+: The agent installation on Kubernetes requires v1.9 or higher because the APIs used to fetch kubernetes metadata are only present in v1.9+.

  • Sysdig account and access key: Request a trial or full account at Sysdig.com and click the Activate Account button. The Getting Started Wizard provides an access key.

  • Port 6443 open for outbound traffic: The agent communicates with the collector on port 6443. If you are using a firewall, you must open port 6443 for outbound traffic for the agent.

  • Kernel headers installed: If a prebuilt kernel probe is not available for your kernel, the kernel headers must be installed in order to build the kernel probe.

  • kubectl installed: All of the installation methods utilize kubectl to install the agent in the cluster.

Kernel Headers

The Sysdig agent requires kernel header files to install successfully on a Kubernetes cluster. If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.

In some cases, the nodes in your Kubernetes environment might use Unix versions that do not match the provided headers, and the agent might fail to install correctly. In those cases, you must install the kernel headers manually on each node.

To do so:

For Debian-style distributions, run the command:

apt-get -y install linux-headers-$(uname -r)

For RHEL-style distributions, run the command:

yum -y install kernel-devel-$(uname -r)

For more information on troubleshooting, see About Kernel Headers and the Kernel Module.

Kubernetes Environments

Some Kubernetes environments require special configuration options to deploy the agent. If you’re installing in one of the following environments, follow the guides specific to those environments to deploy the agent. Otherwise, continue with this topic.

Installation

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Script

Sysdig also provides a script that you can use to install the agent as a DaemonSet.

Installation

  1. Download the script and make it executable.

     wget https://download.sysdig.com/stable/install-agent-kubernetes
     chmod +x install-agent-kubernetes
    
  2. Run the script to install the agent as a DaemonSet.

    ./install-agent-kubernetes -a <ACCESS_KEY> -c <COLLECTOR_URL> -cn <CLUSTER_NAME>
    

Options

Option

Description

-a

The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.

-t

The list of tags to identify the host where the agent is installed. For example: role:webserver, location:europe, role:webserver.

-c

The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations.

-cp

The collector port. The default is 6443.

-cn

If a value is provided, the cluster will be identified with the name provided

-s

Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default.

-cc

Enable strong SSL certificate check. The default is true.

-ns

If a value is provided, the agent will be deployed to the specified namespace/project. The default is sysdig-agent.

-op

If provided, perform the agent installation using the OpenShift command line.

-ac

If a value is provided, the additional configuration will be appended to the agent configuration file.

-av

If a version is provided, use the specified agent version. The default is the latest version.

-r

If a value is provided, the daemonset, configmap, cluster role binding, service acccount and secret associated with the Sysdig Agent will be removed from the specified namespace.

-ae

The api_endpoint is the region-dependent domain for the Sysdig product, without the protocol. E.g. secure.sysdig.com, us2.app.sysdig.com, eu1.app.sysdig.com

-h

Print this usage and exit.

Sysdig Secure Only

-na

If provided, will install the Node Analyzer tools. It is an error to set both -ia and -na.

-ds

The docker socket for Image Analyzer.

-cs

The CRI socket for Image Analyzer.

-cv

The custom volume for Image Analyzer.

-h

Print this usage and exit.

-b

Required in AWS Bottlerocket nodes to determine whether the eBPF should be built. Alternatively, you can use `--bpf`.

Sysdig Secure Only (Legacy)

These values apply to the Node Image Analyzer (v1) in Sysdig Secure.

-am

The Analysis Manager endpoint for Sysdig Secure.

-ia

If provided, will install the Node Image Analyzer (v1). It is an error to set both -ia and -na. The v1 Node Image Analyzer will be deprecated and replaced by the NA tools.

Manifests

To deploy agents using Kubernetes manifests, you can download manifest files, edit them as required, and deploy them using kubectl.

  1. Download the sample files:

    • sysdig-agent-clusterrole.yaml

    • sysdig-agent-daemonset-v2.yaml

    • sysdig-agent-configmap.yaml

    • sysdig-agent-service.yaml

  2. Create a namespace for the Sysdig agent.

    Note: You can use whatever name you prefer. This example uses sysdig-agent for both the namespace and the service account. The default service account name was automatically defined in sysdig-agent-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent

    kubectl create ns sysdig-agent
    
  3. Create a secret key:

    kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
    
  4. Create a cluster role and service account, and define the cluster role binding that grants the Sysdig agent rules in the cluster role:

    kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
    kubectl create serviceaccount sysdig-agent -n sysdig-agent
    kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
    
  5. Edit sysdig-agent-configmap.yaml to add the collector address , port , and the SSL/TLS information:

    collector:
    collector_port:
    ssl: #true or false
    check_certificate: #true or false
    
    • For SaaS, find the collector address for your region.

    • For On-prem, enter the collector endpoint defined in your environment.

    • check_certificate should be set to false if a self-signed certificate or private, and a CA-signed cert is used. See Set Up SSL Connectivity to the Backend for more information.

  6. Apply the sysdig-agent-configmap.yaml file:

    kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
    
  7. Apply the sysdig-agent-service.yaml file:

    kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
    

    This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

  8. Apply the daemonset-v2.yaml file :

    kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
    

Additional Options

Verify Metrics in Sysdig Monitor

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

The steps below give one way to do the check.

  1. Access Sysdig Monitor:

    SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

    For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

    Log in with your Sysdig user name and password.

  2. Select the Explore tab to see if metrics are displayed.

  3. Determine the Kube State Metrics you want to collect.

  4. To verify that kube state metrics and cluster name are working correctly, select the Explore tab and see if your cluster is listed.

Kubernetes metadata (pods, deployments etc.) appear a minute or two later than the nodes/containers themselves; if pod names do not appear immediately, wait and retry the Explore view.

If agents are disconnecting, there could be an issue with your MAC addresses. See Troubleshooting Agent Installation for tips.

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com.

The sysdig-agent-configmap.yaml file can be edited either locally or using the edit command in Kubernetes.

To configure the collector IP in a Kubernetes SaaS instance:

  1. Open sysdig-agent-configmap.yaml in a text editor.

  2. Uncomment the following lines:

    • collector:

    • collector_port

  3. Set the collector: value to collector-static.sysdigcloud.com

  4. Set the collector_port: value to 6443

  5. Save the file.

The example file below shows how the sysdig-agent-configmap.yaml file should look after configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sysdig-agent
data:
  dragent.yaml: |
    ### Agent tags
    # tags: linux:ubuntu,dept:dev,local:nyc

    #### Sysdig Software related config ####

    # Sysdig collector address
    collector: collector-static.sysdigcloud.com

    # Collector TCP port
    collector_port: 6443

    # Whether collector accepts ssl/TLS
    ssl: true

    # collector certificate validation
    ssl_verify_certificate: true

    # Sysdig Secure
    security:
      enabled: true

    #######################################
    # new_k8s: true
    # k8s_cluster_name: production    

3.1.1 - GKE Autopilot

Autopilot is an operation mode for creating and managing clusters in GKE. In brief, with Autopilot, Google configures and manages the underlying node infrastructure for you. This topic helps you use helm to install Sysdig agent on a GKE cluster installed in Autopilot mode.

NodeAnalyzer is not supported on Autopilot environments.

Prerequisites

  1. Install a GKE cluster in Autopilot mode.

  2. Connect the GKE cluster.

  3. Install your workload.

Deploy Sysdig Agent

Sysdig recommends using Helm to install Sysdig agent in kubernetes environments. After connecting to the GKE cluster, use the sysdig-deplpy chart to install Sysdig agent.

To customize the configuration of the agent, see the Sysdig Agent Helm Chart.

Verify Metrics on the Sysdig Monitor UI

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

Given below is one way to do so.

  1. Access Sysdig Monitor:

    SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

    For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysdig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

    Log in with your Sysdig user name and password.

  2. Select the Explore tab to see if metrics are displayed.

  3. Verify that kube state metrics and cluster name are working correctly: select the Explore tab and create a grouping by kube_cluster_name and kube_pod_name.

  4. Select an individual container or pod to see the details.

3.1.2 - GKE Standard

Google Kubernetes Engine (GKE) is a managed environment for running Kubernetes in Google Cloud, in order to deploy containerized applications. Sysdig supports all flavors of GKE, including Ubuntu and GKE’s default Container-Optimized OS(COS).

GKE COS environments require eBPF probe to support agent installation.

Preparation

Open Port 6443 for Agent Egress

Because GKE uses stateful firewalls, you must actively open port 6443 for the Sysdig agent outbound traffic.

In earlier versions, the Sysdig Agent connected to port 6666. This behavior has been deprecated, as the Sysdig agent now connects to port 6443.

GKE COS/eBPF-Specific Requirements

  • Linux kernel version >= 4.14.

  • When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7. Note that only the eBPF probe is supported in GKE COS environments.

Prerequisites

You can review Agent Install: Kubernetes and the Agent Installation Requirements for additional context, if desired.

Installation Steps

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

To deploy agents using Kubernetes manifests, you can download the manifest files, edit them as required, and deploy them using kubectl.

  1. Download the sample files:

    • sysdig-agent-clusterrole.yaml

    • sysdig-agent-daemonset-v2.yaml

    • sysdig-agent-configmap.yaml

    • sysdig-agent-service.yaml

  2. Create a namespace to use for the Sysdig agent.

    You can use whatever name you want. In this document, we used sysdig-agent for both the namespace and the service account.

    kubectl create ns sysdig-agent
    
  3. Create a secret key:

    kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
    
  4. If you are running Kubernetes 1.6 or higher, you must grant your user the ability to create roles in Kubernetes by running the following command:

    kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org
    

    See Google documentation for more information.

    Create a service account for the Sysdig agent using the clusterrole.yaml file.

    The Sysdig agent must be granted read-only access to certain Kubernetes APIs, which the agent uses to populate metadata and provide component metrics.

    You can use the Sysdig-provided, sysdig-agent-clusterrole.yaml file. Deploying this file creates a cluster role and service account in Kubernetes, and defines cluster role binding that grants the Sysdig agent rules in the cluster role.

    Run the following commands (using whatever namespace you’ve defined in Step 2):

    kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
    kubectl create serviceaccount sysdig-agent -n sysdig-agent
    kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
    
  5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information :

    collector:
    collector_port:
    ssl: #true or false
    check_certificate: #true or false
    
  6. Apply the sysdig-agent-configmap.yaml file:

    kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
    
  7. FOR GKE COS ONLY: To enable the eBPF probe required for COS, uncomment the following parameters in  sysdig-agent-daemonset-v2.yaml under the env section:

    env:
      - name: SYSDIG_BPF_PROBE
        value: ""
    
  8. Apply the sysdig-agent-service.yaml file:

    kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
    

    This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

  9. Apply the daemonset-v2.yaml file:

    kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
    

The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Verify Metrics in Sysdig Monitor and optionally, Connect to Sysdig Backend.

3.2 - Steps for OKE

Oracle Kubernetes Engine (OKE) is a managed environment for running Kubernetes in Oracle Cloud, in order to deploy containerized applications. As of Sysdig agent version 12.0.1, Sysdig supports all flavors of OKE.

OKE environments require eBPF probe to support agent installation.

The instructions below describe a standard OKE agent install and call out the special steps needed to install the eBPF probe.

Preparation

Open Port 6443 for Agent Egress

Because OKE uses stateful firewalls, you must actively open port 6443 for the Sysdig agent outbound traffic.

OKE by default allows network access to the sysdig Agent on 6443, but ensure that firewall rules are open and the agent can connect to the Sysdig backends.

eBPF-Specific Requirements

  • Linux kernel version >= 4.14.

  • When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below.

Installation Steps

Identify the appropriate endpoint depending on your Sysdig account region. For more information, see SaaS Regions and IP Ranges. More info here https://docs.sysdig.com/en/docs/administration/saas-regions-and-ip-ranges/

After making clear which region your account belongs to, please choose one of the following methods:

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

To deploy agents using Kubernetes manifests, you can downloadmanifest files, edit them as required, and deploy them using kubectl.

  1. Download the sample files:

    • sysdig-agent-clusterrole.yaml

    • sysdig-agent-daemonset-v2.yaml

    • sysdig-agent-configmap.yaml

    • sysdig-agent-service.yaml

  2. Create a namespace to use for the Sysdig agent.

    Notes: You can use whatever name you want. In this document, we used sysdig-agent for both the namespace and the service account.

    kubectl create ns sysdig-agent
    
  3. Create a secret key:

    kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
    
  4. If you are running Kubernetes 1.6 or higher, you must create a service account for the Sysdig agent by using the clusterrole.yaml file.

    The Sysdig agent must be granted read-only access to certain Kubernetes APIs, which the agent uses to populate metadata and provide component metrics.

    You can use the Sysdig-provided sysdig-agent-clusterrole.yaml file. Deploying it create a cluster role and service account in Kubernetes, and defines cluster role binding that grants the Sysdig agent rules in the cluster role.

    Run the following commands by using the namespace you’ve defined in Step 2:

    kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
    kubectl create serviceaccount sysdig-agent -n sysdig-agent
    kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
    
  5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information :

    collector:
    collector_port:
    ssl: #true or false
    check_certificate: #true or false
    
  6. Apply the sysdig-agent-configmap.yaml file:

    kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
    
  7. To enable the eBPF probe uncomment the following parameters in  sysdig-agent-daemonset-v2.yaml under the env section:

    env:
      - name: SYSDIG_BPF_PROBE
        value: ""
    
  8. Apply the sysdig-agent-service.yaml file:

    kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
    

    This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

  9. Apply the daemonset-v2.yaml file:

    kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
    

    The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Verify Metrics in Sysdig Monitor and optionally, Connect to Sysdig Backend.

3.3 - Steps for OpenShift

You can review Agent Install: Kubernetes and the Agent Installation Requirements for additional context, if desired.

RHCOS environments require eBPF probe to support agent installation.

Preparation

RHCOS/eBPF-Specific Requirements

  • Linux kernel version 4.14 or above.
  • When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below.

Kernel Headers

The Sysdig agent requires kernel header files to install successfully on a Kubernetes cluster. If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.

In some cases, the nodes in your Kubernetes environment might use Unix versions that do not match the provided headers, and the agent might fail to install correctly. In those cases, you must install the kernel headers manually on each node.

To do so:

For Debian-style distributions, run the command:

apt-get -y install linux-headers-$(uname -r)

For RHEL-style distributions, run the command:

yum -y install kernel-devel-$(uname -r)

For more information on troubleshooting, see About Kernel Headers and the Kernel Module.

Configure for OpenShift

If you are using Red Hat OpenShift, these steps are required. They describe how to create a project, assign and label the node selector, create a privileged service account, and add it to a cluster role.

Copy/Paste Sample Code Block

In the example code, this document uses sysdig-agent for the PROJECT NAME (-n) and the SERVICE ACCOUNT (-z).

You can copy and paste the code as is, or follow the steps below to customize your naming conventions.

oc adm new-project sysdig-agent --node-selector=''
oc project sysdig-agent
oc create serviceaccount sysdig-agent
oc adm policy add-scc-to-user privileged -n sysdig-agent -z sysdig-agent -z node-analyzer
oc adm policy add-cluster-role-to-user cluster-reader -n sysdig-agent -z sysdig-agent -z node-analyzer

Customize the Code

You can use your own Project name and Service Account name if desired.

Note that if you use a different Service Account name, you will need to edit the default service account in the Sysdig Installation Steps, below.

  1. Create a new OpenShift project for the Sysdig agent deployment and use an empty string for the node selector:

    oc adm new-project PROJECT-NAME --node-selector=""
    
  2. Change to the new OpenShift Project for the Sysdig agent deployment:

    oc project PROJECT-NAME
    
  3. Create a service account for the project:

    oc create serviceaccount SERVICE-ACCOUNT
    
  4. Add the service account to privileged Security Context Constraints:

    oc adm policy add-scc-to-user privileged -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
    
  5. Add the service account to the cluster-reader Cluster Role:

    oc adm policy add-cluster-role-to-user cluster-reader -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
    

Installation

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

  1. Download the sample files:

    • sysdig-agent-daemonset-v2.yaml

    • sysdig-agent-clusterrole.yaml

    • sysdig-agent-configmap.yaml

    • sysdig-agent-service.yaml

  2. Create the sysdig-agent cluster role and assign it to the service account:

     oc apply -f sysdig-agent-clusterrole.yaml
     oc adm policy add-cluster-role-to-user sysdig-agent -n PROJECT-NAME -z SERVICE-ACCOUNT
    
  3. Create a secret key:

    oc create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n PROJECT-NAME
    
  4. If you created a service account name other than sysdig-agent: Edit sysdig-agent-daemonset-v2.yamlto provide your custom value:``

    serviceAccount: SERVICE-ACCOUNT
    
  5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information:

    collector:
    collector_port:
    ssl: #true or false
    check_certificate: #true or false
    
    • For SaaS, find the collector address for your region.
    • For On-prem, enter the collector endpoint defined in your environment.
    • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Set Up SSL Connectivity to the Backend.
  6. Apply the sysdig-agent-configmap.yaml file:

    oc apply -f sysdig-agent-configmap.yaml -n PROJECT-NAME
    
  7. FOR RHCOS ONLY: To enable the eBPF probe required for COS, uncomment the following parameters in sysdig-agent-daemonset-v2.yaml under the env section:`

    env:
      - name: SYSDIG_BPF_PROBE
        value: ""
    
  8. Apply the sysdig-agent-service.yaml file:

    oc apply -f sysdig-agent-service.yaml -n PROJECT-NAME
    

    This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

  9. Apply the daemonset-v2.yaml file:

    oc apply -f sysdig-agent-daemonset-v2.yaml -n PROJECT-NAME
    

    The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Verify Metrics in Sysdig Monitor and optionally, Connect to Sysdig Backend.

3.4 - Steps for Rancher

Preparation

General Requirements

You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.

Kernel Headers

The Sysdig agent requires a kernel module in order to be installed successfully on a host. On RancherOS distributions, the Unix version does not match the provided headers, and the agent might fail to install correctly. Therefore, you must install the kernel headers manually.

For RancherOS distributions, the kernel headers are available in the form of a system service and therefore are enabled using the ros service command:

$ sudo ros service enable kernel-headers-system-docker
$ sudo ros service up -d kernel-headers-system-docker

Some cloud hosting service providers supply pre-configured Linux instances with customized kernels. You may need to contact your provider’s support desk for instructions on obtaining appropriate header files, or for installing the distribution’s default kernel.

Installation

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

To deploy agents using Kubernetes manifests, you can download the manifest files, edit them as required, and deploy them using kubectl.

  1. Download the sample files:

    • sysdig-agent-clusterrole.yaml

    • sysdig-agent-daemonset-v2.yaml

    • sysdig-agent-configmap.yaml

    • sysdig-agent-service.yaml

  2. Create a namespace to use for the Sysdig agent.

    You can use whatever naming you prefer. This document uses sysdig-agent for both the namespace and the service account.

    The default service account name was automatically defined in sysdig-agent-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent.

    kubectl create ns sysdig-agent
    
  3. Create a secret key:

    kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
    
  4. Create a cluster role and service account, and define the cluster role binding that grants the Sysdig agent rules in the cluster role:

    kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
    kubectl create serviceaccount sysdig-agent -n sysdig-agent
    kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
    
  5. Edit sysdig-agent-configmap.yaml to add the collector address , port , and the SSL/TLS information:

    collector:
    collector_port:
    ssl: #true or false
    check_certificate: #true or false
    
  6. Apply the sysdig-agent-configmap.yaml file:

     kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
    
  7. Apply the sysdig-agent-service.yaml file:

    kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
    

This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

  1. Apply the daemonset-v2.yaml file :

    kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
    

The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Verify Metrics in Sysdig Monitor and optionally, Connect to Sysdig Backend.

3.5 - Steps for MKE

Mirantis Kubernetes Engine (MKE) formerly Docker Enterprise, is a managed environment for running Kubernetes to deploy containerized applications. As of Sysdig agent version 12.0.1, Sysdig supports all flavors of MKE.

MKE environments require eBPF probe to support agent installation.

The instructions below describe a standard MKE agent install and call out the special steps needed to install the eBPF probe.

Preparation

eBPF-Specific Requirements

  • Linux kernel versions 4.14 or above.

  • eBPF probe parameter, --set ebpf.enabled=true to install eBPF probe. See the instructions given below.

Installation Steps

Identify the appropriate endpoint depending on your Sysdig account region. For more information, see SaaS Regions and IP Ranges.

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Make sure to add the eBPF parameter to the helm command:

--set ebpf.enabled=true

Next Steps

You can continue with instructions in Verify Metrics in Sysdig Monitor and optionally, Connect to Sysdig Backend.

3.6 - Using Node Leases

The Sysdig agent uses Kubernetes Lease to control how and when connections are made to the Kubernetes API Server. This mechanism prevents overloading the Kubernetes API server with connection requests during agent bootup.

Kubernetes node leases are automatically created for agent version 12.0.0 and above. On versions prior to 12.0.0, you must configure node leases as given in the KB article.

Prerequisites

  • Sysdig Agent v11.3.0 or above

  • Kubernetes v1.14 or above

Types of Leases

The agent creates the following leases:

Cold Start

During boot up, the Sysdig agent connects to the Kubernetes API server to retrieve Kubernetes metadata and build a cache. The cold-start leases control the number of agents that build up this cache at any given time. An agent will grab a lease, build its cache, and then release the lease so that another agent can build its cache. This mechanism prevents agents from creating a “boot storm” which can overwhelm the API server in large clusters.

Delegation

In Kubernetes environments, two agents are marked as delegated in each cluster. The delegated agents are the designated agents to request more data from the API server and produce KubeState metrics. The delegation leases will not be released until the agent is terminated.

View Leases

To view the leases, run the following:

$ kubectl get leases -n sysdig-agent

You will see an output similar to the following:

NAME           HOLDER             AGE
cold-start-0                      20m
cold-start-1                      20m
cold-start-2                      21m
cold-start-3   ip-10-20-51-167    21m
cold-start-4                      21m
cold-start-5                      21m
cold-start-6                      20m
cold-start-7                      21m
cold-start-8                      20m
cold-start-9   ip-10-20-51-166   21m
delegation-0   ip-10-20-52-53    21m
delegation-1   ip-10-20-51-98    21m

Troubleshoot Leases

Verify Configuration

When lease-based delegation is working as expected, the agent logs show one of the following:

  • Getting pods only for node <node>

  • Getting pods for all nodes.

  • Both (occasionally on the delegated nodes)

Run the following to confirm that it is working:

$ kubectl logs sysdig-agent-9l2gf -n sysdig-agent | grep -i "getting pods"

The configuration is working as expected if the output on a pod is similar to the following:

2021-05-05 02:48:32.877, 15732.15765, Information, cointerface[15738]: Only getting pods for node ip-10-20-51-166.ec2.internal

Unable to Create Leases

The latest Sysdig ClusterRole is required for the agent to create leases. If you do not have the latest ClusterRole or if you have not configured the ClusterRole correctly, the logs show the following error:

Error, lease_pool_manager[2989554]: Cannot access leases objects: leases.coordination.k8s.io is forbidden: User "system:serviceaccount:sysdig-agent:sysdig-agent" cannot list resource "leases" in API group "coordination.k8s.io" in the namespace "sysdig-agent"

Contact Sysdig Support for help.

Optional Agent Configuration

Several configuration options exist for leases. It is recommended to not change the default settings unless prompted by Sysdig Customer Support.

Configuration

Default

Description

k8s_coldstart:
  enabled: <true/false>

true above agent versions 12.0.0

When true, the agent will attempt to create cold-start leases to control the number of agents which are allowed to build their cache at one time.

k8s_coldstart:
  max_parallel_cold_start: <int>

10

The number of cold-start leases to be created. This is the number of agents that can connect to the API Server simultaneously during agent initialization.

k8s_coldstart:
  namespace: <string>

sysdig-agent

The namespace to be created. This shouldn't be needed in agent version 12.0.0 because the DownwardAPI in the ClusterRole will provide the appropriate namespace.

k8s_coldstart:
  enforce_leader_election: <true/false>

false

When true, the agent will not fall back to the previous method if it cannot create leases.This can be useful if the previous method caused API Server problems.

k8s_delegation_election: <true/false>

true above agent versions 12.0.0

When true, the agent will create delegation leases to control which set of agents generate global cluster metrics.

4 - Agent Install: Non-Orchestrated

This section describes how to install the Sysdig agent directly on a Linux host, without using an orchestrator, such as Kubernetes or Mesos.

The agent can be installed in two ways:

  • As a standard container

  • As a non-containerized service

The steps for each flavor differ slightly depending on whether you are using the SaaS or on-premises version of the Sysdig platform.

If you are installing the Sysdig agent in an environment that has Kubernetes, use the Agent Install: Kubernetes instructions instead.

Prerequisites

  • See Agent Installation Requirements for information on the following:

    • Supported Linux distributions

    • Network connection

    • Sysdig access key

    • Cloud service providers (AWS, Google, and Microsoft Azure) and any steps you may need to configure to integrate the Sysdig agent.

  • kernel headers: The Sysdig agent requires kernel header files in order to install successfully on a host, and the agent is delivered with precompiled headers. If the hosts in your environment match the kernel versions included with the agent, no special action is needed .In some cases, the hosts in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernel headers manually. See About Kernel Headers and the Kernel Module for details.

  • Run any commands as root or with the sudo command.

  • Retrieve the Sysdig access key.

  • Collect the configuration parameters.

Configuration Options

OptionDescription
ACCESS_KEYThe agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.
tagsThe list of tags for the host where the agent is installed. For example: role:webserver, location:europe, role:webserver
COLLECTORThe collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges.
collector_portThe default is 6443.
SECUREUse a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default.
CHECK_CERTIFICATE(On-prem) Determines strong SSL certificate check for Sysdig Monitor on-premises installation. Set to true when using SSL/TLS to connect to the collector service to ensure that a valid SSL/TLS certificate is installed. For more information, see Set Up SSL Connectivity to the Backend.
ADDITIONAL_CONFOptional. Use this option to provide custom configuration values to the agent as environment variables. If provided, will be appended to agent configuration file. For example, For example, file log configuration.
bpfEnables eBPF probe. The path to the probe file that is either built or downloaded.

Installing Agent Using Containers

The Sysdig agent can be deployed as a docker container.

The commands below can also be copied from the Get Started page. In that case, your access key will already be included in the command automatically.

SaaS

Installing As Two Containers

The agent is installed by running sysdig/agent-kmodule, followed by running sysdig/agent-slim. See Installation Options for description about agent-slim and agent-kmodule.

Every host restart requires subsequent running of agent-kmodule and agent-slim containers.

  1. Collect the configuration parameters.

  2. Build and load the kernel module:

    If you are not using eBPF, use the following:

    docker run -it --privileged --rm --name sysdig-agent-kmodule \
    -v /usr:/host/usr:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules \
    quay.io/sysdig/agent-kmodule
    

    If you are using eBPF use the following:

    docker run -it --privileged --rm --name sysdig-agent-kmodule \
    -e SYSDIG_BPF_PROBE="" \
    -v /etc/os-release:/host/etc/os-release:ro \
    -v /root/.sysdig:/root/.sysdig \
    -v /usr:/host/usr:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    quay.io/sysdig/agent-kmodule
    
  3. Configure kernel module to load during system boot.

    If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.

    $ sudo mkdir -p /etc/modules-load.d
    $ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
    
  4. Run the agent module providing the access key and, optionally, user-defined tags:

    If you are not using eBPF, use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host \
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
    [-e TAGS=[TAGS]]
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    --shm-size=512m \
    quay.io/sysdig/agent-slim
    

    If you are using eBPF use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host\
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
    [-e TAGS=[TAGS]]
    -e SYSDIG_BPF_PROBE="" \
    -v /sys/kernel/debug:/sys/kernel/debug:ro \
    -v /root/.sysdig:/root/.sysdig \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    --shm-size=512m \
    quay.io/sysdig/agent-slim
    

Installing As Single Container (Legacy)

  1. Collect the configuration parameters.

  2. Run the agent container providing the access key and, optionally, user-defined tags:

    If you are not using eBPF, use the following:

    If you are not using eBPF, use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host\
     -e ACCESS_KEY=[ACCESS_KEY] \
     -e COLLECTOR=[COLLECTOR_ADDRESS] \
    -e TAGS=[TAGS] \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    --shm-size=512m \
    quay.io/sysdig/agent
    

    If you are using eBPF use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host\
     -e ACCESS_KEY=[ACCESS_KEY] \
     -e COLLECTOR=[COLLECTOR_ADDRESS] \
    -e TAGS=[TAGS] \
    -e SYSDIG_BPF_PROBE="" \
    -v /sys/kernel/debug:/sys/kernel/debug:ro \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    --shm-size=512m \
    quay.io/sysdig/agent
    

On-Premises

Installing As Two Containers

  1. Collect the configuration parameters:

  2. Build and load the kernel module:

    If you are not using eBPF, use the following:

    docker run -it --privileged --rm --name sysdig-agent-kmodule \
    -v /usr:/host/usr:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules \
    quay.io/sysdig/agent-kmodule
    

    If you are using eBPF use the following:

    docker run -it --privileged --rm --name sysdig-agent-kmodule \
    -e SYSDIG_BPF_PROBE="" \
    -v /etc/os-release:/host/etc/os-release:ro \
    -v /root/.sysdig:/root/.sysdig \
    -v /usr:/host/usr:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    quay.io/sysdig/agent-kmodule
    
  3. Configure kernel module to load during system boot.

    If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.

    $ sudo mkdir -p /etc/modules-load.d
    $ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
    
  4. Run the agent module providing the access key and, optionally, user-defined tags:

    If you are not using eBPF, use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host \
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
    -e SECURE=true \
    -e CHECK_CERTIFICATE=true \
    [-e TAGS=[TAGS]]
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    --shm-size=512m \
    quay.io/sysdig/agent-slim
    

    If you are using eBPF use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host \
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
    -e SECURE=true \
    -e CHECK_CERTIFICATE=true \
    [-e TAGS=[TAGS]]
    -e SYSDIG_BPF_PROBE="" \
    -v /sys/kernel/debug:/sys/kernel/debug:ro \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    --shm-size=512m \
    quay.io/sysdig/agent-slim
    

Installing As Single Container (Legacy)

  1. Collect the configuration parameters.

  2. Run the agent module providing the access key and, optionally, user-defined tags:

    If you are not using eBPF, use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host \
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
    -e SECURE=true \
    -e CHECK_CERTIFICATE=true \
    [-e TAGS=[TAGS]]
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    --shm-size=512m \
    quay.io/sysdig/agent
    

    If you are using eBPF use the following:

    docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host \
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
    -e SECURE=true \
    -e CHECK_CERTIFICATE=true \
    [-e TAGS=[TAGS]]
    -e SYSDIG_BPF_PROBE="" \
    -v /sys/kernel/debug:/sys/kernel/debug:ro \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    -v /usr:/host/usr:ro \
    --shm-size=512m \
    quay.io/sysdig/agent
    

Installing Agent as a Service on Linux Host

Use these instructions to install the agent on the host itself, not in a container. Install on each host in the environment.

The command lines below can also be copy/pasted from the Welcome wizard or the Settings>Agent Installation page in the Sysdig Monitor interface.

In that case, your access key will already be included in the command automatically.

The Sysdig agent depends on several python modules, some of which might not be installed on the hosts where the agent is running as a service. When the required dependencies are not available, the sdchecks component in the agent will report errors in the log files, such as:

 >> Error, sdchecks[0] ModuleNotFoundError: No module named 'posix_ipc'

To address these errors, install the missing modules using the pip install command.

SaaS

  1. Run the following command:

    curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] [--tags [TAGS]]
    

    Where [ACCESS_KEY] is your unique agent access key string. For example, 1234-your-key-here-1234. TAGS is an optional list of user-defined agent tags. For example, role:webserver,location:europe.

    See SaaS Regions and IP Ranges to find the collector endpoint for your region.

  2. Restart the agent and start the service:

    sudo systemctl enable dragent
    

On-Premises

  1. Run the following command:

    curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] --secure true --check_certificate true [--tags [TAGS]]
    

    For configuration parameters, see Configuration Options.

  2. Restart the agent and start the service:

    sudo systemctl enable dragent
    

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com:

user@host:~$ docker run --name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
-e COLLECTOR=collector-static.sysdigcloud.com \
-e COLLECTOR_PORT=6443 \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
--shm-size=512m \
quay.io/sysdig/agent-slim

Guidelines for Manual Agent Installation

In the following cases, we recommend that you manually install the agent.

  • Full control over the deployment process

  • Integration with configuration management tools

  • Custom kernel

  • Unsupported distribution

See Agent Install: Manual Linux Installation for more information.

5 - Agent Install: Manual Linux Installation

Manual installation of the native Linux agent is recommended in the following cases:

  • Full control over the deployment process

  • Integration with configuration management tools

  • Custom kernel

  • Unsupported distribution (within Debian/Fedora flavors)

Otherwise, you may want to follow Agent Install: Non-Orchestrated

Note: If you are installing the Sysdig agent in an orchestrated infrastructure such as Kubernetes, Mesos/Marathon, use the respective Installation Guides:

Prerequisites

  • Review the Agent Installation Requirements.

  • Collect the configuration parameters:

    • ACCESS_KEY: Your unique access key string. Inability to retrieve the key indicates that the administrator of your instance might have it turned off for non-admin users. Contact your Sysdig administrator to receive the key. If you still have issues please contact Sysdig Support.

    • TAGS: The optional parameter you can use to list one or more tags for this host. Tagging your hosts is highly recommended. Agent Tags allow you to sort nodes of your infrastructure into custom groups in Sysdig Monitor. Replace the [TAGS] parameter above with a comma-separated list of TAG_NAME:TAG_VALUE. For example: role:webserver,location:europe

    • COLLECTOR: The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations.

    • SECURE: Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default.

  • Run the commands as root or with sudo.

Installation

Follow the instructions for the appropriate Linux distribution:

Debian, Ubuntu

  1. Use the Sysdig Monitor GPG key, configure the apt repository, and update the package list:

    curl -s https://download.sysdig.com/DRAIOS-GPG-KEY.public | apt-key add -
    curl -s -o /etc/apt/sources.list.d/draios.list http://download.sysdig.com/stable/deb/draios.list
    apt-get update
    
  2. Install kernel development files.

    Note: The following command might not work with every kernel. Ensure that you customize the name of the package properly.

    apt-get -y install linux-headers-$(uname -r)
    
  3. Install, configure, and restart the Sysdig agent.

    apt-get -y install draios-agent
    echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
    echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
    echo collector: COLLECTOR_URL >> /opt/draios/etc/dragent.yaml
    echo ssl: true >> /opt/draios/etc/dragent.yaml
    service dragent restart
    

    See Prerequisites for the configuration parameters required.

CentOS, RHEL, Fedora, Amazon AMI, Amazon Linux 2

  1. Trust the Sysdig Monitor GPG key, configure the yum repository.

    $ rpm --import https://download.sysdig.com/DRAIOS-GPG-KEY.public
    $ curl -s -o /etc/yum.repos.d/draios.repo http://download.sysdig.com/stable/rpm/draios.repo
    
  2. Install the EPEL repository.

    Note: The following command is required only if DKMS is not available in the distribution. You can verify if DKMS is available with yum list dkms.

    The command below contains a sample release number; be sure to update with the correct release.

    $ rpm -i http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm
    
  3. Install kernel development files.

    Note: The following command might not work with every kernel. Make sure to customize the name of the package properly.

    $ yum -y install kernel-devel-$(uname -r)
    
  4. Install, configure, and start the Sysdig agent.

    $ yum -y install draios-agent
    $ echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
    $ echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
    $ echo collector: COLLECTOR_URL >> /opt/draios/etc/dragent.yaml
    $ echo ssl: true >> /opt/draios/etc/dragent.yaml
    $ echo secure: true >> /opt/draios/etc/dragent.yaml
    $ sudo systemctl enable dragent
    $ sudo systemctl start dragent
    

    See Prerequisites for the configuration parameters required.

  5. If you using a non-systemd Linux distribution, use the service command to start dragent.

    $ service dragent restart
    

Other Linux Distributions

The Sysdig agent is unsupported outside of the Debian, Fedora, and Amazon distributions.

6 - Agent Install: Amazon ECS

Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that helps to easily deploy, manage, and scale containerized applications.

This section describes how to install the Sysdig agent container on each underlying host in your ECS cluster. Once installed, the agent will automatically begin monitoring all of your hosts, service and tasks.

These instructions are valid only for ECS clusters using EC2 instances. For information on ECS Fargate clusters, see AWS Fargate Serverless Agents.

Installation

To install Sysdig agent on ECS, do the following:

  • Create an ECS task definition for the Sysdig agent.

  • Register the task definition in your AWS account.

  • Create a service with the previous task definition to run the Sysdig agent in each of the nodes of your ECS cluster.

Create an ECS Task Definition

  1. Collect the following configuration parameters:
  • ACCESS_KEY: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.
  • COLLECTOR: Use the collector address for your region. For more information, see SaaS Regions and IP Ranges.
  • TAGS: The list of tags for the host where the agent is installed. For example: role:webserver, location:europe, role:webserver
  1. Use the above values to customize the JSON snippet below and save it as a file named sysdig-agent-ecs.json.

Note that memory and cpu have both been set to 1024, depending of the size of your cluster you might want to tune those values. See Tuning Sysdig Agent for more information.

```yaml
{
  "family": "sysdig-agent-ecs",
  "containerDefinitions": [
    {
      "name": "sysdig-agent",
      "image": "quay.io/sysdig/agent-slim",
      "cpu": 1024,
      "memory": 1024,
      "privileged": true,
      "environment": [
        {
          "name": "ACCESS_KEY",
          "value": "$ACCESS_KEY"
        },
        {
          "name": "COLLECTOR",
          "value": "$COLLECTOR"
        },
	      {
          "name": "TAGS",
          "value": "$TAG1,TAG2"
        }
      ],
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/host/boot",
          "sourceVolume": "boot"
        },
        {
          "containerPath": "/host/dev",
          "sourceVolume": "dev"
        },
        {
          "readOnly": true,
          "containerPath": "/host/lib/modules",
          "sourceVolume": "modules"
        },
        {
          "readOnly": true,
          "containerPath": "/host/proc",
          "sourceVolume": "proc"
        },
        {
          "containerPath": "/host/var/run/docker.sock",
          "sourceVolume": "sock"
        },
        {
          "readOnly": true,
          "containerPath": "/host/usr",
          "sourceVolume": "usr"
        }
      ],
      "dependsOn": [
        {
          "containerName": "sysdig-agent-kmodule",
          "condition": "SUCCESS"
        }
      ]
    },
    {
      "name": "sysdig-agent-kmodule",
      "image": "quay.io/sysdig/agent-kmodule",
      "memory": 512,
      "privileged": true,
      "essential": false,
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/host/boot",
          "sourceVolume": "boot"
        },
        {
          "containerPath": "/host/dev",
          "sourceVolume": "dev"
        },
        {
          "readOnly": true,
          "containerPath": "/host/lib/modules",
          "sourceVolume": "modules"
        },
        {
          "readOnly": true,
          "containerPath": "/host/proc",
          "sourceVolume": "proc"
        },
        {
          "containerPath": "/host/var/run/docker.sock",
          "sourceVolume": "sock"
        },
        {
          "readOnly": true,
          "containerPath": "/host/usr",
          "sourceVolume": "usr"
        }
      ]
    }
  ],
  "pidMode": "host",
  "networkMode": "host",
  "volumes": [
    {
      "name": "sock",
      "host": {
        "sourcePath": "/var/run/docker.sock"
      }
    },
    {
      "name": "dev",
      "host": {
        "sourcePath": "/dev/"
      }
    },
    {
      "name": "proc",
      "host": {
        "sourcePath": "/proc/"
      }
    },
    {
      "name": "boot",
      "host": {
        "sourcePath": "/boot/"
      }
    },
    {
      "name": "modules",
      "host": {
        "sourcePath": "/lib/modules/"
      }
    },
    {
      "name": "usr",
      "host": {
        "sourcePath": "/usr/"
      }
    }
  ],
  "requiresCompatibilities": [
    "EC2"
  ]
}
```

Register a Task Definition

Once your task definition is ready, ensure that you register it in your AWS account:

aws ecs register-task-definition \
    --cli-input-json file://sysdig-agent-ecs.json

Run the Agent as an ECS Service

Using the ECS task definition you have created, create a service in the cluster that you want to monitor with Sysdig.

aws ecs create-service \
    --cluster $CLUSTER_NAME \
    --service-name sysdig-agent-svc \
    --launch-type EC2 \
    --task-definition sysdig-agent-ecs \
    --scheduling-strategy DAEMON

With the agent installed, Sysdig will begin auto-discovering your containers and other resources of your ECS environment.

Using ECS Anywhere

If you’re using ECS Anywhere, change the launch type to EXTERNAL when the service is created.

aws ecs create-service \
    --cluster $CLUSTER_NAME \
    --service-name sysdig-agent-svc \
    --launch-type EXTERNAL \
    --task-definition sysdig-agent-ecs \
    --scheduling-strategy DAEMON

Enable Log Driver

You can send the logs from the containers running the ECS tasks to the log groups in CloudWatch Logs. You can send agent container log files to AWS by enabling the log driver, awslogs. To do so:

  1. Add the following section to each of the container definitions you’ve created above:

                "logConfiguration": {
                    "logDriver": "awslogs",
                    "options": {
                        "awslogs-group": "$YOUR_LOG_GROUP",
                        "awslogs-region": "$AWS_REGION",
                        "awslogs-stream-prefix": "sysdig"
                    }
    
  2. Update your task definition and the service to enable the logs.

7 - Agent Install: IKS (IBM Cloud Monitoring)

IBM Cloud maintains the documentation for Sysdig agent installation on IBM Cloud Kubernetes Service (IKS).

For more information, see the IBM Cloud Monitoring documentation:

8 - Agent Install: Mesos | Marathon | DCOS

Marathon is the container orchestration platform for Mesosphere’s Datacenter Operating System (DC/OS) and Apache Mesos.

This guide describes how to install the Sysdig agent container on each underlying host in your Mesos cluster. Once installed, the agent will automatically connect to the Mesos and Marathon APIs to pull relevant metadata about the environment and will begin monitoring all of your hosts, apps, containers, and frameworks.

Prerequisites

  • Review the Agent Installation Requirements.

  • Collect the configuration parameters. You will need them to create the JSON file.

    • ACCESS_KEY: Your unique access key string. Inability to retrieve the key indicates that the administrator of your instance might have it turned off for non-admin users. Contact your Sysdig administrator to receive the key. If you still have issues please contact Sysdig Support.

    • COLLECTOR: The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges.

    • COLLECTOR_PORT: The default is 6443. It is used in environments with Sysdig’s on-premises backend installed.

    • SECURE: Use a secure SSL/TLS connection to send metrics to the collector. It is used in environments with Sysdig’s on-premises backend installed.

    • CHECK_CERT: Determines strong SSL certificate check for Sysdig Monitor on-premises installation. Set to true when using SSL/TLS to connect to the collector service to ensure that a valid SSL/TLS certificate is installed. It is used in environments with Sysdig’s on-premises backend installed.

Installation

In this three-part installation, you:

  • Deploy the Sysdig agent on all Mesos Agent (Slave) nodes, either automatically or by creating and posting a .json file to the leader Marathon API server.

  • Deploy the Sysdig agent on the Mesos Master nodes.

  • Special configuration steps: modify the Sysdig agent config file to monitor Marathon instances.

Deploy the Sysdig Agent on Mesos Agent Nodes

Preferred Option: Automatic install (DC/OS 1.11+)

If you’re using DC/OS 1.8 or higher, then you can find Sysdig agent in the Mesosphere Universe marketplace and install it from there.

It will automatically deploy the Sysdig agent container on each of your Mesos Agent nodes as a Marathon app.

Proceed to Deploy the Sysdig Agent.

Alternate Option: Post a .json file

If you are using a version of DC/OS earlier than 1.8 then:

  1. Create a JSON file for Marathon, in the following format. See configuration parameters for details.

    {
      "backoffFactor": 1.15,
      "backoffSeconds": 1,
      "constraints": [
        [
          "hostname",
          "UNIQUE"
        ]
      ],
      "container": {
        "docker": {
          "forcePullImage": true,
          "image": "sysdig/agent",
          "parameters": [],
          "privileged": true
        },
        "type": "DOCKER",
        "volumes": [
          {
            "containerPath": "/host/var/run/docker.sock",
            "hostPath": "/var/run/docker.sock",
            "mode": "RW"
          },
          {
            "containerPath": "/host/dev",
            "hostPath": "/dev",
            "mode": "RW"
          },
          {
            "containerPath": "/host/proc",
            "hostPath": "/proc",
            "mode": "RO"
          },
          {
            "containerPath": "/host/boot",
            "hostPath": "/boot",
            "mode": "RO"
          },
          {
            "containerPath": "/host/lib/modules",
            "hostPath": "/lib/modules",
            "mode": "RO"
          },
          {
            "containerPath": "/host/usr",
            "hostPath": "/usr",
            "mode": "RO"
          }
        ]
      },
      "cpus": 1,
      "deployments": [],
      "disk": 0,
      "env": {
        "ACCESS_KEY": "ACCESS_KEY=YOUR-ACCESS-KEY-HERE",
        "CHECK_CERT": "false",
        "SECURE": "true",
        "TAGS": "example_tag:example_value",
        "name": "sdc-agent",
        "pid": "host",
        "role": "monitoring",
        "shm-size": "350m"
      },
      "executor": "",
      "gpus": 0,
      "id": "/sysdig-agent",
      "instances": 1,
      "killSelection": "YOUNGEST_FIRST",
      "labels": {},
      "lastTaskFailure": {
        "appId": "/sysdig-agent",
        "host": "YOUR-HOST",
        "message": "Container exited with status 70",
        "slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
        "state": "TASK_FAILED",
        "taskId": "sysdig-agent.3bb0759d-3fa3-11e9-b446-c60a7a2ee871",
        "timestamp": "2019-03-06T00:03:16.234Z",
        "version": "2019-03-06T00:01:57.182Z"
      },
      "maxLaunchDelaySeconds": 3600,
      "mem": 850,
      "networks": [
        {
          "mode": "host"
        }
      ],
      "portDefinitions": [
        {
          "name": "default",
          "port": 10101,
          "protocol": "tcp"
        }
      ],
      "requirePorts": false,
      "tasks": [
        {
          "appId": "/sysdig-agent",
          "healthCheckResults": [],
          "host": "YOUR-HOST-IP",
          "id": "sysdig-agent.0d5436f4-3fa4-11e9-b446-c60a7a2ee871",
          "ipAddresses": [
            {
              "ipAddress": "YOUR-HOST-IP",
              "protocol": "IPv4"
            }
          ],
          "localVolumes": [],
          "ports": [
            4764
          ],
          "servicePorts": [],
          "slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
          "stagedAt": "2019-03-06T00:09:04.232Z",
          "startedAt": "2019-03-06T00:09:06.912Z",
          "state": "TASK_RUNNING",
          "version": "2019-03-06T00:09:04.182Z"
        }
      ],
      "tasksHealthy": 0,
      "tasksRunning": 1,
      "tasksStaged": 0,
      "tasksUnhealthy": 0,
      "unreachableStrategy": {
        "expungeAfterSeconds": 0,
        "inactiveAfterSeconds": 0
      },
      "upgradeStrategy": {
        "maximumOverCapacity": 1,
        "minimumHealthCapacity": 1
      },
      "version": "2019-03-06T00:09:04.182Z",
      "versionInfo": {
        "lastConfigChangeAt": "2019-03-06T00:09:04.182Z",
        "lastScalingAt": "2019-03-06T00:09:04.182Z"
      }
    }
    

    See Environment Variables for Agent Config File for the Sysdig name:value definitions.

    Complete the “cpus”, “mem” and “labels” (i.e. Marathon labels) entries to fit the capacity and requirements of the cluster environment.

  2. Update the created.json file to the leader Marathon API server:

    $ $curl -X POST http://$(hostname -i):8080/v2/apps -d @sysdig.json -H "Content-type: application/json"
    

Deploy the Sysdig Agent on Master Nodes

After deploying the agent to the Mesos Agent nodes, you will install agents on each of the Mesos Master nodes as well.

If any cluster node has both Mesos Master and Mesos Agent roles, do not perform this installation step on that node. It already will have a Sysdig agent installed from the procedure in step A. Running duplicate Sysdig agents on a node will cause errors.

Use the Agent Install: Non-Orchestrated instructions to install the agent directly on each of your Mesos Master nodes.

When the Sysdig agent is successfully installed on the master nodes, it will automatically connect to the local Mesos and Marathon (if available) API servers via http://localhost:5050 and http://localhost:8080 respectively, to collect cluster configuration and current state metadata in addition to host metrics.

Additional Configuration

In certains situations, you may need to add additional configurations to the dragent.yaml file:

  • If the Sysdig agent cannot be run directly on the Mesos API server.

  • If the API server is protected with a username/password.

Descriptions and examples are shown below.

Sysdig Agent Unable to Run on the Mesos API Server

Mesos allows multiple masters. If the API server can not be instrumented with a Sysdig agent, simply delegate ONE other node with an agent installed to remotely receive infrastructure information from the API server.

NOTE: If you manually configure the agent to point to a master with a static configuration file entry, then automatic detection/following of leader changes will no longer be enabled.

Add the following Mesos parameter to the delegated agent’s dragent.yaml file to allow it to connect to the remote API server and authenticate, either by:

a. Directly editing dragent.yaml on the host, or

b. Converting the YAML code to a single-line format and adding it as an ADDITIONAL_CONF argument in a Docker command.

See Understanding the Agent Configuration for details.

Specify the API server’s connection method, address, and port. Also specify credentials if necessary.

YAML example:

mesos_state_uri: http://[acct:passwd@][hostname][:port]
marathon_uris:
  - http://[acct:passwd@][hostname][:port]

Although marathon_uris: is an array, currently only a single “root” Marathon framework per cluster is supported. Multiple side-by-side Marathon frameworks should not be configured in order for our agent to function properly. Multiple side-by-side “root” Marathon frameworks on the same cluster are currently not supported. The only supported multiple-Marathon configuration is with one “root” Marathon and other Marathon frameworks as its apps.

Mesos API Server Requires Authentication

If the agent is installed on the API server but the API server uses a different port or requires authentication, those parameters must be explicitly specified.

Add the following Mesos parameters to the API server’s dragent.yaml to make it connect to the API server and authenticate with any unique account and password, either by:

a. Directly editing dragent.yaml on the host, or

b. Converting the YAML code to a single-line format and adding it as an ADDITIONAL_CONF argument in a Docker command.

See Understanding the Agent Configuration for details.

Specify the API server’s protocol, user credentials, and port:

mesos_state_uri: http://[username:password@][hostname][:port]
marathon_uris:
  - http://[acct:passwd@][hostname][:port]

*HTTPS protocol is also supported.

Troubleshooting: Turning Off Metadata Reception

In troubleshooting cases where auto-detection and reporting of your Mesos infrastructure needs to be temporarily turned off in a designated agent:

  1. Comment out the Mesos parameter entries in the agent’s dragent.yaml file.

    Example parameters to disable: mesos_state_uri, marathon_uris

  2. If the agent is running on the API server (Master node) and auto-detecting a default configuration, you can add the line:

    mesos_autodetect: false

    either directly in the dragent.yaml file or as an ADDITIONAL_CONF parameter in a Docker command.

  3. Restart the agent.

9 - Airgapped Agent Installation

Airgapped environments are those that do not have network access to the public internet.

At startup, the agent will try to compile its own version of the probes, provided kernel header packages are installed on the host. Failing that, the agent will try to download pre-compiled probes, sysdigcloud-probe-<suffix>.ko or sysdigcloud-probe-bpf-<suffix>.o, from the Sysdig download site over the internet.

In an airgapped environemnt, you cannot download these artifacts. Therefore, before installing the agent, you will have to compile sysdigcloud-probe-<suffix> for each kernel version in your environment, and make it available to the installed agents through an internally accessible URL.

Prerequisites

  • A machine with internet access where you can download the required artifacts
  • A machine in your airgapped environment where you can build your probes
  • Tool to transfer artifacts to the machine in your airgapped environment
  • Docker installed

Overview

Sysdig provides a tool, named the probe builder, to help you build the probes for different kernels and for a specific agent version. After downloading the required artifacts on a machine connected to the internet, you can copy them to an airgapped host, build your own probes, and make them available to your agent installations.

On a Machine with Internet Connectivity

Prepare the Sysdig Probe Builder Images

On a machine with internet connectivity, build the Sysdig probe builder container images and create a tar file of the images.

  1. Get the probe builder source code from the repository:

    $ git clone https://github.com/draios/probe-builder
    
  2. Build the container image for the probe builder:

    $ docker build -t airgap/sysdig-probe-builder probe-builder/
    
  3. Build the images for each supported distribution-compiler combination:

    $ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock airgap/sysdig-probe-builder:latest -P -b airgap/
    

    Running this command will create a different image tag for each supported combination of distribution-compiler, with the distro-compiler information suffixed to the image name, airgap/sysdig-probe-builder. For example, airgap/sysdig-probe-builder:centos-gcc4.8.

  4. Save all the above images to a tar archive:

    $ docker save airgap/sysdig-probe-builder | gzip > builders.tar.gz
    
  5. (optional) If you are building probes for the Ubuntu kernels, you will also need an ubuntu:latest image on your airgapped host. You can build it as follows:

    $ docker pull ubuntu
    $ docker save ubuntu | gzip > ubuntu.tar.gz
    

Download the Kernel Packages

Download your kernel packages. For more information, see Download Kernel Packages.

Download Probe Source Code

You need to download the probe source code for a specific agent version you want to build your probes for.

For example, for agent version 12.0.0 you would use:

$ git clone https://github.com/draios/agent-libs
$ cd agent-libs
$ git archive agent/12.0.0 --prefix sysdig/ | gzip > sysdig.tar.gz

Transfer the Downloaded Files

Copy the artifacts you have built to the airgapped host machine:

  • builders.tar.gz
  • ubuntu.tar.gz (if needed, see above)
  • sysdig.tar.gz
  • Kernel packages

On the Airgapped Host

Load the Builder Images

$ zcat builders.tar.gz | docker load

Unpack the Sysdig Source

$ tar xzf sysdig.tar.gz

Running this command will create the sysdig/ directory in the current directory.

Move the Kernel Packages to a Dedicated Location

Make sure you have all the downloaded kernel package artifacts in a single directory, /directory-containing-kernel-packages/, for each distribution you want to support.

Run the Probe Builder

Now that you have all your requirements in place, you can run the main probe builder:

$ docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /a-directory-with-some-free-space/:/workspace \
  -v /wherever-you-unpacked/sysdig/:/sysdig \
  -v /directory-containing-kernel-packages/:/kernels \
  airgap/sysdig-probe-builder:latest -B -b airgap/ -- \
  -p sysdigcloud-probe -v 12.0.0 -k CustomCentOS

The probes will appear in /a-directory-with-some-free-space/output. That directory can be served over HTTP and the URL to the server used as SYSDIG_PROBE_URL when loading the module (e.g. agent-kmodule container). As an example, the following sections describe how you can deploy your own nginx server within your cluster and upload your probes there.

Serve Your Pre-Compiled Probes

Set up a local repository to host the pre-compiled kernel module. Here is an example with nginx:

$ docker run --rm -v /a-directory-with-some-free-space/output:/usr/share/nginx/html/stable/sysdig-probe-binaries -p 80:80 nginx

Note down the URL and use it as the SYSDIG_PROBE_URL while installing the agent.

​ See Run the Probe Builder.

Run the Probe Builder

$ docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /sysdigcloud-probe/:/workspace \
  -v /wherever-you-unpacked/sysdig/:/sysdig \
  -v /directory-containing-kernel-packages/:/kernels \
  airgap/sysdig-probe-builder:latest -B -b airgap/ -- \
  -p sysdigcloud-probe -v 12.0.0 -k CustomCentOS

The probes will appear in /sysdigcloud-probe/output. This directory can be served over HTTP and the URL to the server used as SYSDIG_PROBE_URL when loading the module. For example, agent-kmodule container.

Install Agent in a Docker Environment

  1. Install Sysdig agent by pointing SYSDIG_PROBE_URL to the local repository:

    For docker-based installations:

    $ docker run -d --name sysdig-agent --restart always --privileged --net host --pid host -e ACCESS_KEY=WWWWW-YYYY-XXXX-ZZZZ-123456789 -e SECURE=true -e SYSDIG_PROBE_URL=http://www.mywebserver.net:80/ -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro --shm-size=512m sysdig/agent
    

    Where -e SYSDIG_PROBE_URL=http://www.mywebserver:80/ is the local nginx web server with the loaded module.

    Note: To use HTTPS communication with a self-signed or untrusted certificate, use the -e SYSDIG_PROBE_INSECURE_DOWNLOAD=true environment variable in the above command.

  2. Check the agent log. If the installation is successful, you will see a message as follows:

    Evaluating override of environment variables
    
    Trying to download precompiled module from http://mywebserver:80/stable/sysdig-probe-binaries/sysdigcloud-probe-<version>
    
    Download succeeded
    
  3. Continue with the instructions in Agent Install: Non-Orchestrated.

Install Agent in a Kubernetes Environment

  1. Open your agent daemonset and update the SYSDIG_PROBE_URL to point to the local repository:

    - name: SYSDIG_PROBE_URL
      value: http://www.mywebserver:80/
    

    If you would like to use secure communication with a self-signed or untrusted certificate, apply the SYSDIG_PROBE_INSECURE_DOWNLOAD environment variable.

    - name: SYSDIG_PROBE_INSECURE_DOWNLOAD
      value: true
    
  2. Continue with the instructions in Agent Install: Kubernetes.

10 - Identify Agent Version

Use one of the following methods to determine the version of the agents installed in your environment:

Explore

Segmenting metrics by using agent.version shows the installed versions of agents in your environment. For example, segment the uptime metric across your environment by using agent.version . Hover over the graph to see the list of agent versions.

The image shows the list of agent versions in n/a.

Dashboard

Use the Sysdig Agent Health Dashboard to determine the agent versions:

  1. Log in to the Sysdig Monitor.

  2. Select Dashboards and expand Host Infrastructure Dashboards.

  3. Open the Sysdig Agent Health & Status template or create your own from the template.

    The Sysdig Agent and Health & Status Dashboard shows the agent version corresponding to each host in your environment.