• 1:
• 1.1:
• 1.2:
• 2:
• 3:
• 3.1:
• 3.2:
• 3.3:
• 3.4:
• 4:
• 5:
• 6:
• 7:
• 8:
• 9:
• 10:
• 11:
• 12:

1 -

Host Requirements for Agent Installation

Sysdig agents can be installed on a wide array of Linux hosts. Check your environment to ensure it meets the minimum supported platform, operating system, runtime, and orchestration requirements and uses the appropriate installation instructions.

Versioning Scheme

The versioning scheme for agent releases has been updated with version 9.5.0. Previous versions used the format, <version number><hotfix> such as, 0.94.0.

Sysdig is aligning version numbers to the rest of the product. The new version number reflects the maturity of the agent software over the last several years. Starting v9.5.0, all agent versions are numbered as Major.minor.hotfix. 

We encourage users to be on the latest version of the agent. Starting with v9.6.0, we support n-3 versions backbased on the minor number. For example, if the release is v9.6.0, we will support n-3 versions back, for example, to 9.3.0. The old version scheme is 0.93.0.

Agent Installation Requirements

Cloud Platform or Private Data Center

Cloud Platforms Supported

• AWS AWS Elastic Kubernetes Service (EKS), Elastic Cloud Compute (EC2), AWS Elastic Container Service (ECS) on AWS Cloud or AWS Outpost

• Google Cloud Provider (GCP) and Google Kubernetes Engine (GKE).

• Microsoft Azure and Microsoft Azure Container Service (AKS)

• IBM: Including IBM Cloud Kubernetes Service (IKS)

Private Data Center

See the Supported Linux Distributions table, below.

Container Runtimes

The agent supports the detection of Docker, RKT, LXC, containerd, CRI-O, and Mesos containers.

Prerequisites for CRI-O Environments

Agent artifacts are stored on the Docker registry. In CRI-O and Kubernetes environments, Docker is not a specified registry by default. In order to prevent image pull failure for Sysdig agent installations in CRI-O environments, add docker.ioto the CRI-O configuration file:

1. Edit /etc/crio/crio.conf.

2. Add registries = ["docker.io"] to the crio.conf file.

   For example:

   # registries is used to specify a comma separated list of registries to be used
   # when pulling an unqualified image (e.g. fedora:rawhide).
   registries = [
   “registry.example.xyz”,
   “docker.io”
   ]
   

3. Restart CRI-O.

Prerequisites for Podman Environments

Sysdig agent supports running as a Podman container.

• Enable Podman API service for all the users.

  The agent will not able to collect Podman-managed container metadata, such as the container name, if the API service is not enabled.

• Secure rules and policies that depend on container metadata other than the container ID will not work.

• Pausing and terminating containers will not work because Policy actions for Podman are not supported.

• The containers started as a non-root user will have the podman_owner_uid label associated with it if the API service is enabled for that user. The value of podman_owner_uid will be the numeric user ID corresponding to the user that started the container.

Supported Container Registries

• Quay.io

  For example, to pull the latest agent container from Quay.io:

  docker pull quay.io/sysdig/agent
  

• Docker Hub

  For example, to pull the latest agent container from Docker Hub:

  docker pull sysdig/agent
  

Supported Linux Distributions

Orchestrator: Yes/No

• If NO orchestrator is used, follow the installation instructions for Agent Install: Non-Orchestrated .

• If you ARE using an orchestrator what kind are you using?

Supported Open-Source Orchestrators

Supported Container Platforms

Supported Java Versions and Vendors

The Sysdig agent supports only:

Java versions: 7 and above

Vendors: Oracle, OpenJDK

For Java-based applications (Cassandra, Elasticsearch, Kafka, Tomcat, Zookeeper and etc.), the Sysdig agent requires the Java runtime environment (JRE) to be installed to poll for metrics (beans).

If the Docker-container-based Sysdig agent is installed, the JRE is installed alongside the agent binaries and no further dependencies exist. However, if you are installing the service-based agent (non-container) and you do not see the JVM/JMX metrics reporting, your host may not have the JRE installed or it may not be installed in the expected location: usr/bin/java

Resource Limits

The resource requirements of the agent are subjective to the size and load of the host— more activity equates to more resources required. At a minimum, the agent requires 2% of the total CPU and 512MiB of memory.

It is typical to see between 5-20KiB/s of bandwidth consumed—different variables can increase the throughput required such as the number of metrics, events, Kubernetes objects, and which products and features are enabled. When a Sysdig Capture is being collected, you can expect to see a spike in bandwidth while the capture file is being ingested.

We do not recommend placing bandwidth shaping or caps on the agent to ensure data can be sent to our collection service.

Supported Web Browsers

Sysdig supports, tests, and verifies the latest versions of Chrome and Firefox.

Other browsers may also work, but are not tested in the same way.

Additional Requirements

Access key

The installation of the Sysdig agent requires an access key. This key and the agent installation instructions are presented to you after activating your account and using a web-based wizard upon initial login.

The same information can also be found in the Settings > Agent Installation menu of the web interface after logging in. See Agent Installation: Overview and Key for details.

Network connection

A Sysdig agent (containerized or native) is installed into each host being monitored and will need to be able to connect to the Sysdig Monitor backend servers to report host metrics. The agent must be able to reach the Sysdig Collector addresses. For example, for US East, it is ‘collector.sysdigcloud.com’ (via multiple IPs) over port tcp/6443 . See Sysdig Collector Ports for supported ports for other regions.

The agent supports the HTTP proxy for communicating with Sysdig backend components. For more information, see Enable HTTP Proxy for Agents.

Tags

Tagging your hosts is highly recommended. Tags allow you to sort nodes of your infrastructure into custom groups in Sysdig Monitor.

Replace the [TAGS] parameter in the configuration file with a comma-separated list in the form of TAG_NAME:TAG_VALUE. For example, role:webserver,location:europe.

See Understanding the Agent Config Files.

TracePoints Support

All supported distribution released kernels have this support but if creating a custom kernel, it must support the following options:

CONFIG_TRACEPOINTS

CONFIG_HAVE_SYSCALL_TRACEPOINTS

See Also

Kernel Header Troubleshooting

1.1 -

Tuning Sysdig Agent

The resource requirements for the Sysdig agent are subjective to the size and load of the host. Increased activity equates to higher resource requirements. At a minimum, the agent requires 2% of the total CPU and 512 MiB of memory.

You might see 5 to 20 KiB/s of bandwidth consumed. Different variables can increase the throughput required. For example:

• The number of metrics

• The number of events

• Kubernetes objects

• Products and features enabled

When a Sysdig Capture is being collected, you can expect to see a spike in the bandwidth while the capture file is being ingested.

Sysdig does not recommend placing bandwidth shaping or caps on the agent to ensure that data is sent to the Sysdig Collection service.

In general, in larger clusters, the agent requires more memory, and in servers with a high number of cores, the agent requires more CPU cores to monitor all the system calls. You will use CPU cores on the host and the Kubernetes nodes visible to the agent as proxies for the rate of events processed in the agent.

Similarly, there are different factors that are at play, and considering all the factors, we recommend the following:

Small: CPU core count <= 8. Kubernetes nodes <=10

Medium: 8 < CPU core count <= 32. 10 < Kubernetes nodes <= 100

Large: CPU core count > 32. Kubernetes nodes > 100

While you can expect the behavior with the given numbers to be better than simply using the default values, Sysdig cannot guarantee that resource allocation will be correct for all the cases.

Note that the agent has its own memory watchdog to prevent runaway memory consumption on the host in case of memory leaks. The default values of the watchdog are specified in the following agent configuration file.

watchdog:
  max_memory_usage_mb: 1024
  max_memory_usage_subprocesses:
    sdchecks: 128
    sdjagent: 256
    mountedfs_reader: 32
    statsite_forwarder: 32
    cointerface: 256

All the values are given in MiB.

For example, to match the agent watchdog settings with large values, the agent configuration would be:

watchdog:
  max_memory_usage_mb: 2048
  max_memory_usage_subprocesses:
    sdchecks: 128
    sdjagent: 256
    mountedfs_reader: 32
    statsite_forwarder: 32
    cointerface: 4096

1.2 -

Kernel Header Troubleshooting

In addition to the information on Host Requirements for Agent Installation, this page describes how the agent uses kernel headers and tips on troubleshooting, if needed.

About Kernel Headers and the Kernel Module

The Sysdig agent requires a kernel module in order to install successfully on a host. This can be obtained in three ways:

1. Agent compiles the module using kernel headers.

   If the hosts in your environment already have kernel header files pre-installed, no special action is needed. Or you can install the kernel headers manually; see below.

2. Agent auto-downloads precompiled modules from Sysdig’s AWS storage location.

   If the headers are not already installed but the agent is able to auto-download, no special action is needed. If there is no internet connectivity, you can use method 3 (download from an internal URL).

3. Agent downloads precompiled modules from an internal URL.

   Use the environment variable SYSDIG_PROBE_URL. See also Understanding the Agent Config Files. Contact Sysdig support for assistance.

To Install Kernel Headers

In some cases, the host(s) in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernal headers manually.

Debian-Style

For Debian-syle distributions, run the command:

apt-get -y install linux-headers-$(uname -r)

RHEL-Style

For RHEL-style distributions, run the command:

yum -y install kernel-devel-$(uname -r)

RancherOS

For RancherOS distributions, the kernel headers are available in the form of a system service and therefore are enabled using the ros service command:

sudo ros service enable kernel-headers-system-docker
sudo ros service up -d kernel-headers-system-docker

NOTE: Some cloud hosting service providers supply pre-configured Linux instances with customized kernels. You may need to contact your provider’s support desk for instructions on obtaining appropriate header files, or for installing the distribution’s default kernel.

To Correct Kernel Header Errors in AWS AMI

During an agent installation in an Amazon machine image (AMI) you may encounter the following errors while the installer is trying to compile the Sysdig kernel module:

Errors

• “Unable to find kernel development files for the current kernel version” or

• “FATAL: Module sysdigcloud-probe not found”

This indicates your machine is running a kernel in an older AMI for which the kernel headers are no longer available in the configured repositories. The issue has to do with Amazon purging packages in the yum repository when new Amazon Linux machine images are released.

The solution is either to update your kernel to a version for which header files are readily available (recommended), or perform a one-time installation of the kernel headers for your older AMI.

Option 1: Upgrade Your Host’s Kernel

First install a new kernel and reboot your instance:

sudo yum -y install kernel
sudo reboot

After rebooting, check to see if the host is reporting metrics to your Sysdig account. If not, you may need to issue three more commands to install the required header files:

sudo yum -y install kernel-devel-$(uname -r)
sudo /usr/lib/dkms/dkms_autoinstaller start
sudo service dragent restart

Option 2: Install Older Kernel Headers

Although it is recommended to upgrade to the latest kernel for security and performance reasons, you can alternatively install the older headers for your AMI.

Find the the AMI version string and install the appropriate headers with the commands:

releasever=$(cat /etc/os-release | grep 'VERSION_ID' | grep -Eo "[0-9]{4}\.[0-9]{2}")
sudo yum -y --releasever=${releasever} install kernel-devel-$(uname -r)

Issue the remaining commands to allow the Sydig Agent to start successfully:

sudo /usr/lib/dkms/dkms_autoinstaller start
sudo service dragent restart

Reference: Find Your AWS Instance Image Version

The file /etc/image-id shows information about the original machine image with which your instance was set up:

[ec2-user ~]$ cat /etc/image-id
image_name="amzn-ami-hvm"
image_version="2017.03"
image_arch="x86_64"
image_file="amzn-ami-hvm-2017.03.0.20170401-x86_64.ext4.gpt"
image_stamp="26a3-ed31"
image_date="20170402053945"
recipe_name="amzn ami"
recipe_id="47cfa924-413c-d460-f4f2-2af7-feb6-9e37-7c9f1d2b"

This file will not change as you install updates from the yum repository.

The file /etc/system-release will tell what version of the AWS image is currently installed:

[ec2-user ~]$ cat /etc/system-release
Amazon Linux AMI release 2017.03

2 -

Quick Install Sysdig Agent on Kubernetes

Sysdig provides a single-line Sysdig Agent installer to help you get started quickly in Kubernetes environments. The code for using the installer is also presented in the Get Started pages of the Sysdig Monitor and Sysdig Secure UIs, with some of the values auto-completed.

This page documents the single-line installer options in more detail.

You can also access the help by using the following command:

 $ ./install-agent-kubernetes --help

Access from Get Started Pages

To access the quick-install code pre-filled with some of your environment variables:

1. Log in as admin to Sysdig Monitor or Sysdig Secure.

2. Select the Get Started page (rocket icon).

3. Choose Install the Agent, select the appropriate deployment type (e.g. Helm or Kubernetes), and copy the auto-generated code, filling in remaining variable values as required.

   

Sample Usage

$ install-agent-kubernetes [-a | --access_key <value>] [-t | --tags <value>] [-c | --collector <value>] [-cp | --collector_port <value>] [-s | --secure <value>] [-cc | --check_certificate <value>] [-ns | --namespace | --project <value>] [-ac | --additional_conf <value>] [-op | --openshift] [-as | --agent_slim] [-av | --agent_version <value>] [-ae | --api_endpoint <value> ] [-na | --nodeanalyzer ] [-ia | --imageanalyzer ] [-am | --analysismanager <value>] [-ds | --dockersocket <value>] [-cs | --crisocket <value>] [-cv | --customvolume <value>] [-cn | --cluster_name <value>] [-r | --remove ] [-h | --help]

Options

3 -

Agent Install: Kubernetes

This document describes how to install a regular Sysdig agent container in a Kubernetes environment. This document assumes you will run the agent container as a Kubernetes pod, which then enables the Sysdig agent automatically to detect and monitor your Kubernetes environment.

If you want to install a slim agent, see Install Slim Agent.

It is relevant for any platform where Kubernetes is deployed, including Amazon environments (EKS, EC2, ECS), Azure Container Service (AKS), Google Kubernetes Engine (GKE), Red Hat OpenShift, and IBM Cloud Kubernetes Service (IKS).

You use DaemonSets to deploy agents on every node in your Kubernetes environment. Once deployed, Sysdig Monitor automatically begins monitoring all of your hosts, apps, pods, and services and automatically connects to the Kubernetes API server to pull relevant metadata about the environment. If licensed, Sysdig Secure launches with default policies that you can view and configure to suit your needs. You can access the front-end web interfaces for Sysdig Monitor and Sysdig Secure immediately.

Prerequisites

• A supported distribution. See Host Requirements for Agent Installation for details.

• Kubernetes v 1.9+: The agent installation on Kubernetes requires using v1.9 or higher because the APIs used to fetch kubernetes metadata are only present in v1.9+.

• Sysdig account and access key: Request a trial or full account at Sysdig.com and click the Activate Account button. You create a Sysdig user name and password.

  The Getting Started Wizard provides an access key.

Runtime Support: CRI-O and Containerd

By default, Sysdig agents deployed in Kubernetes automatically detect metadata from containerd and CRI-O (in addition to Docker), as long as the prerequisites are fulfilled.

After reviewing the information on this page, continue with the Sysdig agent installation steps: Kubernetes Agent Installation Steps.

Containerd Support

As of agent version 0.88.1, the Sysdig agent will automatically detect containerd metadata (as well as any Docker metadata) in your environment, as long as the prerequisites are fulfilled.

Prerequisites

• Agent version: Sysdig agent version 0.88.1 or higher

  NOTE: If you are upgrading from an earlier version of the agent, you must also download the latest sysdig-agent-daemonset-v2.yamlfrom GitHub.

• Configuration parameter: In the agent config file, new_k8s: true must be set.

  As of agent 9.6.0, new_k8s is enabled by default.

  See Enable Kube State Metrics and Cluster Name below for details on editing the config file.

• Kubernetes-only: The containerd API must support CRI (a Kubernetes runtime interface).

Results in the Sysdig Monitor UI

If the Sysdig agent detects containerd metadata, it will be reported in the front end as follows:

• Explore/Dashboard views: The icon next to container-specific items (container.name, container.id, etc.) shows whether it’s a Docker or containerd object.

  

• Spotlight: Updated for containerd display.

• Events: Containerd events die and oom are enabled by default.

  Events create and exit are also supported.

  

CRI-O Support

The Sysdig agent will automatically detect CRI-O metadata (as well as any Docker and/or containerd metadata) in your environment, as long as the Prerequisites are fulfilled.

Prerequisites

• Platform version: Sysdig SaaS March 2019or higher

• Agent version: Sysdig agent v 0.89.4 March 27, 2019 or higher.

  NOTE: If you are upgrading from an earlier version of the agent, you must also download the latest sysdig-agent-daemonset-v2.yamlfrom GitHub.

• Configuration parameter: In the agent config file, new_k8s: true must be set.

  See Enable Kube State Metrics and Cluster Name below for details on editing the config file.

• Kubernetes-only: The API must support CRI (a Kubernetes runtime interface).

Results in the Sysdig Monitor UI

• Events: There are no CRI-O events, so the Events pane remains unchanged.

• Explore/Dashboard views: The icon next to container-specific items (container.name, container.id, etc.) shows CRI-O type.

• Supported Metrics: By default, the same metrics are supported for CRI-O as for Docker and containerd, except for image id (container.image.id).

Enable Image ID Metrics with cri: extra_queries

To enable image id metrics, edit the agent configuration file dragent.yaml to contain the following:

cri:
  extra_queries: true

See Understanding the Agent Config Files for more information on editing dragent.yaml.

Complete the Installation

Choose the appropriate link to complete the installation steps:

• Steps for Kubernetes (Vanilla)

  All environments except IKS, GKE, or those using OpenShift.

• Steps for GKE

• Steps for Rancher

• Steps for OpenShift

3.1 -

Steps for Kubernetes (Vanilla)

Preparation

Kernel Headers

The Sysdig agent requires kernel header files to install successfully on a host.

If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.

In some cases, the host(s) in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernel headers manually.

To do so:

For Debian-style distributions, run the command:

apt-get -y install linux-headers-$(uname -r)

For RHEL-style distributions, run the command:

yum -y install kernel-devel-$(uname -r)

Background info: see also About Kernel Headers and the Kernel Module.

Background Info

You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Host Requirements for Agent Installation for additional context, if desired.

Installation Steps

To deploy agents using Kubernetes daemonsets, you will download the following configuration files, edit them as required, and deploy them.

• sysdig-agent-clusterrole.yaml

• sysdig-agent-service.yaml

• sysdig-agent-daemonset-v2.yaml

• sysdig-agent-configmap.yaml

Deploy the Agents

1. Download the sample files:

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create a namespace to use for the Sysdig agent.

   You can use whatever naming you prefer. In this document, we used sysdig-agent for both the namespace and the service account.

   The default service account name was automatically defined in sysdig-agent-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent.

   kubectl create ns sysdig-agent
   

3. Create a secret key:

   kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. Create a cluster role and service account, and define the cluster role bindingthat grants the Sysdig agent rules in the cluster role, using the commands:

   kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   kubectl create serviceaccount sysdig-agent -n sysdig-agent
   kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector address , port , and the SSL/TLS information:

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Step 5 Set Up SSL Connectivity to the Backend.

6. (All installs) Apply the sysdig-agent-configmap.yaml file:

   kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. (All installs) Apply the sysdig-agent-service.yaml file:

   kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
   

   This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

8. (All installs) Apply the daemonset-v2.yaml file :

   kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
   

The agents will be deployed. See Getting Started with Sysdig Monitor to view some metrics in the Sysdig Monitor UI. You can make further edits to the configmap as described below.Getting Started with Sysdig Monitor

Enable Kube State Metrics and Cluster Name

These steps are optional but recommended.

1. Edit sysdig-agent-configmap.yaml to uncomment the line: new_k8s: true

   This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.

   For more information, see the Kube State Metrics entry in the Sysdig blog.

   As of agent 9.6.0, new_k8s is enabled by default.

2. Edit sysdig-agent-configmap.yaml to uncomment the line: **k8s_cluster_name: **and add your cluster name.

   Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.

   Note: Alternatively, if you assign a tag with “cluster” in the tag name, Sysdig Monitor will display that as the Kubernetes cluster name.

3. Apply the configmap changes using the command:

   kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

4. Proceed to verify the metrics in the Sysdig Monitor UI.

Additional Options

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com.

To configure the collector IP in a Kubernetes SaaS instance:

1. Open sysdig-agent-configmap.yaml in a text editor.

2. Uncomment the following lines:

   • collector:

   • collector_port

3. Set the collector: value to collector-static.sysdigcloud.com

   See SaaS Regions and IP Ranges and identify the correct URL associated with your Sysdig collector and region.

4. Set the collector_port: value to 6443

5. Save the file.

The example file below shows how the sysdig-agent-configmap.yaml file should look after configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sysdig-agent
data:
  dragent.yaml: |
    ### Agent tags
    # tags: linux:ubuntu,dept:dev,local:nyc

    #### Sysdig Software related config ####

    # Sysdig collector address
    collector: collector-static.sysdigcloud.com

    # Collector TCP port
    collector_port: 6443

    # Whether collector accepts ssl/TLS
    ssl: true

    # collector certificate validation
    ssl_verify_certificate: true

    # Sysdig Secure
    security:
                enabled: true

    #######################################
    # new_k8s: true
    # k8s_cluster_name: production

Verify Metrics in Sysdig Monitor UI

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

The steps below give one way to do the check.

1. Access Sysdig Monitor:

   SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

   For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

   Log in with your Sysdig user name and password.

2. Select the Explore tab to see if metrics are displayed.

3. (Once you have enabled new_k8s:true): To verify that kube state metrics and cluster name are working correctly: Select the Explore tab and create a grouping by kubernetes.cluster.name and kubernetes.pod.name.

   As of agent 9.6.0, new_k8s is enabled by default.

   

4. Select an individual container or pod to see details.

   

3.2 -

Steps for GKE

Google Kubernetes Engine (GKE) is a managed environment for running Kubernetes in Google Cloud, in order to deploy containerized applications. As of Sysdig agent version 0.88, Sysdig supports all flavors of GKE, including Ubuntu and GKE’s default Container-Optimized OS (COS).

Note that the standard Sysdig agent cannot be installed on GKE COS because Sysdig relies on a kernel module that COS does not allow. To accommodate this limitation Sysdig has developed an alternate probe built on eBPF, a “universal in-kernel virtual machine.”

The instructions below describe a standard GKE agent install and call out the special steps needed to install the eBPF probe if you are using COS.

Preparation

Open Port 6443 for Agent Egress

Because GKE uses stateful firewalls, you must actively open port 6443 for the Sysdig agent outbound traffic.

GKE COS/eBPF-Specific Requirements

• Linux kernel version >= 4.14.

• When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below. Note that the eBPF probe is supported only in GKE COS environments.

Background Info

You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Host Requirements for Agent Installation for additional context, if desired.

Installation Steps

To deploy agents using Kubernetes daemonsets, you will download the following configuration files, edit them as required, and deploy them.

• sysdig-agent-clusterrole.yaml

• sysdig-agent-daemonset-v2.yaml

• sysdig-agent-configmap.yaml

Deploy the Agents

1. Download the sample files:

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create a namespace to use for the Sysdig agent.

   You can use whatever name you want. In this document, we used sysdig-agent for both the namespace and the service account.

   kubectl create ns sysdig-agent
   

3. Create a secret key:

   kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. If you are running Kubernetes 1.6 or higher, you must

   Grant your user the ability to create roles in Kubernetes by running the following command (see Google documentation for more):

   kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org
   

   Create a service account for the Sysdig agent using the clusterrole.yaml file.

   The Sysdig agent must be granted read-only access to certain Kubernetes APIs, which the agent uses to populate metadata and provide component metrics.

   Sysdig provides a config file in GitHub. Deploying this file creates a cluster role and service account in Kubernetes, and defines cluster role binding that grants the Sysdig agent rules in the cluster role.

   Run the following commands (using whatever namespace you defined in Step 2):

   kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   kubectl create serviceaccount sysdig-agent -n sysdig-agent
   kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information :

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Step 5 Set Up SSL Connectivity to the Backend.

6. (All installs) Apply the sysdig-agent-configmap.yaml file using the command:

   kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. FOR GKE COS ONLY: To enable the eBPF probe required for COS, uncomment the following parameters in  sysdig-agent-daemonset-v2.yaml under the env section:

   env:
     - name: SYSDIG_BPF_PROBE
       value: ""
   

8. Apply the sysdig-agent-service.yaml file:

   kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
   

   This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

9. (All installs) Apply the daemonset-v2.yaml file using the command:

   kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
   

The agents will be deployed and you can see Getting Started with Sysdig Monitor to view some metrics in the Sysdig Monitor UI. You can make further edits to the configmap as described below.Getting Started with Sysdig Monitor

Enable Kube State Metrics and Cluster Name

These steps are optional but recommended.

1. Edit sysdig-agent-configmap.yaml to uncomment the line: new_k8s: true

   This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.

   For more information, see the Kube State Metrics entry in the Sysdig blog.

   As of agent 9.6.0, new_k8s is enabled by default.

2. Edit sysdig-agent-configmap.yaml to uncomment the line: **k8s_cluster_name: **and add your cluster name.

   Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.

   Note: Alternatively, if you assign a tag with “cluster” in the tag name, Sysdig Monitor will display that as the Kubernetes cluster name.

3. Apply the configmap changes using the command:

   kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

4. Proceed to verify the metrics in the Sysdig Monitor UI.

Verify Metrics in Sysdig Monitor UI

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

The steps below give one way to do the check.

1. Access Sysdig Monitor:

   SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

   For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

   Log in with your Sysdig user name and password.

2. Select the Explore tab to see if metrics are displayed.

3. (Once you have enabled new_k8s:true): To verify that kube state metrics and cluster name are working correctly: Select the Explore tab and create a grouping by kubernetes.cluster.name and kubernetes.pod.name.

   As of agent 9.6.0, new_k8s is enabled by default.

   

4. Select an individual container or pod to see details.

   

Additional Options

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com.

To configure the collector IP in a Kubernetes SaaS instance:

1. Open sysdig-agent-configmap.yaml in a text editor.

2. Uncomment the following lines:

   • collector:

   • collector_port

3. Set the collector: value to collector-static.sysdigcloud.com

4. Set the collector_port: value to 6443

5. Save the file.

The example file below shows how the sysdig-agent-configmap.yaml file should look after configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sysdig-agent
data:
  dragent.yaml: |
    ### Agent tags
    # tags: linux:ubuntu,dept:dev,local:nyc

    #### Sysdig Software related config ####

    # Sysdig collector address
    collector: collector-static.sysdigcloud.com

    # Collector TCP port
    collector_port: 6443

    # Whether collector accepts ssl/TLS
    ssl: true

    # collector certificate validation
    ssl_verify_certificate: true

    # Sysdig Secure
    security:
      enabled: true

    #######################################
    # new_k8s: true
    # k8s_cluster_name: production

3.3 -

Steps for OpenShift

Review the Prerequisites in Agent Install: Kubernetes | GKE | OpenShift | IBM and the Host Requirements for Agent Installation, then proceed with the installation.

Kernel Headers

The Sysdig agent requires kernel header files to install successfully on a host.

If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.

In some cases, the host(s) in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernel headers manually.

To do so:

For Debian-style distributions, run the command:

apt-get -y install linux-headers-$(uname -r)

For RHEL-style distributions, run the command:

yum -y install kernel-devel-$(uname -r)

Background info: see also About Kernel Headers and the Kernel Module.

Configure for OpenShift

If you are using Red Hat OpenShift, these steps are required. They describe how to create a project, assign and label the node selector, create a privileged service account, and add it to a cluster role.

Copy/Paste Sample Code Block

You can copy-paste the code as-is, or follow the steps below to customize your naming conventions.

oc adm new-project sysdig-agent --node-selector='app=sysdig-agent'
oc label node --all "app=sysdig-agent"
oc project sysdig-agent
oc create serviceaccount sysdig-agent
oc adm policy add-scc-to-user privileged -n sysdig-agent -z sysdig-agent
oc adm policy add-cluster-role-to-user cluster-reader -n sysdig-agent -z sysdig-agent

Customize the Code

You can use your own Project Name and Node Selector names if desired.

Note that if you use a different Service Acccount name, you will need to edit the default service account in the Sysdig Installation Steps, below.

1. Create a new OpenShift project for the Sysdig agent deployment and assign the node selector:

   oc adm new-project PROJECT-NAME --node-selector="app=APP-NAME"
   

2. Label the node with the node selector:

   oc label node --all "app=APP-NAME"
   

3. Change to the new OpenShift Project for the Sysdig agent deployment:

   oc project PROJECT-NAME
   

4. Create a service account for the project:

   oc create serviceaccount SERVICE-ACCOUNT
   

5. Add the service account to privileged Security Context Constraints:

   oc adm policy add-scc-to-user privileged -n PROJECT-NAME -z SERVICE-ACCOUNT
   

6. Add the service account to the cluster-reader Cluster Role:

   oc adm policy add-cluster-role-to-user cluster-reader -n PROJECT-NAME -z SERVICE-ACCOUNT
   

Sysdig Installation Steps

To deploy agents using Kubernetes daemonsets, you download the configuration files, edit them as required, and deploy them.

• sysdig-agent-daemonset-v2.yaml

• sysdig-agent-clusterrole.yaml

• sysdig-agent-configmap.yaml

• sysdig-agent-service.yaml

Deploy the Agents

1. Download the sample files:

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create the sysdig-agent cluster role and assign it to the service account:

    oc apply -f sysdig-agent-clusterrole.yaml
    oc adm policy add-cluster-role-to-user sysdig-agent -n PROJECT-NAME -z SERVICE-ACCOUNT
   

3. Create a secret key using the command:

   oc create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. If you created a service account name other than sysdig-agent: Edit sysdig-agent-daemonset-v2.yamlto provide your custom value:``

   serviceAccount: sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information:

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Step 5 Set Up SSL Connectivity to the Backend.

6. (All installs) Apply the sysdig-agent-configmap.yaml file using the command:

   oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. Apply the sysdig-agent-service.yaml file:

   oc apply -f sysdig-agent-service.yaml -n sysdig-agent
   

   This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

8. (All installs) Apply the daemonset-v2.yaml file:

   oc apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
   

The agents will be deployed and you can see Getting Started with Sysdig Monitor to view some metrics in the Sysdig Monitor UI. You can make further edits to the configmap as described below.Getting Started with Sysdig Monitor

Enable Kube State Metrics and Cluster Name

These steps are optional but recommended.

1. Edit sysdig-agent-configmap.yaml to uncomment the line: new_k8s: true

   This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.

   For more information, see the Kube State Metrics entry in the Sysdig blog.

   As of agent 9.6.0, new_k8s is enabled by default.

2. Edit sysdig-agent-configmap.yaml to uncomment the line: **k8s_cluster_name: **and add your cluster name.

   Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.

   Note: Alternatively, if you assign a tag with “cluster” in the tag name, Sysdig Monitor will display that as the Kubernetes cluster name.

3. Apply the configmap changes using the command:

   oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

4. Proceed to verify the metrics in the Sysdig Monitor UI.

Verify Metrics in Sysdig Monitor UI

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

The steps below give one way to do the check.

1. Access Sysdig Monitor:

   SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

   For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

   Log in with your Sysdig user name and password.

2. Select the Explore tab to see if metrics are displayed.

3. (Once you have enabled new_k8s:true): To verify that kube state metrics and cluster name are working correctly: Select the Explore tab and create a grouping by kubernetes.cluster.name and kubernetes.pod.name.

   As of agent 9.6.0, new_k8s is enabled by default.

   

4. Select an individual container or pod to see details.

   

Additional Options

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com.

To configure the collector IP in a Kubernetes SaaS instance:

1. Open sysdig-agent-configmap.yaml in a text editor.

2. Uncomment the following lines:

   • collector:

   • collector_port

3. Set the collector: value to collector-static.sysdigcloud.com

4. Set the collector_port: value to 6443

5. Save the file.

The example file below shows how the sysdig-agent-configmap.yaml file should look after configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sysdig-agent
data:
  dragent.yaml: |
    ### Agent tags
    # tags: linux:ubuntu,dept:dev,local:nyc

    #### Sysdig Software related config ####

    # Sysdig collector address
    collector: collector-static.sysdigcloud.com

    # Collector TCP port
    collector_port: 6443

    # Whether collector accepts ssl/TLS
    ssl: true

    # collector certificate validation
    ssl_verify_certificate: true

    # Sysdig Secure
    security:
      enabled: true

    #######################################
    # new_k8s: true
    # k8s_cluster_name: production

3.4 -

Steps for Rancher

Preparation

General Requirements

You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Host Requirements for Agent Installation for additional context if desired.

Kernel Headers

The Sysdig agent requires a kernel module in order to be installed successfully on a host. On RancherOS distributions, the Unix version does not match the provided headers, and the agent might fail to install correctly. Therefore, you must install the kernel headers manually.

For RancherOS distributions, the kernel headers are available in the form of a system service and therefore are enabled using the ros service command:

$ sudo ros service enable kernel-headers-system-docker
$ sudo ros service up -d kernel-headers-system-docker

Some cloud hosting service providers supply pre-configured Linux instances with customized kernels. You may need to contact your provider’s support desk for instructions on obtaining appropriate header files, or for installing the distribution’s default kernel.

Install Agent

To deploy agents using Kubernetes daemonsets, download the following configuration files, edit them as required, and deploy them.

• sysdig-agent-clusterrole.yaml

• sysdig-agent-service.yaml

• sysdig-agent-daemonset-v2.yaml

• sysdig-agent-configmap.yaml

Deploy Agent

• Download the sample files:

  • sysdig-agent-clusterrole.yaml

  • sysdig-agent-daemonset-v2.yaml

  • sysdig-agent-configmap.yaml

  • sysdig-agent-service.yaml

• Create a namespace to use for the Sysdig agent.

  You can use whatever naming you prefer. In this document, we used sysdig-agent for both the namespace and the service account.

  The default service account name was automatically defined in sysdig-agent-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent.

  kubectl create ns sysdig-agent
  

• Create a secret key:

  kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
  

• Create a cluster role and service account, and define the cluster role bindingthat grants the Sysdig agent rules in the cluster role, using the commands:

  kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
  kubectl create serviceaccount sysdig-agent -n sysdig-agent
  kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
  

• Edit sysdig-agent-configmap.yaml to add the collector address , port , and the SSL/TLS information:

  collector:
  collector_port:
  ssl: #true or false
  check_certificate: #true or false
  

  • For SaaS, find the collector address for your region.

  • For On-prem, enter the collector endpoint defined in your environment.

  • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Step 5 Set Up SSL Connectivity to the Backend.

• (All installs) Apply the sysdig-agent-configmap.yaml file:

  kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
  

• (All installs) Apply the sysdig-agent-service.yaml file:

  kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
  

  This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

• (All installs) Apply the daemonset-v2.yaml file :

  kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
  

The agents will be deployed. See Getting Started with Sysdig Monitor to view some metrics in the Sysdig Monitor UI. You can make further edits to the configmap as described below.Getting Started with Sysdig Monitor

Enable Kube State Metrics and Cluster Name

These steps are optional but recommended.

• Edit sysdig-agent-configmap.yaml to uncomment the line: new_k8s: true

  This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.

  For more information, see the Kube State Metrics entry in the Sysdig blog.

  As of agent 9.6.0, new_k8s is enabled by default.

• Edit sysdig-agent-configmap.yaml to uncomment the line: **k8s_cluster_name: **and add your cluster name.

  Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.

  Note: Alternatively, if you assign a tag with “cluster” in the tag name, Sysdig Monitor will display that as the Kubernetes cluster name.

• Apply the configmap changes using the command:

  kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
  

• Proceed to verify the metrics in the Sysdig Monitor UI.

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com.

To configure the collector IP in a Kubernetes SaaS instance:

• Open sysdig-agent-configmap.yaml in a text editor.

• Uncomment the following lines:

  • collector:

  • collector_port

• Set the collector: value to collector-static.sysdigcloud.com

  See SaaS Regions and IP Ranges and identify the correct URL associated with your Sysdig collector and region.

• Set the collector_port: value to 6443

• Save the file.

The example file below shows how the sysdig-agent-configmap.yaml file should look after configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sysdig-agent
data:
  dragent.yaml: |
    ### Agent tags
    # tags: linux:ubuntu,dept:dev,local:nyc

    #### Sysdig Software related config ####

    # Sysdig collector address
    collector: collector-static.sysdigcloud.com

    # Collector TCP port
    collector_port: 6443

    # Whether collector accepts ssl/TLS
    ssl: true

    # collector certificate validation
    ssl_verify_certificate: true

    # Sysdig Secure
    security:
                enabled: true

    #######################################
    # new_k8s: true
    # k8s_cluster_name: production

Verify Metrics in Sysdig Monitor UI

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

• Access Sysdig Monitor:

  SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

  For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

  Log in with your Sysdig user name and password.

• Select the Explore tab to see if metrics are displayed.

• (Once you have enabled new_k8s:true): To verify that kube state metrics and cluster name are working correctly: Select the Explore tab and create a grouping by kubernetes.cluster.name and kubernetes.pod.name.

  As of agent 9.6.0, new_k8s is enabled by default.

  

• Select an individual container or pod to see details.

  

4 -

Agent Install: Non-Orchestrated

This section describes how to install the Sysdig agent directly on a Linux host, without using an orchestrator, such as Kubernetes or Mesos.

The agent can be installed in two ways:

• As a standard container

  If you want to install the lighter version of the Sysdig agent, see Install Slim Agent.

• As a non-containerized service

The steps for each flavor differ slightly depending on whether you are using the SaaS or on-premises version of the Sysdig platform.

Prerequisites

• See Host Requirements for Agent Installation. There you can check the requirements concerning:

  • Supported Linux distributions

  • Network connection

  • Sysdig access key

  • Cloud service providers (AWS, Google, and Microsoft Azure) and any steps you may need to configure to integrate the Sysdig agent.

• Run any commands as root or with the sudo command.

• Have your Sysdig access key on hand.

  If you launch an agent install from www.sysdig.com, the welcome wizard will present an access key.

Docker Container Agent Installation

The Sysdig agent can be deployed as a Docker container.

SaaS

Run the agent image, providing the access key and (optional) user-defined tags:

docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host\
 -e ACCESS_KEY=[ACCESS_KEY] \
 -e COLLECTOR=[COLLECTOR_ADDRESS] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro --shm-size=512m sysdig/agent

For the COLLECTOR, find the address for your region.

On-Premises

Provide collector and SSL/TLS information in addition to access key and optional tags:

docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro --shm-size=512m sysdig/agent

Service Agent Installation on Linux Host

Use these instructions to install the agent on the host itself, not in a container. Install on each host in the environment.

The command lines below can also be copy/pasted from the Welcome wizard or the Settings>Agent Installation page in the Sysdig Monitor interface.

In that case, your access key will already be included in the command automatically.

 >> Error, sdchecks[0] ModuleNotFoundError: No module named 'posix_ipc'

SaaS

1. Run the following command:

   curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] [--tags [TAGS]]
   

   Where [ACCESS_KEY] is your unique agent access key string. For example, 1234-your-key-here-1234. [TAGS] is an optional list of user-defined agent tags. For example, role:webserver,location:europe.

   Find the collector endpoint for your region listed here.

2. Make sure restarting the agent results in starting the service:

   sudo systemctl enable dragent
   

On-Premises

1. Run the following command:

   curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] --secure true --check_certificate true [--tags [TAGS]]
   

   check_certificate should be set to false if a self-signed certificate, a private, or a CA-signed certificate is used. See information about SSL in on-premises here.

2. Make sure restarting the agent results in starting the service:

   sudo systemctl enable dragent
   

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com:

user@host:~$ docker run --name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
-e COLLECTOR=collector-static.sysdigcloud.com \
-e COLLECTOR_PORT=6443 \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
--shm-size=512m \
sysdig/agent

Note on Manual Agent Installation

In the following cases, it may be preferable to perform a manual installation.

• Full control over the deployment process

• Integration with configuration management tools

• Custom kernel

• Unsupported distribution

If desired, see Agent Install: Manual Linux Installation.

5 -

Install Slim Agent

The slim agent is a lighter version of the Sysdig agent that is created by splitting the regular agent image into two components responsible for different functions. The slim agent reduces the surface area of attack for potential vulnerabilities and is, therefore, more secure.

You install the slim agent package as two separate containers:

• agent-kmodule: Responsible for downloading and building the kernel module. The image is short-lived. The container exits after the kernel module is loaded. The transient nature of the container reduces the time and opportunities for exploiting any potential vulnerabilities present in the container image.

  Prerequisites: The package depends on Dynamic Kernel Module Support (DKMS) and requires the compiler and kernel headers installed if you are using the agent-kmodule to build the kernel probe. Alternatively, you can use it without the kernel headers. In such cases, the agent-kmodule will attempt to download a pre-built kernel probe if it is present in the Sysdig probe repository.

  The module contains:

  • The driver sources

  • A post-install script that builds the module upon installation

• agent-slim: Responsible for running the agent module once the kernel module has been loaded. When the slim agent is up and running it functions the same way as the regular agent.

Install Slim Agent in a Non-Orchestrated Environment

The agent is installed by running sysdig/agent-kmodule first, followed by running sysdig/agent-slim.

1. Build and load the kernel module:

   docker run -it --privileged --rm --name sysdig-agent-kmodule -v /usr:/host/usr:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro sysdig/agent-kmodule
   

2. Run the agent module:

   docker run -d --name sysdig-agent \
   --privileged \
   --net host \
   --pid host\
   -e ACCESS_KEY=[ACCESS_KEY] \
   -e COLLECTOR=[COLLECTOR_ADDRESS] \
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   sysdig/agent-slim
   

Install Slim Agent on Kubernetes

The agent is installed by scheduling both the agent-kmodule and agent-slim containers into a single daemonset. The agent-kmodule container is defined as an init container, which ensures that it runs first and must succeed in order for the other containers to run.

1. Download sysdig-agent-slim-daemonset-v2.yaml, edit it as required, and deploy.

   An example daemonset is given below:

   ### WARNING: this file is supported from Sysdig Agent 0.80.0
   # apiVersion: extensions/v1beta1  # If you are in Kubernetes version 1.8 or less please use this line instead of the following one
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: sysdig-agent
     labels:
       app: sysdig-agent
   spec:
     selector:
       matchLabels:
         app: sysdig-agent
     updateStrategy:
       type: RollingUpdate
     template:
       metadata:
         labels:
           app: sysdig-agent
       spec:
         volumes:
         - name: modprobe-d
           hostPath:
             path: /etc/modprobe.d
         - name: dshm
           emptyDir:
             medium: Memory
         - name: dev-vol
           hostPath:
             path: /dev
         - name: proc-vol
           hostPath:
             path: /proc
         - name: boot-vol
           hostPath:
             path: /boot
         - name: modules-vol
           hostPath:
             path: /lib/modules
         - name: usr-vol
           hostPath:
             path: /usr
         - name: run-vol
           hostPath:
             path: /run
         - name: varrun-vol
           hostPath:
             path: /var/run
         - name: sysdig-agent-config
           configMap:
             name: sysdig-agent
             optional: true
         - name: sysdig-agent-secrets
           secret:
             secretName: sysdig-agent
         hostNetwork: true
         hostPID: true
         tolerations:
           - effect: NoSchedule
             key: node-role.kubernetes.io/master
         # The following line is necessary for RBAC
         serviceAccount: sysdig-agent
         terminationGracePeriodSeconds: 5
         initContainers:
         - name: sysdig-agent-kmodule
           image: sysdig/agent-kmodule
           imagePullPolicy: Always
           securityContext:
             privileged: true
           resources:
             requests:
               cpu: 1000m
               memory: 384Mi
             limits:
               memory: 512Mi
           volumeMounts:
           - mountPath: /etc/modprobe.d
             name: modprobe-d
             readOnly: true
           - mountPath: /host/boot
             name: boot-vol
             readOnly: true
           - mountPath: /host/lib/modules
             name: modules-vol
             readOnly: true
           - mountPath: /host/usr
             name: usr-vol
             readOnly: true
         containers:
         - name: sysdig-agent
           # WARNING: the agent-slim release is currently dependent on the above
           # initContainer and thus only functions correctly in a kubernetes cluster
           image: sysdig/agent-slim
           imagePullPolicy: Always
           securityContext:
             privileged: true
           resources:
             # Resources needed are subjective to the actual workload.
             # Please refer to Sysdig Support for more info.
             requests:
               cpu: 600m
               memory: 512Mi
             limits:
               cpu: 2000m
               memory: 1536Mi
           readinessProbe:
             exec:
               command: [ "test", "-e", "/opt/draios/logs/running" ]
             initialDelaySeconds: 10
           volumeMounts:
           - mountPath: /host/dev
             name: dev-vol
             readOnly: false
           - mountPath: /host/proc
             name: proc-vol
             readOnly: true
           - mountPath: /host/run
             name: run-vol
           - mountPath: /host/var/run
             name: varrun-vol
           - mountPath: /dev/shm
             name: dshm
           - mountPath: /opt/draios/etc/kubernetes/config
             name: sysdig-agent-config
           - mountPath: /opt/draios/etc/kubernetes/secrets
             name: sysdig-agent-secrets
   

   See Sysdig Cloud Scripts for the latest daemonset.

2. Create a namespace to use for the Sysdig agent.

   # kubectl create ns sysdig-agent
   

   You can use whatever naming you prefer. In this document, we used sysdig-agent for both the namespace and the service account. The default service account name was automatically defined in sysdig-agent-slim-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent.

3. Create a secret key:

   # kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. Create a cluster role and service account, and define the cluster role binding that grants the Sysdig agent rules in the cluster role, using the commands:

   # kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   # kubectl create serviceaccount sysdig-agent -n sysdig-agent
   # kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector``address``` and portand theSSL/TLS` information :

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Step 5 Set Up SSL Connectivity to the Backend.

6. Apply the configuration changes:

   # kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. Deploy the kernel module and slim agent containers using the daemonset:

   # kubectl apply -f sysdig-agent-slim-daemonset-v2.yaml -n sysdig-agent
   

The agents will be deployed and you can see Getting Started with Sysdig Monitor to view some metrics in the Sysdig Monitor UI. You can make further edits to the configmap as described in the following sections:Getting Started with Sysdig Monitor

• Enable Kube State Metrics and Cluster Name

• Connect to the Sysdig Backend via Static IPs (SaaS only)

Install Slim Agent on GKE

The agent is installed by scheduling both the agent-kmodule and agent-slim containers into a single daemonset. The agent-kmodule container is defined as an init container, which ensures that it runs first and must succeed in order for the other containers to run.

1. Download sysdig-agent-slim-daemonset-v2.yaml.

   An example daemonset is given below:

   ### WARNING: this file is supported from Sysdig Agent 0.80.0
   # apiVersion: extensions/v1beta1  # If you are in Kubernetes version 1.8 or less please use this line instead of the following one
   apiVersion: apps/v1
   kind: DaemonSet
   metadata:
     name: sysdig-agent
     labels:
       app: sysdig-agent
   spec:
     selector:
       matchLabels:
         app: sysdig-agent
     updateStrategy:
       type: RollingUpdate
     template:
       metadata:
         labels:
           app: sysdig-agent
       spec:
         volumes:
         - name: modprobe-d
           hostPath:
             path: /etc/modprobe.d
   ### uncomment for minikube
   #      - name: etc-version
   #        hostPath:
   #          path: /etc/VERSION
   #          type: FileOrCreate
         - name: dshm
           emptyDir:
             medium: Memory
         - name: dev-vol
           hostPath:
             path: /dev
         - name: proc-vol
           hostPath:
             path: /proc
         - name: boot-vol
           hostPath:
             path: /boot
         - name: modules-vol
           hostPath:
             path: /lib/modules
         - name: usr-vol
           hostPath:
             path: /usr
         - name: run-vol
           hostPath:
             path: /run
         - name: varrun-vol
           hostPath:
             path: /var/run
         - name: sysdig-agent-config
           configMap:
             name: sysdig-agent
             optional: true
         - name: sysdig-agent-secrets
           secret:
             secretName: sysdig-agent
         # This section is for eBPF support. Please refer to Sysdig Support before
         # uncommenting, as eBPF is recommended for only a few configurations.
         - name: bpf-probes
           emptyDir: {}
         - name: osrel
           hostPath:
             path: /etc/os-release
             type: FileOrCreate
         hostNetwork: true
         hostPID: true
         tolerations:
           - effect: NoSchedule
             key: node-role.kubernetes.io/master
         # The following line is necessary for RBAC
         serviceAccount: sysdig-agent
         terminationGracePeriodSeconds: 5
         initContainers:
         - name: sysdig-agent-kmodule
           image: quay.io/sysdig/agent-kmodule
           imagePullPolicy: Always
           securityContext:
             privileged: true
           resources:
             requests:
               cpu: 1000m
               memory: 384Mi
             limits:
               memory: 512Mi
           # This section is for eBPF support. Please refer to Sysdig Support before
           # uncommenting, as eBPF is recommended for only a few configurations.
           env:
             - name: SYSDIG_BPF_PROBE
               value: ""
           volumeMounts:
           - mountPath: /etc/modprobe.d
             name: modprobe-d
             readOnly: true
   ### uncomment for minikube
   #        - mountPath: /host/etc/VERSION
   #          name: etc-version
   #          readOnly: true
           - mountPath: /host/boot
             name: boot-vol
             readOnly: true
           - mountPath: /host/lib/modules
             name: modules-vol
             readOnly: true
           - mountPath: /host/usr
             name: usr-vol
             readOnly: true
           # This section is for eBPF support. Please refer to Sysdig Support before
           # uncommenting, as eBPF is recommended for only a few configurations.
           - mountPath: /root/.sysdig
             name: bpf-probes
           - mountPath: /host/etc/os-release
             name: osrel
             readOnly: true
         containers:
         - name: sysdig-agent
           # WARNING: the agent-slim release is currently dependent on the above
           # initContainer and thus only functions correctly in a kubernetes cluster
           image: quay.io/sysdig/agent-slim
           imagePullPolicy: Always
           securityContext:
             privileged: true
           resources:
             # Resources needed are subjective to the actual workload.
             # Please refer to Sysdig Support for more info.
             requests:
               cpu: 600m
               memory: 512Mi
             limits:
               cpu: 2000m
               memory: 1536Mi
           readinessProbe:
             exec:
               command: [ "test", "-e", "/opt/draios/logs/running" ]
             initialDelaySeconds: 10
           # This section is for eBPF support. Please refer to Sysdig Support before
           # uncommenting, as eBPF is recommended for only a few configurations.
           env:
            - name: SYSDIG_BPF_PROBE
               value: ""
           volumeMounts:
           - mountPath: /host/dev
             name: dev-vol
             readOnly: false
           - mountPath: /host/proc
             name: proc-vol
             readOnly: true
           - mountPath: /host/run
             name: run-vol
           - mountPath: /host/var/run
             name: varrun-vol
           - mountPath: /dev/shm
             name: dshm
           - mountPath: /opt/draios/etc/kubernetes/config
             name: sysdig-agent-config
           - mountPath: /opt/draios/etc/kubernetes/secrets
             name: sysdig-agent-secrets
           # This section is for eBPF support. Please refer to Sysdig Support before
           # uncommenting, as eBPF is recommended for only a few configurations.
           - mountPath: /root/.sysdig
             name: bpf-probes
   

2. Either use the single-line command from the Getting Started section of the Sysdig application or continue with the step 3 through 7.

   $ curl -s https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s -- --access_key 84d1d241-cde3-4ecc-9ecf-9a735ed0df45 --collector collector-staging.sysdigcloud.com --collector_port 6443 --nodeanalyzer --api_endpoint secure-staging.sysdig.com
   

3. Ensure that you uncomment the following sections:

   • eBPF Probes under spec volume:

     - name: bpf-probes
       emptyDir: {}
     - name: osrel
       hostPath:
          path: /etc/os-release
          type: FileOrCreate
     

   • Environment variable for eBPF under initContainers:

     env:
       - name: SYSDIG_BPF_PROBE
         value: ""
     

   • Mount path for eBPF under initContainers:

     - mountPath: /root/.sysdig
       name: bpf-probes
     - mountPath: /host/etc/os-release
       name: osrel
       readOnly: true
     

   • Environment variable for eBPF under sysdig-agent:

     env:
      - name: SYSDIG_BPF_PROBE
        value: ""
     

   • Mount path for eBPF under volume mounts

     - mountPath: /root/.sysdig
       name: bpf-probesenv:
     

4. Create a namespace to use for the Sysdig agent.

   $ kubectl create ns sysdig-agent
   

   You can use whatever naming you prefer. In this document, we used sysdig-agent for both the namespace and the service account. The default service account name was automatically defined in sysdig-agent-slim-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent.

5. Create a secret key:

   $ kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

6. Create a cluster role and service account, and define the cluster role binding that grants the Sysdig agent rules in the cluster role, using the commands:

   $ kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   $ kubectl create serviceaccount sysdig-agent -n sysdig-agent
   $ kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

7. Edit sysdig-agent-configmap.yaml to add the collector``address``` and portand theSSL/TLS` information :

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Step 5 Set Up SSL Connectivity to the Backend.

8. Apply the configuration changes:

   $ kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

9. Deploy the kernel module and slim agent containers using the daemonset:

   # kubectl apply -f sysdig-agent-slim-daemonset-v2.yaml -n sysdig-agent
   

The agents will be deployed and you can see Getting Started with Sysdig Monitor to view some metrics in the Sysdig Monitor UI. You can make further edits to the configmap as described in the following sections:Getting Started with Sysdig Monitor

• Enable Kube State Metrics and Cluster Name

• Connect to the Sysdig Backend via Static IPs (SaaS only)

6 -

Agent Install: Manual Linux Installation

Manual installation of the native Linux agent is recommended in the following cases:

• Full control over the deployment process

• Integration with configuration management tools

• Custom kernel

• Unsupported distribution (within Debian/Fedora flavors)

Otherwise, you may want to just follow the standard Installation Guide:

• Agent Install: Non-Orchestrated

NOTE: If you are installing the Sysdig agent in an orchestrated infrastructure such as Kubernetes, Mesos/Marathon, use the respective Installation Guides:

• Agent Install: Kubernetes

• Agent Install: Mesos | Marathon | DCOS

Installation Options

Review the Host Requirements for Agent Installation. Then follow the steps for the appropriate Linux distribution, below.

Debian, Ubuntu

1. Trust the Sysdig Monitor GPG key, configure the apt repository, and update the package list:

   curl -s https://download.sysdig.com/DRAIOS-GPG-KEY.public | apt-key add -
   curl -s -o /etc/apt/sources.list.d/draios.list http://download.sysdig.com/stable/deb/draios.list
   apt-get update
   

2. Install kernel development files.

   The following command might not work with every kernel. Make sure to customize the name of the package properly.

   apt-get -y install linux-headers-$(uname -r)
   

3. Install, configure, and restart the Sysdig agent.

   apt-get -y install draios-agent
   echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
   echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
   service dragent restart
   

   Replace ACCESS_KEY with your unique access key string. Inability to retrieve the key indicates that the administrator of your instance might have it turned off for non-admin users. Please contact your Sysdig administrator to receive the key. If you still have issues please contact Sysdig support.

   [TAGS] is an optional parameter you can use to list one or more tags for this host (see below).

CentOS, RHEL, Fedora, Amazon AMI, Amazon Linux 2

1. Trust the Sysdig Monitor GPG key, configure the yum repository.

   $ rpm --import https://download.sysdig.com/DRAIOS-GPG-KEY.public
   $ curl -s -o /etc/yum.repos.d/draios.repo http://download.sysdig.com/stable/rpm/draios.repo
   

2. Install the EPEL repository

   The following command is required only if DKMS is not available in the distribution. You can verify if DKMS is available with yum list dkms

   The command below contains a sample release number; be sure to update with the correct release.

   $ rpm -i http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm
   

3. Install kernel development files.

   The following command might not work with every kernel. Make sure to customize the name of the package properly.

   $ yum -y install kernel-devel-$(uname -r)
   

4. Install, configure, and start the Sysdig agent.

   $ yum -y install draios-agent
   $ echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
   $ echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
   $ sudo systemctl enable dragent
   $ sudo systemctl start dragent
   

   Replace ACCESS_KEY with your unique access key string. Inability to retrieve the key indicates that the administrator of your instance might have it turned off for non-admin users. Please contact your Sysdig administrator to receive the key. If you still have issues please contact Sysdig support.

   [TAGS] is an optional parameter you can use to list one or more tags for this host (see below).

   If you using a non-systemd Linux distribution, use the service command to start dragent.

   $ service dragent restart
   

Other Linux Distributions

The Sysdig Agent is unsupported outside of the Debian, Fedora, and Amazon distributions.

Agent Tags

Tagging your hosts is highly recommended. Agent Tags allow you to sort nodes of your infrastructure into custom groups in Sysdig Monitor.

Replace the [TAGS] parameter above with a comma-separated list of TAG_NAME:TAG_VALUE .

For example: role:webserver,location:europe

7 -

Agent Install: IKS (IBM Cloud with Sysdig)

IBM maintains the documentation for Sysdig agent installation on IBM Cloud Kubernetes Service (IKS).

For more information, see the IBM Cloud Monitoring with Sysdig documentation:

• Getting Started

• Configuring a Sysdig Agent on a Standard Kubernetes Cluster

8 -

Agent Install: Mesos | Marathon | DCOS

Marathon is the container orchestration platform for Mesosphere’s Datacenter Operating System (DC/OS) and Apache Mesos.

This guide describes how to install the Sysdig agent container on each underlying host in your Mesos cluster. Once installed, the agent will automatically connect to the Mesos and Marathon APIs to pull relevant metadata about the environment and will begin monitoring all of your hosts, apps, containers, and frameworks.

Standard Installation Instructions

Review the Host Requirements for Agent Installation.

In this three-part installation, you:

• Deploy the Sysdig agent on all Mesos Agent (aka “Slave”) nodes, either automatically or by creating and posting a .json file to the leader Marathon API server.

• Deploy the Sysdig agent on the Mesos Master nodes.

• Special configuration steps: modify the Sysdig agent config file to monitor Marathon instances.

Deploy the Sysdig agent on your Mesos Agent nodes

Preferred Option: Automatic install (DC/OS 1.11+)

If you’re using DC/OS 1.8 or higher, then you can find Sysdig in the Mesosphere Universe marketplace and install it from there.

It will automatically deploy the Sysdig agent container on each of your Mesos Agent nodes as a Marathon app.

Proceed to Deploy the Sysdig Agent.

Alternate Option: Post a .json file

If you are using a version of DC/OS earlier than 1.8 then:

1. Create a JSON file for Marathon, in the following format.

   The COLLECTOR address comes from your own environment in on-prem installations. For SaaS installations, find the collector endpoint for your region listed here.

   COLLECTOR_PORT, SECURE, and CHECK_CERT are used in environments with Sysdig’s on-premises backend installed.

   {
     "backoffFactor": 1.15,
     "backoffSeconds": 1,
     "constraints": [
       [
         "hostname",
         "UNIQUE"
       ]
     ],
     "container": {
       "docker": {
         "forcePullImage": true,
         "image": "sysdig/agent",
         "parameters": [],
         "privileged": true
       },
       "type": "DOCKER",
       "volumes": [
         {
           "containerPath": "/host/var/run/docker.sock",
           "hostPath": "/var/run/docker.sock",
           "mode": "RW"
         },
         {
           "containerPath": "/host/dev",
           "hostPath": "/dev",
           "mode": "RW"
         },
         {
           "containerPath": "/host/proc",
           "hostPath": "/proc",
           "mode": "RO"
         },
         {
           "containerPath": "/host/boot",
           "hostPath": "/boot",
           "mode": "RO"
         },
         {
           "containerPath": "/host/lib/modules",
           "hostPath": "/lib/modules",
           "mode": "RO"
         },
         {
           "containerPath": "/host/usr",
           "hostPath": "/usr",
           "mode": "RO"
         }
       ]
     },
     "cpus": 1,
     "deployments": [],
     "disk": 0,
     "env": {
       "ACCESS_KEY": "ACCESS_KEY=YOUR-ACCESS-KEY-HERE",
       "CHECK_CERT": "false",
       "SECURE": "true",
       "TAGS": "example_tag:example_value",
       "name": "sdc-agent",
       "pid": "host",
       "role": "monitoring",
       "shm-size": "350m"
     },
     "executor": "",
     "gpus": 0,
     "id": "/sysdig-agent",
     "instances": 1,
     "killSelection": "YOUNGEST_FIRST",
     "labels": {},
     "lastTaskFailure": {
       "appId": "/sysdig-agent",
       "host": "YOUR-HOST",
       "message": "Container exited with status 70",
       "slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
       "state": "TASK_FAILED",
       "taskId": "sysdig-agent.3bb0759d-3fa3-11e9-b446-c60a7a2ee871",
       "timestamp": "2019-03-06T00:03:16.234Z",
       "version": "2019-03-06T00:01:57.182Z"
     },
     "maxLaunchDelaySeconds": 3600,
     "mem": 850,
     "networks": [
       {
         "mode": "host"
       }
     ],
     "portDefinitions": [
       {
         "name": "default",
         "port": 10101,
         "protocol": "tcp"
       }
     ],
     "requirePorts": false,
     "tasks": [
       {
         "appId": "/sysdig-agent",
         "healthCheckResults": [],
         "host": "YOUR-HOST-IP",
         "id": "sysdig-agent.0d5436f4-3fa4-11e9-b446-c60a7a2ee871",
         "ipAddresses": [
           {
             "ipAddress": "YOUR-HOST-IP",
             "protocol": "IPv4"
           }
         ],
         "localVolumes": [],
         "ports": [
           4764
         ],
         "servicePorts": [],
         "slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
         "stagedAt": "2019-03-06T00:09:04.232Z",
         "startedAt": "2019-03-06T00:09:06.912Z",
         "state": "TASK_RUNNING",
         "version": "2019-03-06T00:09:04.182Z"
       }
     ],
     "tasksHealthy": 0,
     "tasksRunning": 1,
     "tasksStaged": 0,
     "tasksUnhealthy": 0,
     "unreachableStrategy": {
       "expungeAfterSeconds": 0,
       "inactiveAfterSeconds": 0
     },
     "upgradeStrategy": {
       "maximumOverCapacity": 1,
       "minimumHealthCapacity": 1
     },
     "version": "2019-03-06T00:09:04.182Z",
     "versionInfo": {
       "lastConfigChangeAt": "2019-03-06T00:09:04.182Z",
       "lastScalingAt": "2019-03-06T00:09:04.182Z"
     }
   }
   
   

   See Table 1: Environment Variables for Agent Config Filef or the Sysdig name:value definitions.

   Complete the “cpus”, “mem” and “labels” (i.e. Marathon labels) entries to fit the capacity and requirements of the cluster environment.

2. Update the created.json file to the leader Marathon API server:

   $ $curl -X POST http://$(hostname -i):8080/v2/apps -d @sysdig.json -H "Content-type: application/json"
   

Deploy the Sysdig Agent

After deploying the agent to the Mesos Agent nodes, you will install agents on each of the Mesos Master nodes as well.

Use the Agent Install: Non-Orchestrated instructions to install the agent directly on each of your Mesos Master nodes.

When the Sysdig agent is successfully installed on the master nodes, it will automatically connect to the local Mesos and Marathon (if available) API servers via http://localhost:5050 and http://localhost:8080 respectively, to collect cluster configuration and current state metadata in addition to host metrics.

Special Configuration Steps

In certains situations, you may need to add additional configurations to the dragent.yaml file:

• If the Sysdig agent cannot be run directly on the Mesos API server

• If the API server is protected with a username/password.

Descriptions and examples are shown below.

If the Sysdig Agent Cannot Run On the Mesos API Server

Mesos allows multiple masters. If the API server can not be instrumented with a Sysdig agent, simply delegate ONE other node with an agent installed to remotely receive infrastructure information from the API server.

NOTE: If you manually configure the agent to point to a master with a static configuration file entry, then automatic detection/following of leader changes will no longer be enabled.

Add the following Mesos parameter to the delegated agent’s dragent.yaml file to allow it to connect to the remote API server and authenticate, either by:

a. Directly editing dragent.yaml on the host, or

b. Converting the YAML code to a single-line format and adding it as an ADDITIONAL_CONF argument in a Docker command.

See Understanding the Agent Config Files for details.

Specify the API server’s connection method, address, and port. Also specify credentials if necessary.

YAML example:

mesos_state_uri: http://[acct:passwd@][hostname][:port]
marathon_uris:
  - http://[acct:passwd@][hostname][:port]

If the Mesos API server requires authentication

If the agent is installed on the API server but the API server uses a different port or requires authentication, those parameters must be explicitly specified.

Add the following Mesos parameters to the API server’s dragent.yaml to make it connect to the API server and authenticate with any unique account and password, either by:

a. Directly editing dragent.yaml on the host, or

b. Converting the YAML code to a single-line format and adding it as an ADDITIONAL_CONF argument in a Docker command.

See Understanding the Agent Config Files for details.

Specify the API server’s protocol, user credentials, and port:

mesos_state_uri: http://[username:password@][hostname][:port]
marathon_uris:
  - http://[acct:passwd@][hostname][:port]

*HTTPS protocol is also supported.

Troubleshooting: Turning Off Metadata Reception

In troubleshooting cases where auto-detection and reporting of your Mesos infrastructure needs to be temporarily turned off in a designated agent:

1. Comment out the Mesos parameter entries in the agent’s dragent.yaml file.

   Example parameters to disable: mesos_state_uri, marathon_uris

2. If the agent is running on the API server (Master node) and auto-detecting a default configuration, you can add the line:

   mesos_autodetect: false

   either directly in the dragent.yaml file or as an ADDITIONAL_CONF parameter in a Docker command.

3. Restart the agent.

9 -

Airgapped Agent Installation

Airgapped environments are those that do not have the network access to pull images from the container repository. Agent installation requires sysdigcloud-probe and you cannot download a pre-compiled module in an airgapped environment. Therefore, ensure that you compile your own sysdigcloud-probe before installing the agent.

Prepare the Sysdig Probe Builder Images

On a machine with internet connectivity, build the Sysdig probe container and create a tar file of the image.

1. Get the probe builder artifacts from the repository:

   $ git clone https://github.com/draios/sysdig
   $ git checkout probe-builder
   $ cd sysdig
   

2. Build the container image:

   $ docker build -t airgap/sysdig-probe-builder probe-builder/
   

3. Create the container and run:

   $ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock airgap/sysdig-probe-builder:latest -P -b airgap/
   

4. Save the images to a tar archive:

   $ docker save airgap/sysdig-probe-builder | gzip > builders.tar.gz
   

   Ensure that you make this tar available to the airgapped machines where you intend to install the Sysdig agent.

Set Up Kernel Module

1. Set up a local repository to host the pre-compiled kernel module:

   $ kubectl run my-nginx --image=nginx --port=80
   $ kubectl expose deployment my-nginx --port=80 --type=NodePort
   

2. Copy sysdigcloud-probe to the repository you have created:

   $ kubectl cp sysdigcloud-probe-<version> my-nginx-xxxxxxxx-xxxx:/usr/share/nginx
   

Install Agent in Docker Environment

1. Install Sysdig agent by pointing SYSDIG_PROBE_URL to the local repository:

   For docker-based installations:

   $ docker run -d --name sysdig-agent --restart always --privileged --net host --pid host -e ACCESS_KEY=WWWWW-YYYY-XXXX-ZZZZ-123456789 -e SECURE=true -e SYSDIG_PROBE_URL=http://www.mywebserver.net:80/ -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro --shm-size=512m sysdig/agent
   

   Where -e SYSDIG_PROBE_URL=http://www.mywebserver:80/ is the local nginx pod with the loaded module.

   To use secure communication with a self-signed or untrusted certificate, apply the -e SYSDIG_PROBE_INSECURE_DOWNLOAD=true environment variable.

2. Check the agent log. You will see a similar message:

   Found custom module URL http://mywebserver:80/, will use it * Trying to download precompiled module from http://mywebserver:80/sysdigcloud-probe-<version>
   

3. Continue with the instructions in Agent Install: Non-Orchestrated.

Install Agent in Kubernetes Environment

1. Open your agent daemonset and update the SYSDIG_PROBE_URL to point to the local repository:

   - name: SYSDIG_PROBE_URL
     value: http://www.mywebserver:80/
   

   If you would like to use secure communication with a self-signed or untrusted certificate, apply the SYSDIG_PROBE_INSECURE_DOWNLOAD environment variable.

   - name: SYSDIG_PROBE_INSECURE_DOWNLOAD
     value: true
   

2. Continue with the instructions in Agent Install: Kubernetes.

10 -

Troubleshooting Agent Installation

This section describes methods for troubleshooting two types of issue:

• Disconnecting Agents

• Can’t See Metrics After Agent Install

Disconnecting Agents

If agents are disconnecting, there could be problems with addresses that need to be resolved in the agent configuration files. See also Understanding the Agent Config Files.

Check for Duplicate MAC addresses

The Sysdig agent will use the eth0 MAC address to identify the different hosts within an infrastructure. In a virtualized environment, you should confirm each of your VM’s eth0 MAC addresses are unique.

If a unique address cannot be configured, you can supply an additional parameter in the Sysdig agent’s dragent.yaml configuration file: machine_id_prefix: prefix

The prefix text can be any string and will be prepended to the MAC address as reported in the Sysdig Monitor web interface’s Explore tables.

Example: (using ADDITIONAL_CONF rather than Kubernetes Configmap)

Here is an example Docker run command installing the parameter via the ADDITIONAL_CONF parameter

docker run --name sysdig-agent --privileged --net host --pid host -e ACCESS_KEY=abc123-1234-abcd-4321-abc123def456 -e TAGS=tag1:value1 -e ADDITIONAL_CONF="machine_id_prefix: MyPrefix123-" -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/agent

The resulting /opt/draios/etc/dragent.yaml config file would look like this:

customerid:abc123-1234-abcd-4321-abc123def456
tags: tag1:value1
machine_id_prefix: MyPrefix123-

You will then see all of your hosts, provided that all the prefixes are unique. The prefix will be visible whenever the MAC address is displayed in any view.

See also: Agent Configuration.

Check for Conflicting MAC addresses in GKE environments

In Google Container Engine (GKE) environments, MAC addresses could be repeated across multiple hosts. This would cause some hosts running Sysdig agents not to appear in your web interface.

To address this, add a unique machine ID prefix to each config you use to deploy the agent to a given cluster (i.e. each sysdig-daemonset.yaml file).

Note: This example uses the (v1) ADDITIONAL_CONF, rather than (v2) Configmap method.

- name: ADDITIONAL_CONF value: "machine_id_prefix: mycluster1-prefix-"

Can’t See Metrics After Agent Install

If agents were successfully installed, you could log in to the Sysdig Monitor UI, but no metrics are displayed in the Explore panel, first confirm that the agent license count has not been exceeded. Then check for any proxy, firewall, or host security policies preventing proper agent communication to the Sysdig Monitor backend infrastructure.

Check License Count

If network connectivity is good, the agent will connect to the backend but will be disconnected after a few seconds if the license count has been exceeded.

To check whether you are over-subscribed, go to Settings > Subscription.

See Subscription for details.

Check Network Policy

Agent Connection Port

Check your service provider VPC security groups to verify that network ACLs are set to allow the agent’s outbound traffic over TCP ports. See Sysdig Collector Ports for the supported TCP ports for each region.

Outbound IP Addresses

Due to the distributed nature of the Sysdig Monitor infrastructure, the agent must be open for outbound connections to collector.sysdigcloud.com on all outbound IP addresses.

Check Amazon’s public IP ranges file to see all the potential IP addresses the Sysdig agent can use to communicate with the Sysdig backend databases.

AWS Metadata Endpoint

AWS metadata is used for gathering information about the instance itself, such as instance id, public IP address, etc.

When running on an AWS instance, access to the following AWS metadata endpoint is also needed: 169.254.169.254

Check Local Host Policy

The agent requires access to the following local system resources in order to gather metrics:

• Read/Write access to /dev/sysdig devices.

• Read access to all the files under /proc file system.

• For container support, the Docker API endpoint /var/run/docker.sock

If any settings or firewall modifications are made, you may need to restart the agent service. In a shell on the affected instances issue the following command:

sudo service dragent restart

11 -

Identify Agent Version

Use one of the following methods to determine the version of the agents installed in your environment:

Explore

Segmenting metrics by using agent.version shows the installed versions of agents in your environment. For example, segment the uptime metric across your environment by using agent.version . Hover over the graph to see the list of agent versions.

The image shows the list of agent versions in n/a.

Dashboard

Use the Sysdig Agent Health Dashboard to determine the agent versions:

1. Log in to the Sysdig Monitor.

2. Select Dashboards and expand Host Infrastructure Dashboards.

3. Open the Sysdig Agent Health & Status template or create your own from the template.

   

   The Sysdig Agent and Health & Status Dashboard shows the agent version corresponding to each host in your environment.

12 -

Using Node Leases

The Sysdig agent uses Kubernetes Lease to control how and when connections are made to the Kubernetes API Server. This mechanism prevents overloading the Kubernetes API server with connection requests during agent bootup.

Kubernetes node leases are automatically created for agent version 12.0.0 and above. On versions prior to 12.0.0, you must configure node leases as given in the KB article.

Prerequisites

• Sysdig Agent v11.3.0 or above

• Kubernetes v1.14 or above

Types of Leases

The agent creates the following leases:

Cold Start

During boot up, the Sysdig agent connects to the Kubernetes API server to retrieve Kubernetes metadata and build a cache. The cold-start leases control the number of agents that build up this cache at any given time. An agent will grab a lease, build its cache, and then release the lease so that another agent can build its cache. This mechanism prevents agents from creating a “boot storm” which can overwhelm the API server in large clusters.

Delegation

In Kubernetes environments, two agents are marked as delegated in each cluster. The delegated agents are the designated agents to request more data from the API server and produce KubeState metrics. The delegation leases will not be released until the agent is terminated.

View Leases

To view the leases, run the following:

$ kubectl get leases -n sysdig-agent

You will see an output similar to the following:

NAME           HOLDER             AGE
cold-start-0                      20m
cold-start-1                      20m
cold-start-2                      21m
cold-start-3   ip-10-20-51-167    21m
cold-start-4                      21m
cold-start-5                      21m
cold-start-6                      20m
cold-start-7                      21m
cold-start-8                      20m
cold-start-9   ip-10-20-51-166   21m
delegation-0   ip-10-20-52-53    21m
delegation-1   ip-10-20-51-98    21m

Troubleshoot Leases

Verify Configuration

When lease-based delegation is working as expected, the agent logs show one of the following:

• Getting pods only for node <node>

• Getting pods for all nodes.

• Both (occasionally on the delegated nodes)

Run the following to confirm that it is working:

$ kubectl logs sysdig-agent-9l2gf -n sysdig-agent | grep -i "getting pods"

The configuration is working as expected if the output on a pod is similar to the following:

2021-05-05 02:48:32.877, 15732.15765, Information, cointerface[15738]: Only getting pods for node ip-10-20-51-166.ec2.internal

Unable to Create Leases

The latest Sysdig ClusterRole is required for the agent to create leases. If you do not have the latest ClusterRole or if you have not configured the ClusterRole correctly, the logs show the following error:

Error, lease_pool_manager[2989554]: Cannot access leases objects: leases.coordination.k8s.io is forbidden: User "system:serviceaccount:sysdig-agent:sysdig-agent" cannot list resource "leases" in API group "coordination.k8s.io" in the namespace "sysdig-agent"

Contact Sysdig Support for help.

Optional Agent Configuration

Several configuration options exist for leases. It is recommended to not change the default settings unless prompted by Sysdig Customer Support.

k8s_coldstart:
  enabled: <true/false>

k8s_coldstart:
  max_parallel_cold_starts: <int>

k8s_coldstart:
  namespace: <string>

k8s_coldstart:
  enforce_leader_election: <true/false>

k8s_delegation_election: <true/false>