Installation

• 1: Sysdig Agent
• 1.1: Agent Installation
• 1.1.1: Quick Install Sysdig Agent
• 1.1.2: Agent Installation Requirements
• 1.1.3: Agent Install: Kubernetes
• 1.1.3.1: Steps for GKE
• 1.1.3.1.1: GKE Autopilot
• 1.1.3.1.2: GKE Standard
• 1.1.3.2: Steps for OKE
• 1.1.3.3: Steps for OpenShift
• 1.1.3.4: Steps for Rancher
• 1.1.3.5: Steps for MKE
• 1.1.3.6: Using Node Leases
• 1.1.4: Agent Install: Non-Orchestrated
• 1.1.5: Agent Install: Manual Linux Installation
• 1.1.6: Agent Install: Amazon ECS
• 1.1.7: Agent Install: IKS (IBM Cloud Monitoring)
• 1.1.8: Agent Install: Mesos | Marathon | DCOS
• 1.1.9: Airgapped Agent Installation
• 1.1.10: Identify Agent Version
• 1.2: Agent Configuration
• 1.2.1: Understand the Agent Configuration
• 1.2.2: Configure Agent Modes
• 1.2.2.1: Metrics Available in Monitor Light
• 1.2.2.2: Additional Metrics Values Available in Troubleshooting
• 1.2.2.3: Metrics Not Available in Essentials Mode
• 1.2.3: Tune Agent
• 1.2.3.1: Enable HTTP Proxy for Agents
• 1.2.3.2: Manage Agent Log Levels
• 1.2.3.2.1: Change Agent Log Level Globally
• 1.2.3.2.2: Manage File Logging for Agent Components
• 1.2.3.2.3: Manage Console Logging for Agent Components
• 1.2.3.2.4: Change the Agent Log Directory
• 1.2.3.2.5: Control Disk Usage by Agent Logs
• 1.2.3.3: Disable Captures
• 1.2.3.4: Blacklist Ports
• 1.3: Agent Upgrade
• 1.4: Agent Uninstall
• 1.5: Agent Troubleshooting
• 1.5.1: Kernel Header Troubleshooting
• 1.5.2: Using the Agent Console
• 2: Serverless Agents
• 2.1: AWS Fargate Serverless Agents
• 2.1.1: Configure Workload Starting Policy
• 2.1.2: Manage Serverless Agent Logs
• 3: Prometheus Remote Write
• 4: Sysdig Secure for cloud
• 4.1: AWS Deployment
• 4.2: GCP Deployment
• 4.3: Azure Deployment
• 4.4: Agentless (CSPM Only)
• 5: Rapid Response: Installation
• 6: Node Analyzer: Multi-Feature Installation
• 7: Install Admission Controller
• 8: KSPM Components

1 - Sysdig Agent

Sysdig agents are simple to deploy and upgrade, and out of the box, they will gather and report on a wide variety of pre-defined metrics. Agent installation can be as simple as copying and pasting a few lines of code from a Wizard prompt, or you can follow step-by-step instructions to check supported environments and distributions, review installation options, and customize the agent configuration file post-install.

About Sysdig Agent

Sysdig agent is a lightweight host component that processes syscall events, creates capture files, and performs auditing and compliance. It is a platform that supports monitoring, securing, and troubleshooting cloud-native environments

In addition, the agent performs the following:

• Metrics processing, analysis, aggregation, and transmission

• Policy monitoring and alert notification through bidirectional communication with the Sysdig collector

• Integration with third-party software for consolidating customer ecosystem data

• Full assimilation into containerized and orchestrated environment

• Matches runtime policies against the syscall events and generates policy events

Learn More

• Agent Installation Requirements: Determine supported platforms and system requirements for installing Sysdig Agent.

• Agent Licensing: Retrieve the agent access key and understand the subscription model.

• Using Agent Modes: Edit the agent configuration file to configure agent modes.

1.1 - Agent Installation

Sysdig agents are delivered as either a container or a service and can be deployed with or without an orchestrator such as Kubernetes or Mesos.

A quick install involves just a few lines of code from the Getting Started wizard copied into a shell. The complete install instructions address checking for and installing kernel headers if needed, any prerequisite permissions settings for particular environments, and tips about updating the configuration files after initial installation.

Plan the Installation

Installation Options

In the default mode of agent installation, you install the agent package as two containers, each container responsible for different functions as given below. The agent-slim reduces the surface area of attack for potential vulnerabilities and is, therefore, more secure.

• agent-kmodule: Responsible for downloading and building the kernel module. The image is short-lived. The container exits after the kernel module is loaded. The transient nature of the container reduces the time and opportunities for exploiting any potential vulnerabilities present in the container image.

  Prerequisites: The package depends on Dynamic Kernel Module Support (DKMS) and requires the compiler and kernel headers installed if you are using the agent-kmodule to build the kernel probe. Alternatively, you can use it without the kernel headers. In such cases, the agent-kmodule will attempt to download a pre-built kernel probe if it is present in the Sysdig probe repository.

  The module contains:

  • The driver sources

  • A post-install script that builds the module upon installation

• agent-slim: Responsible for running the agent module once the kernel module has been loaded. Slim agent functions the same way as the regular agent and retains the feature parity.

Use the instruction below to install agent on your chosen environment:

Legacy Agent: The legacy agent can be run as a single container or a service. It includes the components for downloading and building the kernel module, as well as for gathering and reporting on a wide variety of pre-defined metrics. For more information, see Installing Agent as a Single Container.

Helm

Helm is the preferred way of installing Sysdig agent. It is used in most cloud environments, for example, Amazon EKS or EC2 on AWS Cloud or AWS Outpost, EC2, and Azure AKS.

• Deploy Agent on Kubernetes Using Helm Charts
• Deploy Agent on OpenShift Using Helm Charts
• Deploy Agent on GKE Using Helm Charts
• Deploy Agent on MKE Using Helm Charts
• Deploy Agent on Rancher Using Helm Charts
• Deploy Agent on OKE Using Helm Charts

Manual

With the Getting Started wizard, you can copy a simple line of code to deploy agents in a variety of environments.

Behind the scenes, the wizard auto-detects and completes configuration items such as the required access key and port information. The wizard can also be launched from the Start a Free Trial button at sysdig.com.

After the first install, Sysdig Secure and Monitor users can access the wizard at any time from the Rocket icon on the navigation bar.

1.1.1 - Quick Install Sysdig Agent

Sysdig provides you with quick-install commands pre-filled with some of your environment variables to get started with Sysdig agent. You choose the deployment type and Sysdig gives you auto-generated commands to ease your installation experience.

Access from Get Started or Data Sources

1. Log in as the administrator to Sysdig Monitor or Sysdig Secure.

2. Select the Get Started page in Monitor or the Data Sources page in Sysdig Secure.

3. Click Install the Agent, select the appropriate deployment type, and copy the auto-generated code, filling in remaining variable values as required.

Sample Usage

Kubernetes

Helm is the recommended option for installing agents on Kubernetes. For the latest helm-based installation instructions and configuration options, see sysdig-deploy.

If you cannot utilize helm, we also provide a script which will download and apply Kubernetes manifests to deploy the agent as a DaemonSet. The script requires curl and kubectl installed in the $PATH on the host in which it is run.

Sysdig Monitor

• Helm (SaaS)
• Script (SaaS)

  kubectl create ns sysdig-agent
  helm repo add sysdig https://charts.sysdig.com
  helm repo update

  helm install sysdig sysdig/sysdig-deploy  
  --namespace sysdig-agent \
  --set global.sysdig.accessKey=ACCESS_KEY \
  --set global.sysdig.region=REGION \
  --set global.clusterConfig.name=CLUSTER_NAME \
  --set nodeAnalyzer.enabled=false \
   
  

 curl -s https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s -- \
 --access_key <ACCESS_KEY>  \
 --collector <COLLECTOR_ADDRESS> --collector_port <COLLECTOR_PORT> \
 

Sysdig Secure

• Helm (SaaS)
• Script (SaaS)

  kubectl create ns sysdig-agent
  helm repo add sysdig https://charts.sysdig.com
  helm repo update

  helm install sysdig sysdig/sysdig-deploy 
  --namespace sysdig-agent \
  --set global.sysdig.accessKey=ACCESS_KEY \
  --set global.sysdig.region=REGION \
  --set global.clusterConfig.name=CLUSTER_NAME \
  --set global.kspm.deploy=true \
  --set nodeAnalyzer.secure.vulnerabilityManagement.newEngineOnly=true \
  --set nodeAnalyzer.nodeAnalyzer.benchmarkRunner.deploy=false \
    
  

 curl -s https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s -- \
 --access_key <ACCESS_KEY>  \
 --collector <COLLECTOR_ADDRESS> --collector_port <COLLECTOR_PORT> \
 --nodeanalyzer --api_endpoint <SECURE_ENDPOINT_ADDRESS>
 

Docker

• STEP 1
• STEP 2
• STEP 3

   #Build and load the kernel module 
  docker run -it --privileged --rm --name sysdig-agent-kmodule \
    -v /usr:/host/usr:ro \
    -v /boot:/host/boot:ro \
    -v /lib/modules:/host/lib/modules:ro \
    quay.io/sysdig/agent-kmodule
  

  #Configure the Kernel module to load during system boot.
  sudo mkdir -p /etc/modules-load.d
  sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
  

  #Run the sysdig-agent
  docker run -d --name sysdig-agent \
    --restart always \
    --privileged \
    --net host \
    --pid host \
    -e ACCESS_KEY=<ACCESS_KEY> \
    -e COLLECTOR=<COLLECTOR_URL> \
    -e SECURE=true \
    -e TAGS=<LIST_OF_TAGS> \
    -e ADDITIONAL_CONF= <LIST_OF_CONFIG> \
    -v /var/run/docker.sock:/host/var/run/docker.sock \
    -v /dev:/host/dev \
    -v /proc:/host/proc:ro \
    -v /boot:/host/boot:ro \
    --shm-size=512m \
    quay.io/sysdig/agent-slim
  

Linux

• Linux
• Example

  curl -s https://download.sysdig.com/stable/install-agent | sudo bash -c -- \
  --access_key [-t | --tags <value>] [-c | --collector <value>] \
  [-cp | --collector_port <value>] [-s | --secure <value>] \
  [-cc | --check_certificate]  [-ac | --additional_conf <value>] \
  [-b | --bpf] [-h | --help]
  

  curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- \
  --access_key <ACCESS_KEY> --collector <COLLECTOR_URL> \
  --secure true
  

Configuration Options

1.1.2 - Agent Installation Requirements

Sysdig agents can be installed on a wide array of Linux hosts. Check your environment to ensure it meets the minimum supported platform, operating system, runtime, and orchestration requirements and uses the appropriate installation instructions.

Versioning Scheme

We recommend that you use the latest version of the agent. Sysdig supports n-3 versions back based on the minor number. For example, if the latest release is v12.0.0, we will support n-3 versions back, up to v11.2.0.

End of Support

Sysdig agents that are older than version 0.85.1, released October 1, 2018, will no longer connect to the Sysdig US-East SaaS platform with default agent values.

Going forward all the agent releases will have a 3-year deprecation policy. This implies:

• Sysdig Support might not be able to help you troubleshoot or address the problems with agents past the deprecation date.

• Sysdig will no longer provide prebuilt kernel probe binaries for these agent releases. You need to build the kernel probe binaries on the fly by using the hosts kernel headers.

  These changes is effective starting Sysdig agent v12.1.0.

Agent Installation Requirements

Orchestration Platforms

Tested and Supported

Sysdig agent has been tested on the following list of latest Kubernetes versions. The matrix provides a single view into the supported operating systems, architecture, and runtime versions for different flavors of Kubernetes orchestrators.

Supported

Sysdig agent is supported on the following orchestration platforms:

If you are not using an orchestrator in your environment, follow the instructions for Agent Install Non-Orchestrated.

Note: Installing the Sysdig agent into a namespace managed by Istio and configured for sidecar auto-injection is not supported. For example, setting kubectl label namespace sysdig-agent istio-injection=enabled. Because the agent behaves more like a host component, it is required to be part of the host PID and network namespace to function correctly. Due to this requirement, deploying the Sysdig agent in Istio with an Envoy sidecar is not supported. However, running the Sysdig agent in a non-injected namespace where Istio is installed and managing other namespaces is fully supported. See Istio integration for more details on using the Sysdig agent to monitor Istio control plane and sidecar metrics.

Linux Distributions and Kernels

Tested and Supported

Sysdig agent (installed as a service) has been tested on the following list of latest linux distros:

Supported

Sysdig agent is supported on the following Linux distributions:

Container Runtimes

Sysdig agent supports the detection of the following:

• Docker
• LXC
• CRI-O
• containerd
• Podman
• Mesos

Tested and Supported for Docker

Sysdig agent has been tested on the following list of latest linux distros:

Prerequisites for Podman Environments

Sysdig agent supports running as a Podman container.

• Enable Podman API Service for all the users.

  The agent will not able to collect Podman-managed container metadata, such as the container name, if the API service is not enabled.

• Secure rules and policies that depend on container metadata other than the container ID will not work.

• Pausing and terminating containers will not work because Policy actions for Podman are not supported.

• The containers started as a non-root user will have the podman_owner_uid label associated with it if the API service is enabled for that user. The value of podman_owner_uid will be the numeric user ID corresponding to the user that started the container.

Container Registries

Quay.io

For example, to pull the latest agent container from Quay.io:

docker pull quay.io/sysdig/agent

CPU Architectures

x86

Supported Agent Containers

• agent
• agent-slim
• agent-kmodule

ARM (aarch64)

Supported kernel versions are v4.17 and above

Unsupported Features

• Pre-built probes
• Activity Audit
• Sysdig agent installation using the agent container

s390x (zLinux)

Unsupported Features

Probes

No support for pre-built probes on zLinux. For kernel instrumentation, use the kernel module. eBPF probes are not supported on zLinux.

Captures

Capture is not supported on zLinux.

Legacy Agent Installation

Sysdig agent installation using agent container is not supported.

Java Versions and Vendors

Sysdig agent supports the following:

• Java versions: v7 and above
• Vendors: Oracle, OpenJDK

For Java-based applications (Cassandra, Elasticsearch, Kafka, Tomcat, Zookeeper and etc.), the Sysdig agent requires the Java runtime environment (JRE) to be installed to poll for metrics (beans).

If the Docker-container-based Sysdig agent is installed, the JRE is installed alongside the agent binaries and no further dependencies exist. However, if you are installing the service-based agent (non-container) and you do not see the JVM/JMX metrics reporting, your host may not have the JRE installed or it may not be installed in the expected location: usr/bin/java

Minimum Resource Requirements

The resource requirements of the agent are subjective to the size and load of the host— more activity equates to more resources required.

It is typical to see between 5-20KiB/s of bandwidth consumed—different variables can increase the throughput required such as the number of metrics, events, Kubernetes objects, and which products and features are enabled. When a Sysdig Capture is being collected, you can expect to see a spike in bandwidth while the capture file is being ingested.

We do not recommend placing bandwidth shaping or caps on the agent to ensure data can be sent to our collection service.

Additional Requirements

Access key

The installation of the Sysdig agent requires an access key.

This key and the agent installation instructions are presented to you after activating your account and using a web-based wizard upon initial login.

The same information can also be found in the Settings > Agent Installation menu of the web interface after logging in. See Agent Installation: Overview and Key for details.

Network connection

A Sysdig agent (containerized or native) is installed into each host being monitored and will need to be able to connect to the Sysdig Monitor backend servers to report host metrics. The agent must be able to reach the Sysdig Collector addresses. For example, for US East, it is ‘collector.sysdigcloud.com’ (via multiple IPs) over port tcp/6443 . See Sysdig Collector Ports for supported ports for other regions.

The agent supports the HTTP proxy for communicating with Sysdig backend components. For more information, see Enable HTTP Proxy for Agents.

1.1.3 - Agent Install: Kubernetes

The recommended method to monitor Kubernetes environments is to deploy the Sysdig agent using the helm chart. Alternatively, you can install the agent container using DaemonSet. This section helps you install the agent in both the methods.

Installing the agent using helm or as a daemonSet will deploy agent containers on every node in your Kubernetes environment. Once the agent is installed, Sysdig Monitor automatically begins monitoring all of your hosts, apps, pods, and services and automatically connects to the Kubernetes API server to pull relevant metadata about the environment. If licensed, Sysdig Secure launches with default policies that you can view and configure to suit your needs. You can access the front-end web interfaces for Sysdig Monitor and Sysdig Secure immediately.

Sysdig supports monitoring numerous Kubernetes platforms, including the following:

• Amazon environments (EKS,EC2, ECS)
• Google Kubernetes Engine (GKE)
• Red Hat OpenShift
• Oracle Kubernetes Engine (OKE)
• IBM Cloud Kubernetes Service (IKS)
• Rancher
• Mirantis Kubernetes Engine (MKE)
• Azure Container Service (AKS)

Prerequisites

• A supported distribution: See Agent Installation Requirements for details.

• Kubernetes v1.9+: The agent installation on Kubernetes requires v1.9 or higher because the APIs used to fetch kubernetes metadata are only present in v1.9+.

• Sysdig account and access key: Request a trial or full account at Sysdig.com and click the Activate Account button. The Get Started page in Sysdig Monitor or the Data Sources page in Secure provide an access key.

• Port 6443 open for outbound traffic: The agent communicates with the collector on port 6443. If you are using a firewall, you must open port 6443 for outbound traffic for the agent.

• Kernel headers installed: If a prebuilt kernel probe is not available for your kernel, the kernel headers must be installed in order to build the kernel probe.

• kubectl installed: All of the installation methods utilize kubectl to install the agent in the cluster.

Kernel Headers

The Sysdig agent requires kernel header files to install successfully on a Kubernetes cluster. If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.

In some cases, the nodes in your Kubernetes environment might use Unix versions that do not match the provided headers, and the agent might fail to install correctly. In those cases, you must install the kernel headers manually on each node.

To do so:

For Debian-style distributions, run the command:

apt-get -y install linux-headers-$(uname -r)

For RHEL-style distributions, run the command:

yum -y install kernel-devel-$(uname -r)

For more information on troubleshooting, see About Kernel Headers and the Kernel Module.

Kubernetes Environments

Some Kubernetes environments require special configuration options to deploy the agent. If you’re installing in one of the following environments, follow the guides specific to those environments to deploy the agent. Otherwise, continue with this topic.

• IBM Cloud Kubernetes Service (IKS)
• Google Kubernetes Engine (GKE)
• Oracle Kubernetes Engine (OKE)
• OpenShift
• Rancher

Installation

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Script

Sysdig also provides a script that you can use to install the agent as a DaemonSet.

Installation

1. Download the script and make it executable.

    wget https://download.sysdig.com/stable/install-agent-kubernetes
    chmod +x install-agent-kubernetes
   

2. Run the script to install the agent as a DaemonSet.

   ./install-agent-kubernetes -a <ACCESS_KEY> -c <COLLECTOR_URL> -cn <CLUSTER_NAME>
   

Options

Manifests

To deploy agents using Kubernetes manifests, you can download manifest files, edit them as required, and deploy them using kubectl.

1. Download the sample files:

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create a namespace for the Sysdig agent.

   Note: You can use whatever name you prefer. This example uses sysdig-agent for both the namespace and the service account. The default service account name was automatically defined in sysdig-agent-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent

   kubectl create ns sysdig-agent
   

3. Create a secret key:

   kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. Create a cluster role and service account, and define the cluster role binding that grants the Sysdig agent rules in the cluster role:

   kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   kubectl create serviceaccount sysdig-agent -n sysdig-agent
   kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector address , port , and the SSL/TLS information:

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, and a CA-signed cert is used. See Set Up SSL Connectivity to the Backend for more information.

6. Apply the sysdig-agent-configmap.yaml file:

   kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. Apply the sysdig-agent-service.yaml file:

   kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
   

   This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

8. Apply the daemonset-v2.yaml file :

   kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
   

Additional Options

Verify Metrics in Sysdig Monitor

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

The steps below give one way to do the check.

1. Access Sysdig Monitor:

   SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

   For other regions, the format is https://<region>.app.sysdig.com). Replace ; with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

   Log in with your Sysdig user name and password.

2. Select the Explore tab to see if metrics are displayed.

3. Determine the Kube State Metrics you want to collect.

4. To verify that kube state metrics and cluster name are working correctly, select the Explore tab and see if your cluster is listed.

   

Kubernetes metadata (pods, deployments etc.) appear a minute or two later than the nodes/containers themselves; if pod names do not appear immediately, wait and retry the Explore view.

If agents are disconnecting, there could be an issue with your MAC addresses. See Troubleshooting Agent Installation for tips.

Verify Agent Installation in Sysdig Secure

1. Log in to Sysdig Secure.

   See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region.

2. Navigate to Integrations > Sysdig Agents.

   The Sysdig Agents page list all the agents installed your environment. For more information, see Sysdig Agents.

1.1.3.1 - Steps for GKE

• Installing on GKE Standard
• Installing on GKE Autopilot

1.1.3.1.1 - GKE Autopilot

Autopilot is an operation mode for creating and managing clusters in GKE. In brief, with Autopilot, Google configures and manages the underlying node infrastructure for you. This topic helps you use helm to install Sysdig agent on a GKE cluster installed in Autopilot mode.

NodeAnalyzer is not supported on Autopilot environments.

Prerequisites

1. Install a GKE cluster in Autopilot mode.

2. Connect the GKE cluster.

3. Install your workload.

Deploy Sysdig Agent

Sysdig recommends using Helm to install Sysdig agent in kubernetes environments. After connecting to the GKE cluster, use the sysdig-deplpy chart to install Sysdig agent.

To customize the configuration of the agent, see the Sysdig Agent Helm Chart.

Verify Agent Installation in Sysdig Secure

1. Log in to Sysdig Secure.

   See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region.

2. Navigate to Integrations > Sysdig Agents.

   The Sysdig Agents page list all the agents installed your environment. For more information, see Sysdig Agents.

Verify Metrics on the Sysdig Monitor UI

Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.

Given below is one way to do so.

1. Access Sysdig Monitor:

   SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.

   For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysdig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.

   Log in with your Sysdig user name and password.

2. Select the Explore tab to see if metrics are displayed.

3. Verify that kube state metrics and cluster name are working correctly: select the Explore tab and create a grouping by kube_cluster_name and kube_pod_name.

4. Select an individual container or pod to see the details.

1.1.3.1.2 - GKE Standard

Google Kubernetes Engine (GKE) is a managed environment for running Kubernetes in Google Cloud, in order to deploy containerized applications. Sysdig supports all flavors of GKE, including Ubuntu and GKE’s default Container-Optimized OS(COS).

GKE COS environments require eBPF probe to support agent installation.

Preparation

Open Port 6443 for Agent Egress

Because GKE uses stateful firewalls, you must actively open port 6443 for the Sysdig agent outbound traffic.

In earlier versions, the Sysdig Agent connected to port 6666. This behavior has been deprecated, as the Sysdig agent now connects to port 6443.

GKE COS/eBPF-Specific Requirements

• Linux kernel version >= 4.14.

• When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7. Note that only the eBPF probe is supported in GKE COS environments.

Prerequisites

You can review Agent Install: Kubernetes and the Agent Installation Requirements for additional context, if desired.

Installation Steps

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

To deploy agents using Kubernetes manifests, you can download the manifest files, edit them as required, and deploy them using kubectl.

1. Download the sample files:

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create a namespace to use for the Sysdig agent.

   You can use whatever name you want. In this document, we used sysdig-agent for both the namespace and the service account.

   kubectl create ns sysdig-agent
   

3. Create a secret key:

   kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. If you are running Kubernetes 1.6 or higher, you must grant your user the ability to create roles in Kubernetes by running the following command:

   kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org
   

   See Google documentation for more information.

   Create a service account for the Sysdig agent using the clusterrole.yaml file.

   The Sysdig agent must be granted read-only access to certain Kubernetes APIs, which the agent uses to populate metadata and provide component metrics.

   You can use the Sysdig-provided, sysdig-agent-clusterrole.yaml file. Deploying this file creates a cluster role and service account in Kubernetes, and defines cluster role binding that grants the Sysdig agent rules in the cluster role.

   Run the following commands (using whatever namespace you’ve defined in Step 2):

   kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   kubectl create serviceaccount sysdig-agent -n sysdig-agent
   kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information :

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used.

     See Set Up SSL Connectivity to the Backend for more information.

6. Apply the sysdig-agent-configmap.yaml file:

   kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. FOR GKE COS ONLY: To enable the eBPF probe required for COS, uncomment the following parameters in  sysdig-agent-daemonset-v2.yaml under the env section:

   env:
     - name: SYSDIG_BPF_PROBE
       value: ""
   

8. Apply the sysdig-agent-service.yaml file:

   kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
   

   This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

9. Apply the daemonset-v2.yaml file:

   kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
   

The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Additional Options.

1.1.3.2 - Steps for OKE

Oracle Kubernetes Engine (OKE) is a managed environment for running Kubernetes in Oracle Cloud, in order to deploy containerized applications. As of Sysdig agent version 12.0.1, Sysdig supports all flavors of OKE.

OKE environments require eBPF probe to support agent installation.

The instructions below describe a standard OKE agent install and call out the special steps needed to install the eBPF probe.

Preparation

Open Port 6443 for Agent Egress

Because OKE uses stateful firewalls, you must actively open port 6443 for the Sysdig agent outbound traffic.

OKE by default allows network access to the sysdig Agent on 6443, but ensure that firewall rules are open and the agent can connect to the Sysdig backends.

eBPF-Specific Requirements

• Linux kernel version >= 4.14.

• When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below.

Installation Steps

Identify the appropriate endpoint depending on your Sysdig account region. For more information, see SaaS Regions and IP Ranges. More info here https://docs.sysdig.com/en/docs/administration/saas-regions-and-ip-ranges/

After making clear which region your account belongs to, please choose one of the following methods:

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

To deploy agents using Kubernetes manifests, you can downloadmanifest files, edit them as required, and deploy them using kubectl.

1. Download the sample files:

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create a namespace to use for the Sysdig agent.

   Notes: You can use whatever name you want. In this document, we used sysdig-agent for both the namespace and the service account.

   kubectl create ns sysdig-agent
   

3. Create a secret key:

   kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. If you are running Kubernetes 1.6 or higher, you must create a service account for the Sysdig agent by using the clusterrole.yaml file.

   The Sysdig agent must be granted read-only access to certain Kubernetes APIs, which the agent uses to populate metadata and provide component metrics.

   You can use the Sysdig-provided sysdig-agent-clusterrole.yaml file. Deploying it create a cluster role and service account in Kubernetes, and defines cluster role binding that grants the Sysdig agent rules in the cluster role.

   Run the following commands by using the namespace you’ve defined in Step 2:

   kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   kubectl create serviceaccount sysdig-agent -n sysdig-agent
   kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information :

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Set Up SSL Connectivity to the Backend.

6. Apply the sysdig-agent-configmap.yaml file:

   kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. To enable the eBPF probe uncomment the following parameters in  sysdig-agent-daemonset-v2.yaml under the env section:

   env:
     - name: SYSDIG_BPF_PROBE
       value: ""
   

8. Apply the sysdig-agent-service.yaml file:

   kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
   

   This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

9. Apply the daemonset-v2.yaml file:

   kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
   

   The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Additional Options.

1.1.3.3 - Steps for OpenShift

You can review Agent Install: Kubernetes and the Agent Installation Requirements for additional context, if desired.

RHCOS environments require eBPF probe to support agent installation.

Preparation

RHCOS/eBPF-Specific Requirements

• Linux kernel version 4.14 or above.
• When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below.

Kernel Headers

The Sysdig agent requires kernel header files to install successfully on a Kubernetes cluster. If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.

In some cases, the nodes in your Kubernetes environment might use Unix versions that do not match the provided headers, and the agent might fail to install correctly. In those cases, you must install the kernel headers manually on each node.

To do so:

For RHEL-style distributions, run:

yum -y install kernel-devel-$(uname -r)

For more information on troubleshooting, see About Kernel Headers and the Kernel Module.

Configure for OpenShift

If you are using Red Hat OpenShift, these steps are required. They describe how to create a project, assign and label the node selector, create a privileged service account, and add it to a cluster role.

Copy/Paste Sample Code Block

In the example code, this document uses sysdig-agent for the PROJECT NAME (-n) and the SERVICE ACCOUNT (-z).

You can copy and paste the code as is, or follow the steps below to customize your naming conventions.

oc adm new-project sysdig-agent --node-selector=''
oc project sysdig-agent
oc create serviceaccount sysdig-agent
oc adm policy add-scc-to-user privileged -n sysdig-agent -z sysdig-agent -z node-analyzer
oc adm policy add-cluster-role-to-user cluster-reader -n sysdig-agent -z sysdig-agent -z node-analyzer

Customize the Code

You can use your own Project name and Service Account name if desired.

Note that if you use a different Service Account name, you will need to edit the default service account in the Sysdig Installation Steps, below.

1. Create a new OpenShift project for the Sysdig agent deployment and use an empty string for the node selector:

   oc adm new-project PROJECT-NAME --node-selector=""
   

2. Change to the new OpenShift Project for the Sysdig agent deployment:

   oc project PROJECT-NAME
   

3. Create a service account for the project:

   oc create serviceaccount SERVICE-ACCOUNT
   

4. Add the service account to privileged Security Context Constraints:

   oc adm policy add-scc-to-user privileged -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
   

5. Add the service account to the cluster-reader Cluster Role:

   oc adm policy add-cluster-role-to-user cluster-reader -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
   

Installation

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

1. Download the sample files:

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create the sysdig-agent cluster role and assign it to the service account:

    oc apply -f sysdig-agent-clusterrole.yaml
    oc adm policy add-cluster-role-to-user sysdig-agent -n PROJECT-NAME -z SERVICE-ACCOUNT
   

3. Create a secret key:

   oc create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n PROJECT-NAME
   

4. If you created a service account name other than sysdig-agent: Edit sysdig-agent-daemonset-v2.yamlto provide your custom value:``

   serviceAccount: SERVICE-ACCOUNT
   

5. Edit sysdig-agent-configmap.yaml to add the collector address, port, and the SSL/TLS information:

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.
   • For On-prem, enter the collector endpoint defined in your environment.
   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Set Up SSL Connectivity to the Backend.

6. Apply the sysdig-agent-configmap.yaml file:

   oc apply -f sysdig-agent-configmap.yaml -n PROJECT-NAME
   

7. FOR RHCOS ONLY: To enable the eBPF probe required for COS, uncomment the following parameters in sysdig-agent-daemonset-v2.yaml under the env section:`

   env:
     - name: SYSDIG_BPF_PROBE
       value: ""
   

8. Apply the sysdig-agent-service.yaml file:

   oc apply -f sysdig-agent-service.yaml -n PROJECT-NAME
   

   This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

9. Apply the daemonset-v2.yaml file:

   oc apply -f sysdig-agent-daemonset-v2.yaml -n PROJECT-NAME
   

   The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Additional Options.

1.1.3.4 - Steps for Rancher

Preparation

General Requirements

You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.

Kernel Headers

The Sysdig agent requires a kernel module in order to be installed successfully on a host. On RancherOS distributions, the Unix version does not match the provided headers, and the agent might fail to install correctly. Therefore, you must install the kernel headers manually.

For RancherOS distributions, the kernel headers are available in the form of a system service and therefore are enabled using the ros service command:

$ sudo ros service enable kernel-headers-system-docker
$ sudo ros service up -d kernel-headers-system-docker

Some cloud hosting service providers supply pre-configured Linux instances with customized kernels. You may need to contact your provider’s support desk for instructions on obtaining appropriate header files, or for installing the distribution’s default kernel.

Installation

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Manifests

To deploy agents using Kubernetes manifests, you can download the manifest files, edit them as required, and deploy them using kubectl.

1. Download the sample files:

   • sysdig-agent-clusterrole.yaml

   • sysdig-agent-daemonset-v2.yaml

   • sysdig-agent-configmap.yaml

   • sysdig-agent-service.yaml

2. Create a namespace to use for the Sysdig agent.

   You can use whatever naming you prefer. This document uses sysdig-agent for both the namespace and the service account.

   The default service account name was automatically defined in sysdig-agent-daemonset-v2.yaml, at the line: serviceAccount: sysdig-agent.

   kubectl create ns sysdig-agent
   

3. Create a secret key:

   kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
   

4. Create a cluster role and service account, and define the cluster role binding that grants the Sysdig agent rules in the cluster role:

   kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
   kubectl create serviceaccount sysdig-agent -n sysdig-agent
   kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
   

5. Edit sysdig-agent-configmap.yaml to add the collector address , port , and the SSL/TLS information:

   collector:
   collector_port:
   ssl: #true or false
   check_certificate: #true or false
   

   • For SaaS, find the collector address for your region.

   • For On-prem, enter the collector endpoint defined in your environment.

   • check_certificate should be set to false if a self-signed certificate or private, CA-signed cert is used. See also Set Up SSL Connectivity to the Backend.

6. Apply the sysdig-agent-configmap.yaml file:

    kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
   

7. Apply the sysdig-agent-service.yaml file:

   kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
   

This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.

8. Apply the daemonset-v2.yaml file :

   kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
   

The agents will be deployed and you can see some metrics in the Sysdig Monitor UI.

Next Steps

You can continue with instructions in Additional Options.

1.1.3.5 - Steps for MKE

Mirantis Kubernetes Engine (MKE) formerly Docker Enterprise, is a managed environment for running Kubernetes to deploy containerized applications. As of Sysdig agent version 12.0.1, Sysdig supports all flavors of MKE.

MKE environments require eBPF probe to support agent installation.

The instructions below describe a standard MKE agent install and call out the special steps needed to install the eBPF probe.

Preparation

eBPF-Specific Requirements

• Linux kernel versions 4.14 or above.

• eBPF probe parameter, --set ebpf.enabled=true to install eBPF probe. See the instructions given below.

Installation Steps

Identify the appropriate endpoint depending on your Sysdig account region. For more information, see SaaS Regions and IP Ranges.

Helm

Sysdig recommends using helm charts to install Sysdig agent in Kubernetes environments. For the latest chart and installation instructions, see sysdig-deploy.

Make sure to add the eBPF parameter to the helm command:

--set ebpf.enabled=true

Next Steps

You can continue with instructions in Additional Options.

1.1.3.6 - Using Node Leases

The Sysdig agent uses Kubernetes Lease to control how and when connections are made to the Kubernetes API Server. This mechanism prevents overloading the Kubernetes API server with connection requests during agent bootup.

Kubernetes node leases are automatically created for agent version 12.0.0 and above. On versions prior to 12.0.0, you must configure node leases as given in the KB article.

Prerequisites

• Sysdig Agent v11.3.0 or above

• Kubernetes v1.14 or above

Types of Leases

The agent creates the following leases:

Cold Start

During boot up, the Sysdig agent connects to the Kubernetes API server to retrieve Kubernetes metadata and build a cache. The cold-start leases control the number of agents that build up this cache at any given time. An agent will grab a lease, build its cache, and then release the lease so that another agent can build its cache. This mechanism prevents agents from creating a “boot storm” which can overwhelm the API server in large clusters.

Delegation

In Kubernetes environments, two agents are marked as delegated in each cluster. The delegated agents are the designated agents to request more data from the API server and produce KubeState metrics. The delegation leases will not be released until the agent is terminated.

View Leases

To view the leases, run the following:

$ kubectl get leases -n sysdig-agent

You will see an output similar to the following:

NAME           HOLDER             AGE
cold-start-0                      20m
cold-start-1                      20m
cold-start-2                      21m
cold-start-3   ip-10-20-51-167    21m
cold-start-4                      21m
cold-start-5                      21m
cold-start-6                      20m
cold-start-7                      21m
cold-start-8                      20m
cold-start-9   ip-10-20-51-166   21m
delegation-0   ip-10-20-52-53    21m
delegation-1   ip-10-20-51-98    21m

Troubleshoot Leases

Verify Configuration

When lease-based delegation is working as expected, the agent logs show one of the following:

• Getting pods only for node <node>

• Getting pods for all nodes.

• Both (occasionally on the delegated nodes)

Run the following to confirm that it is working:

$ kubectl logs sysdig-agent-9l2gf -n sysdig-agent | grep -i "getting pods"

The configuration is working as expected if the output on a pod is similar to the following:

2021-05-05 02:48:32.877, 15732.15765, Information, cointerface[15738]: Only getting pods for node ip-10-20-51-166.ec2.internal

Unable to Create Leases

The latest Sysdig ClusterRole is required for the agent to create leases. If you do not have the latest ClusterRole or if you have not configured the ClusterRole correctly, the logs show the following error:

Error, lease_pool_manager[2989554]: Cannot access leases objects: leases.coordination.k8s.io is forbidden: User "system:serviceaccount:sysdig-agent:sysdig-agent" cannot list resource "leases" in API group "coordination.k8s.io" in the namespace "sysdig-agent"

Contact Sysdig Support for help.

Optional Agent Configuration

Several configuration options exist for leases. It is recommended to not change the default settings unless prompted by Sysdig Customer Support.

k8s_coldstart:
  enabled: <true/false>

k8s_coldstart:
  max_parallel_cold_start: <int>

k8s_coldstart:
  namespace: <string>

k8s_coldstart:
  enforce_leader_election: <true/false>

k8s_delegation_election: <true/false>

1.1.4 - Agent Install: Non-Orchestrated

This section describes how to install the Sysdig agent directly on a Linux host, without using an orchestrator, such as Kubernetes or Mesos.

The agent can be installed in two ways:

• As a standard container

• As a non-containerized service

The steps for each flavor differ slightly depending on whether you are using the SaaS or on-premises version of the Sysdig platform.

If you are installing the Sysdig agent in an environment that has Kubernetes, use the Agent Install: Kubernetes instructions instead.

Prerequisites

• See Agent Installation Requirements for information on the following:

  • Supported Linux distributions

  • Network connection

  • Sysdig access key

  • Cloud service providers (AWS, Google, and Microsoft Azure) and any steps you may need to configure to integrate the Sysdig agent.

• kernel headers: The Sysdig agent requires kernel header files in order to install successfully on a host, and the agent is delivered with precompiled headers. If the hosts in your environment match the kernel versions included with the agent, no special action is needed .In some cases, the hosts in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernel headers manually. See About Kernel Headers and the Kernel Module for details.

• Run any commands as root or with the sudo command.

• Retrieve the Sysdig access key.

• Collect the configuration parameters.

Configuration Options

Installing Agent Using Containers

The Sysdig agent can be deployed as a docker container.

The commands below can also be copied from the Get Started page. In that case, your access key will already be included in the command automatically.

SaaS

Installing As Two Containers

The agent is installed by running sysdig/agent-kmodule, followed by running sysdig/agent-slim. See Installation Options for description about agent-slim and agent-kmodule.

Every host restart requires subsequent running of agent-kmodule and agent-slim containers.

1. Collect the configuration parameters.

2. Build and load the kernel module:

   If you are not using eBPF, use the following:

   docker run -it --privileged --rm --name sysdig-agent-kmodule \
   -v /usr:/host/usr:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules \
   quay.io/sysdig/agent-kmodule
   

   If you are using eBPF use the following:

   docker run -it --privileged --rm --name sysdig-agent-kmodule \
   -e SYSDIG_BPF_PROBE="" \
   -v /etc/os-release:/host/etc/os-release:ro \
   -v /root/.sysdig:/root/.sysdig \
   -v /usr:/host/usr:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   quay.io/sysdig/agent-kmodule
   

3. Configure kernel module to load during system boot.

   If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.

   $ sudo mkdir -p /etc/modules-load.d
   $ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
   

4. Run the agent module providing the access key and, optionally, user-defined tags:

   If you are not using eBPF, use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host \
   -e ACCESS_KEY=[ACCESS_KEY] \
   -e COLLECTOR=[COLLECTOR_ADDRESS] \
   [-e TAGS=[TAGS]]
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   --shm-size=512m \
   quay.io/sysdig/agent-slim
   

   If you are using eBPF use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host\
   -e ACCESS_KEY=[ACCESS_KEY] \
   -e COLLECTOR=[COLLECTOR_ADDRESS] \
   [-e TAGS=[TAGS]]
   -e SYSDIG_BPF_PROBE="" \
   -v /sys/kernel/debug:/sys/kernel/debug:ro \
   -v /root/.sysdig:/root/.sysdig \
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   --shm-size=512m \
   quay.io/sysdig/agent-slim
   

Installing As Single Container (Legacy)

1. Collect the configuration parameters.

2. Run the agent container providing the access key and, optionally, user-defined tags:

   If you are not using eBPF, use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host\
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
   -e TAGS=[TAGS] \
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   -v /usr:/host/usr:ro \
   --shm-size=512m \
   quay.io/sysdig/agent
   

   If you are using eBPF use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host\
    -e ACCESS_KEY=[ACCESS_KEY] \
    -e COLLECTOR=[COLLECTOR_ADDRESS] \
   -e TAGS=[TAGS] \
   -e SYSDIG_BPF_PROBE="" \
   -v /sys/kernel/debug:/sys/kernel/debug:ro \
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   -v /usr:/host/usr:ro \
   --shm-size=512m \
   quay.io/sysdig/agent
   

On-Premises

Installing As Two Containers

1. Collect the configuration parameters:

2. Build and load the kernel module:

   If you are not using eBPF, use the following:

   docker run -it --privileged --rm --name sysdig-agent-kmodule \
   -v /usr:/host/usr:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules \
   quay.io/sysdig/agent-kmodule
   

   If you are using eBPF use the following:

   docker run -it --privileged --rm --name sysdig-agent-kmodule \
   -e SYSDIG_BPF_PROBE="" \
   -v /etc/os-release:/host/etc/os-release:ro \
   -v /root/.sysdig:/root/.sysdig \
   -v /usr:/host/usr:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   quay.io/sysdig/agent-kmodule
   

3. Configure kernel module to load during system boot.

   If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.

   $ sudo mkdir -p /etc/modules-load.d
   $ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
   

4. Run the agent module providing the access key and, optionally, user-defined tags:

   If you are not using eBPF, use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host \
   -e ACCESS_KEY=[ACCESS_KEY] \
   -e COLLECTOR=[COLLECTOR_ADDRESS] \
   -e SECURE=true \
   -e CHECK_CERTIFICATE=true \
   [-e TAGS=[TAGS]]
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   -v /usr:/host/usr:ro \
   --shm-size=512m \
   quay.io/sysdig/agent-slim
   

   If you are using eBPF use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host \
   -e ACCESS_KEY=[ACCESS_KEY] \
   -e COLLECTOR=[COLLECTOR_ADDRESS] \
   -e SECURE=true \
   -e CHECK_CERTIFICATE=true \
   [-e TAGS=[TAGS]]
   -e SYSDIG_BPF_PROBE="" \
   -v /sys/kernel/debug:/sys/kernel/debug:ro \
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   -v /usr:/host/usr:ro \
   --shm-size=512m \
   quay.io/sysdig/agent-slim
   

Installing As Single Container (Legacy)

1. Collect the configuration parameters.

2. Run the agent module providing the access key and, optionally, user-defined tags:

   If you are not using eBPF, use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host \
   -e ACCESS_KEY=[ACCESS_KEY] \
   -e COLLECTOR=[COLLECTOR_ADDRESS] \
   -e SECURE=true \
   -e CHECK_CERTIFICATE=true \
   [-e TAGS=[TAGS]]
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   -v /usr:/host/usr:ro \
   --shm-size=512m \
   quay.io/sysdig/agent
   

   If you are using eBPF use the following:

   docker run -d --name sysdig-agent \
   --restart always \
   --privileged \
   --net host \
   --pid host \
   -e ACCESS_KEY=[ACCESS_KEY] \
   -e COLLECTOR=[COLLECTOR_ADDRESS] \
   -e SECURE=true \
   -e CHECK_CERTIFICATE=true \
   [-e TAGS=[TAGS]]
   -e SYSDIG_BPF_PROBE="" \
   -v /sys/kernel/debug:/sys/kernel/debug:ro \
   -v /var/run/docker.sock:/host/var/run/docker.sock \
   -v /dev:/host/dev \
   -v /proc:/host/proc:ro \
   -v /boot:/host/boot:ro \
   -v /lib/modules:/host/lib/modules:ro \
   -v /usr:/host/usr:ro \
   --shm-size=512m \
   quay.io/sysdig/agent
   

Installing Agent as a Service on Linux Host

Use these instructions to install the agent on the host itself, not in a container. Install on each host in the environment.

The command lines below can also be copy/pasted from the Welcome wizard or the Settings>Agent Installation page in the Sysdig Monitor interface.

In that case, your access key will already be included in the command automatically.

 >> Error, sdchecks[0] ModuleNotFoundError: No module named 'posix_ipc'

SaaS

1. Run the following command:

   curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] [--tags [TAGS]]
   

   Where [ACCESS_KEY] is your unique agent access key string. For example, 1234-your-key-here-1234. TAGS is an optional list of user-defined agent tags. For example, role:webserver,location:europe.

   See SaaS Regions and IP Ranges to find the collector endpoint for your region.

2. Restart the agent and start the service:

   sudo systemctl enable dragent
   

On-Premises

1. Run the following command:

   curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] --secure true --check_certificate true [--tags [TAGS]]
   

   For configuration parameters, see Configuration Options.

2. Restart the agent and start the service:

   sudo systemctl enable dragent
   

Connect to the Sysdig Backend via Static IPs (SaaS only)

Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com:

user@host:~$ docker run --name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
-e COLLECTOR=collector-static.sysdigcloud.com \
-e COLLECTOR_PORT=6443 \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
--shm-size=512m \
quay.io/sysdig/agent-slim

Guidelines for Manual Agent Installation

In the following cases, we recommend that you manually install the agent.

• Full control over the deployment process

• Integration with configuration management tools

• Custom kernel

• Unsupported distribution

See Agent Install: Manual Linux Installation for more information.

1.1.5 - Agent Install: Manual Linux Installation

Manual installation of the native Linux agent is recommended in the following cases:

• Full control over the deployment process

• Integration with configuration management tools

• Custom kernel

• Unsupported distribution (within Debian/Fedora flavors)

Otherwise, you may want to follow Agent Install: Non-Orchestrated

Note: If you are installing the Sysdig agent in an orchestrated infrastructure such as Kubernetes, Mesos/Marathon, use the respective Installation Guides:

• Agent Install: Kubernetes

• Agent Install: Mesos | Marathon | DCOS

Prerequisites

• Review the Agent Installation Requirements.

• Collect the configuration parameters:

  • ACCESS_KEY: Your unique access key string. Inability to retrieve the key indicates that the administrator of your instance might have it turned off for non-admin users. Contact your Sysdig administrator to receive the key. If you still have issues please contact Sysdig Support.

  • TAGS: The optional parameter you can use to list one or more tags for this host. Tagging your hosts is highly recommended. Agent Tags allow you to sort nodes of your infrastructure into custom groups in Sysdig Monitor. Replace the [TAGS] parameter above with a comma-separated list of TAG_NAME:TAG_VALUE. For example: role:webserver,location:europe

  • COLLECTOR: The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the Monitor UI or the Data Sources page in Secure. It is a custom value in on-prem installations.

  • SECURE: Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default.

• Run the commands as root or with sudo.

Installation

Follow the instructions for the appropriate Linux distribution:

Debian, Ubuntu

1. Use the Sysdig Monitor GPG key, configure the apt repository, and update the package list:

   curl -s https://download.sysdig.com/DRAIOS-GPG-KEY.public | apt-key add -
   curl -s -o /etc/apt/sources.list.d/draios.list http://download.sysdig.com/stable/deb/draios.list
   apt-get update
   

2. Install kernel development files.

   Note: The following command might not work with every kernel. Ensure that you customize the name of the package properly.

   apt-get -y install linux-headers-$(uname -r)
   

3. Install, configure, and restart the Sysdig agent.

   apt-get -y install draios-agent
   echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
   echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
   echo collector: COLLECTOR_URL >> /opt/draios/etc/dragent.yaml
   echo ssl: true >> /opt/draios/etc/dragent.yaml
   service dragent restart
   

   See Prerequisites for the configuration parameters required.

CentOS, RHEL, Fedora, Amazon AMI, Amazon Linux 2

1. Trust the Sysdig Monitor GPG key, configure the yum repository.

   $ rpm --import https://download.sysdig.com/DRAIOS-GPG-KEY.public
   $ curl -s -o /etc/yum.repos.d/draios.repo http://download.sysdig.com/stable/rpm/draios.repo
   

2. Install the EPEL repository.

   Note: The following command is required only if DKMS is not available in the distribution. You can verify if DKMS is available with yum list dkms.

   The command below contains a sample release number; be sure to update with the correct release.

   $ rpm -i http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm
   

3. Install kernel development files.

   Note: The following command might not work with every kernel. Make sure to customize the name of the package properly.

   $ yum -y install kernel-devel-$(uname -r)
   

4. Install, configure, and start the Sysdig agent.

   $ yum -y install draios-agent
   $ echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
   $ echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
   $ echo collector: COLLECTOR_URL >> /opt/draios/etc/dragent.yaml
   $ echo ssl: true >> /opt/draios/etc/dragent.yaml
   $ echo secure: true >> /opt/draios/etc/dragent.yaml
   $ sudo systemctl enable dragent
   $ sudo systemctl start dragent
   

   See Prerequisites for the configuration parameters required.

5. If you using a non-systemd Linux distribution, use the service command to start dragent.

   $ service dragent restart
   

Other Linux Distributions

The Sysdig agent is unsupported outside of the Debian, Fedora, and Amazon distributions.

1.1.6 - Agent Install: Amazon ECS

Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that helps to easily deploy, manage, and scale containerized applications.

This section describes how to install the Sysdig agent container on each underlying host in your ECS cluster. Once installed, the agent will automatically begin monitoring all of your hosts, service and tasks.

Installation

To install Sysdig agent on ECS, do the following:

• Create an ECS task definition for the Sysdig agent.

• Register the task definition in your AWS account.

• Create a service with the previous task definition to run the Sysdig agent in each of the nodes of your ECS cluster.

Create an ECS Task Definition

1. Collect the following configuration parameters:

• ACCESS_KEY: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.
• COLLECTOR: Use the collector address for your region. For more information, see SaaS Regions and IP Ranges.
• TAGS: The list of tags for the host where the agent is installed. For example: role:webserver, location:europe, role:webserver

2. Use the above values to customize the JSON snippet below and save it as a file named sysdig-agent-ecs.json.

Note that memory and cpu have both been set to 1024, depending on the size of your cluster you might want to tune those values.

{
  "family": "sysdig-agent-ecs",
  "containerDefinitions": [
    {
      "name": "sysdig-agent",
      "image": "quay.io/sysdig/agent-slim",
      "cpu": 1024,
      "memory": 1024,
      "privileged": true,
      "environment": [
        {
          "name": "ACCESS_KEY",
          "value": "$ACCESS_KEY"
        },
        {
          "name": "COLLECTOR",
          "value": "$COLLECTOR"
        },
        {
          "name": "TAGS",
          "value": "$TAG1,TAG2"
        }
      ],
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/host/boot",
          "sourceVolume": "boot"
        },
        {
          "containerPath": "/host/dev",
          "sourceVolume": "dev"
        },
        {
          "readOnly": true,
          "containerPath": "/host/lib/modules",
          "sourceVolume": "modules"
        },
        {
          "readOnly": true,
          "containerPath": "/host/proc",
          "sourceVolume": "proc"
        },
        {
          "containerPath": "/host/var/run/docker.sock",
          "sourceVolume": "sock"
        },
        {
          "readOnly": true,
          "containerPath": "/host/usr",
          "sourceVolume": "usr"
        }
      ],
      "dependsOn": [
        {
          "containerName": "sysdig-agent-kmodule",
          "condition": "SUCCESS"
        }
      ]
    },
    {
      "name": "sysdig-agent-kmodule",
      "image": "quay.io/sysdig/agent-kmodule",
      "memory": 512,
      "privileged": true,
      "essential": false,
      "mountPoints": [
        {
          "readOnly": true,
          "containerPath": "/host/boot",
          "sourceVolume": "boot"
        },
        {
          "containerPath": "/host/dev",
          "sourceVolume": "dev"
        },
        {
          "readOnly": true,
          "containerPath": "/host/lib/modules",
          "sourceVolume": "modules"
        },
        {
          "readOnly": true,
          "containerPath": "/host/proc",
          "sourceVolume": "proc"
        },
        {
          "containerPath": "/host/var/run/docker.sock",
          "sourceVolume": "sock"
        },
        {
          "readOnly": true,
          "containerPath": "/host/usr",
          "sourceVolume": "usr"
        }
      ]
    }
  ],
  "pidMode": "host",
  "networkMode": "host",
  "volumes": [
    {
      "name": "sock",
      "host": {
        "sourcePath": "/var/run/docker.sock"
      }
    },
    {
      "name": "dev",
      "host": {
        "sourcePath": "/dev/"
      }
    },
    {
      "name": "proc",
      "host": {
        "sourcePath": "/proc/"
      }
    },
    {
      "name": "boot",
      "host": {
        "sourcePath": "/boot/"
      }
    },
    {
      "name": "modules",
      "host": {
        "sourcePath": "/lib/modules/"
      }
    },
    {
      "name": "usr",
      "host": {
        "sourcePath": "/usr/"
      }
    }
  ],
  "requiresCompatibilities": [
    "EC2"
  ]
}

Register a Task Definition

Once your task definition is ready, ensure that you register it in your AWS account:

aws ecs register-task-definition \
    --cli-input-json file://sysdig-agent-ecs.json

Run the Agent as an ECS Service

Using the ECS task definition you have created, create a service in the cluster that you want to monitor with Sysdig.

aws ecs create-service \
    --cluster $CLUSTER_NAME \
    --service-name sysdig-agent-svc \
    --launch-type EC2 \
    --task-definition sysdig-agent-ecs \
    --scheduling-strategy DAEMON

With the agent installed, Sysdig will begin auto-discovering your containers and other resources of your ECS environment.

Using ECS Anywhere

If you’re using ECS Anywhere, change the launch type to EXTERNAL when the service is created.

aws ecs create-service \
    --cluster $CLUSTER_NAME \
    --service-name sysdig-agent-svc \
    --launch-type EXTERNAL \
    --task-definition sysdig-agent-ecs \
    --scheduling-strategy DAEMON

Enable Log Driver

You can send the logs from the containers running the ECS tasks to the log groups in CloudWatch Logs. You can send agent container log files to AWS by enabling the log driver, awslogs. To do so:

1. Add the following section to each of the container definitions you’ve created above:

               "logConfiguration": {
                   "logDriver": "awslogs",
                   "options": {
                       "awslogs-group": "$YOUR_LOG_GROUP",
                       "awslogs-region": "$AWS_REGION",
                       "awslogs-stream-prefix": "sysdig"
                   }
   

2. Update your task definition and the service to enable the logs.

1.1.7 - Agent Install: IKS (IBM Cloud Monitoring)

IBM Cloud maintains the documentation for Sysdig agent installation on IBM Cloud Kubernetes Service (IKS).

For more information, see the IBM Cloud Monitoring documentation:

• Getting Started

• Deploying a Monitoring Agent on a Standard Kubernetes Cluster

• Forwarding cluster and app metrics to IBM Cloud Monitoring

1.1.8 - Agent Install: Mesos | Marathon | DCOS

Marathon is the container orchestration platform for Mesosphere’s Datacenter Operating System (DC/OS) and Apache Mesos.

This guide describes how to install the Sysdig agent container on each underlying host in your Mesos cluster. Once installed, the agent will automatically connect to the Mesos and Marathon APIs to pull relevant metadata about the environment and will begin monitoring all of your hosts, apps, containers, and frameworks.

Prerequisites

• Review the Agent Installation Requirements.

• Collect the configuration parameters. You will need them to create the JSON file.

  • ACCESS_KEY: Your unique access key string. Inability to retrieve the key indicates that the administrator of your instance might have it turned off for non-admin users. Contact your Sysdig administrator to receive the key. If you still have issues please contact Sysdig Support.

  • COLLECTOR: The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the Monitor UI or the Data Sources page in Secure. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges.

  • COLLECTOR_PORT: The default is 6443. It is used in environments with Sysdig’s on-premises backend installed.

  • SECURE: Use a secure SSL/TLS connection to send metrics to the collector. It is used in environments with Sysdig’s on-premises backend installed.

  • CHECK_CERT: Determines strong SSL certificate check for Sysdig Monitor on-premises installation. Set to true when using SSL/TLS to connect to the collector service to ensure that a valid SSL/TLS certificate is installed. It is used in environments with Sysdig’s on-premises backend installed.

Installation

In this three-part installation, you:

• Deploy the Sysdig agent on all Mesos Agent (Slave) nodes, either automatically or by creating and posting a .json file to the leader Marathon API server.

• Deploy the Sysdig agent on the Mesos Master nodes.

• Special configuration steps: modify the Sysdig agent config file to monitor Marathon instances.

Deploy the Sysdig Agent on Mesos Agent Nodes

Preferred Option: Automatic install (DC/OS 1.11+)

If you’re using DC/OS 1.8 or higher, then you can find Sysdig agent in the Mesosphere Universe marketplace and install it from there.

It will automatically deploy the Sysdig agent container on each of your Mesos Agent nodes as a Marathon app.

Proceed to Deploy the Sysdig Agent.

Alternate Option: Post a .json file

If you are using a version of DC/OS earlier than 1.8 then:

1. Create a JSON file for Marathon, in the following format. See configuration parameters for details.

   {
     "backoffFactor": 1.15,
     "backoffSeconds": 1,
     "constraints": [
       [
         "hostname",
         "UNIQUE"
       ]
     ],
     "container": {
       "docker": {
         "forcePullImage": true,
         "image": "sysdig/agent",
         "parameters": [],
         "privileged": true
       },
       "type": "DOCKER",
       "volumes": [
         {
           "containerPath": "/host/var/run/docker.sock",
           "hostPath": "/var/run/docker.sock",
           "mode": "RW"
         },
         {
           "containerPath": "/host/dev",
           "hostPath": "/dev",
           "mode": "RW"
         },
         {
           "containerPath": "/host/proc",
           "hostPath": "/proc",
           "mode": "RO"
         },
         {
           "containerPath": "/host/boot",
           "hostPath": "/boot",
           "mode": "RO"
         },
         {
           "containerPath": "/host/lib/modules",
           "hostPath": "/lib/modules",
           "mode": "RO"
         },
         {
           "containerPath": "/host/usr",
           "hostPath": "/usr",
           "mode": "RO"
         }
       ]
     },
     "cpus": 1,
     "deployments": [],
     "disk": 0,
     "env": {
       "ACCESS_KEY": "ACCESS_KEY=YOUR-ACCESS-KEY-HERE",
       "CHECK_CERT": "false",
       "SECURE": "true",
       "TAGS": "example_tag:example_value",
       "name": "sdc-agent",
       "pid": "host",
       "role": "monitoring",
       "shm-size": "350m"
     },
     "executor": "",
     "gpus": 0,
     "id": "/sysdig-agent",
     "instances": 1,
     "killSelection": "YOUNGEST_FIRST",
     "labels": {},
     "lastTaskFailure": {
       "appId": "/sysdig-agent",
       "host": "YOUR-HOST",
       "message": "Container exited with status 70",
       "slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
       "state": "TASK_FAILED",
       "taskId": "sysdig-agent.3bb0759d-3fa3-11e9-b446-c60a7a2ee871",
       "timestamp": "2019-03-06T00:03:16.234Z",
       "version": "2019-03-06T00:01:57.182Z"
     },
     "maxLaunchDelaySeconds": 3600,
     "mem": 850,
     "networks": [
       {
         "mode": "host"
       }
     ],
     "portDefinitions": [
       {
         "name": "default",
         "port": 10101,
         "protocol": "tcp"
       }
     ],
     "requirePorts": false,
     "tasks": [
       {
         "appId": "/sysdig-agent",
         "healthCheckResults": [],
         "host": "YOUR-HOST-IP",
         "id": "sysdig-agent.0d5436f4-3fa4-11e9-b446-c60a7a2ee871",
         "ipAddresses": [
           {
             "ipAddress": "YOUR-HOST-IP",
             "protocol": "IPv4"
           }
         ],
         "localVolumes": [],
         "ports": [
           4764
         ],
         "servicePorts": [],
         "slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
         "stagedAt": "2019-03-06T00:09:04.232Z",
         "startedAt": "2019-03-06T00:09:06.912Z",
         "state": "TASK_RUNNING",
         "version": "2019-03-06T00:09:04.182Z"
       }
     ],
     "tasksHealthy": 0,
     "tasksRunning": 1,
     "tasksStaged": 0,
     "tasksUnhealthy": 0,
     "unreachableStrategy": {
       "expungeAfterSeconds": 0,
       "inactiveAfterSeconds": 0
     },
     "upgradeStrategy": {
       "maximumOverCapacity": 1,
       "minimumHealthCapacity": 1
     },
     "version": "2019-03-06T00:09:04.182Z",
     "versionInfo": {
       "lastConfigChangeAt": "2019-03-06T00:09:04.182Z",
       "lastScalingAt": "2019-03-06T00:09:04.182Z"
     }
   }
   

   See Environment Variables for Agent Config File for the Sysdig name:value definitions.

   Complete the “cpus”, “mem” and “labels” (i.e. Marathon labels) entries to fit the capacity and requirements of the cluster environment.

2. Update the created.json file to the leader Marathon API server:

   $ $curl -X POST http://$(hostname -i):8080/v2/apps -d @sysdig.json -H "Content-type: application/json"
   

Deploy the Sysdig Agent on Master Nodes

After deploying the agent to the Mesos Agent nodes, you will install agents on each of the Mesos Master nodes as well.

Use the Agent Install: Non-Orchestrated instructions to install the agent directly on each of your Mesos Master nodes.

When the Sysdig agent is successfully installed on the master nodes, it will automatically connect to the local Mesos and Marathon (if available) API servers via http://localhost:5050 and http://localhost:8080 respectively, to collect cluster configuration and current state metadata in addition to host metrics.

Additional Configuration

In certains situations, you may need to add additional configurations to the dragent.yaml file:

• If the Sysdig agent cannot be run directly on the Mesos API server.

• If the API server is protected with a username/password.

Descriptions and examples are shown below.

Sysdig Agent Unable to Run on the Mesos API Server

Mesos allows multiple masters. If the API server can not be instrumented with a Sysdig agent, simply delegate ONE other node with an agent installed to remotely receive infrastructure information from the API server.

NOTE: If you manually configure the agent to point to a master with a static configuration file entry, then automatic detection/following of leader changes will no longer be enabled.

Add the following Mesos parameter to the delegated agent’s dragent.yaml file to allow it to connect to the remote API server and authenticate, either by:

a. Directly editing dragent.yaml on the host, or

b. Converting the YAML code to a single-line format and adding it as an ADDITIONAL_CONF argument in a Docker command.

See Understanding the Agent Configuration for details.

Specify the API server’s connection method, address, and port. Also specify credentials if necessary.

YAML example:

mesos_state_uri: http://[acct:passwd@][hostname][:port]
marathon_uris:
  - http://[acct:passwd@][hostname][:port]

Mesos API Server Requires Authentication

If the agent is installed on the API server but the API server uses a different port or requires authentication, those parameters must be explicitly specified.

Add the following Mesos parameters to the API server’s dragent.yaml to make it connect to the API server and authenticate with any unique account and password, either by:

a. Directly editing dragent.yaml on the host, or

b. Converting the YAML code to a single-line format and adding it as an ADDITIONAL_CONF argument in a Docker command.

See Understanding the Agent Configuration for details.

Specify the API server’s protocol, user credentials, and port:

mesos_state_uri: http://[username:password@][hostname][:port]
marathon_uris:
  - http://[acct:passwd@][hostname][:port]

*HTTPS protocol is also supported.

Troubleshooting: Turning Off Metadata Reception

In troubleshooting cases where auto-detection and reporting of your Mesos infrastructure needs to be temporarily turned off in a designated agent:

1. Comment out the Mesos parameter entries in the agent’s dragent.yaml file.

   Example parameters to disable: mesos_state_uri, marathon_uris

2. If the agent is running on the API server (Master node) and auto-detecting a default configuration, you can add the line:

   mesos_autodetect: false

   either directly in the dragent.yaml file or as an ADDITIONAL_CONF parameter in a Docker command.

3. Restart the agent.

1.1.9 - Airgapped Agent Installation

Airgapped environments are those that do not have network access to the public internet.

At startup, the agent will try to compile its own version of the probes, provided kernel header packages are installed on the host. Failing that, the agent will try to download pre-compiled probes, sysdigcloud-probe-<suffix>.ko or sysdigcloud-probe-bpf-<suffix>.o, from the Sysdig download site over the internet.

In an airgapped environemnt, you cannot download these artifacts. Therefore, before installing the agent, you will have to compile sysdigcloud-probe-<suffix> for each kernel version in your environment, and make it available to the installed agents through an internally accessible URL.

Prerequisites

• A machine with internet access where you can download the required artifacts
• A machine in your airgapped environment where you can build your probes
• Tool to transfer artifacts to the machine in your airgapped environment
• Docker installed

Overview

Sysdig provides a tool, named the probe builder, to help you build the probes for different kernels and for a specific agent version. After downloading the required artifacts on a machine connected to the internet, you can copy them to an airgapped host, build your own probes, and make them available to your agent installations.

On a Machine with Internet Connectivity

Prepare the Sysdig Probe Builder Images

On a machine with internet connectivity, build the Sysdig probe builder container images and create a tar file of the images.

1. Get the probe builder source code from the repository:

   $ git clone https://github.com/draios/probe-builder
   

2. Build the container image for the probe builder:

   $ docker build -t airgap/sysdig-probe-builder probe-builder/
   

3. Build the images for each supported distribution-compiler combination:

   $ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock airgap/sysdig-probe-builder:latest -P -b airgap/
   

   Running this command will create a different image tag for each supported combination of distribution-compiler, with the distro-compiler information suffixed to the image name, airgap/sysdig-probe-builder. For example, airgap/sysdig-probe-builder:centos-gcc4.8.

4. Save all the above images to a tar archive:

   $ docker save airgap/sysdig-probe-builder | gzip > builders.tar.gz
   

5. (optional) If you are building probes for the Ubuntu kernels, you will also need an ubuntu:latest image on your airgapped host. You can build it as follows:

   $ docker pull ubuntu
   $ docker save ubuntu | gzip > ubuntu.tar.gz
   

Download the Kernel Packages

Download your kernel packages. For more information, see Download Kernel Packages.

Download Probe Source Code

You need to download the probe source code for a specific agent version you want to build your probes for.

For example, for agent version 12.0.0 you would use:

$ git clone https://github.com/draios/agent-libs
$ cd agent-libs
$ git archive agent/12.0.0 --prefix sysdig/ | gzip > sysdig.tar.gz

Transfer the Downloaded Files

Copy the artifacts you have built to the airgapped host machine:

• builders.tar.gz
• ubuntu.tar.gz (if needed, see above)
• sysdig.tar.gz
• Kernel packages

On the Airgapped Host

Load the Builder Images

$ zcat builders.tar.gz | docker load

Unpack the Sysdig Source

$ tar xzf sysdig.tar.gz

Running this command will create the sysdig/ directory in the current directory.

Move the Kernel Packages to a Dedicated Location

Make sure you have all the downloaded kernel package artifacts in a single directory, /directory-containing-kernel-packages/, for each distribution you want to support.

Run the Probe Builder

Now that you have all your requirements in place, you can run the main probe builder:

$ docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /a-directory-with-some-free-space/:/workspace \
  -v /wherever-you-unpacked/sysdig/:/sysdig \
  -v /directory-containing-kernel-packages/:/kernels \
  airgap/sysdig-probe-builder:latest -B -b airgap/ -- \
  -p sysdigcloud-probe -v 12.0.0 -k CustomCentOS

The probes will appear in /a-directory-with-some-free-space/output. That directory can be served over HTTP and the URL to the server used as SYSDIG_PROBE_URL when loading the module (e.g. agent-kmodule container). As an example, the following sections describe how you can deploy your own nginx server within your cluster and upload your probes there.

Serve Your Pre-Compiled Probes

Set up a local repository to host the pre-compiled kernel module. Here is an example with nginx:

$ docker run --rm -v /a-directory-with-some-free-space/output:/usr/share/nginx/html/stable/sysdig-probe-binaries -p 80:80 nginx

Note down the URL and use it as the SYSDIG_PROBE_URL while installing the agent.

​ See Run the Probe Builder.

Run the Probe Builder

$ docker run --rm \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /sysdigcloud-probe/:/workspace \
  -v /wherever-you-unpacked/sysdig/:/sysdig \
  -v /directory-containing-kernel-packages/:/kernels \
  airgap/sysdig-probe-builder:latest -B -b airgap/ -- \
  -p sysdigcloud-probe -v 12.0.0 -k CustomCentOS

The probes will appear in /sysdigcloud-probe/output. This directory can be served over HTTP and the URL to the server used as SYSDIG_PROBE_URL when loading the module. For example, agent-kmodule container.

Install Agent in a Docker Environment

1. Install Sysdig agent by pointing SYSDIG_PROBE_URL to the local repository:

   For docker-based installations:

   $ docker run -d --name sysdig-agent --restart always --privileged --net host --pid host -e ACCESS_KEY=WWWWW-YYYY-XXXX-ZZZZ-123456789 -e SECURE=true -e SYSDIG_PROBE_URL=http://www.mywebserver.net:80/ -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro --shm-size=512m sysdig/agent
   

   Where -e SYSDIG_PROBE_URL=http://www.mywebserver:80/ is the local nginx web server with the loaded module.

   Note: To use HTTPS communication with a self-signed or untrusted certificate, use the -e SYSDIG_PROBE_INSECURE_DOWNLOAD=true environment variable in the above command.

2. Check the agent log. If the installation is successful, you will see a message as follows:

   Evaluating override of environment variables
   
   Trying to download precompiled module from http://mywebserver:80/stable/sysdig-probe-binaries/sysdigcloud-probe-<version>
   
   Download succeeded
   

3. Continue with the instructions in Agent Install: Non-Orchestrated.

Install Agent in a Kubernetes Environment

1. Open your agent daemonset and update the SYSDIG_PROBE_URL to point to the local repository:

   - name: SYSDIG_PROBE_URL
     value: http://www.mywebserver:80/
   

   If you would like to use secure communication with a self-signed or untrusted certificate, apply the SYSDIG_PROBE_INSECURE_DOWNLOAD environment variable.

   - name: SYSDIG_PROBE_INSECURE_DOWNLOAD
     value: true
   

2. Continue with the instructions in Agent Install: Kubernetes.

1.1.10 - Identify Agent Version

Use one of the following methods to determine the version of the agents installed in your environment:

Sysdig Secure

The Sysdig Agents page displays all the agents deployed your environment. You can determine the version number as well as the agents that are up-to-date, out of date, or approaching being out of date.

1. Log in to Sysdig Secure.
2. Select Integrations > Sysdig Agents.

For more information, see Sysdig Agents.

Sysdig Monitor

Explore

Segmenting metrics by using agent.version shows the installed versions of agents in your environment. For example, segment the uptime metric across your environment by using agent.version . Hover over the graph to see the list of agent versions.

The image shows the list of agent versions in n/a.

Dashboard

Use the Sysdig Agent Health Dashboard to determine the agent versions:

1. Log in to the Sysdig Monitor.

2. Select Dashboards and expand Host Infrastructure Dashboards.

3. Open the Sysdig Agent Health & Status template or create your own from the template.

   

   The Sysdig Agent and Health & Status Dashboard shows the agent version corresponding to each host in your environment.

1.2 - Agent Configuration

Out of the box, the Sysdig agent will gather and report on a wide variety of pre-defined metrics from a range of platforms and applications. It can also accommodate any number of custom parameters for additional metrics collection.

You can edit the agent configuration file to extend the default behavior, including additional metrics for JMX, StatsD, Prometheus, or a wide range of other monitoring integrations.

Use this section when you need to change the default or pre-defined settings by editing the agent configuration files.

For the latest helm-based installation instructions and configuration options, see sysdig-deploy.

1.2.1 - Understand the Agent Configuration

Out of the box, the Sysdig agent will gather and report on a wide variety of predefined metrics. It can also accommodate any number of custom parameters for additional metrics collection.

The agent relies on a configuration file named dragent.yaml to define metrics collection parameters. This file is located in the /opt/draios/etc/ directory. You can add configuration parameters directly in YAML as key-value pairs, or using the environment variable such as ADDITIONAL_CONF.

The dragent.yaml file can be accessed and edited in several ways, depending on how the agent was installed. This document describes how to modify dragent.yaml.

Environments

For more information about configuring each of the three environments listed in this section, see Edit the Configuration File.

Kubernetes

If Sysdig agent is installed in a Kubernetes environment, you can edit the dragent.yaml file using one of the following options:

• values.yaml

• ConfigMap

• sysdig-deploy Helm chart

Non-Orchestrated

If Sysdig agent is installed in a non-orchestrated environment such as Docker, you can edit the dragent.yaml file using one of the following options:

• dragent.yaml
• docker run

Linux

If Sysdig agent is installed in a Linux host, edit the dragent.yaml file directly.

Edit the Configuration File

dragent.yaml

1. Log in to the host where the agent is installed.

2. Open /opt/draios/etc/dragent.yaml.

3. Edit the file using proper YAML syntax. See the examples at the bottom of the page.

4. Restart the agent for changes to take effect.

   • Linux: service dragent restart

   • Non-orchestrated: docker restart sysdig-agent

configmap.yaml

If you install the agent using DaemonSets on a Kubernetes cluster, you use configmap.yaml to connect with and manipulate the underlying dragent.yaml file.

Use the following ways to add parameters to configmap.yaml:

• Directly edit configmap.yaml in YAML as key-value pairs.

• Use environment variables such as ADDITIONAL_CONF.

You can edit the files locally and apply the changes with kubectl -f:

1. Open the configmap.yaml file.

2. Edit the file as needed.

3. Apply the changes:

   kubectl apply -f sysdig-agent-configmap.yaml

All the running agents will automatically pick the new configuration after Kubernetes pushes the changes across all the nodes in the cluster.

docker run

Run the docker run command with -e ADDITIONAL_CONF="<VARIABLES>" where <VARIABLES> contains all the customized parameters you want to include.

Convert YAML Parameters to Single-Line Format

To insert ADDITIONAL_CONF parameters in a docker run command or a DaemonSet file, you must convert the YAML code into a single line. You can do the conversion manually for short snippets. To convert longer portions of YAML, use echo|sed commands.

1. Write your configuration in YAML, as it would be entered directly in dragent.yaml.

2. In a Bash shell, use echo and sed to convert to a single line:

   echo "<YAML_CONTENT>" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'

3. Insert the resulting line into the docker run command or add it to the DaemonSet file as an ADDITIONAL_CONF.

Examples

Disable StatsD Collection

This example shows how to turn off StatsD collection and blacklist port 6443.

Sysdig agent uses port 6443 for both inbound and outbound communication with the Sysdig backend. The agent initiates a request and keeps a connection open with the Sysdig backend for the backend to push configurations, Falco rules, policies, and so on.

Ensure that you allow the agents’ inbound and outbound communication on TCP 6443 from the respective IP addresses associated with your SaaS Regions. Note that you are allowing the agent to send communication outbound on TCP 6443 to the inbound IP ranges listed in the SaaS Regions.

YAML Format

statsd:
    enabled: false
    blacklisted_ports:
    - 6443

Single-Line Format

Use spaces, hyphens, and \n correctly when manually converting to a single line:

ADDITIONAL_CONF="statsd:\n enabled: false\n blacklisted_ports:\n - 6443"

You can run a full agent startup Docker command in a single line as follows:

docker run
  --name sysdig-agent \
  --privileged \
  --net host \
  --pid host \
  -e ACCESS_KEY=1234-your-key-here-1234 \
  -e TAGS=dept:sales,local:NYC \
  -e ADDITIONAL_CONF="statsd:\n    enabled: false\n    blacklisted_ports:\n    - 6443" \
  -v /var/run/docker.sock:/host/var/run/docker.sock \
  -v /dev:/host/dev \
  -v /proc:/host/proc:ro \
  -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro \
  -v /usr:/host/usr:ro \
  quay.io/sysdig/agent

Add RabbitMQ App Check

This example helps you override the default configuration for a RabbitMQ app check.

YAML Format

app_checks:
  - name: rabbitmq
    pattern:
      port: 15672
    conf:
      rabbitmq_api_url: "http://localhost:15672/api/"
      rabbitmq_user: myuser
      rabbitmq_pass: mypassword
      queues:
        - MyQueue1
        - MyQueue2

Single-Line Format (echo | sed)

From a Bash shell, issue the echo command and sed script.

echo "app_checks:
  - name: rabbitmq
    pattern:
      port: 15672
    conf:
      rabbitmq_api_url: "http://localhost:15672/api/"
      rabbitmq_user: myuser
      rabbitmq_pass: mypassword
      queues:
        - MyQueue1
        - MyQueue2
" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'

This results in the single-line format to be used with ADDITIONAL_CONF in a Docker command or DaemonSet file.

"app_checks:\n - name: rabbitmq\n  pattern:\n    port: 15672\n  conf:\n    rabbitmq_api_url: http://localhost:15672/api/\n    rabbitmq_user: myuser\n    rabbitmq_pass: mypassword\n    queues:\n      - MyQueue1\n      - MyQueue2\n"

helm install

If you installed the Sysdig agent in Kubernetes using the Helm chart, then no configmap.yaml file was downloaded. You can edit dragent.yaml using the Helm syntax:

helm install \
  --namespace sysdig-agent \
  --set agent.sysdig.settings.tags='linux:ubuntu\,dept:dev\,local:nyc' \
  --set global.clusterConfig.name='my_cluster' \
  sysdig/sysdig-deploy

This command will be translated into the following:

data:
  dragent.yaml: |
    tags: linux:ubuntu,dept:dev,local:nyc
    k8s_cluster_name: my_cluster    

Environment Variables for Agent Configuration File

Here is a sample Docker command using environment variables:

docker run \
  --name sysdig-agent \
  --privileged \
  --net host \
  --pid host \
  -e ACCESS_KEY=3e762f9a-3936-4c60-9cf4-c67e7ce5793b \
  -e COLLECTOR=mycollector.elb.us-west-1.amazonaws.com \
  -e COLLECTOR_PORT=6443 \
  -e CHECK_CERTIFICATE=false \
  -e TAGS=my_tag:some_value \
  -e ADDITIONAL_CONF="log:\n file_priority: debug\n console_priority: error" \
  -v /var/run/docker.sock:/host/var/run/docker.sock \
  -v /dev:/host/dev \
  -v /proc:/host/proc:ro \
  -v /boot:/host/boot:ro \
  -v /lib/modules:/host/lib/modules:ro \
  -v /usr:/host/usr:ro \
  --shm-size=350m \
  quay.io/sysdig/agent

1.2.2 - Configure Agent Modes

Agent modes provide the ability to control metric collection to fit your scale and specific requirement. You can choose one of the following modes to do so:

• Monitor

• Monitor Light

• Troubleshooting

• Secure

• Secure Light

• Custom Metrics Only

Using a stripped-down mode limits collection of unneeded metrics, which in turn prevents the consumption of excess resources and helps reduce expenses.

Monitor

The Monitor mode offers an extensive collection of metrics. We recommend this mode to monitor enterprise environments.

monitor is the default mode if you are running the Enterprise tier. To switch back to the Monitor mode from a different mode, do one of the following:

• Add the following to the dragent.yaml file and restart the agent:

  feature:
    mode: monitor
  

• Remove the parameter related to the existing mode from the dragent.yaml file and restart the agent. For example, to switch from troubleshooting mode to monitor, delete the following lines:

  feature:
    mode: troubleshooting
  

Monitor Light

Monitor Light caters to the users that run agents in a resource-restrictive environment, or to those who are interested only in a limited set of metrics.

Monitor Light provides CPU, Memory, File, File system, and Network metrics. For more information, see Metrics Available in Monitor Light.

Enable Monitor Light Mode

To switch to the Monitor Light mode, edit the dragent.yaml file:

1. Open the dragent.yaml file.

2. Add the following configuration parameter:

   feature:
     mode: monitor_light
   

3. Restart the agent.

Troubleshooting

Troubleshooting mode offers sophisticated metrics with detailed diagnostic capabilities. Some of these metrics are heuristic in nature.

In addition to the extensive metrics available in the Monitor mode, Troubleshooting mode provides additional metrics such as net.sql and additional segmentation for file and network metrics. For more information, see Additional Metrics Values Available in Troubleshooting.

Enable Troubleshooting Mode

To switch to the Troubleshooting mode, edit the dragent.yaml file:

1. Open the dragent.yaml file.

2. Add the following configuration parameter:

   feature:
     mode: troubleshooting
   

3. Restart the agent.

Secure

The secure mode supports only Sysdig Secure features.

Sysdig agent collects no metrics in the secure mode, which, in turn, minimizes network consumption and storage requirement in the Sysdig backend. Lower resource usage can help reduce costs and improve performance.

In the Secure mode, the Monitor UI shows no data because no metrics are sent to the collector.

This feature requires agent v10.5.0 or above.

Enable Secure Mode

1. Open the dragent.yaml file.

2. Add the following:

   feature:
     mode: secure
   

3. Restart the agent.

Secure Light

The secure light mode supports only the following Sysdig Secure features:

• Runtime Policies
• Activity Audit (only executed commands)
• Captures

Sysdig agent running in secure_light mode consumes fewer resources than that of running in the secure mode.

This feature requires agent v12.10.0 or above.

Enable Secure Light

1. Open the dragent.yaml file.

2. Add the following:

   feature:
     mode: secure_light
   

3. Restart the agent.

Custom Metrics Only Mode

Custom Metrics Only mode collects the same metrics as the Monitor Light mode, but also adds the ability to collect the following:

• Custom Metrics: StatsD, JMX, App Checks, and Prometheus
• Kubernetes State Metrics

As such, Custom Metrics Only mode is suitable if would like to use most of the features of Monitor mode but are limited in resources.

This mode is not compatible with Secure. If your account is configured for Secure, you must explicitly disable Secure in the agent configuration if you wish to use this mode.

This mode requires agent v12.4.0 or above.

Enable Custom Metrics Only Mode

1. Open the dragent.yaml file.

2. Add the following configuration parameter:

   feature:
     mode: custom-metrics-only
   

3. If your account is enabled for Secure, add the following:

   security:
     enabled: false
   secure_audit_streams:
     enabled: false
   falcobaseline:
     enabled: false
   

   This configuration explicitly disables the Secure features in the agent. If you do not disable Secure, the agent will not start due to incompatiblity issues.

4. Restart the agent.

1.2.2.1 - Metrics Available in Monitor Light

Monitor Light provides cpu, memory, file, file system, and network metrics.