Welcome to the Sysdig Documentation Hub!
Explore Sysdig’s brand-new documentation site using the table of contents placed at the left of this paragraph.
We encourage you to send your feedback at doc-feedback@sysdig.com.
This the multi-page printable view of this section. Click here to print.
Welcome to the Sysdig Documentation Hub!
Explore Sysdig’s brand-new documentation site using the table of contents placed at the left of this paragraph.
We encourage you to send your feedback at doc-feedback@sysdig.com.
This guide describes deployment options for various Sysdig components:
Serverless Agents used for container-based cloud environments such as Fargate
Sysdig Secure for cloud connection components: CIS Benchmarks, threat detection based on cloud provider native logs and compute resources and registry image scanning.
Rapid Response component, allowing designated Advanced Users to remote connect into a host (available for Sysdig Secure on-prem only)
Node Analyzer: Multi-Feature Installation for benchmarks, host scanning, and the image analyzer.
Admission Controller: Installation for enhanced scanning, where container images that do not fulfill the configured admission policies will be rejected from the cluster before being assigned to a node and allowed to run.
Sysdig agents are simple to deploy and upgrade, and out of the box, they will gather and report on a wide variety of pre-defined metrics. Agent installation can be as simple as copying and pasting a few lines of code from a Wizard prompt, or you can follow step-by-step instructions to check supported environments and distributions, review installation options, and customize the agent configuration file post-install.
Sysdig agent is a lightweight host component that processes syscall, creates capture files, and performs auditing and compliance. It is a platform that supports monitoring, securing, and troubleshooting cloud-native environments
In addition, the agent performs the following:
Metrics processing, analysis, aggregation, and transmission
Policy monitoring and alert notification through bidirectional communication with the Sysdig collector
Integration with third-party software for consolidating customer ecosystem data
Full assimilation into containerized and orchestrated environment
Matches runtime policies against the syscall events and generate policy events
Agent Installation Requirements: Determine supported platforms and system requirements for installing Sysdig Agent.
Agent Licensing: Retrieve the agent access key and understand the subscription model.
Installing Agent: Install Sysdig agents on supported platforms.
Configuring Agent: Edit the agent configuration file to filter incoming metrics, change log levels, and more.
Using Agent Modes: Edit the agent configuration file to configure agent modes.
Upgrading Sysdig Agent: Upgrade Sysdig agent installed as a service or as a container.
Troubleshooting Sysdig Agent: Troubleshoot common Sysdig agent issues.
Sysdig agents are delivered as either a container or a service and can be deployed with or without an orchestrator such as Kubernetes or Mesos.
A quick install involves just a few lines of code from the Getting Started wizard copied into a shell. The complete install instructions address checking for and installing kernel headers if needed, any prerequisite permissions settings for particular environments, and tips about updating the configuration files after initial installation.
Topic | Description |
---|---|
Review the platforms, runtimes, Linux distributions, orchestration, browsers, etc. that are supported. | |
An agent access key is provided with a Sysidg trial | |
Different ways in which you can install Sysdig agent. | |
Troubleshooting tips for agent installation, tuning agents, and compiling kernel modules. |
In the default mode of agent installation, you install the agent package as two containers, each container responsible for different functions as given below. The agent-slim
reduces the surface area of attack for potential vulnerabilities and is, therefore, more secure.
agent-kmodule
: Responsible for downloading and building the kernel
module. The image is short-lived. The container exits after the
kernel module is loaded. The transient nature of the container
reduces the time and opportunities for exploiting any potential
vulnerabilities present in the container image.
Prerequisites: The package depends on Dynamic Kernel Module
Support (DKMS) and requires the compiler and kernel headers
installed if you are using the agent-kmodule
to build the kernel
probe. Alternatively, you can use it without the kernel headers. In
such cases, the agent-kmodule
will attempt to download a pre-built
kernel probe if it is present in the Sysdig probe repository.
The module contains:
The driver sources
A post-install script that builds the module upon installation
agent-slim
: Responsible for running the agent module once the
kernel module has been loaded. Slim agent functions the same way as the regular agent and retains the feature parity.
Use the instruction below to install agent on your chosen environment:
Environment | Flavor | Installation | Install Instructions |
---|---|---|---|
Kubernetes | Open Source | Helm and manual | Install Agent on Kubernetes |
OpenShift | Helm and manual | Install Agent on OpenShift | |
GKE | Helm and manual | Install Agent on GKE | |
Rancher | Helm and manual | Install Agent on Rancher | |
OKE | Helm and manual | Install Agent on OKE | |
Non-Orchestrated | Manual | Install Agent on Non-Orchestrated Environment |
Legacy Agent: The legacy agent can be run as a single container or a service. It includes the components for downloading and building the kernel module, as well as for gathering and reporting on a wide variety of pre-defined metrics. For more information, see Installing Agent as a Single Container.
Helm is the preferred way of installing Sysdig agent. It is used in most cloud environments, for example, Amazon EKS or EC2 on AWS Cloud or AWS Outpost, EC2, and Azure AKS.
With the Getting Started wizard, you can copy a simple line of code to deploy agents in a variety of environments.
Behind the scenes, the wizard auto-detects and completes configuration items such as the required access key and port information. The wizard can also be launched from the Start a Free Trial button at sysdig.com.
After the first install, Sysdig Secure and Monitor users can access the wizard at any time from the Rocket icon on the navigation bar.
Environment | Flavor | Install Instructions |
---|---|---|
Kubernetes | Open Source | Helm is the preferred way of installing Sysdig agent. It is used in most cloud environments, for example, Amazon EKS or EC2 on AWS Cloud or AWS Outpost, EC2, and Azure AKS. |
OpenShift | ||
GKE | Used for Google Kubernetes Service environment. | |
Rancher | ||
OKE | ||
IKS | IBM manages and documents Sysdig installs as part of IKS. IBM Cloud Monitoring | |
Non-Orchestrated | Used when there is no orchestrator such as Kubernetes. Install Agent on Non-Orchestrated Environment. | |
Linux | Rare, used with custom kernel headers, unique use cases Agent Install: Manual Linux Installation. | |
Mesos |Marathon|DCOS | Agent Install: Mesos |Marathon |DCOS. |
Sysdig agents can be installed on a wide array of Linux hosts. Check your environment to ensure it meets the minimum supported platform, operating system, runtime, and orchestration requirements and uses the appropriate installation instructions.
We recommend that you use the latest version of the agent. Sysdig supports n-3 versions back based on the minor number. For example, if the latest release is v12.0.0
, we will support n-3 versions back, up to v11.2.0
.
Sysdig agents that are older than version 0.85.1, released October 1, 2018, will no longer connect to the Sysdig US-East SaaS platform with default agent values.
Going forward all the agent releases will have a 3-year deprecation policy. This implies:
Sysdig Support might not be able to help you troubleshoot or address the problems with agents past the deprecation date.
Sysdig will no longer provide prebuilt kernel probe binaries for these agent releases. You need to build the kernel probe binaries on the fly by using the hosts kernel headers.
These changes is effective starting Sysdig agent v12.1.0.
Sysdig agent versions 12.4.0 and above has been tested on the following list of latest Kubernetes versions. The matrix provides a single view into the supported operating systems, architecture, and runtime versions for different flavors of Kubernetes orchestrators.
Cluster | Operating System | Kubernetes Version | Architecture | Runtime |
---|---|---|---|---|
RedHat OpenShift Kubernetes Service (ROKS) | Ubuntu 18.04.6 LTS | v1.22.7+IKS | x86_64 | containerd |
Rancher | SUSE Linux Enterprise Server 15 SP2 | v1.20.4 | x86_64 | docker |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 46.82.202110110956-0 (Ootpa) | v1.19.14+fcff70a | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 47.84.202202070903-0 (Ootpa) | v1.20.11+e880017 | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 48.84.202202142303-0 (Ootpa) | v1.21.6+4b61f94 | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 49.84.202202230006-0 (Ootpa) | v1.22.3+b93fd35 | x86_64 | cri-o |
OpenShift (okd4) | Red Hat Enterprise Linux CoreOS 49.84.202203081945-0 (Ootpa) | v1.22.5+5c84e52 | x86_64 | cri-o |
OpenShift (okd3) | CentOS Linux 7 (Core) | v1.11.0+d4cacc0 | x86_64 | docker |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.20.0 | x86_64, arm64 | containerd |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.21.0 | x86_64, arm64 | containerd |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.22.0 | x86_64, arm64 | containerd |
Kubernetes Operations (kops) | Ubuntu 20.04.4 LTS | v1.21.9 | x86_64 | containerd |
Kubernetes | Ubuntu 20.04.2 LTS | v1.23.0 | x86_64 | docker |
Kubernetes | Ubuntu 20.04.2 LTS | v1.21.0 | x86_64 | docker |
IBM Cloud Kubernetes Service (IKS) | Ubuntu 18.04.6 LTS | v1.22.7+IKS | x86_64 | containerd |
Google Kubernetes Engine (GKE) | Container-Optimized OS from Google | v1.21.6-gke.1503 | x86_64 | containerd |
Google Kubernetes Engine (GKE) | Container-Optimized OS from Google | v1.21.9-gke.1002 | x86_64 | containerd |
Amazon Elastic Kubernetes Service (EKS) | Amazon Linux 2 | v1.21.5-eks-9017834 | x86_64 | docker |
Orchestration Platforms | Documentation |
---|---|
Oracle Kubernetes Engine (OKE) | Steps for OKE |
Microsoft Azure Cloud Services | Agent Install: Non-Orchestrated |
Microsoft Azure Kubernetes Service (AKS) | Agent Install: Kubernetes |
Amazon Elastic Container Service (Amazon ECS) | Agent Install: Non-Orchestrated + AWS Integration Instructions |
RancherOS | Agent Install: Non-Orchestrated |
Mesos/Marathon | Agent Install: Mesos/Marathon |
Docker Datacenter (DDC) | Agent Install: Non-Orchestrated |
If you are not using an orchestrator in your environment, follow the instructions for Agent Install Non-Orchestrated .
Sysdig agent has been tested on the following linux destros:
Operating System | Architecture |
---|---|
Amazon Linux 2 | x86_64 |
CentOS Linux 7 (Core) | x86_64 |
Fedora 33 (Cloud Edition) | x86_64 |
Fedora 34 (Cloud Edition) | x86_64 |
Red Hat Enterprise Linux 8.5 (Ootpa) | x86_64 |
Red Hat Enterprise Linux Server 7.9 (Maipo) | x86_64 |
Ubuntu 16.04.7 LTS (Xenial Xerus) | x86_64 |
Ubuntu 18.04.6 LTS (Bionic Beaver) | x86_64 |
Ubuntu 20.04.4 LTS | x86_64 |
Sysdig agent is supported on the following Linux distributions:
Platforms | Linux Distribution |
---|---|
Core Set |
|
AWS EC2 |
|
GCP |
|
Azure |
|
Sysdig agent supports the detection of the following:
Operating System | Architecture |
---|---|
Amazon Linux 2 | x86_64, arm64 |
CentOS Linux 7 (Core) | x86_64 |
Debian GNU/Linux 9 (stretch) | x86_64 |
Debian GNU/Linux 9.13 (stretch) | x86_64 |
Fedora 34 (Cloud Edition) | x86_64 |
Fedora Linux 35 (Cloud Edition) | x86_64 |
Red Hat Enterprise Linux 8.5 (Ootpa) | x86_64, arm64 |
Red Hat Enterprise Linux Server 7.9 (Maipo) | x86_64 |
Ubuntu 16.04.7 LTS (Xenial Xerus) | x86_64, arm64 |
Ubuntu 18.04.6 LTS (Bionic Beaver) | x86_64, arm64 |
Ubuntu 20.04.4 LTS (Focal Fossa) | x86_64, arm64 |
Sysdig agent supports running as a Podman container.
Enable Podman API Service for all the users.
The agent will not able to collect Podman-managed container metadata, such as the container name, if the API service is not enabled.
Secure rules and policies that depend on container metadata other than the container ID will not work.
Pausing and terminating containers will not work because Policy actions for Podman are not supported.
The containers started as a non-root user will have the
podman_owner_uid
label associated with it if the API service is
enabled for that user. The value of podman_owner_uid
will be the
numeric user ID corresponding to the user that started the
container.
For example, to pull the latest agent container from Quay.io:
docker pull quay.io/sysdig/agent
agent
agent-slim
agent-kmodule
agent
containerNo support for pre-built probes on zLinux. For kernel instrumentation, use the kernel module. eBPF probes are not supported on zLinux.
Capture is not supported on zLinux.
Sysdig agent installation using agent
container is not supported.
Sysdig agent supports the following:
For Java-based applications (Cassandra, Elasticsearch, Kafka, Tomcat, Zookeeper and etc.), the Sysdig agent requires the Java runtime environment (JRE) to be installed to poll for metrics (beans).
If the Docker-container-based Sysdig agent is installed, the JRE is
installed alongside the agent binaries and no further dependencies
exist. However, if you are installing the service-based agent
(non-container) and you do not see the JVM/JMX metrics reporting, your
host may not have the JRE installed or it may not be installed in the
expected location: usr/bin/java
The resource requirements of the agent are subjective to the size and load of the host— more activity equates to more resources required.
It is typical to see between 5-20KiB/s of bandwidth consumed—different variables can increase the throughput required such as the number of metrics, events, Kubernetes objects, and which products and features are enabled. When a Sysdig Capture is being collected, you can expect to see a spike in bandwidth while the capture file is being ingested.
We do not recommend placing bandwidth shaping or caps on the agent to ensure data can be sent to our collection service. For more information, see Tuning Sysdig Agent.
The installation of the Sysdig agent requires an access key.
This key and the agent installation instructions are presented to you after activating your account and using a web-based wizard upon initial login.
The same information can also be found in the
Settings > Agent Installation
menu of the web interface after logging
in. See Agent Installation: Overview and
Key for details.
A Sysdig agent (containerized or native) is installed into each host
being monitored and will need to be able to connect to the Sysdig
Monitor backend servers to report host metrics. The agent must be able
to reach the Sysdig Collector addresses. For example, for US East, it is
‘collector.sysdigcloud.com’ (via
multiple IPs) over port tcp/6443
. See Sysdig Collector
Ports for supported ports
for other regions.
The agent supports the HTTP proxy for communicating with Sysdig backend components. For more information, see Enable HTTP Proxy for Agents.
Sysdig provides you with quick-install commands pre-filled with some of your environment variables to get started with Sysdig agent. You choose the deployment type and Sysdig gives you auto-generated command to ease your installation experience.
Log in as admin to Sysdig Monitor or Sysdig Secure.
Select the Get Started page.
Click Install the Agent, select the appropriate deployment type, and copy the auto-generated code, filling in remaining variable values as required.
Helm is the recommended option for installing agents on Kubernetes.
helm install sysdig-agent --namespace <value> --set sysdig.accessKey=<value> \
--set sysdig.settings.collector=<value> --set sysdig.settings.collector_port=<value> \
--set nodeAnalyzer.apiEndpoint=<value> --set secure.vulnerabilityManagement.newEngineOnly=true sysdig/sysdig
kubectl create ns sysdig-agent
helm repo add sysdig https://charts.sysdig.com
helm repo update
helm install
--namespace=`dev`
--sysdig.accessKey=`1234-your-key-here-1234`
--sysdig.settings.collector='mycollector.elb.us-west-1.amazonaws.com'
--sysdig.settings.collector_port=`6443`
--set sysdig.settings.tags='linux:ubuntu,dept:dev,local:nyc'
--set sysdig.settings.k8s_cluster_name='my_cluster'
sysdig/sysdig
Option | Description |
---|---|
namespace | If a value is provided, the agent will be deployed to the specified Kubernetes namespace. The default is sysdig-agent. |
accessKey | The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure. |
collector | The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. See SaaS Regions and IP Ranges for more information. It is a custom value in on-prem installations. |
collector_port | The default is 6443. |
nodeAnalyzer.apiEndpoint | Node analyzer Endpoint. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges for more information. |
secure.vulnerabilityManagement.newEngineOnly | The default is false . Installs the vulnerabilitiy management engine and the runtime scanner.If you are still using the legacy engine, set secure.vulnerabilityManagement.newEngineOnly to false ; otherwise the Get Started code snippet will prompt you to use true . It is still possible to deploy the Runtime scanner by setting nodeAnalyzer.runtimeScanner.deploy to true . See Helm Charts for more information. |
nodeAnalyzer.runtimeScanner.settings.eveEnabled | The default is false . Enables Sysdig EVE, a requirement for the runtime feature Risk Spotlight. |
nodeAnalyzer.runtimeScanner.deploy | The default is false . Deploys the Runtime Scanner. This option works only if the secure.vulnerabilityManagement.newEngineOnly flag is set to false . |
install-agent-kubernetes \
[-a | --access_key <value>] [-t | --tags <value>] \
[-c | --collector <value>] [-cp | --collector_port <value>] [-s | --secure <value>] \
[-cc | --check_certificate <value>] [-ns | --namespace | --project <value>] \
[-ac | --additional_conf <value>] [-op | --openshift] [-as | --agent_slim] \
[-av | --agent_version <value>] [-ae | --api_endpoint <value> ] [-na | --nodeanalyzer ] \
[-ia | --imageanalyzer ] [-am | --analysismanager <value>] [-ds | --dockersocket <value>] \
[-cs | --crisocket <value>] [-cv | --customvolume <value>] \
[-cn | --cluster_name <value>] [-r | --remove ] [-h | --help]
curl -s https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s -- \
--access_key 1234-your-key-here-1234 \
--collector collector-staging.sysdigcloud.com --collector_port 6443 \
--nodeanalyzer --api_endpoint secure-staging.sysdig.com
Option | Description |
---|---|
| The agent access key. You can retrieve this from |
| The list of tags to identify the host where the agent is installed. For example: |
| The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. |
| The collector port. The default is 6443. |
| Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default. |
| Enable strong SSL certificate check. The default is true. |
| If a value is provided, the agent will be deployed to the specified namespace/project. The default is |
| If provided, perform the agent installation using the OpenShift command line. |
| If a value is provided, the additional configuration will be appended to the agent configuration file. |
| If a version is provided, use the specified agent version. The default is the latest version. |
| If a value is provided, the daemonset, configmap, cluster role binding, service acccount and secret associated with the Sysdig Agent will be removed from the specified namespace. |
| The |
| Print this usage and exit. |
Sysdig Secure Only | |
| If provided, will install the Node Analyzer tools. It is an error to set both -ia and -na. |
| The docker socket for Image Analyzer. |
| The CRI socket for Image Analyzer. |
| The custom volume for Image Analyzer. |
| Print this usage and exit. |
Sysdig Secure Only (Legacy) These values apply to the Node Image Analyzer (v1) in Sysdig Secure. | |
| The Analysis Manager endpoint for Sysdig Secure. |
| If provided, will install the Node Image Analyzer (v1). It is an error to set both -ia and -na. The v1 Node Image Analyzer will be deprecated and replaced by the NA tools. |
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=<ACCESS_KEY> \
-e COLLECTOR=<COLLECTOR_URL> \
-e SECURE=true \
[-e TAGS=<LIST_OF_TAGS>] \
-e ADDITIONAL_CONF= <LIST_OF_CONFIG> \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
docker run \
--name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=1234-your-key-here-1234 \
-e COLLECTOR=mycollector.elb.us-west-1.amazonaws.com \
-e COLLECTOR_PORT=6443 \
-e CHECK_CERTIFICATE=false \
-e TAGS=my_tag:some_value \
-e ADDITIONAL_CONF="log:\n file_priority: debug\n console_priority: error" \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=350m \
quay.io/sysdig/agent-slim
Option | Description |
---|---|
ACCESS_KEY | The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure. |
tags | Enter meaningful tags you want applied to your instances. |
COLLECTOR | The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges. |
collector_port | The default is 6443. |
SECURE | Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default. |
CHECK_CERTIFICATE | (On-prem) Determines strong SSL certificate check for Sysdig Monitor on-premises installation. Set to true when using SSL/TLS to connect to the collector service to ensure that a valid SSL/TLS certificate is installed. |
ADDITIONAL_CONF | Optional. Use this option to provide custom configuration values to the agent as environment variables. If provided, will be appended to agent configuration file. For example, For example, file log configuration. |
bpf | Enables eBPF probe. |
$ curl -s https://download.sysdig.com/stable/install-agent -a | \
--access_key <value> [-t | --tags <value>] [-c | --collector <value>] \
[-cp | --collector_port <value>] [-s | --secure <value>] \
[-cc | --check_certificate] [-ac | --additional_conf <value>] \
[-b | --bpf] [-h | --help]
curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- \
--access_key <ACCESS_KEY> --collector collector-staging.sysdigcloud.com \
--secure true
Option | Description |
---|---|
access-key | The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure. |
tags | Enter meaningful tags you want applied to your instances. |
collector | The collector URL for Sysdig Monitor or Sysdig Secure. This value is region-dependent in SaaS and is auto-completed on the Get Started page in the UI. It is a custom value in on-prem installations. See SaaS Regions and IP Ranges. |
collector_port | The default is 6443. |
secure | Use a secure SSL/TLS connection to send metrics to the collector. This option is enabled by default. |
check_certificate | Disables strong SSL certificate check for Sysdig Monitor on-premises installation. |
additional_conf | Optional. Use this option to provide custom configuration values to the agent as environment variables. If provided, the value will be appended to agent configuration file. For example, file log configuration. |
bpf | Enables eBPF probe. |
This document describes how to install Sysdig agent in a Kubernetes environment. This document assumes you will run the agent container as a Kubernetes pod, which then enables the Sysdig agent automatically to detect and monitor your Kubernetes environment.
It is relevant for any platform where Kubernetes is deployed, including Amazon environments (EKS, EC2, ECS), Azure Container Service (AKS), Google Kubernetes Engine (GKE), Red Hat OpenShift, Oracle Kubernetes Engine (OKE), and IBM Cloud Kubernetes Service (IKS).
You can also use DaemonSets to deploy agents on every node in your Kubernetes environment. Once deployed, Sysdig Monitor automatically begins monitoring all of your hosts, apps, pods, and services and automatically connects to the Kubernetes API server to pull relevant metadata about the environment. If licensed, Sysdig Secure launches with default policies that you can view and configure to suit your needs. You can access the front-end web interfaces for Sysdig Monitor and Sysdig Secure immediately.
A supported distribution. See Agent Installation Requirements for details.
Kubernetes v 1.9+: The agent installation on Kubernetes requires using v1.9 or higher because the APIs used to fetch kubernetes metadata are only present in v1.9+.
Sysdig account and access key: Request a trial or full account at Sysdig.com and click the Activate Account button. You create a Sysdig user name and password.
The Getting Started Wizard provides an access key.
By default, Sysdig agents deployed in Kubernetes automatically detect metadata from containerd and CRI-O (in addition to Docker), as long as the prerequisites are fulfilled.
After reviewing the information on this page, continue with the Sysdig agent installation steps: Kubernetes Agent Installation Steps.
As of agent version 0.88.1, the Sysdig agent will automatically detect containerd metadata (as well as any Docker metadata) in your environment, as long as the prerequisites are fulfilled.
Agent version: Sysdig agent version 0.88.1 or higher
NOTE: If you are upgrading from an earlier version of the agent,
you must also download the latest
sysdig-agent-daemonset-v2.yaml
from GitHub.
Configuration parameter: In the agent config file,
new_k8s: true
must be set.
As of agent 9.6.0, new_k8s
is enabled by default.
See Enable Kube State Metrics and Cluster Name below for details on editing the config file.
Kubernetes-only: The containerd API must support CRI (a Kubernetes runtime interface).
If the Sysdig agent detects containerd metadata, it will be reported in the front end as follows:
Explore/Dashboard views: The icon next to container-specific items (container.name, container.id, etc.) shows whether it’s a Docker or containerd object.
Spotlight: Updated for containerd display.
Events: Containerd events die
and oom
are enabled by
default.
Events create
and exit
are also supported.
The Sysdig agent will automatically detect CRI-O metadata (as well as any Docker and/or containerd metadata) in your environment, as long as the Prerequisites are fulfilled.
Platform version: Sysdig SaaS March 2019or higher
Agent version: Sysdig agent v 0.89.4 March 27, 2019 or higher.
NOTE: If you are upgrading from an earlier version of the agent,
you must also download the latest
sysdig-agent-daemonset-v2.yaml
from GitHub.
Configuration parameter: In the agent config file,
new_k8s: true
must be set.
See Enable Kube State Metrics and Cluster Name below for details on editing the config file.
Kubernetes-only: The API must support CRI (a Kubernetes runtime interface).
Events: There are no CRI-O events, so the Events pane remains unchanged.
Explore/Dashboard views: The icon next to container-specific items (container.name, container.id, etc.) shows CRI-O type.
Supported Metrics: By default, the same metrics are supported
for CRI-O as for Docker and containerd, except for image id
(container.image.id
).
As of agent version 0.92.1, this setting is enabled by default.
To enable image id metrics, edit the agent configuration file
dragent.yaml
to contain the following:
cri:
extra_queries: true
See Understanding the Agent Config
Files for more information
on editing dragent.yaml
.
Choose the appropriate link to complete the installation steps:
Steps for Kubernetes (Vanilla) All environments except IKS, GKE, or those using OpenShift.
The Sysdig agent requires kernel header files to install successfully on a host.
This setup step is required for some environments and not others, as noted.
If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.
In some cases, the host(s) in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernel headers manually.
To do so:
For Debian-style distributions, run the command:
apt-get -y install linux-headers-$(uname -r)
For RHEL-style distributions, run the command:
yum -y install kernel-devel-$(uname -r)
Background info: see also About Kernel Headers and the Kernel Module.
You can review Agent Install: Kubernetes | GKE | OpenShift |IBM and the Agent Installation Requirements for additional context, if desired.
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export CLUSTER_NAME=my-cluster # Kubernetes cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information, see charts.
To deploy agents using Kubernetes daemonsets, you will download the following configuration files, edit them as required, and deploy them.
sysdig-agent-clusterrole.yaml
sysdig-agent-service.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
Download the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever naming you prefer. In this document, we used
sysdig-agent
for both the namespace and the service account.
The default service account name was automatically defined in
sysdig-agent-daemonset-v2.yaml, at
the line:
serviceAccount: sysdig-agent.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
Create a cluster role and service account, and define the cluster role bindingthat grants the Sysdig agent rules in the cluster role, using the commands:
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information:
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
For SaaS, find the collector address for your region.
For On-prem, enter the collector endpoint defined in your environment.
check_certificate
should be set to false
if a self-signed
certificate or private, CA-signed cert is used. See also Step 5
Set Up SSL Connectivity to the
Backend.
(All installs) Apply the sysdig-agent-configmap.yaml
file:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
(All installs) Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file :
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed. See Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State Metrics entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
There are two ways to update the agent configuration
Option 1: Edit the files locally and apply the changes with
kubectl apply -f
:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Option 2: Use kubectl edit
to edit files on the fly:
kubectl edit configmap sysdig-agent
-n sysdig-agent
Running agents will automatically pick the new configuration after Kubernetes pushes the changes across all the nodes in the cluster.
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
collector:
collector_port
Set the collector: value to collector-static.sysdigcloud.com
See SaaS Regions and IP Ranges and identify the correct URL associated with your Sysdig collector and region.
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.
Select an individual container or pod to see details.
Kubernetes metadata (pods, deployments etc.) appear a minute or two later than the nodes/containers themselves; if pod names do not appear immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC addresses. See Troubleshooting Agent Installation for tips.
Autopilot is an operation mode for creating and managing clusters in GKE. In brief, with Autopilot, Google configures and manages the underlying node infrastructure for you. This topic helps you use helm to install Sysdig agent on a GKE cluster installed in Autopilot mode.
NodeAnalyzer is not supported on Autopilot environments.
After connecting to the GKE cluster, you can install the pre-configured Sysdig agent using helm.
To customize the configuration of the agent, see the Sysdig Agent Helm Chart.
Add the Sysdig Agent Helm Chart repository:
$ helm repo add sysdig https://charts.sysdig.com/ `
Create a namespace for the Sysdig agent:
$ kubectl create ns sysdig
Install the Sysdig agent chart:
$ helm install agent --namespace sysdig --set sysdig.accessKey=$SYSDIG_AGENT_KEY --set sysdig.settings.collector=$COLLECTOR_URL sysdig/sysdig --set gke.autopilot=true
Replace the values as follows:
$SYSDIG_AGENT_KEY
: Use the Sysdig agent key for your Sysdig Platform.
$COLLECTOR_URL
: The URL is region-dependent in Sysdig SaaS. See Regions and IP Ranges. The Collector URL is custom for on-prem installations.
Once the installation is complete, you can get started with Sysdig Secure and Sysdig Monitor.
Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.
Given below is one way to do so.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysdig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
Verify that kube state metrics and cluster name are working correctly: select the Explore
tab and create a grouping by kube_cluster_name
and
kube_pod_name
.
Select an individual container or pod to see the details.
Google Kubernetes Engine (GKE) is a managed environment for running Kubernetes in Google Cloud, in order to deploy containerized applications. As of Sysdig agent version 0.88, Sysdig supports all flavors of GKE, including Ubuntu and GKE’s default Container-Optimized OS (COS).
GKE COS environments require eBPF probe to support agent installation.
Because GKE uses stateful firewalls, you must actively open port 6443 for the Sysdig agent outbound traffic.
In earlier versions, the Sysdig Agent connected to port 6666. This behavior has been deprecated, as the Sysdig agent now connects to port 6443.
Linux kernel version >= 4.14.
When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below. Note that the eBPF probe is supported only in GKE COS environments.
You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.
To deploy agent using Helm charts, run the following:
Create environment variables for each value you will need during the installation phases:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (Integrations > Agent Installation > Your access key).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export GKE_CLUSTER_NAME=my-cluster # GKE cluster name
See SaaS Regions and IP Ranges for configuring collector URL and node analyzer URL for your region if you are using the SaaS version. This is a custom valyue in on-prem installations.
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$GKE_CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL --set ebpf.enabled=true
For more information,charts.
To deploy agents using Kubernetes daemonsets, you will download the following configuration files, edit them as required, and deploy them.
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
Download the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever name you want. In this document, we used
sysdig-agent
for both the namespace and the service account.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
If you are running Kubernetes 1.6 or higher, you must
Grant your user the ability to create roles in Kubernetes by running the following command (see Google documentation for more):
kubectl create clusterrolebinding your-user-cluster-admin-binding --clusterrole=cluster-admin --user=your.google.cloud.email@example.org
Create a service account for the Sysdig agent using the
clusterrole.yaml
file.
The Sysdig agent must be granted read-only access to certain Kubernetes APIs, which the agent uses to populate metadata and provide component metrics.
Sysdig provides a config file in GitHub. Deploying this file creates a cluster role and service account in Kubernetes, and defines cluster role binding that grants the Sysdig agent rules in the cluster role.
Run the following commands (using whatever namespace you defined in Step 2):
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information :
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
For SaaS, find the collector address for your region.
For On-prem, enter the collector endpoint defined in your environment.
check_certificate
should be set to false
if a self-signed
certificate or private, CA-signed cert is used. See also Step 5
Set Up SSL Connectivity to the
Backend.
(All installs) Apply the sysdig-agent-configmap.yaml
file using
the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
FOR GKE COS ONLY: To enable the eBPF probe required for COS,
uncomment the following parameters in
sysdig-agent-daemonset-v2.yaml
under the env section:
env:
- name: SYSDIG_BPF_PROBE
value: ""
Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file using the command:
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed and you can see some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State Metrics entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
There are two ways to update the agent configuration
Option 1: Edit the files locally and apply the changes with
kubectl apply -f
:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Option 2: Use kubectl edit
to edit files on the fly:
kubectl edit configmap sysdig-agent
-n sysdig-agent
Running agents will automatically pick the new configuration after Kubernetes pushes the changes across all the nodes in the cluster.
Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.
Select an individual container or pod to see details.
Kubernetes metadata (pods, deployments etc.) appear a minute or two later than the nodes/containers themselves; if pod names do not appear immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC addresses. See Troubleshooting Agent Installation for tips.
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
collector:
collector_port
Set the collector: value to collector-static.sysdigcloud.com
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
Oracle Kubernetes Engine (OKE) is a managed environment for running Kubernetes in Oracle Cloud, in order to deploy containerized applications. As of Sysdig agent version 12.0.1, Sysdig supports all flavors of OKE.
OKE environments require eBPF probe to support agent installation.
The instructions below describe a standard OKE agent install and call out the special steps needed to install the eBPF probe.
Because OKE uses stateful firewalls, you must actively open port 6443 for the Sysdig agent outbound traffic.
OKE by default allows network access to the sysdig Agent on 6443, but ensure that firewall rules are open and the agent can connect to the Sysdig backends.
Linux kernel version >= 4.14.
When performing the installation steps, you will add one additional parameter to install the eBPF probe. See Step 7, below.
Identify the appropriate endpoint depending on your Sysdig account region. For more information, see SaaS Regions and IP Ranges. More info here https://docs.sysdig.com/en/docs/administration/saas-regions-and-ip-ranges/
After making clear which region your account belongs to, please choose one of the following methods:
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export OKE_CLUSTER_NAME=my-cluster # OKE cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$OKE_CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information,charts.
Download the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever name you want. In this document, we used
sysdig-agent
for both the namespace and the service account.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
If you are running Kubernetes 1.6 or higher, you must Create a service account for the Sysdig agent by using the clusterrole.yaml
file.
The Sysdig agent must be granted read-only access to certain Kubernetes APIs, which the agent uses to populate metadata and provide component metrics.
Sysdig provides a config file in GitHub. Deploying this file creates a cluster role and service account in Kubernetes, and defines cluster role binding that grants the Sysdig agent rules in the cluster role.
Run the following commands by using the namespace you defined in Step 2:
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information :
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
For SaaS, find the collector address for your region.
For On-prem, enter the collector endpoint defined in your environment.
check_certificate
should be set to false
if a self-signed
certificate or private, CA-signed cert is used. See also Step 5
Set Up SSL Connectivity to the
Backend.
(All installs) Apply the sysdig-agent-configmap.yaml
file using
the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
To enable the eBPF probe uncomment the following parameters in
sysdig-agent-daemonset-v2.yaml
under the env section:
env:
- name: SYSDIG_BPF_PROBE
value: ""
Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file using the command:
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed and you can see Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.
Select an individual container or pod to see details.
Kubernetes metadata (pods, deployments etc.) appear a minute or two later than the nodes/containers themselves; if pod names do not appear immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC addresses. See Troubleshooting Agent Installation for tips.
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
collector:
collector_port
Set the collector: value to collector-static.sysdigcloud.com
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.
RHCOS environments require eBPF probe to support agent installation.
The Sysdig agent requires kernel header files to install successfully on a host.
This setup step is required for some environments and not others, as noted.
If the hosts in your environment match the pre-compiled kernel modules available from Sysdig, no special action is required.
In some cases, the host(s) in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernel headers manually.
To do so:
For Debian-style distributions, run the command:
apt-get -y install linux-headers-$(uname -r)
For RHEL-style distributions, run the command:
yum -y install kernel-devel-$(uname -r)
Background info: see also About Kernel Headers and the Kernel Module.
If you are using Red Hat OpenShift, these steps are required. They describe how to create a project, assign and label the node selector, create a privileged service account, and add it to a cluster role.
In the example code, this document uses sysdig-agent
for the
PROJECT NAME (-n
), the SERVICE ACCOUNT (-z
), and the NODE SELECTOR.
You can copy-paste the code as is, or follow the steps below to customize your naming conventions.
oc adm new-project sysdig-agent --node-selector=''
oc project sysdig-agent
oc create serviceaccount sysdig-agent
oc adm policy add-scc-to-user privileged -n sysdig-agent -z sysdig-agent -z node-analyzer
oc adm policy add-cluster-role-to-user cluster-reader -n sysdig-agent -z sysdig-agent -z node-analyzer
You can use your own Project name and Service Account name if desired.
Note that if you use a different Service Account name, you will need to edit the default service account in the Sysdig Installation Steps, below.
Create a new OpenShift project for the Sysdig agent deployment and use an empty string for the node selector:
oc adm new-project PROJECT-NAME --node-selector=""
Change to the new OpenShift Project for the Sysdig agent deployment:
oc project PROJECT-NAME
Create a service account for the project:
oc create serviceaccount SERVICE-ACCOUNT
Add the service account to privileged Security Context Constraints:
oc adm policy add-scc-to-user privileged -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
Add the service account to the cluster-reader
Cluster Role:
oc adm policy add-cluster-role-to-user cluster-reader -n PROJECT-NAME -z SERVICE-ACCOUNT -z node-analyzer
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export CLUSTER_NAME=my-cluster # OpenShift cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information,charts.
To deploy agents using Kubernetes daemonsets, you download the configuration files, edit them as required, and deploy them.
sysdig-agent-daemonset-v2.yaml
sysdig-agent-clusterrole.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Download the sample files:
sysdig-agent-daemonset-v2.yaml
sysdig-agent-clusterrole.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create the sysdig-agent
cluster role and assign it to the service account:
oc apply -f sysdig-agent-clusterrole.yaml
oc adm policy add-cluster-role-to-user sysdig-agent -n PROJECT-NAME -z SERVICE-ACCOUNT
Create a secret key using the command:
oc create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
If you created a service account name other than sysdig-agent
:
Edit sysdig-agent-daemonset-v2.yaml
to provide your custom value:``
serviceAccount: sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information:
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
check_certificate
should be set to false
if a self-signed
certificate or private, CA-signed cert is used. See also Step 5
Set Up SSL Connectivity to the
Backend.(All installs) Apply the sysdig-agent-configmap.yaml
file using
the command:
oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
FOR RHCOS ONLY: To enable the eBPF probe required for COS, uncomment the following parameters in sysdig-agent-daemonset-v2.yaml
under the env section:`
env:
- name: SYSDIG_BPF_PROBE
value: ""
Apply the sysdig-agent-service.yaml
file:
oc apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file:
oc apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed and you can see Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State Metrics entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
There are two ways to update the agent configuration
Option 1: Edit the files locally and apply the changes with
oc apply -f
:
oc apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Option 2: Use oc edit
to edit files on the fly:
oc edit configmap sysdig-agent
-n sysdig-agent
Running agents will automatically pick the new configuration after Kubernetes pushes the changes across all the nodes in the cluster.
Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.
The steps below give one way to do the check.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.
Select an individual container or pod to see details.
Kubernetes metadata (pods, deployments etc.) appear a minute or two later than the nodes/containers themselves; if pod names do not appear immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC addresses. See Troubleshooting Agent Installation for tips.
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
collector:
collector_port
Set the collector: value to collector-static.sysdigcloud.com
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
You can review Agent Install: Kubernetes | GKE | OpenShift | IBM and the Agent Installation Requirements for additional context, if desired.
The Sysdig agent requires a kernel module in order to be installed successfully on a host. On RancherOS distributions, the Unix version does not match the provided headers, and the agent might fail to install correctly. Therefore, you must install the kernel headers manually.
For RancherOS distributions, the kernel headers are available in the
form of a system service and therefore are enabled using the ros
service command:
$ sudo ros service enable kernel-headers-system-docker
$ sudo ros service up -d kernel-headers-system-docker
Some cloud hosting service providers supply pre-configured Linux instances with customized kernels. You may need to contact your provider’s support desk for instructions on obtaining appropriate header files, or for installing the distribution’s default kernel.
To deploy agent using Helm charts, run the following:
Export the access token and the name of the OKE cluster:
export SDC_ACCESS_TOKEN=xxxx # Get it from the UI (User > Settings > Sysdig Secure API Token).
export SDC_COLLECTOR_URL=collector-static.sysdigcloud.com # us-west by default. Please check the right region.
export SDC_NODEANALYZER_URL=secure.sysdig.com # us-east by default. Please check the right region.
export CLUSTER_NAME=my-cluster # Rancher cluster name
Create a namespace to use for the Sysdig agent:
kubectl create ns sysdig-agent
Set up the helm repo:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the agent:
helm install sysdig-agent --namespace sysdig-agent --set sysdig.accessKey=$SDC_ACCESS_TOKEN --set sysdig.settings.collector=$SDC_COLLECTOR_URL --set sysdig.settings.collector_port=6443 --set clusterName=$CLUSTER_NAME sysdig/sysdig --set nodeAnalyzer.apiEndpoint=$SDC_NODEANALYZER_URL
For more information,charts.
To deploy agents using Kubernetes daemonsets, download the following configuration files, edit them as required, and deploy them.
sysdig-agent-clusterrole.yaml
sysdig-agent-service.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
HELM CHART OPTION Kubernetes also offers a package manager, Helm, which uses charts to simplify this process.
If you are using Helm charts in your K8s environment, we recommend using them to deploy Sysdig agents, as described here.
Download the sample files:
sysdig-agent-clusterrole.yaml
sysdig-agent-daemonset-v2.yaml
sysdig-agent-configmap.yaml
sysdig-agent-service.yaml
Create a namespace to use for the Sysdig agent.
You can use whatever naming you prefer. In this document, we used
sysdig-agent
for both the namespace and the service account.
The default service account name was automatically defined in
sysdig-agent-daemonset-v2.yaml, at
the line:
serviceAccount: sysdig-agent.
kubectl create ns sysdig-agent
Create a secret key:
kubectl create secret generic sysdig-agent --from-literal=access-key=<your sysdig access key> -n sysdig-agent
Create a cluster role and service account, and define the cluster role bindingthat grants the Sysdig agent rules in the cluster role, using the commands:
kubectl apply -f sysdig-agent-clusterrole.yaml -n sysdig-agent
kubectl create serviceaccount sysdig-agent -n sysdig-agent
kubectl create clusterrolebinding sysdig-agent --clusterrole=sysdig-agent --serviceaccount=sysdig-agent:sysdig-agent
Edit sysdig-agent-configmap.yaml
to add the collector address
,
port
, and the SSL/TLS
information:
collector:
collector_port:
ssl: #true or false
check_certificate: #true or false
For SaaS, find the collector address for your region.
For On-prem, enter the collector endpoint defined in your environment.
check_certificate
should be set to false
if a self-signed
certificate or private, CA-signed cert is used. See also Step 5
Set Up SSL Connectivity to the
Backend.
(All installs) Apply the sysdig-agent-configmap.yaml
file:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
(All installs) Apply the sysdig-agent-service.yaml
file:
kubectl apply -f sysdig-agent-service.yaml -n sysdig-agent
This allows the agent to receive Kubernetes audit events from the Kubernetes API server. See Kubernetes Audit Logging for information on enabling Kubernetes audit logging.
(All installs) Apply the daemonset-v2.yaml
file :
kubectl apply -f sysdig-agent-daemonset-v2.yaml -n sysdig-agent
The agents will be deployed. See Getting Started with Sysdig
Monitor
to view some metrics in the Sysdig Monitor UI. You can make further
edits to the configmap
as described below.Getting
Started with Sysdig Monitor
These steps are optional but recommended.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
new_k8s: true
This allows kube state metrics to be automatically detected, monitored, and displayed in Sysdig Monitor.
For more information, see the Kube State Metrics entry in the Sysdig blog.
As of agent 9.6.0, new_k8s
is enabled by default.
Edit sysdig-agent-configmap.yaml
to uncomment the line:
**k8s_cluster_name:
**and add your cluster name.
Setting cluster name here allows you to view, scope, and segment metrics in the Sysdig Monitor UI by the Kubernetes cluster.
Note: Alternatively, if you assign a tag with “cluster
” in the
tag name, Sysdig Monitor will display that as the Kubernetes cluster
name.
Apply the configmap changes using the command:
kubectl apply -f sysdig-agent-configmap.yaml -n sysdig-agent
Proceed to verify the metrics in the Sysdig Monitor UI.
Sysdig provides a list of static IP addresses that can be whitelisted in
a Sysdig environment, allowing users to establish a network connection
to the Sysdig backend without opening complete network connectivity.
This is done by setting the Collector IP to
collector-static.sysdigcloud.com
.
The sysdig-agent-configmap.yaml
file can be edited either locally or
using the edit command in Kubernetes. refer to the section above for
more information.
To configure the collector IP in a Kubernetes SaaS instance:
Open sysdig-agent-configmap.yaml
in a text editor.
Uncomment the following lines:
collector:
collector_port
Set the collector: value to collector-static.sysdigcloud.com
See SaaS Regions and IP Ranges and identify the correct URL associated with your Sysdig collector and region.
Set the collector_port: value to 6443
Save the file.
The example file below shows how the sysdig-agent-configmap.yaml
file
should look after configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-agent
data:
dragent.yaml: |
### Agent tags
# tags: linux:ubuntu,dept:dev,local:nyc
#### Sysdig Software related config ####
# Sysdig collector address
collector: collector-static.sysdigcloud.com
# Collector TCP port
collector_port: 6443
# Whether collector accepts ssl/TLS
ssl: true
# collector certificate validation
ssl_verify_certificate: true
# Sysdig Secure
security:
enabled: true
#######################################
# new_k8s: true
# k8s_cluster_name: production
Log in to Sysdig Monitor to verify that the agent deployed and the metrics are detected and collected appropriately.
Kubernetes metadata (pods, deployments etc.) appear a minute or two later than the nodes/containers themselves; if pod names do not appear immediately, wait and retry the Explore view.
If agents are disconnecting, there could be an issue with your MAC addresses. See Troubleshooting Agent Installation for tips.
Access Sysdig Monitor:
SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for US East, the URL is https://app.sysdigcloud.com.
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
Log in with your Sysdig user name and password.
Select the Explore
tab to see if metrics are displayed.
(Once you have enabled new_k8s:true
): To verify that kube state
metrics and cluster name are working correctly: Select the Explore
tab and create a grouping by kubernetes.cluster.name
and
kubernetes.pod.name
.
As of agent 9.6.0, new_k8s
is enabled by default.
Select an individual container or pod to see details.
The Sysdig agent uses Kubernetes Lease to control how and when connections are made to the Kubernetes API Server. This mechanism prevents overloading the Kubernetes API server with connection requests during agent bootup.
Kubernetes node leases are automatically created for agent version 12.0.0 and above. On versions prior to 12.0.0, you must configure node leases as given in the KB article.
Sysdig Agent v11.3.0 or above
Kubernetes v1.14 or above
The agent creates the following leases:
During boot up, the Sysdig agent connects to the Kubernetes API server
to retrieve Kubernetes metadata and build a cache. The cold-start
leases control the number of agents that build up this cache at any
given time. An agent will grab a lease, build its cache, and then
release the lease so that another agent can build its cache. This
mechanism prevents agents from creating a “boot storm” which can
overwhelm the API server in large clusters.
In Kubernetes environments, two agents are marked as delegated
in each
cluster. The delegated
agents are the designated agents to request
more data from the API server and produce KubeState metrics. The
delegation
leases will not be released until the agent is terminated.
To view the leases, run the following:
$ kubectl get leases -n sysdig-agent
You will see an output similar to the following:
NAME HOLDER AGE
cold-start-0 20m
cold-start-1 20m
cold-start-2 21m
cold-start-3 ip-10-20-51-167 21m
cold-start-4 21m
cold-start-5 21m
cold-start-6 20m
cold-start-7 21m
cold-start-8 20m
cold-start-9 ip-10-20-51-166 21m
delegation-0 ip-10-20-52-53 21m
delegation-1 ip-10-20-51-98 21m
When lease-based delegation is working as expected, the agent logs show one of the following:
Getting pods only for node <node>
Getting pods for all nodes.
Both (occasionally on the delegated nodes)
Run the following to confirm that it is working:
$ kubectl logs sysdig-agent-9l2gf -n sysdig-agent | grep -i "getting pods"
The configuration is working as expected if the output on a pod is similar to the following:
2021-05-05 02:48:32.877, 15732.15765, Information, cointerface[15738]: Only getting pods for node ip-10-20-51-166.ec2.internal
The latest Sysdig ClusterRole is required for the agent to create leases. If you do not have the latest ClusterRole or if you have not configured the ClusterRole correctly, the logs show the following error:
Error, lease_pool_manager[2989554]: Cannot access leases objects: leases.coordination.k8s.io is forbidden: User "system:serviceaccount:sysdig-agent:sysdig-agent" cannot list resource "leases" in API group "coordination.k8s.io" in the namespace "sysdig-agent"
Contact Sysdig Support for help.
Several configuration options exist for leases. It is recommended to not change the default settings unless prompted by Sysdig Customer Support.
Configuration | Default | Description |
---|---|---|
|
| When true, the agent will attempt to create |
| 10 | The number of |
|
| The namespace to be created. This shouldn't be needed in agent version 12.0.0 because the DownwardAPI in the ClusterRole will provide the appropriate namespace. |
|
| When |
|
| When |
This section describes how to install the Sysdig agent directly on a Linux host, without using an orchestrator, such as Kubernetes or Mesos.
The agent can be installed in two ways:
As a standard container
As a non-containerized service
The steps for each flavor differ slightly depending on whether you are using the SaaS or on-premises version of the Sysdig platform.
If you are installing the Sysdig agent in an environment that has Kubernetes, use the Agent Install: Kubernetes instructions instead.
See Agent Installation Requirements for information on the following:
Supported Linux distributions
Network connection
Sysdig access key
Cloud service providers (AWS, Google, and Microsoft Azure) and any steps you may need to configure to integrate the Sysdig agent.
On kernel headers: The Sysdig agent requires kernel header files in order to install successfully on a host, and the agent is delivered with precompiled headers. If the hosts in your environment match the kernel versions included with the agent, no special action is needed.
In some cases, the host(s) in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernel headers manually. See About Kernel Headers and the Kernel Module for details.
Run any commands as root or with the sudo
command.
Have your Sysdig access key on hand.
If you launch an agent install from www.sysdig.com, the welcome wizard will present an access key.
The Sysdig agent can be deployed as a Docker container.
The commands below can also be copied from the Get Started page. In that case, your access key will already be included in the command automatically.
The agent is installed by running sysdig/agent-kmodule
, followed by running sysdig/agent-slim
.
Every host restart requires subsequent running of agent-kmodule
and
agent-slim
containers.
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the address for your region.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that is either built or downloaded.Build and load the kernel module:
If you are not using eBPF, use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules \
quay.io/sysdig/agent-kmodule
If you are using eBPF use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-e SYSDIG_BPF_PROBE="" \
-v /etc/os-release:/host/etc/os-release:ro \
-v /root/.sysdig:/root/.sysdig \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
Configure kernel module to load during system boot.
If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.
$ sudo mkdir -p /etc/modules-load.d
$ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
Run the agent module providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
[-e TAGS=[TAGS]]
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host\
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
[-e TAGS=[TAGS]]
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /root/.sysdig:/root/.sysdig \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the address for your region.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that was either built or downloaded.Run the agent container providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host\
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host\
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e TAGS=[TAGS] \
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the URL of your environment.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that was either built or downloaded.CHECK_CERTIFICATE
: Set to false
if a self-signed certificate or private CA-signed certificate is used. For more information, see Set Up SSL Connectivity to the Backend.Build and load the kernel module:
If you are not using eBPF, use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules \
quay.io/sysdig/agent-kmodule
If you are using eBPF use the following:
docker run -it --privileged --rm --name sysdig-agent-kmodule \
-e SYSDIG_BPF_PROBE="" \
-v /etc/os-release:/host/etc/os-release:ro \
-v /root/.sysdig:/root/.sysdig \
-v /usr:/host/usr:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
quay.io/sysdig/agent-kmodule
Configure kernel module to load during system boot.
If you are not using eBPF, use the following commands to configure the Linux system to automatically load the kernel module during system boot.
$ sudo mkdir -p /etc/modules-load.d
$ sudo bash -c "echo sysdigcloud-probe > /etc/modules-load.d/sysdigcloud-probe.conf"
Run the agent module providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent-slim
Collect the following configuration parameters:
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the URL of your environment.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
SYSDIG_BPF_PROBE
: The path to the probe file that was either built or downloaded.CHECK_CERTIFICATE
: Set to false
if a self-signed certificate or private CA-signed certificate is used. For more information, see Set Up SSL Connectivity to the Backend.Run the agent module providing the access key and, optionally, user-defined tags:
If you are not using eBPF, use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
If you are using eBPF use the following:
docker run -d --name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e COLLECTOR=[COLLECTOR_ADDRESS] \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
[-e TAGS=[TAGS]]
-e SYSDIG_BPF_PROBE="" \
-v /sys/kernel/debug:/sys/kernel/debug:ro \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=512m \
quay.io/sysdig/agent
Use these instructions to install the agent on the host itself, not in a container. Install on each host in the environment.
The command lines below can also be copy/pasted from the Welcome wizard
or the Settings>Agent Installation
page in the Sysdig Monitor
interface.
In that case, your access key will already be included in the command automatically.
The Sysdig agent depends on several python modules, some of which might
not be installed on the hosts where the agent is running as a service.
When the required dependencies are not available, the sdchecks
component in the agent will report errors in the log files, such as:
>> Error, sdchecks[0] ModuleNotFoundError: No module named 'posix_ipc'
To address these errors, install the missing modules using the
pip install
command.
Run the following command:
curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] [--tags [TAGS]]
Where
[ACCESS_KEY]
is your unique agent access key string. For example, 1234-your-key-here-1234.
TAGS
is an optional list of user-defined agent tags. For example, role:webserver,location:europe
.
See SaaS Regions and IP Ranges to find the collector endpoint for your region.
Restart the agent and start the service:
sudo systemctl enable dragent
Run the following command:
curl -s https://download.sysdig.com/stable/install-agent | sudo bash -s -- --access_key [ACCESS_KEY] --collector [COLLECTOR_ADDRESS] --secure true --check_certificate true [--tags [TAGS]]
Where
[ACCESS_KEY]
is your unique agent access key string. For example, 1234-your-key-here-1234.
TAGS
is an optional list of user-defined agent tags. For example, role:webserver,location:europe
.
check_certificate
: Set it to false
if a self-signed certificate, a private, or a CA-signed certificate is used. See Set Up SSL Connectivity to the Backend for more information.
Restart the agent and start the service:
sudo systemctl enable dragent
Sysdig provides a list of static IP addresses that can be whitelisted in a Sysdig environment, allowing users to establish a network connection to the Sysdig backend without opening complete network connectivity. This is done by setting the Collector IP to collector-static.sysdigcloud.com:
user@host:~$ docker run --name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=[ACCESS_KEY] \
-e TAGS=[TAGS] \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
-e COLLECTOR=collector-static.sysdigcloud.com \
-e COLLECTOR_PORT=6443 \
-e SECURE=true \
-e CHECK_CERTIFICATE=true \
--shm-size=512m \
quay.io/sysdig/agent-slim
In the following cases, we recommend that you manually install the agent.
Full control over the deployment process
Integration with configuration management tools
Custom kernel
Unsupported distribution
See Agent Install: Manual Linux Installation for more information.
Manual installation of the native Linux agent is recommended in the following cases:
Full control over the deployment process
Integration with configuration management tools
Custom kernel
Unsupported distribution (within Debian/Fedora flavors)
Otherwise, you may want to just follow the standard Installation Guide:
NOTE: If you are installing the Sysdig agent in an orchestrated infrastructure such as Kubernetes, Mesos/Marathon, use the respective Installation Guides:
Run the commands as root or with sudo.
Review the Host Requirements for Agent Installation. Then follow the steps for the appropriate Linux distribution, below.
Trust the Sysdig Monitor GPG key, configure the apt repository, and update the package list:
curl -s https://download.sysdig.com/DRAIOS-GPG-KEY.public | apt-key add -
curl -s -o /etc/apt/sources.list.d/draios.list http://download.sysdig.com/stable/deb/draios.list
apt-get update
Install kernel development files.
The following command might not work with every kernel. Make sure to customize the name of the package properly.
apt-get -y install linux-headers-$(uname -r)
Install, configure, and restart the Sysdig agent.
apt-get -y install draios-agent
echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
service dragent restart
Replace ACCESS_KEY
with your unique access key
string.
Inability to retrieve the key indicates that the administrator of
your instance might have it turned off for non-admin users. Please
contact your Sysdig administrator to receive the key. If you still
have issues please contact Sysdig
support.
[TAGS]
is an optional parameter you can use to list one or more
tags for this host (see below).
Trust the Sysdig Monitor GPG key, configure the yum repository.
$ rpm --import https://download.sysdig.com/DRAIOS-GPG-KEY.public
$ curl -s -o /etc/yum.repos.d/draios.repo http://download.sysdig.com/stable/rpm/draios.repo
Install the EPEL repository
The following command is required only if DKMS is not available in
the distribution. You can verify if DKMS is available with
yum list dkms
The command below contains a sample release number; be sure to update with the correct release.
$ rpm -i http://mirror.us.leaseweb.net/epel/6/i386/epel-release-6-8.noarch.rpm
Install kernel development files.
The following command might not work with every kernel. Make sure to customize the name of the package properly.
$ yum -y install kernel-devel-$(uname -r)
Install, configure, and start the Sysdig agent.
$ yum -y install draios-agent
$ echo customerid: ACCESS_KEY >> /opt/draios/etc/dragent.yaml
$ echo tags: [TAGS] >> /opt/draios/etc/dragent.yaml
$ sudo systemctl enable dragent
$ sudo systemctl start dragent
Replace ACCESS_KEY
with your unique access key
string.
Inability to retrieve the key indicates that the administrator of
your instance might have it turned off for non-admin users. Please
contact your Sysdig administrator to receive the key. If you still
have issues please contact Sysdig
support.
[TAGS]
is an optional parameter you can use to list one or more
tags for this host (see below).
If you using a non-systemd Linux distribution, use the service
command to start dragent
.
$ service dragent restart
The Sysdig Agent is unsupported outside of the Debian, Fedora, and Amazon distributions.
Tagging your hosts is highly recommended. Agent Tags allow you to sort nodes of your infrastructure into custom groups in Sysdig Monitor.
Replace the [TAGS] parameter above with a comma-separated list of
TAG_NAME:TAG_VALUE
.
For example: role:webserver,location:europe
Amazon Elastic Container Service (ECS) is a fully managed container orchestration service that helps to easily deploy, manage, and scale containerized applications.
This section describes how to install the Sysdig agent container on each underlying host in your ECS cluster. Once installed, the agent will automatically begin monitoring all of your hosts, service and tasks.
These instructions are valid only for ECS clusters using EC2 instances. For information on ECS Fargate clusters, see AWS Fargate Serverless Agents.
To install Sysdig agent on ECS, do the following:
Create an ECS task definition for the Sysdig agent.
Register the task definition in your AWS account.
Create a service with the previous task definition to run the Sysdig agent in each of the nodes of your ECS cluster.
ACCESS_KEY
: The agent access key. You can retrieve this from Settings > Agent Installation in either Sysdig Monitor or Sysdig Secure.COLLECTOR
: Use the collector address for your region. For more information, see SaaS Regions and IP Ranges.TAGS
: The list of tags for the host where the agent is installed. For example: role:webserver
, location:europe
, role:webserver
Use the above values to customize the JSON snippet below and save it as a file named sysdig-agent-ecs.json
. Note that memory and cpu have both been set to 1024, depending of the size of your cluster you might want to tune those values, see Tuning Sysdig Agent for more information.
{
"family": "sysdig-agent-ecs",
"containerDefinitions": [
{
"name": "sysdig-agent",
"image": "quay.io/sysdig/agent-slim",
"cpu": 1024,
"memory": 1024,
"privileged": true,
"environment": [
{
"name": "ACCESS_KEY",
"value": "$ACCESS_KEY"
},
{
"name": "COLLECTOR",
"value": "$COLLECTOR"
},
{
"name": "TAGS",
"value": "$TAG1,TAG2"
}
],
"mountPoints": [
{
"readOnly": true,
"containerPath": "/host/boot",
"sourceVolume": "boot"
},
{
"containerPath": "/host/dev",
"sourceVolume": "dev"
},
{
"readOnly": true,
"containerPath": "/host/lib/modules",
"sourceVolume": "modules"
},
{
"readOnly": true,
"containerPath": "/host/proc",
"sourceVolume": "proc"
},
{
"containerPath": "/host/var/run/docker.sock",
"sourceVolume": "sock"
},
{
"readOnly": true,
"containerPath": "/host/usr",
"sourceVolume": "usr"
}
],
"dependsOn": [
{
"containerName": "sysdig-agent-kmodule",
"condition": "SUCCESS"
}
]
},
{
"name": "sysdig-agent-kmodule",
"image": "quay.io/sysdig/agent-kmodule",
"memory": 512,
"privileged": true,
"essential": false,
"mountPoints": [
{
"readOnly": true,
"containerPath": "/host/boot",
"sourceVolume": "boot"
},
{
"containerPath": "/host/dev",
"sourceVolume": "dev"
},
{
"readOnly": true,
"containerPath": "/host/lib/modules",
"sourceVolume": "modules"
},
{
"readOnly": true,
"containerPath": "/host/proc",
"sourceVolume": "proc"
},
{
"containerPath": "/host/var/run/docker.sock",
"sourceVolume": "sock"
},
{
"readOnly": true,
"containerPath": "/host/usr",
"sourceVolume": "usr"
}
]
}
],
"pidMode": "host",
"networkMode": "host",
"volumes": [
{
"name": "sock",
"host": {
"sourcePath": "/var/run/docker.sock"
}
},
{
"name": "dev",
"host": {
"sourcePath": "/dev/"
}
},
{
"name": "proc",
"host": {
"sourcePath": "/proc/"
}
},
{
"name": "boot",
"host": {
"sourcePath": "/boot/"
}
},
{
"name": "modules",
"host": {
"sourcePath": "/lib/modules/"
}
},
{
"name": "usr",
"host": {
"sourcePath": "/usr/"
}
}
],
"requiresCompatibilities": [
"EC2"
]
}
Once your task definition is ready, ensure that you register it in your AWS account:
aws ecs register-task-definition \
--cli-input-json file://sysdig-agent-ecs.json
Using the ECS task definition you have created, create a service in the cluster that you want to monitor with Sysdig.
aws ecs create-service \
--cluster $CLUSTER_NAME \
--service-name sysdig-agent-svc \
--launch-type EC2 \
--task-definition sysdig-agent-ecs \
--scheduling-strategy DAEMON
With the agent installed, Sysdig will begin auto-discovering your containers and other resources of your ECS environment.
If you’re using ECS Anywhere, change the launch type to EXTERNAL
when the service is created.
aws ecs create-service \
--cluster $CLUSTER_NAME \
--service-name sysdig-agent-svc \
--launch-type EXTERNAL \
--task-definition sysdig-agent-ecs \
--scheduling-strategy DAEMON
awslogs
Log Driver for the Sysdig Agent ContainersYou can send the logs from the containers running in ECS tasks to log groups in CloudWatch Logs. To send Sysdig container logs, do the following:
Add the following section to each of the container definitions described above:
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "$YOUR_LOG_GROUP",
"awslogs-region": "$AWS_REGION",
"awslogs-stream-prefix": "sysdig"
}
Update your task definition and the service to enable the logs.
IBM Cloud maintains the documentation for Sysdig agent installation on IBM Cloud Kubernetes Service (IKS).
For more information, see the IBM Cloud Monitoring documentation:
Marathon is the container orchestration platform for Mesosphere’s Datacenter Operating System (DC/OS) and Apache Mesos.
This guide describes how to install the Sysdig agent container on each underlying host in your Mesos cluster. Once installed, the agent will automatically connect to the Mesos and Marathon APIs to pull relevant metadata about the environment and will begin monitoring all of your hosts, apps, containers, and frameworks.
Review the Host Requirements for Agent Installation.
In this three-part installation, you:
Deploy the Sysdig agent on all Mesos Agent (aka “Slave”) nodes,
either automatically or by creating and posting a .json
file to
the leader Marathon API server.
Deploy the Sysdig agent on the Mesos Master nodes.
Special configuration steps: modify the Sysdig agent config file to monitor Marathon instances.
If you’re using DC/OS 1.8 or higher, then you can find Sysdig in the Mesosphere Universe marketplace and install it from there.
It will automatically deploy the Sysdig agent container on each of your Mesos Agent nodes as a Marathon app.
Proceed to Deploy the Sysdig Agent.
If you are using a version of DC/OS earlier than 1.8 then:
Create a JSON file for Marathon, in the following format.
The COLLECTOR
address comes from your own environment in on-prem
installations. For SaaS installations, find the collector endpoint
for your region listed
here.
COLLECTOR_PORT, SECURE,
and CHECK_CERT
are used in environments
with Sysdig’s on-premises backend installed.
{
"backoffFactor": 1.15,
"backoffSeconds": 1,
"constraints": [
[
"hostname",
"UNIQUE"
]
],
"container": {
"docker": {
"forcePullImage": true,
"image": "sysdig/agent",
"parameters": [],
"privileged": true
},
"type": "DOCKER",
"volumes": [
{
"containerPath": "/host/var/run/docker.sock",
"hostPath": "/var/run/docker.sock",
"mode": "RW"
},
{
"containerPath": "/host/dev",
"hostPath": "/dev",
"mode": "RW"
},
{
"containerPath": "/host/proc",
"hostPath": "/proc",
"mode": "RO"
},
{
"containerPath": "/host/boot",
"hostPath": "/boot",
"mode": "RO"
},
{
"containerPath": "/host/lib/modules",
"hostPath": "/lib/modules",
"mode": "RO"
},
{
"containerPath": "/host/usr",
"hostPath": "/usr",
"mode": "RO"
}
]
},
"cpus": 1,
"deployments": [],
"disk": 0,
"env": {
"ACCESS_KEY": "ACCESS_KEY=YOUR-ACCESS-KEY-HERE",
"CHECK_CERT": "false",
"SECURE": "true",
"TAGS": "example_tag:example_value",
"name": "sdc-agent",
"pid": "host",
"role": "monitoring",
"shm-size": "350m"
},
"executor": "",
"gpus": 0,
"id": "/sysdig-agent",
"instances": 1,
"killSelection": "YOUNGEST_FIRST",
"labels": {},
"lastTaskFailure": {
"appId": "/sysdig-agent",
"host": "YOUR-HOST",
"message": "Container exited with status 70",
"slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
"state": "TASK_FAILED",
"taskId": "sysdig-agent.3bb0759d-3fa3-11e9-b446-c60a7a2ee871",
"timestamp": "2019-03-06T00:03:16.234Z",
"version": "2019-03-06T00:01:57.182Z"
},
"maxLaunchDelaySeconds": 3600,
"mem": 850,
"networks": [
{
"mode": "host"
}
],
"portDefinitions": [
{
"name": "default",
"port": 10101,
"protocol": "tcp"
}
],
"requirePorts": false,
"tasks": [
{
"appId": "/sysdig-agent",
"healthCheckResults": [],
"host": "YOUR-HOST-IP",
"id": "sysdig-agent.0d5436f4-3fa4-11e9-b446-c60a7a2ee871",
"ipAddresses": [
{
"ipAddress": "YOUR-HOST-IP",
"protocol": "IPv4"
}
],
"localVolumes": [],
"ports": [
4764
],
"servicePorts": [],
"slaveId": "1fa6f2fc-95b0-445f-8b97-7f91c1321250-S2",
"stagedAt": "2019-03-06T00:09:04.232Z",
"startedAt": "2019-03-06T00:09:06.912Z",
"state": "TASK_RUNNING",
"version": "2019-03-06T00:09:04.182Z"
}
],
"tasksHealthy": 0,
"tasksRunning": 1,
"tasksStaged": 0,
"tasksUnhealthy": 0,
"unreachableStrategy": {
"expungeAfterSeconds": 0,
"inactiveAfterSeconds": 0
},
"upgradeStrategy": {
"maximumOverCapacity": 1,
"minimumHealthCapacity": 1
},
"version": "2019-03-06T00:09:04.182Z",
"versionInfo": {
"lastConfigChangeAt": "2019-03-06T00:09:04.182Z",
"lastScalingAt": "2019-03-06T00:09:04.182Z"
}
}
See Table 1: Environment Variables for Agent Config
Filef
or the Sysdig name:value
definitions.
Complete the “cpus
”, “mem
” and “labels
” (i.e. Marathon labels)
entries to fit the capacity and requirements of the cluster
environment.
Update the created.json
file to the leader Marathon API server:
$ $curl -X POST http://$(hostname -i):8080/v2/apps -d @sysdig.json -H "Content-type: application/json"
After deploying the agent to the Mesos Agent nodes, you will install agents on each of the Mesos Master nodes as well.
If any cluster node has both Mesos Master and Mesos Agent roles, do not perform this installation step on that node. It already will have a Sysdig agent installed from the procedure in step A. Running duplicate Sysdig agents on a node will cause errors.
Use the Agent Install: Non-Orchestrated instructions to install the agent directly on each of your Mesos Master nodes.
When the Sysdig agent is successfully installed on the master nodes, it
will automatically connect to the local Mesos and Marathon (if
available) API servers via http://localhost:5050
and
http://localhost:8080
respectively, to collect cluster configuration
and current state metadata in addition to host metrics.
In certains situations, you may need to add additional configurations to
the dragent.yaml
file:
If the Sysdig agent cannot be run directly on the Mesos API server
If the API server is protected with a username/password.
Descriptions and examples are shown below.
Mesos allows multiple masters. If the API server can not be instrumented with a Sysdig agent, simply delegate ONE other node with an agent installed to remotely receive infrastructure information from the API server.
NOTE: If you manually configure the agent to point to a master with a static configuration file entry, then automatic detection/following of leader changes will no longer be enabled.
Add the following Mesos parameter to the delegated agent’s
dragent.yaml
file to allow it to connect to the remote API server and
authenticate, either by:
a. Directly editing dragent.yaml
on the host, or
b. Converting the YAML code to a single-line format and adding it as an
ADDITIONAL_CONF
argument in a Docker command.
See Understanding the Agent Config Files for details.
Specify the API server’s connection method, address, and port. Also specify credentials if necessary.
YAML example:
mesos_state_uri: http://[acct:passwd@][hostname][:port]
marathon_uris:
- http://[acct:passwd@][hostname][:port]
Although marathon_uris:
is an array, currently only a single “root”
Marathon framework per cluster is supported. Multiple side-by-side
Marathon frameworks should not be configured in order for our agent to
function properly. Multiple side-by-side “root” Marathon frameworks on
the same cluster are currently not supported. The only supported
multiple-Marathon configuration is with one “root” Marathon and other
Marathon frameworks as its apps.
If the agent is installed on the API server but the API server uses a different port or requires authentication, those parameters must be explicitly specified.
Add the following Mesos parameters to the API server’s dragent.yaml
to
make it connect to the API server and authenticate with any unique
account and password, either by:
a. Directly editing dragent.yaml
on the host, or
b. Converting the YAML code to a single-line format and adding it as an
ADDITIONAL_CONF
argument in a Docker command.
See Understanding the Agent Config Files for details.
Specify the API server’s protocol, user credentials, and port:
mesos_state_uri: http://[username:password@][hostname][:port]
marathon_uris:
- http://[acct:passwd@][hostname][:port]
*HTTPS protocol is also supported.
In troubleshooting cases where auto-detection and reporting of your Mesos infrastructure needs to be temporarily turned off in a designated agent:
Comment out the Mesos parameter entries in the agent’s dragent.yaml file.
Example parameters to disable: mesos_state_uri, marathon_uris
If the agent is running on the API server (Master node) and auto-detecting a default configuration, you can add the line:
mesos_autodetect: false
either directly in the dragent.yaml file or as an ADDITIONAL_CONF
parameter in a Docker command.
Restart the agent.
Airgapped environments are those that do not have the network access to
pull images from the container repository. Agent installation requires
sysdigcloud-probe
and you cannot download a pre-compiled module in an
airgapped environment. Therefore, ensure that you compile your own
sysdigcloud-probe
before installing the agent.
On a machine with internet connectivity, build the Sysdig probe container and create a tar file of the image.
Get the probe builder artifacts from the repository:
$ git clone https://github.com/draios/sysdig
$ git checkout probe-builder
$ cd sysdig
Build the container image:
$ docker build -t airgap/sysdig-probe-builder probe-builder/
Create the container and run:
$ docker run --rm -v /var/run/docker.sock:/var/run/docker.sock airgap/sysdig-probe-builder:latest -P -b airgap/
Save the images to a tar archive:
$ docker save airgap/sysdig-probe-builder | gzip > builders.tar.gz
Ensure that you make this tar available to the airgapped machines where you intend to install the Sysdig agent.
Set up a local repository to host the pre-compiled kernel module:
$ kubectl run my-nginx --image=nginx --port=80
$ kubectl expose deployment my-nginx --port=80 --type=NodePort
Copy sysdigcloud-probe
to the repository you have created:
$ kubectl cp sysdigcloud-probe-<version> my-nginx-xxxxxxxx-xxxx:/usr/share/nginx
Install Sysdig agent by pointing SYSDIG_PROBE_URL
to the local
repository:
For docker-based installations:
$ docker run -d --name sysdig-agent --restart always --privileged --net host --pid host -e ACCESS_KEY=WWWWW-YYYY-XXXX-ZZZZ-123456789 -e SECURE=true -e SYSDIG_PROBE_URL=http://www.mywebserver.net:80/ -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro --shm-size=512m sysdig/agent
Where -e SYSDIG_PROBE_URL=http://www.mywebserver:80/
is the local
nginx
pod with the loaded module.
To use secure communication with a self-signed or untrusted
certificate, apply the -e SYSDIG_PROBE_INSECURE_DOWNLOAD=true
environment variable.
Check the agent log. You will see a similar message:
Found custom module URL http://mywebserver:80/, will use it * Trying to download precompiled module from http://mywebserver:80/sysdigcloud-probe-<version>
Continue with the instructions in Agent Install: Non-Orchestrated.
Open your agent daemonset and update the SYSDIG_PROBE_URL
to point
to the local repository:
- name: SYSDIG_PROBE_URL
value: http://www.mywebserver:80/
If you would like to use secure communication with a self-signed or
untrusted certificate, apply the SYSDIG_PROBE_INSECURE_DOWNLOAD
environment variable.
- name: SYSDIG_PROBE_INSECURE_DOWNLOAD
value: true
Continue with the instructions in Agent Install: Kubernetes.
Use one of the following methods to determine the version of the agents installed in your environment:
Segmenting metrics by using agent.version
shows the installed versions
of agents in your environment. For example, segment the uptime
metric
across your environment by using agent.version
. Hover over the graph
to see the list of agent versions.
The image shows the list of agent versions in n/a.
Use the Sysdig Agent Health Dashboard to determine the agent versions:
Log in to the Sysdig Monitor.
Select Dashboards and expand Host Infrastructure Dashboards.
Open the Sysdig Agent Health & Status template or create your own from the template.
The Sysdig Agent and Health & Status Dashboard shows the agent version corresponding to each host in your environment.
Out of the box, the Sysdig agent will gather and report on a wide variety of pre-defined metrics. It can also accommodate any number of custom parameters for additional metrics collection.
Use this section when you need to change the default or pre-defined settings by editing the agent configuration files, or for other special circumstances.
Understanding the Agent Config Files: Describes the core configuration files, how to access them, and how to add custom environment variables.
Optional: Agent Auto-Config: Describes how to use the Sysdig REST APIs and Python client wrappers to auto-orchestrate changes to all agents in an environment, in the (rare) situation in which a standard orchestration tool such as Kubernetes, Chef, or Ansible is not being used.
Monitoring Integrations also require editing the agent config files.
By default, the Sysdig agent is configured to collect metric data from a range of platforms and applications. You can edit the agent config files to extend the default behavior, including additional metrics for JMX, StatsD, Prometheus, or a wide range of other applications. You can also monitor log files for targeted text strings.
Out of the box, the Sysdig agent will gather and report on a wide variety of pre-defined metrics. It can also accommodate any number of custom parameters for additional metrics collection.
The agent relies on a pair of configuration files to define metrics collection parameters:
| The core configuration file. You can look at it to understand more about the default configurations provided. Location: CAUTION. This file should never be edited. |
| The configuration file where parameters can be added, either directly in YAML as |
The dragent.yaml
file can be accessed and edited in several ways,
depending on how the agent was installed. This document describes how to
modify dragent.yaml
.
One additional file, dragent.auto.yaml
is also created and used in
special circumstances. See Optional: Agent
Auto-Config for more
detail.
There are various ways to add or edit parameters indragent.yaml
.
It is possible to edit the container’s file directly on the host.
Add parameters directly in YAML.
Access dragent.yaml
directly
at"/opt/draios/etc/dragent.yaml
."
Edit the file. Use proper YAML syntax.
See the examples at the bottom of the page.
Restart the agent for changes to take effect
Native agent: service dragent restart
Container agent: docker restart sysdig-agent
Configmap.yaml
is the configuration file where parameters can be added,
either directly in YAML as name/value pairs, or using environment
variables such as ‘ADDTIONAL_CONF."
If you install agents as DaemonSets on a system running Kubernetes, you
use configmap.yaml
to connect with and manipulate the
underlyingdragent.yaml
file.
See also: Agent Install: Kubernetes | GKE | OpenShift | IBM
Add parameters directly in YAML.
Edit the files locally and apply with the changes withkubectl -f.
Access theconfigmap.yaml
.
Edit the file as needed.
Apply the changes:
kubectl apply -f sysdig-agent-configmap.yaml
Running agents will automatically pick the new configuration after Kubernetes pushes the changes across all the nodes in the cluster.
Add -e ADDITIONAL_CONF="<VARIABLES>"
to a Docker run command, where
<VARIABLES>
contains all the customized parameters you want to
include, in a single-line format.
To insert ADDITIONAL_CONF
parameters in a Docker run command or a
daemonset
file, you must convert the YAML code into a single-line
format.
You can do the conversion manually for short snippets. To convert longer
portions of YAML, use echo|sed
commands.
In earlier versions, the Sysdig Agent connected to port 6666. This behavior has been deprecated, as the Sysdig agent now connects to port 6443.
The basic procedure:
Write your configuration in YAML, as it would be entered directly in
dragent.yaml
.
In a bash shell, use echo
and sed
to convert to a single
line.
sed
script: echo "" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'
Insert the resulting line into a Docker run command or add it to the
daemonset
file as an ADDITIONAL_CONF
.
Insert parameters to turn off StatsD collection and blacklist port 6443.
Sysdig agent uses port 6443 for both inbound and outbound communication with the Sysdig backend. The agent initiates a request and keeps a connection open with the Sysdig backend for the backend to push configurations, Falco rules, policies, and so on. Ensure that you allow the agents’ inbound and outbound communication on TCP 6443 from the respective IPs associated with your SaaS Regions. Note that you are allowing the agent to send communication outbound on TCP 6443 to the inbound IP ranges listed in the SaaS Regions.
YAML format
statsd:
enabled: false
blacklisted_ports:
- 6443
Single-line format (manual)
Use spaces, hyphens, and \n
correctly when manually converting to a
single line:
ADDITIONAL_CONF="statsd:\n enabled: false\n blacklisted_ports:\n - 6443"
Here the single line is incorporated into a full agent startup Docker command.
docker run
--name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=1234-your-key-here-1234 \
-e TAGS=dept:sales,local:NYC \
-e ADDITIONAL_CONF="statsd:\n enabled: false\n blacklisted_ports:\n - 6443" \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
quay.io/sysdig/agent
Insert parameters to override the default configuration for a RabbitMQ app check.
YAML format
app_checks:
- name: rabbitmq
pattern:
port: 15672
conf:
rabbitmq_api_url: "http://localhost:15672/api/"
rabbitmq_user: myuser
rabbitmq_pass: mypassword
queues:
- MyQueue1
- MyQueue2
Single-line format (echo |sed)
From a bash shell, issue the echo command and sed script.
echo "app_checks:
- name: rabbitmq
pattern:
port: 15672
conf:
rabbitmq_api_url: "http://localhost:15672/api/"
rabbitmq_user: myuser
rabbitmq_pass: mypassword
queues:
- MyQueue1
- MyQueue2
" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g'
This results in the single-line format to be used with ADDITIONAL_CONF in a Docker command or daemonset file.
"app_checks:\n - name: rabbitmq\n pattern:\n port: 15672\n conf:\n rabbitmq_api_url: http://localhost:15672/api/\n rabbitmq_user: myuser\n rabbitmq_pass: mypassword\n queues:\n - MyQueue1\n - MyQueue2\n"
If you installed the Sysdig agent in Kubernetes using a Helm
chart, then no
configmap.yaml
file was downloaded. You edit dragent.yaml
using Helm
syntax:
$ helm install
--name sysdig-agent
--set sysdig.settings.tags='linux:ubuntu\,dept:dev\,local:nyc'
--set clusterName='my_cluster'
sysdig/sysdig
Will be transformed into
data:
dragent.yaml: |
tags: linux:ubuntu,dept:dev,local:nyc
k8s_cluster_name: my_cluster
Name | Value | Description |
---|---|---|
|
| Required |
|
| Optional. These are displayed in Sysdig Monitor for ease of use. For example:
|
|
| Enter the host name or IP address of the Sysdig collector service. Note that when used within For SaaS regions, see: SaaS Regions and IP Ranges. |
|
| On-prem only. The port used by the Sysdig collector service; default 6443. |
|
| On-prem only. If using SSL/TLS to connect to collector service value = "true" otherwise "false." |
|
| On-prem only. Set to "true" when using SSL/TLS to connect to the collector service and should check for valid SSL/TLS certificate. |
| Optional. A place to provide custom configuration values to the agent as environment variables . | |
| Optional. An alternative URL to download precompiled kernel module. |
docker run \
--name sysdig-agent \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=3e762f9a-3936-4c60-9cf4-c67e7ce5793b \
-e COLLECTOR=mycollector.elb.us-west-1.amazonaws.com \
-e COLLECTOR_PORT=6443 \
-e CHECK_CERTIFICATE=false \
-e TAGS=my_tag:some_value \
-e ADDITIONAL_CONF="log:\n file_priority: debug\n console_priority: error" \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
--shm-size=350m \
quay.io/sysdig/agent
Agent modes provide the ability to control metric collection to fit your scale and specific requirement. You can choose one of the following modes to do so:
Monitor
Monitor Light
Troubleshooting
Secure
Custom Metrics Only
Using a stripped-down mode limits collection of unneeded metrics, which in turn prevents the consumption of excess resources and helps reduce expenses.
The Monitor mode offers an extensive collection of metrics. We recommend this mode to monitor enterprise environments.
monitor
is the default mode if you are running the Enterprise
tier. To switch back to the
Monitor mode from a different mode, do one of the following:
Add the following to the dragent.yaml
file and restart the agent:
feature:
mode: monitor
Remove the parameter related to the existing mode from the
dragent.yaml
file and restart the agent. For example, to switch
from troubleshooting
mode to monitor
, delete the following
lines:
feature:
mode: troubleshooting
Monitor Light caters to the users that run agents in a resource-restrictive environment, or to those who are interested only in a limited set of metrics.
Monitor Light provides CPU, Memory, File, File system, and Network metrics. For more information, see Metrics Available in Monitor Light.
To switch to the Monitor Light mode, edit the dragent.yaml
file:
Open the dragent.yaml
file.
Add the following configuration parameter:
feature:
mode: monitor_light
Restart the agent.
Troubleshooting mode offers sophisticated metrics with detailed diagnostic capabilities. Some of these metrics are heuristic in nature.
In addition to the extensive metrics available in the Monitor mode,
Troubleshooting mode provides additional metrics such as net.sql
and
additional segmentation for file and network metrics. For more
information, see Additional Metrics Values Available in
Troubleshooting.
To switch to the Troubleshooting mode, edit the dragent.yaml
file:
Open the dragent.yaml
file.
Add the following configuration parameter:
feature:
mode: troubleshooting
Restart the agent.
The secure mode supports only Sysdig Secure features.
Sysdig agent collects no metrics in the secure mode, which, in turn, minimizes network consumption and storage requirement in the Sysdig backend. Lower resource usage can help reduce costs and improve performance.
In the Secure mode, the Monitor UI shows no data because no metrics are sent to the collector.
This feature requires agent v10.5.0 or above.
Open the dragent.yaml
file.
Add the following:
feature:
mode: secure
Restart the agent.
Custom Metrics Only mode collects the same metrics as the Monitor Light mode, but also adds the ability to collect the following:
As such, Custom Metrics Only mode is suitable if would like to use most of the features of Monitor mode but are limited in resources.
This mode is not compatible with Secure. If your account is configured for Secure, you must explicitly disable Secure in the agent configuration if you wish to use this mode.
This mode requires agent v12.4.0 or above.
Open the dragent.yaml
file.
Add the following configuration parameter:
feature:
mode: custom-metrics-only
If your account is enabled for Secure, add the following:
security:
enabled: false
secure_audit_streams:
enabled: false
falcobaseline:
enabled: false
This configuration explicitly disables the Secure features in the agent. If you do not disable Secure, the agent will not start due to incompatiblity issues.
Restart the agent.
Monitor Light provides cpu, memory, file, file system, and network metrics.
Metrics | Description |
---|---|
cpu.cores.used | See System. |
cpu.cores.used.percent | |
cpu.idle.percent | |
cpu.iowait.percent | |
cpu.nice.percent | |
cpu.stolen.percent | |
cpu.system.percent | |
cpu.used.percent | |
cpu.user.percent | |
load.average.percpu.1m | |
load.average.percpu.5m | |
load.average.percpu.15m | |
memory.bytes.available | |
memory.bytes.total | |
memory.bytes.used | |
memory.bytes.used | |
memory.bytes.virtual | |
memory.pageFault.major | |
memory.pageFault.minor | |
memory.swap.bytes.available | |
memory.swap.bytes.total | |
memory.swap.bytes.used | |
memory.swap.used.percent | |
memory.used.percent | |
file.bytes.in | |
file.bytes.out | |
file.bytes.total | |
file.iops.in | |
file.iops.out | |
file.iops.total | |
file.open.count | |
file.time.in | |
file.time.out | |
file.time.total | |
fs.bytes.free | |
fs.bytes.total | |
fs.bytes.used | |
fs.free.percent | |
fs.inodes.total.count | |
fs.inodes.used.count | |
fs.inodes.used.percent | |
fs.largest.used.percent | |
fs.root.used.percent | |
fs.used.percent | |
net.bytes.in | |
net.bytes.out | |
net.bytes.total | |
proc.count | |
thread.count | |
container.count | |
system.uptime | |
uptime |
In addition to the extensive set of metrics available in the monitor
mode, additional metrics, such as net.sql
and net.mongodb
, as well
as additional segmentations for file and network metrics are available.
Metrics | Additional Metrics Values Available When Segmented by | Supported Agent Versions |
---|---|---|
file.error.total.count | file.name and file.mount labels | Version 10.1.0 or above |
file.bytes.total | ||
file.bytes.in | ||
file.bytes.out | ||
file.open.count | ||
file.time.total | ||
host.count | ||
host.error.count | ||
proc.count | ||
proc.start.count | ||
net.mongodb.collection | all | Version 10.2.0 or above |
net.mongodb.error.count | ||
net.mongodb.operation | ||
net.mongodb.request.count | ||
net.mongodb.request.time | ||
net.sql.query | all | |
net.sql.error.count | ||
net.sql.query.type | ||
net.sql.request.count | ||
net.sql.request.time | ||
net.sql.table | ||
net.http.error.count | net.http.url | Version 10.3.0 or above |
net.http.method | ||
net.http.request.count | ||
net.http.request.time | ||
net.bytes.in | ||
net.bytes.out | ||
net.request.time.worst.out | ||
net.request.count | ||
net.request.time | ||
net.bytes.total | ||
net.http.request.time.worst | all |
The following metrics will not be reported in the essentials
mode when
compared with monitor
mode:
Metrics | Segmented By |
---|---|
net.bytes.in | net.connection.server , net.connection.direction , net.connection.l4proto , and net.connection.client labels |
net.bytes.out | |
net.connection.count.total | |
net.connection.count.in | |
net.connection.count.out | |
net.request.count | |
net.request.count.in | |
net.request.count.out | |
net.request.time | |
net.request.time.in | |
net.request.time.out | |
net.bytes.total | |
net.mongodb.collection | all |
net.mongodb.error.count | |
net.mongodb.operation | |
net.mongodb.request.count | |
net.mongodb.request.time | |
net.sql.query | all |
net.sql.error.count | |
net.sql.query.type | |
net.sql.request.count | |
net.sql.request.time | |
net.sql.table | |
net.sql.query | all |
net.sql.error.count | |
net.sql.query.type | |
net.sql.request.count | |
net.sql.request.time | |
net.sql.table | |
net.http.method | |
net.http.request.count | |
net.http.request.time | |
net.http.statusCode | |
net.http.url |
You can configure the agent to allow it to communicate with the Sysdig collector through an HTTP proxy. HTTP proxy is usually configured to offer greater visibility and better management of the network.
The agent can connect to the collector through an HTTP proxy by sending an HTTP CONNECT message and receiving a response. The proxy then initiates a TCP connection to the collector. These two connections form a tunnel that acts like one logical connection.
By default, the agent will encrypt all messages sent through this
tunnel. This means that after the initial CONNECT message and response,
all the communication on that tunnel is encrypted by SSL end-to-end.
This encryption is controlled by the top-level ssl
parameter in the
agent configuration.
Optionally, the agent can add a second layer of encryption, securing the
CONNECT message and response. This second layer of encryption may be
desired in the case of HTTP authentication if there is a concern that
network packet sniffing could be used to determine the user’s
credentials. This second layer of encryption is enabled by setting the
ssl
parameter to true in the http_proxy
section of the agent
configuration. See
Examples
for details.
You specify the following parameters at the same level as http_proxy
in the dragent.yaml
file. These existing configuration options affect
the communication between the agent and collector (both with and without
a proxy).
ssl
: If set to false
, the metrics sent from the agent to the
collector are unencrypted (default is true
).
ssl_verify_certificate
: Determines whether the agent verifies the
SSL certificate sent from the collector (default is true
).
The following configuration options affect the behavior of the HTTP
Proxy setting. You specify them under the http_proxy
heading in the
dragent.yaml
file.
proxy_host
: Indicates the hostname of the proxy server. The
default is an empty string, which implies communication through an
HTTP proxy is disabled.
proxy_port
: Specifies the port on the proxy server the agent
should connect to. The default is 0, which indicates that the HTTP
proxy is disabled.
proxy_user
: Required if HTTP authentication is configured. This
option specifies the username for the HTTP authentication. The
default is an empty string, which indicates that authentication is
not configured.
proxy_password
: Required if HTTP authentication is configured.
This option specifies the password for the HTTP authentication. The
default is an empty string. Specifying proxy_user
with no
proxy_password
is allowed.
ssl
: If set to true, the connection between the agent and the
proxy server is encrypted.
Note that this parameter requires the top-level ssl
parameter to
be enabled, as the agent does not support SSL to the proxy but
unencrypted traffic to the collector. This additional security
prevents you from misconfiguring the agent assuming the metrics are
as well encrypted end-to-end when they are not.
ssl_verify_certificate
: Determines whether the agent will verify
the certificate presented by the proxy.
This option is configured independently of the top-level
ssl_verify_certificate
parameter. This option is enabled by
default. If the provided certificate is not correct, this option can
cause the connection to the proxy server to fail.
ca_certificate
: The path to the CA certificate for the proxy
server. If ssl_verify_certificate
is enabled, the CA certificate
must be signed appropriately.
The following example shows no SSL connection between the agent and the proxy server as well as between the proxy server and the collector.
collector_port: 6667
ssl: false
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3128
ssl: false
In this example, SSL is enabled only between the proxy server and the collector.
collector_port: 6443
ssl: true
ssl_verify_certificate: true
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3128
The following example shows SSL is enabled between the agent and the proxy server as well as between the proxy server and the collector.
collector_port: 6443
ssl: true
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3129
ssl: true
ssl_verify_certificate: true
ca_certificate: /usr/proxy/proxy.crt
The following configuration instructs the agent to connect to a proxy
server located at squid.yourdomain.com
on port 3128
. The agent will
request the proxy server to establish an HTTP tunnel to the Sysdig
collector at collector-your.sysdigcloud.com
on port 6443. The agent
will authenticate with the proxy server using the given user and
password combination.
collector: collector-your.sysdigcloud.com
collector_port: 6443
http_proxy:
proxy_host: squid.yourdomain.com
proxy_port: 3128
proxy_user: sysdig_customer
proxy_password: 12345
ssl: true
ssl_verify_certificate: true
ca_certificate: /usr/proxy/proxy_cert.crt
The dragent.yaml
file elements are wide-reaching. This section
describes the parameters to edit in dragent.yaml
to perform a range of
activities:
Use the blacklisted_ports
parameter in the agent configuration file to
block network traffic and metrics from unnecessary network ports.
Note: Port 53 (DNS) is always blacklisted.
Access the agent configuration file, using one of the options listed.
Add blacklisted_ports
with desired port numbers.
Example (YAML):
blacklisted_ports:
- 6443
- 6379
Restart the agent (if editing dragent.yaml
file directly), using
either the service dragent restart
or
docker restart sysdig-agent
command as appropriate.
Sysdig Monitor supports event integrations with certain applications by default. The Sysdig agent will automatically discover these services and begin collecting event data from them.
The following applications are currently supported:
Docker
Kubernetes
Other methods of ingesting custom events into Sysdig Monitor are touched upon in Custom Events.
By default, only a limited set of events is collected for a supported
application, and are listed in the agent’s default settings
configuration file (/opt/draios/etc/dragent.default.yaml
).
To enable collecting other supported events, add an events
entry to
dragent.yaml
.
You can also change log
entry in dragent.yaml
to filter events by
severity.
Learn more about it in the following sections.
Events marked with *
are enabled by default; see the
dragent.default.yaml
file.
The following Docker events are supported.
docker:
container:
- attach # Container Attached (information)
- commit # Container Committed (information)
- copy # Container Copied (information)
- create # Container Created (information)
- destroy # Container Destroyed (warning)
- die # Container Died (warning)
- exec_create # Container Exec Created (information)
- exec_start # Container Exec Started (information)
- export # Container Exported (information)
- kill # Container Killed (warning)*
- oom # Container Out of Memory (warning)*
- pause # Container Paused (information)
- rename # Container Renamed (information)
- resize # Container Resized (information)
- restart # Container Restarted (warning)
- start # Container Started (information)
- stop # Container Stopped (information)
- top # Container Top (information)
- unpause # Container Unpaused (information)
- update # Container Updated (information)
image:
- delete # Image Deleted (information)
- import # Image Imported (information)
- pull # Image Pulled (information)
- push # Image Pushed (information)
- tag # Image Tagged (information)
- untag # Image Untaged (information)
volume:
- create # Volume Created (information)
- mount # Volume Mounted (information)
- unmount # Volume Unmounted (information)
- destroy # Volume Destroyed (information)
network:
- create # Network Created (information)
- connect # Network Connected (information)
- disconnect # Network Disconnected (information)
- destroy # Network Destroyed (information)
The following Kubernetes events are supported.
kubernetes:
node:
- TerminatedAllPods # Terminated All Pods (information)
- RegisteredNode # Node Registered (information)*
- RemovingNode # Removing Node (information)*
- DeletingNode # Deleting Node (information)*
- DeletingAllPods # Deleting All Pods (information)
- TerminatingEvictedPod # Terminating Evicted Pod (information)*
- NodeReady # Node Ready (information)*
- NodeNotReady # Node not Ready (information)*
- NodeSchedulable # Node is Schedulable (information)*
- NodeNotSchedulable # Node is not Schedulable (information)*
- CIDRNotAvailable # CIDR not Available (information)*
- CIDRAssignmentFailed # CIDR Assignment Failed (information)*
- Starting # Starting Kubelet (information)*
- KubeletSetupFailed # Kubelet Setup Failed (warning)*
- FailedMount # Volume Mount Failed (warning)*
- NodeSelectorMismatching # Node Selector Mismatch (warning)*
- InsufficientFreeCPU # Insufficient Free CPU (warning)*
- InsufficientFreeMemory # Insufficient Free Mem (warning)*
- OutOfDisk # Out of Disk (information)*
- HostNetworkNotSupported # Host Ntw not Supported (warning)*
- NilShaper # Undefined Shaper (warning)*
- Rebooted # Node Rebooted (warning)*
- NodeHasSufficientDisk # Node Has Sufficient Disk (information)*
- NodeOutOfDisk # Node Out of Disk Space (information)*
- InvalidDiskCapacity # Invalid Disk Capacity (warning)*
- FreeDiskSpaceFailed # Free Disk Space Failed (warning)*
pod:
- Pulling # Pulling Container Image (information)
- Pulled # Ctr Img Pulled (information)
- Failed # Ctr Img Pull/Create/Start Fail (warning)*
- InspectFailed # Ctr Img Inspect Failed (warning)*
- ErrImageNeverPull # Ctr Img NeverPull Policy Violate (warning)*
- BackOff # Back Off Ctr Start, Image Pull (warning)
- Created # Container Created (information)
- Started # Container Started (information)
- Killing # Killing Container (information)*
- Unhealthy # Container Unhealthy (warning)
- FailedSync # Pod Sync Failed (warning)
- FailedValidation # Failed Pod Config Validation (warning)
- OutOfDisk # Out of Disk (information)*
- HostPortConflict # Host/Port Conflict (warning)*
replicationController:
- SuccessfulCreate # Pod Created (information)*
- FailedCreate # Pod Create Failed (warning)*
- SuccessfulDelete # Pod Deleted (information)*
- FailedDelete # Pod Delete Failed (warning)*
To customize the default events collected for a specific application (by
either enabling or disabling events), add an events
entry to
dragent.yaml
as described in the examples below.
An entry in a section in dragent.yaml
overrides the entire section
in the default configuration.
For example, the Pulling
entry below will permit only kubernetes pod
Pulling
events to be collected and all other kubernetes pod events
settings in dragent.default.yaml
will be ignored.
However, other kubernetes sections - node
and replicationController
-
remain intact and will be used as specified in dragent.default.yaml.
Collect only ‘Pulling’ events from Kubernetes for pods:
events:
kubernetes:
pod:
- Pulling
To disable all events in a section, set the event section to none
:
events:
kubernetes: none
docker: none
These methods can be combined. For example, disable all kubernetes node
and docker image events and limit docker container events to
[attach, commit, copy]
(components events in other sections will be
collected as specified by default):
events:
kubernetes:
node: none
docker:
image: none
container:
- attach
- commit
- copy
In addition to bulleted lists, sequences can also be specified in a bracketed single line, eg.:
events:
kubernetes:
pod: [Pulling, Pulled, Failed]
So, the following two settings are equivalent, permitting only
Pulling, Pulled, Failed
events for pods to be emitted:
events:
kubernetes:
pod: [Pulling, Pulled, Failed]
events:
kubernetes:
pod:
- Pulling
- Pulled
- Failed
Events are limited globally at the agent level based on severity, using
the log
settings in dragent.yaml
.
The default setting for the events severity filter is information
(only warning and higher severity events are transmitted).
Valid severity levels are:
none, emergency, alert, critical, error, warning, notice, information, debug
.
Block all low-severity messages (notice, information, debug
):
log:
event_priority: warning
Block all event collection:
log:
event_priority: none
For other uses of the log
settings see Optional: Change the Agent Log
Level.
It is possible to filter custom metrics in the following ways:
Ability to include/exclude custom metrics using configurable patterns,
Ability to log which custom metrics are exceeding limits
After you identify those key custom metrics that must be received, use the new ‘include’ and ’exclude’ filtering parameters to make sure you receive them before the metrics limit is hit.
Here is an example configuration entry that would be put into the agent config file: (/opt/draios/etc/dragent.yaml)
metrics_filter:
- include: test.*
- exclude: test.*
- include: haproxy.backend.*
- exclude: haproxy.*
- exclude: redis.*
Given the config entry above, this is the action for these metrics:
test.* → send
haproxy.backend.request → send
haproxy.frontend.bytes → drop
redis.keys → drop
The semantic is: whenever the agent is reading metrics, they are filtered according to configured filters and the filtering rule order - the first rule that matches will be applied. Thus since the inclusion item for test.* was listed first it will be followed and that second ’exclude’ rule for the same exact metric entry will be ignored.
Logging is disabled by default. You can enable logging to see which metrics are accepted or dropped by adding the following configuration entry into the dragent.yaml config file:
metrics_excess_log: true
When logging of excess metrics is enabled, logging occurs at INFO-level,
every 30 seconds and lasts for 10 seconds. The entries that can be seen
in /opt/draios/logs/draios.log
will be formatted like this:
+/-[type] [metric included/excluded]: metric.name (filter: +/-[metric.filter])
The first ‘+’ or ‘-’, followed by ’type’ provides an easy way to quickly scan the list of metrics and spot which are included or excluded (’+’ means “included”, ‘-’ means “excluded”).
The second entry specifies metric type (“statsd”, “app_check”, “service_check”, or “jmx”).
A third entry spells out whether “included” or “excluded”, followed by the metric name. Finally, inside the last entry (in parentheses), there is information about filter applied and its effect (’+’ or ‘-’, meaning “include” or “exclude”).
With this example filter rule set:
metrics_filter:
- include: mongo.statsd.net*
- exclude: mongo.statsd.*
We might see the following INFO-level log entries (timestamps stripped):
-[statsd] metric excluded: mongo.statsd.vsize (filter: -[mongo.statsd.*])
+[statsd] metric included: mongo.statsd.netIn (filter: +[mongo.statsd.net*])
To get the most out of Sysdig Monitor, you may want to customize the way in which container data is prioritized and reported. Use this page to understand the default behavior and sorting rules, and to implement custom behavior when and where you need it. This can help reduce agent and backend load by not monitoring unnecessary containers, or– if encountering backend limits for containers– you can filter to ensure that the important containers are always reported.
By default, a Sysdig agent will collect metrics from all containers it detects in an environment. When reporting to the Monitor interface, it uses default sorting behavior to prioritize what container information to display first.
Out of the box, it chooses the containers with the highest
CPU
Memory
File IO
Net IO
and allocates approximately 1/4 of the total limit to each stat type.
As of agent version 0.86,
it is possible set a use_container_filter
parameter in the agent
config
file, tag/label specific containers, and set include/exclude
rules to push those containers to the top of the reporting hierarchy.
This is an effective sorting tool when:
You can manually mark each container with an include
or exclude
tag, AND
The number of includes is small (say, less than 100)
In this case, the containers that explicitly match the include
rules
will take top priority.
In some enterprises, the number of containers is too high to tag with
simple filtering rules, and/or the include_all
group is too large to
ensure that the most-desired containers are consistently reported. As of
Sysdig agent version 0.91,
you can append another parameter to the agent config file,
smart_container_reporting
.
This is an effective sorting tool when:
The number of containers is large and you can’t or won’t mark each one with include/exclude tags, AND
There are certain containers you would like to always prioritize
This helps ensure that even when there are thousands of containers in an environment, the most-desired containers are consistently reported.
Container filtering and smart container reporting affect the monitoring of all the processes/metrics within a container, including StatsD, JMX, app-checks, and built-in metrics.
Prometheus metrics are attached to processes, rather than containers, and are therefore handled differently.
The container limit is set in dragent.yaml under containers:limit:
The sydig_aggregated
parameter is automatically activated when smart
container reporting is enabled, to capture the most-desired metrics from
the containers that were excluded by smart filtering and report them
under a single entity. It appears like any other container in the Sysdig
Monitor UI, with the name “sysdig_aggregated.
”
Sysdig_aggregated
can report on a wide array of metrics; see
Sysdig_aggregated Container
Metrics. However, because
this is not a regular container, certain limitations apply:
container_id and container_image do not exist.
The aggregated container cannot be segmented by certain metrics that are excluded, such as process.
Some default dashboards associated with the aggregated container may have some empty graphs.
By default, the filtering feature is turned off. It can be enabled by adding the following line to the agent configuration:
use_container_filter: true
When enabled, the agent will follow include/exclude filtering rules based on:
container image
container name
container label
Kubernetes annotation or label
The default behavior in default.dragent.yaml
excludes based on a
container label (com.sysdig.report
) and/or a Kubernetes pod
annotation (.sysdig.com/report
).
The condition parameters are described in the following table:
Pattern name | Description | Example |
---|---|---|
| Matches if the process is running inside a container running the specified image |
|
| Matches if the process is running inside a container with the specified name |
|
| Matches if the process is running in a container that has a Label matching the given value |
|
| Matches if the process is attached to a Kubernetes object (Pod, Namespace, etc.) that is marked with the Annotation/Label matching the given value. |
|
all | Matches all. Use as last rule to determine default behavior. |
|
Once enabled (when use_container_filter: true
is set), the agent will
follow filtering rules from the container_filter
section.
Each rule is an include
or exclude
rule which can contain one or
more conditions.
The first matching rule in the list will determine if the container is included or excluded.
The conditions consist of a key name and a value. If the given key for a container matches the value, the rule will be matched.
If a rule contains multiple conditions they all need to match for the rule to be considered a match.
The dragent.default.yaml
contains the following default configuration
for container filters:
use_container_filter: false
container_filter:
- include:
container.label.com.sysdig.report: true
- exclude:
container.label.com.sysdig.report: false
- include:
kubernetes.pod.annotation.sysdig.com/report: true
- exclude:
kubernetes.pod.annotation.sysdig.com/report: false
- include:
all
Note that it excludes via a container.label
and by a
kubernetes.pod.annotation.
The examples on this page show how to edit in the dragent.yaml
file
directly. Convert the examples to Docker or Helm commands, if applicable
for your situation.
To enable container filtering using the default configuration in
default.dragent.yaml
(above), follow the steps below.
To set up, decide which containers should be excluded from automatic monitoring.
Apply the container label .com.sysdig.report
and/or the Kubernetes
pod annotation sysdig.com/report
to the designated containers.
Add the following line to dragent.yaml
to turn on the default
functionality:
use_container_filter: true
You can also edit dragent.yaml
to apply your own container filtering
rules.
To set up, decide which containers should be excluded from automatic monitoring.
Note the image, name, label, or Kubernetes pod information as appropriate, and build your rule set accordingly.
For example:
use_container_filter: true
container_filter:
- include:
container.name: my-app
- include:
container.label.com.sysdig.report: true
- exclude:
kubernetes.namespace.name: kube-system
container.image: "gcr.io*"
- include:
all
The above example shows a container_filter
with 3 include rules and 1
exclude rule.
If the container name is “my-app
” it will be included.
Likewise, if the container has a label with the key
“com.sysdig.report
” and with the value “true
”.
If neither of those rules is true, and the container is part of a
Kubernetes hierarchy within the “kube-system
” namespace and the
container image starts with “gcr.io
”, it will be excluded.
The last rule includes all, so any containers not matching an earlier rule will be monitored and metrics for them will be sent to the backend.
As of Sysdig agent version
0.91, you can add another
parameter to the config file: smart_container_reporting = true
This enables several new prioritization checks:
container_filter (you would enable and set include/exclude rules, as described above)
container age
high stats
legacy patterns
The sort is modified with the following rules in priority order:
User-specified containers come before others
Containers reported previously should be reported before those which have never been reported
Containers with higher usage by each of the 4 default stats should come before those with lower usage
Set up any simple container filtering rules you need, following either Option 1 or Option 2, above.
Edit the agent configuration:
smart_container_reporting: true
This turns on both smart_container_reporting
and
sysdig_aggregated
. The changes will be visible in the Sysdig
Monitor UI.
See also Sysdig_aggregated Container Metrics..
When the log level is set to DEBUG, the following messages may be found in the logs:
message | meaning |
---|---|
container <id>, no filter configured | container filtering is not enabled |
container <id>, include in report | container is included |
container <id>, exclude in report | container is excluded |
Not reporting thread <thread-id> in container <id> | Process thread is excluded |
See also: Optional: Change the Agent Log Level.
Sysdig_aggregated containers can report on the following metrics:
tcounters
other
time_ns
time_percentage
count
io_file
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
io_net
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
processing
time_ns
time_percentage
count
reqcounters
other
time_ns
time_percentage
count
io_file
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
io_net
time_ns_in
time_ns_out
time_ns_other
time_percentage_in
time_percentage_out
time_percentage_other
count_in
count_out
count_other
bytes_in
bytes_out
bytes_other
processing
time_ns
time_percentage
count
max_transaction_counters
time_ns_in
time_ns_out
count_in
count_out
resource_counters
connection_queue_usage_pct
fd_usage_pct
cpu_pct
resident_memory_usage_kb
swap_memory_usage_kb
major_pagefaults
minor_pagefaults
fd_count
cpu_shares
memory_limit_kb
swap_limit_kb
count_processes
proc_start_count
threads_count
syscall_errors
count
count_file
count_file_opened
count_net
protos
http
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
mysql
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
postgres
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
mongodb
server_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
client_totals
ncalls
time_tot
time_max
bytes_in
bytes_out
nerrors
names
transaction_counters
time_ns_in
time_ns_out
count_in
count_out
In addition to filtering data by container, it is also possible to filter independently by process. Broadly speaking, this refinement helps ensure that relevant data is reported while noise is reduced. More specifically, use cases for process filtering may include:
Wanting to alert reliably whenever a given process goes down. The total number of processes can exceed the reporting limit; when that happens, some processes are not reported. In this case, an unreported process could be misinterpreted as being “down.” Specify a filter for 30-40 processes to guarantee that they will always be reported.
Wanting to limit the number of noisy but inessential processes being reported, for example: sed, awk, grep, and similar tools that may be used infrequently.
Wanting to prioritize workload-specific processes, perhaps from integrated applications such as NGINX, Supervisord or PHP-FPM.
Note that you can report on processes and containers independently; the including/excluding of one does not affect the including/excluding of the other.
This feature requires the following Sysdig component versions:
Sysdig agent version 0.91 or higher
For on-premises installations: version 3.2.0.2540 or higher
By default, processes are reported according to internal criteria such as resource usage (CPU/memory/file and net IO) and container count.
If you choose to enable process filtering, processes in the include list will be given preference over other internal criteria.
Processes are filtered based on a standard priority filter description already used in Sysdig yaml files. It is comprised of -include and -exclude statements which are matched in order, with evaluation ceasing with the first matched statement. Statements are considered matched if EACH of the conditions in the statement is met.
Edit dragent.yaml per the following patterns to implement the filtering you need.
The process:
condition parameters and rules are described below.
Name | Value | Description |
---|---|---|
app_checks_always_send: | true/false | Legacy config that causes the agent to emit any process with app check. With process filtering, this translates to an extra “include” clause at the head of the process filter which matches a process with any app check, thereby overriding any exclusions. Still subject to limit. |
flush_filter: | Definition of process filter to be used if flush_filter_enabled == true. Defaults to -include all | |
flush_filter_enabled: | true/false | Defaults to false (default process reporting behavior). Set to true to use the rest of the process filtering options. |
limit: | N (chosen number) | Defines the approximate limit of processes to emit to the backend, within 10 processes or so. Default is 250 processes. |
top_n_per_container: | N (chosen number) | Defines how many of the top processes per resource category per emitted container to report after included processes. Still subject to limit. Defaults to 1. |
top_n_per_host: | N (chosen number) | Defines how many of the top processes per resource category per host are reported before included processes. Still subject to limit. Defaults to 1. |
The process: Condition Parameters
Rules
container.image: my_container_image
Validates whether the
container image associated with the process is a wild card match of
the provided image name
container.name: my_container_name Validates whether the container name associated with the process is a wild card match of the provided image name
container.label.XYZ: value
Validates whether the label XYZ of the
container associated with the process is a wildcard match of the
provided value
process.name: my_process_name
Validates whether the name of the
process is a wild card match of the provided value
process.cmdline: value
Checks whether the executable name of a
process contains the specified value, or any argument to the process
is a wildcard match of the provided value
appcheck.match: value
Checks whether the process has any appcheck
which is a wildcard match of the given value
all
Matches all processes, but does not whitelist them, nor does
it blacklist them. If no filter is provided, the default
is -include all
. However, if a filter is provided and no match is
made otherwise, then all unmatched processes will be blacklisted. In
most cases, the definition of a process filter should end
with -include: all
.
Block all processes from a given container. No processes from some_container_name will be reported.
process:
flush_filter_enabled: true
flush_filter:
- exclude:
container.name: some_container_name
- include:
allprocess: flush_filter: - exclude: container.name: some_container_name - include: all
Send all processes from a given container at high priority.
process:
flush_filter_enabled: true
flush_filter:
- include:
container.name: some_container_name
- include:
all
Send all processes that contain ‘java" in the name at high priority.
process:
flush_filter_enabled: true
flush_filter:
- include:
process.name: java
- include:
all
Send processes containing “java” from a given container at high priority.
process:
flush_filter_enabled: true
flush_filter:
- include:
container.name: some_container_name
process.name: java
- include:
all
Send all processes that contain “java” in the name that are not in
container some_container_nane
.
process:
flush_filter_enabled: true
flush_filter:
- exclude:
container.name: some_container_name
- include:
process.name: java
- include:
all
Send all processes containing “java” in the name. If a process does not contain “java” in the name and if the container within which the process runs is named some_container_name, then exclude it.
Note that each include/exclude rule is handled sequentially and hierarchically so that even if the container is excluded, it can still report “java” processes.
flush_filter:
- flush_filter_enabled: true
- include:
process.name: java
- exclude:
container.name: some_container_name
- include:
all
Send Java processes from one container and SQL processes from another at high priority.
process:
flush_filter:
- flush_filter_enabled: true
- include:
container.name: java_container_name
process.name: java
- include
container.name: sql_container_name
process.name: sql
- include
all
Only send processes running in a container with a given label.
process:
flush_filter:
- flush_filter_enabled: true
- include:
=container.label.report_processes_from_this_container_example_label: true
- exclude:
all
Sysdig agent does not automatically discover and collect metrics from
external file systems, such as NFS, by default. To enable collecting
these metrics, add the following entry to the dragent.yaml
file:
remotefs = true
In addition to the remote file systems, the following mount types are also excluded because they cause high load.
mounts_filter:
- exclude: "*|autofs|*"
- exclude: "*|proc|*"
- exclude: "*|cgroup|*"
- exclude: "*|subfs|*"
- exclude: "*|debugfs|*"
- exclude: "*|devpts|*"
- exclude: "*|fusectl|*"
- exclude: "*|mqueue|*"
- exclude: "*|rpc_pipefs|*"
- exclude: "*|sysfs|*"
- exclude: "*|devfs|*"
- exclude: "*|devtmpfs|*"
- exclude: "*|kernfs|*"
- exclude: "*|ignore|*"
- exclude: "*|rootfs|*"
- exclude: "*|none|*"
- exclude: "*|tmpfs|*"
- exclude: "*|pstore|*"
- exclude: "*|hugetlbfs|*"
- exclude: "*|*|/etc/resolv.conf"
- exclude: "*|*|/etc/hostname"
- exclude: "*|*|/etc/hosts"
- exclude: "*|*|/var/lib/rkt/pods/*"
- exclude: "overlay|*|/opt/stage2/*"
- exclude: "/dev/mapper/cl-root*|*|/opt/stage2/*"
- exclude: "*|*|/dev/termination-log*"
- include: "*|*|/var/lib/docker"
- exclude: "*|*|/var/lib/docker/*"
- exclude: "*|*|/var/lib/kubelet/pods/*"
- exclude: "*|*|/run/secrets"
- exclude: "*|*|/run/containerd/*"
- include: "*|*|*"
To include a mount type:
Open the dragent.yaml
file.
Remove the corresponding line from the exclude list in the
mount_filter
.
Add the file mount to the include list under mount_filter
.
The format is:
# format of a mount filter is:
# ```
# mounts_filter:
# - exclude: "device|filesystem|mount_directory"
# - include: "pattern1|pattern2|pattern3"
For example:
mounts_filter:
- include: "*|autofs|*"mounts_filter:
- include: "overlay|*|/opt/stage2/*"
- include: "/dev/mapper/cl-root*|*|/opt/stage2/*"
Save the configuration changes and restart the agent.
Sometimes, security requirements dictate that capture functionality should NOT be triggered at all (for example, PCI compliance for payment information).
To disable Captures altogether:
Access using one of the options listed.
This example accesses dragent.yaml
directly. ``
Set the parameter:
sysdig_capture_enabled: false
Restart the agent, using the command: ``
service dragent restart
See Captures for more information on the feature
Sysdig provides a configuration option called thin cointerface to reduce
the memory footprint in the agent. When the agent is installed as a
Kubernetes daemonset, you can optionally enable the thin cointerface in
the sysdig-agent configmap
.
Pros
Cons
How It Works
In a typical Kubernetes cluster, two instances of agent daemonset are installed to retrieve the data. They are automatically connected to the Kubernetes API server to retrieve the metadata associated with the entities running on the cluster and sends the global Kubernetes state to the Sysdig backend. Sysdig uses this data to generate kube state metrics.
A delegated agent will not have a higher CPU or memory footprint than a non-delegated agent.
On very large Kubernetes clusters (in the range of 10,000 pods) or clusters with several replication controllers, the agent’s data ingestion can have a significant memory footprint on itself and on the Kubernetes API server. Thin cointerface is provided to reduce this impact.
Enabling this option changes the way the agent communicates with the API server and reduces the need to cache data, which in turn reduces the overall memory usage. Thin cointerface does this by moving some processing from the agent’s cointerface process to the dragent process. This change does not alter the data which is ultimately sent to the backend nor will it impact any Sysdig feature.
The thin cointerface feature is disabled by default.
To Enable:
Add the following in either the sysdig-agent’s configmap
or via
the dragent.yaml
file:
thin_cointerface_enabled: true
HPA kube state metrics are no longer collected by default. To enable the agent to collect HPA kube state metrics, you must edit the agent configuration file, dragent.yaml
, and include it along with the other resources you would like to collect.
For example, to collect all supported resources including HPAs, add the following to dragent.yaml
:
k8s_extra_resources:
include:
- services
- resourcequotas
- persistentvolumes
- persistentvolumeclaims
- horizontalpodautoscalers
The Sysdig agent collects HPA, PVS, PV, Resourcequota, and Services kube state metrics by default.
To disable some of them, you must edit the agent config file, dragent.yaml
, as follows:
k8s_extra_resources:
include:
- services
- resourcequotas
- persistentvolumes
- persistentvolumeclaims
- horizontalpodautoscalers
The above list includes all the supported resources so you must remove the resources you are not interested in. For example, if you wanted to disable Services, it should look like the following:
k8s_extra_resources:
include:
- resourcequotas
- persistentvolumes
- persistentvolumeclaims
- horizontalpodautoscalers
For more information, see Understanding the Agent Configuration Files.
Required: Sysdig agent version 92.1 or higher.
As of agent version 9.5.0, go_k8s_user_events:true
is the default
setting. Set to false
to use the older, C++-based version.
To streamline Sysdig agent processing times and reduce CPU load, you can use an updated processing engine written in Go.
To do so, edit the following code in dragent.yaml
:
go_k8s_user_events: true
The agent listens on /k8s-audit
for Kubernetes audit events. Configure
the path using the following configuration option:
security:{k8s_audit_server_path_uris: [path1, path2]}
For more information, see Kubernetes Audit Logging.
If you have containerd using a custom socket, you can specify this parameter in the agent configuration to correctly capture the containers’ metadata:
cri:
socket_path: /run/k3s/containerd/containerd.sock
Sysdig allows you to configure file log levels for agents globally and granularly.
The Sysdig agent generates log entries in /opt/draios/logs/draios.log
.
The agent will rotate the log file when it reaches 10MB in size, keeping
the 10 most recent log files archived with a date-stamp appended to the
filename.
In order of increasing detail, the log levels available are: [ none | critical| error | warning |notice | info | debug | trace ].
The default level (info) creates an entry for each aggregated metrics transmission to the backend servers, once per second, in addition to entries for any warnings and errors.
Setting the value lower than info
may prohibit troubleshooting
agent-related issues.
The type and amount of logging can be changed by adding parameters and log level arguments shown below to the agent’s user settings configuration file here:
/opt/draios/etc/dragent.yaml
After editing the dragent.yaml
file, restart the agent at the shell
with: service dragent restart
to affect changes.
Note that dragent.yaml
code can be written in both YAML and JSON. The
examples below use YAML.
When troubleshooting agent behavior, increase the logging to debug for full detail:
log:
file_priority: debug
If you wish to reduce log messages going to the
/opt/draios/logs/draios.log
file, add the log:
parameter with one of
the following arguments under it and indented two spaces: [ none |
error | warning | info | debug | trace ]
log:
file_priority: error
If you are running the containerized agent, you can also reduce
container console output by adding the additional parameter
console_priority:
with the same arguments [ none | error | warning
| info | debug | trace ]
log:
console_priority: warning
Note that troubleshooting a host with less than the default ‘info’ level will be more difficult or not possible. You should revert to ‘info’ when you are done troubleshooting the agent.
A level of ’error’ will generate the fewest log entries, a level of ’trace’ will give the most, ‘info’ is the default if no entry exists.
helm install ... \
--set sysdig.settings.log.file_priority=debug \
--set sysdig.settings.log.console_priority=debug
sysdig:
settings:
log:
file_priority: debug
console_priority: debug
customerid: 831f3-Your-Access-Key-9401
tags: local:sf,acct:eng,svc:websvr
log:
file_priority: warning
console_priority: info
OR
customerid: 831f3-Your-Access-Key-9401
tags: local:sf,acct:eng,svc:websvr
log: { file_priority: debug, console_priority: debug }
If you are using the “ADDITIONAL_CONF” parameter to start a Docker containerized agent, you would specify this entry in the Docker run command:
-e ADDITIONAL_CONF="log: { file_priority: error, console_priority: none }"
-e ADDITIONAL_CONF="log:\n file_priority: error\n console_priority: none"
When running in a Kubernetes infrastructure (installed using the v1
method, comment in the “ADDITIONAL_CONF” line in the agent
sysdig-daemonset.yaml
manifest file, and modify as needed:
- name: ADDITIONAL_CONF #OPTIONAL pass additional parameters to the agent
value: "log:\n file_priority: debug\n console_priority: error"
Sysdig Agent provides the ability to set component-wise log levels that
override the global file logging level controlled by the file_priority
configuration option. The components represent internal software modules
and can be found in /opt/draios/logs/draios.log
.
By controlling logging at the fine-grained component level, you can
avoid excessive logging from certain components in draios.log
or
enable extra logging from specific components for troubleshooting.
Components can also have an optional feature level logging that can provide a way to control the logging for a particular feature in Sysdig Agent.
To set feature-level or component-level logging:
Determine the agent feature or component you want to set the log level:
To do so,
Open the /opt/draios/logs/draios.log
file.
Copy the component name.
The format of the log entry is:
<timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
For example, the given snippet from a sample log file shows log
messages from promscrape
featture, sdjagent
, mountedfs_reader
,
watchdog_runnable
, protobuf_file_emitter
,
connection_manager
, and dragent
.
2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
2020-09-07 18:52:40.065, 27979.27980, Debug, promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
To set feature-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info.
log:
file_priority: notice
file_priority_by_component:
- "promscrape: info"
The log levels specified for feature override global settings.
To set component-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info, sdjagent
, mountedfs_reader
component log level to debug, watchdog_runnable
component log level
to warning and promscrape:prom_emitter
component log level to debug.
log:
file_priority: notice
file_priority_by_component:
- "promscrape: info"
- "promscrape:prom_emitter: debug"
- "watchdog_runnable: warning"
- "sdjagent: debug"
- "mountedfs_reader: debug"
The log levels specified for feature override global settings. The log levels specified for component overide feature and global settings.
Restart the agent.
For example, if you have installed the agent as a service, then run:
$ service dragent restart
Sysdig Agent provides the ability to set component-wise log levels that
override the global console logging level controlled by the
console_priority
configuration option. The components represent
internal software modules and can be found in
/opt/draios/logs/draios.log
.
By controlling logging at the fine-grained component level, you can
avoid excessive logging from certain components in draios.log
or
enable extra logging from specific components for troubleshooting.
Components can also have an optional feature level logging that can provide a way to control the logging for a particular feature in Sysdig Agent.
To set feature-level or component-level logging:
Determine the agent component you want to set the log level:
To do so,
Look at the console output.
If you’re using an orchestrator like Kubernetes, the log viewer
facility, such as the kubectl
log command, shows the console
log output.
Copy the component name.
The format of the log entry is:
<timestamp>, <<pid>.<tid>>, <log level>, [feature]:<component>[pid]:[line]: <message>
For example, the given snippet from a sample log file shows log
messages from promscrape
featture, sdjagent
, mountedfs_reader
,
watchdog_runnable
, protobuf_file_emitter
,
connection_manager
, and dragent
.
2020-09-07 17:56:01.173, 27979.28018, Information, sdjagent[27980]: Java classpath: /opt/draios/share/sdjagent.jar
2020-09-07 17:56:01.173, 27979.28018, Information, mountedfs_reader: Starting mounted_fs_reader with pid 27984
2020-09-07 17:56:01.174, 27979.28019, Information, watchdog_runnable:105: connection_manager starting
2020-09-07 17:56:01.174, 27979.28019, Information, protobuf_file_emitter:64: Will save protobufs for all message types
2020-09-07 17:56:01.174, 27979.28019, Information, connection_manager:282: Initiating connection to collector
2020-09-07 17:56:01.175, 27979.27979, Information, dragent:1243: Created Sysdig inspector
2020-09-07 18:52:40.065, 27979.27980, Debug, promscrape:prom_emitter:72: Sent 927 Prometheus metrics of 7297 total
2020-09-07 18:52:41.129, 27979.27981, Information, promscrape:prom_stats:45: Prometheus timeseries statistics, 5 endpoints
To set feature-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info.
log:
console_priority: notice
console_priority_by_component:
- "promscrape: info"
The log levels specified for feature override global settings.
To set component-level logging:
Open /opt/draios/etc/dragent.yaml
.
Edit the dragent.yaml
file and add the desired feature:
In this example, you are setting the global level to notice and
promscrape
feature level to info, sdjagent
, mountedfs_reader
component log level to debug, watchdog_runnable
component log level
to warning and promscrape:prom_emitter
component log level to debug.
log:
console_priority: notice
console_priority_by_component:
- "promscrape: info"
- "promscrape:prom_emitter: debug"
- "watchdog_runnable: warning"
- "sdjagent: debug"
- "mountedfs_reader: debug"
The log levels specified for feature override global settings. The log levels specified for component overide feature and global settings.
Restart the agent.
For example, if you have installed the agent as a service, then run:
$ service dragent restart
If you want to maintain centralized control over the configuration of your Sysdig agents, one of the following approaches is typically ideal:
Via an orchestration system, such as using Kubernetes or Mesos/Marathon.
Using a configuration management system, such as Chef or Ansible.
However, if these approaches are not viable for your environment, or to further augment your Agent configurations via central control, Sysdig Monitor provides an Auto-Config option for agents. The feature allows you to upload fragments of YAML configuration to Sysdig Monitor that will be automatically pushed and applied to some/all of your Agents based on your requirements.
Independent of the Auto-Config feature, typical Agent configuration lives in /opt/draios/etc and is derived from a combination of base config in the dragent.default.yaml file and any overrides that may be present in dragent.yaml. See also Understanding the Agent Config Files.
Agent Auto-Config adds a middle layer of possible overrides in an additional file dragent.auto.yaml.When present, the the order of config application from highest precedence to lowest now becomes:
dragent.yaml
dragent.auto.yaml
dragent.default.yaml
While all Agents are by default prepared to receive and make use of Auto-Config data, the file dragent.auto.yaml will not be present on an Agent until you’ve pushed central Auto-Config data to be applied to that Agent.
Auto-Config settings are performed via Sysdig Monitor’s REST API. Simplified examples are available that use the Python client library to get or set current Auto-Config settings. Detailed examples using the REST API are shown below.
The REST endpoint for Auto-Config is /api/agents/config. Use the GET method to review the current configuration. The following example shows the initial empty settings that result in no dragent.auto.yaml files being present on your Agents.
curl -X GET \
--header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
https://app.sysdigcloud.com/api/agents/config
Output:
{
"files": []
}
Use the PUT method to centrally push YAML that will be distributed and applied to your Agents as dragent.auto.yaml files. The content parameter must contain syntactically-correct YAML. The filter option is used to specify if the config should be sent to one agent or all of them, such as in this example to globally enable Debug logging on all Agents:
curl -X PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
https://app.sysdigcloud.com/api/agents/config -d '
{
"files": [
{
"filter": "*",
"content": "log:\n console_priority: debug"
}
]
}'
Alternatively, the filter can specify a hardware MAC address for a single Agent that should receive a certain YAML config. All MAC-specific configs should appear at the top of the JSON object and are not additive to any global Auto-Config specified with “filter”: “*” at the bottom. For example, when the following config is applied, the one Agent that has the MySQL app check configured would not have Debug logging enabled, but all others would.
curl -X PUT \
--header "Content-Type: application/json" \
--header "Authorization: Bearer xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" \
https://app.sysdigcloud.com/api/agents/config -d '
{
"files": [
{
"filter": "host.mac = \"08:00:27:de:5b:b9\"",
"content": "app_checks:\n - name: mysql\n pattern:\n comm: mysqld\n conf:\n server: 127.0.0.1\n user: sysdig-cloud\n pass: sysdig-cloud-password"
},
{
"filter": "*",
"content": "log:\n console_priority: debug"
}
]
}'
To update the active central Auto-Config settings, simply PUT a complete replacement JSON object.
All connected Agents will receive centrally-pushed Auto-Config updates that apply to them based on the filter settings. Any Agent whose Auto-Config is enabled/disabled/changed based on the centrally-pushed settings will immediately restart, putting the new configuration into effect. Any central Auto-Config settings that would result in a particular Agent’s Auto-Config remaining the same will not trigger a restart.
To clear all Agent Auto-Configs, use the PUT method to upload the original blank config setting of ’{ “files”: [] }’.
It is also possible to override active Auto-Config on an individual Agent. To do so, follow these steps for your Agent:
Add the following config directly to the dragent.yaml file: auto_config: false.
Delete the file /opt/draios/etc/dragent.auto.yaml.
Restart the Agent.
For such an Agent to opt-in to Auto-Config again, remove auto_config: false from the dragent.yaml and restart the Agent.
To prevent the possibility of pushing Auto-Config that would damage an Agent’s ability to connect, the following keys will not be accepted in the centrally-pushed YAML.
auto_config
customerid
collector
collector_port
ssl
ssl_verify_certificate
ca_certificate
compression
Sysdig provides an Agent Console to interact with the Sysdig agent. This is a troubleshooting tool to help you view configuration files and investigate agent configuration problems quickly.
From Explore click the Groupings drop-down.
Select Hosts & Container or Nodes.
Click the desired host to investigate the corresponding agent configuration.
Click Options (three dots) on the right upper corner of the Explore tab.
Click Agent Console.
The ?
command displays the commands to manage Prometheus configuration
and targets monitored by the Sysdig agent.
$ prometheus ?
$ prometheus config ?
$ prometheus config show ?
The syntax of the Agent Console commands is as follows:
directory command
directory sub-directory command
directory sub-directory sub-sub-directory command
Run the following to find the version of the agent running in your environment:
$ version
An example output:
12.0.0
These commands help troubleshoot Prometheus targets configured in your environment.
For example, the following commands display and scrape the Prometheus endpoints respectively.
$ prometheus target show
$ prometheus target scrape
The Promscrape CLI consists of the following sections.
config
: Manages Sysdig agent-specific Prometheus configuration.
metadata
: Manages metadata associated with the Prometheus targets
monitored by the Sysdig agent.
stats
: Helps view the global- and job-specific Prometheus
statistics.
target
: Manages Prometheus endpoints monitored by Sysdig agent.
The show
command displays the information about the subsection. For
example, the following example displays the configuration of the
Prometheus server.
$ prometheus config show
5 Configuration Value
6 Enabled True
7 Target discovery Prometheus service discovery
8 Scraper Promscrape v2
9 Ingest raw True
10 Ingest calculated True
11 Metric limit 2000
The scrape
command scrapes a Prometheus target and displays the
information. The syntax is:
$ prometheus target scrape -url <URL>
For example:
$ prometheus target scrape -url http://99.99.99.3:10055/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
7 # TYPE go_gc_duration_seconds summary
8 go_gc_duration_seconds{quantile="0"} 7.5018e-05
9 go_gc_duration_seconds{quantile="0.25"} 0.000118155
10 go_gc_duration_seconds{quantile="0.5"} 0.000141586
11 go_gc_duration_seconds{quantile="0.75"} 0.000171626
12 go_gc_duration_seconds{quantile="1"} 0.00945638
13 go_gc_duration_seconds_sum 0.114420898
14 go_gc_duration_seconds_count 607
The Agent configuration commands have a different syntax.
Run the following to view the configuration of the agent running in your environment:
$ configuration show-dragent-yaml
$ configuration show-configmap-yaml
$ configuration show-default-yaml
$ configuration show-backend-yaml
The output displays the configuration file. Sensitive data, such as the credentials, are obfuscated.
customerid: "********"
watchdog:
max_memory_usage_mb: 2048
User-sensitive configuration is obfuscated and not visible through the CLI.
All the information is read-only. You cannot currently change any configuration by using the Agent console.
Runs completely inside the agent. It does not use bash or any other Linux terminals to prevent the risk of command injection.
Runs only via a TLS connection with the Sysdig backend.
This is currently turned on by default. To turn off Agent Console for a particular team:
Navigate to Settings > Teams.
Select the team that you want to disable Agent Console for.
From Additional Permissions, Deselect Agent CLI.
Click Save.
To turn it off in your environment, edit the following in the
dragent.yaml
file:
command_line:
enabled: false
The steps to upgrade an agent differ depending on whether the agent was originally installed as a container or as a service.
Follow the upgrade best practices for a smooth upgrade and to maximize the value of Sysdig applications:
Keep upgrades current
Upgrade progressively without skipping versions
Test upgrades in a non-mission-critical or staging environment before rolling in to production.
This section describes how to check the current version of the installed agents, and then how to upgrade them.
If the agent is installed in a Kubernetes environment, run:
kubectl get pods -n sysdig-agent -o=jsonpath='{.items[0].spec.containers[:1].image}'
If the agent is installed as container, run:
docker exec sysdig-agent /opt/draios/bin/dragent --version
If the agent is installed as a service, run:
/opt/draios/bin/dragent --version
The agent version can also be found in the agent log file:
/opt/draios/logs/draios.log.
Look for the “Agent starting” message, which is logged whenever the agent restarts.
Update the containerized agent version as you normally update any container; the basic steps are below.
Use the full run command as shown in the Settings > Agent Installation tab of your account.
To see which agent versions are available see tags.
Update the chart:
helm repo update
Do one of the following:
values.yaml
file, modify or add (if it’s missing) the image.tag
field and run:helm upgrade --namespace sysdig-agent sysdig-agent -f values.yaml sysdig/sysdig
helm upgrade --namespace sysdig-agent --set image.tag=<latest_version> --reuse-values sysdig-agent sysdig/sysdig
Replace <latest_version> with the latest version number of Sysdig agent.For more information on using Helm, see Helm Charts.
Check whether .yaml
files must be updated:
Updating the agent image does not overwrite the daemonset.yaml
and
sysdig-agent-configmap.yaml
on your local system. Check the Sysdig
Agent Release Notes to see
if you need to download the latest .yaml
files from
GitHub.
Perform update:
kubectl set image ds/sysdig-agent sysdig-agent=sysdig/agent:<tag>
Watch update status:
kubectl rollout status ds/sysdig-agent
Basic Steps: stop the agent, remove it, pull the new agent, and install it.
The exact Docker command can also be found in the Sysdig Settings > Agent Installation menu.
docker stop sysdig-agent
docker rm sysdig-agent
docker pull sysdig/agent
docker run . . .
For service (non-containerized) agent installations, updates are
installed as part of the normal system upgrade available with
apt-get
or yum
.
apt-get update
apt-get -y install draios-agent
yum clean expire-cache
yum -y install draios-agent
This section describes uninstalling the Sysdig agent when it was installed as a service.
If the agent was installed as a container, remove it using standard container commands.
If the agent was installed by an orchestrator, such as Kubernetes, remove it by using the standard orchestrator commands.
To uninstall the agent from Debian Linux distributions, including Ubuntu:
As the sudo
user, run the following command in a terminal on each
host:
sudo apt-get remove draios-agent
To uninstall the agent from Fedora Linux distributions, including CentOS, Red Hat Enterprise Linux, as well as Amazon AMI and Amazon Linux 2:
As the sudo
user, run the following command in a terminal on each
host:
sudo yum erase draios-agent
This section describes methods for troubleshooting two types of issue:
Disconnecting Agents
Can’t See Metrics After Agent Install
If agents are disconnecting, there could be problems with addresses that need to be resolved in the agent configuration files. See also Understanding the Agent Config Files.
The Sysdig agent will use the eth0
MAC address to identify the
different hosts within an infrastructure. In a virtualized environment,
you should confirm each of your VM’s eth0
MAC addresses are unique.
If a unique address cannot be configured, you can supply an additional
parameter in the Sysdig agent’s dragent.yaml
configuration file:
machine_id_prefix: prefix
The prefix text can be any string and will be prepended to the MAC address as reported in the Sysdig Monitor web interface’s Explore tables.
Example: (using ADDITIONAL_CONF
rather than Kubernetes
Configmap
)
Here is an example Docker run command installing the parameter via the
ADDITIONAL_CONF
parameter
docker run --name sysdig-agent --privileged --net host --pid host -e ACCESS_KEY=abc123-1234-abcd-4321-abc123def456 -e TAGS=tag1:value1 -e ADDITIONAL_CONF="machine_id_prefix: MyPrefix123-" -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro sysdig/agent
The resulting /opt/draios/etc/dragent.yaml
config file would look like
this:
customerid:abc123-1234-abcd-4321-abc123def456
tags: tag1:value1
machine_id_prefix: MyPrefix123-
You will then see all of your hosts, provided that all the prefixes are unique. The prefix will be visible whenever the MAC address is displayed in any view.
See also: Agent Configuration.
In Google Container Engine (GKE) environments, MAC addresses could be repeated across multiple hosts. This would cause some hosts running Sysdig agents not to appear in your web interface.
To address this, add a unique machine ID prefix to each config you use to deploy the agent to a given cluster (i.e. each sysdig-daemonset.yaml file).
Note: This example uses the
(v1) ADDITIONAL_CONF
,
rather than (v2)
Configmap
method.
- name: ADDITIONAL_CONF value: "machine_id_prefix: mycluster1-prefix-"
If agents were successfully installed, you could log in to the Sysdig Monitor UI, but no metrics are displayed in the Explore panel, first confirm that the agent license count has not been exceeded. Then check for any proxy, firewall, or host security policies preventing proper agent communication to the Sysdig Monitor backend infrastructure.
If network connectivity is good, the agent will connect to the backend but will be disconnected after a few seconds if the license count has been exceeded.
To check whether you are over-subscribed, go to
Settings > Subscription
.
See Subscription for details.
Check your service provider VPC security groups to verify that network ACLs are set to allow the agent’s outbound traffic over TCP ports. See Sysdig Collector Ports for the supported TCP ports for each region.
Due to the distributed nature of the Sysdig Monitor infrastructure, the agent must be open for outbound connections to collector.sysdigcloud.com on all outbound IP addresses.
Check Amazon’s public IP ranges file to see all the potential IP addresses the Sysdig agent can use to communicate with the Sysdig backend databases.
AWS metadata is used for gathering information about the instance itself, such as instance id, public IP address, etc.
When running on an AWS instance, access to the following AWS metadata endpoint is also needed: 169.254.169.254
The agent requires access to the following local system resources in order to gather metrics:
Read/Write access to /dev/sysdig
devices.
Read access to all the files under /proc
file system.
For container support, the Docker API endpoint
/var/run/docker.sock
If any settings or firewall modifications are made, you may need to restart the agent service. In a shell on the affected instances issue the following command:
sudo service dragent restart
In addition to the information on Host Requirements for Agent Installation, this page describes how the agent uses kernel headers and tips on troubleshooting, if needed.
The Sysdig agent requires a kernel module in order to install successfully on a host. This can be obtained in three ways:
Agent compiles the module using kernel headers.
If the hosts in your environment already have kernel header files pre-installed, no special action is needed. Or you can install the kernel headers manually; see below.
Agent auto-downloads precompiled modules from Sysdig’s AWS storage location.
If the headers are not already installed but the agent is able to auto-download, no special action is needed. If there is no internet connectivity, you can use method 3 (download from an internal URL).
Agent downloads precompiled modules from an internal URL.
Use the environment variable SYSDIG_PROBE_URL
. See also
Understanding the Agent Config
Files. Contact Sysdig
support for assistance.
On Fedora 35 or similar SELinux-enabled distributions with default restrictive policies, the agent init container, agent-kmodule
, will not install the downloaded kernel module raising an error similar to the following:
insmod: ERROR: could not insert module /root/.sysdig/sysdigcloud-probe-12.3.1-x86_64-5.16.11-200.fc35.x86_64-67098c7fdcc97105d4b9fd0bb2341888.ko: Permission denied
In such cases, we recommend that you use eBPF option while running agent-kmodule
instead.
All supported distribution released kernels have this support but if creating a custom kernel, it must support the following options:
CONFIG_TRACEPOINTS
CONFIG_HAVE_SYSCALL_TRACEPOINTS
In some cases, the host(s) in your environment may use Unix versions that do not match the provided headers, and the agent may fail to install correctly. In those cases, you must install the kernal headers manually.
For Debian-syle distributions, run the command:
apt-get -y install linux-headers-$(uname -r)
For RHEL-style distributions, run the command:
yum -y install kernel-devel-$(uname -r)
For RancherOS distributions, the kernel headers are available in the
form of a system service and therefore are enabled using the
ros
service command:
sudo ros service enable kernel-headers-system-docker
sudo ros service up -d kernel-headers-system-docker
NOTE: Some cloud hosting service providers supply pre-configured Linux instances with customized kernels. You may need to contact your provider’s support desk for instructions on obtaining appropriate header files, or for installing the distribution’s default kernel.
During an agent installation in an Amazon machine image (AMI) you may encounter the following errors while the installer is trying to compile the Sysdig kernel module:
Errors
“Unable to find kernel development files for the current kernel version” or
“FATAL: Module sysdigcloud-probe not found”
This indicates your machine is running a kernel in an older AMI for which the kernel headers are no longer available in the configured repositories. The issue has to do with Amazon purging packages in the yum repository when new Amazon Linux machine images are released.
The solution is either to update your kernel to a version for which header files are readily available (recommended), or perform a one-time installation of the kernel headers for your older AMI.
First install a new kernel and reboot your instance:
sudo yum -y install kernel
sudo reboot
After rebooting, check to see if the host is reporting metrics to your Sysdig account. If not, you may need to issue three more commands to install the required header files:
sudo yum -y install kernel-devel-$(uname -r)
sudo /usr/lib/dkms/dkms_autoinstaller start
sudo service dragent restart
Although it is recommended to upgrade to the latest kernel for security and performance reasons, you can alternatively install the older headers for your AMI.
Find the the AMI version string and install the appropriate headers with the commands:
releasever=$(cat /etc/os-release | grep 'VERSION_ID' | grep -Eo "[0-9]{4}\.[0-9]{2}")
sudo yum -y --releasever=${releasever} install kernel-devel-$(uname -r)
Issue the remaining commands to allow the Sydig Agent to start successfully:
sudo /usr/lib/dkms/dkms_autoinstaller start
sudo service dragent restart
The file /etc/image-id
shows information about the original machine
image with which your instance was set up:
[ec2-user ~]$ cat /etc/image-id
image_name="amzn-ami-hvm"
image_version="2017.03"
image_arch="x86_64"
image_file="amzn-ami-hvm-2017.03.0.20170401-x86_64.ext4.gpt"
image_stamp="26a3-ed31"
image_date="20170402053945"
recipe_name="amzn ami"
recipe_id="47cfa924-413c-d460-f4f2-2af7-feb6-9e37-7c9f1d2b"
This file will not change as you install updates from the yum repository.
The file /etc/system-release
will tell what version of the AWS image
is currently installed:
[ec2-user ~]$ cat /etc/system-release
Amazon Linux AMI release 2017.03
The resource requirements for the Sysdig agent are subjective to the size and load of the host. Increased activity equates to higher resource requirements.
You might see 5 to 20 KiB/s of bandwidth consumed. Different variables can increase the throughput required. For example:
The number of metrics
The number of events
Kubernetes objects
Products and features enabled
When a Sysdig Capture is being collected, you can expect to see a spike in the bandwidth while the capture file is being ingested.
Sysdig does not recommend placing bandwidth shaping or caps on the agent to ensure that data is sent to the Sysdig Collection service.
In general, in larger clusters, the agent requires more memory, and in servers with a high number of cores, the agent requires more CPU cores to monitor all the system calls. You will use CPU cores on the host and the Kubernetes nodes visible to the agent as proxies for the rate of events processed in the agent.
Similarly, there are different factors that are at play, and considering all the factors, we recommend the following:
Small: CPU core count <= 8. Kubernetes nodes <=10
Medium: 8 < CPU core count <= 32. 10 < Kubernetes nodes <= 100
Large: CPU core count > 32. Kubernetes nodes > 100
While you can expect the behavior with the given numbers to be better than simply using the default values, Sysdig cannot guarantee that resource allocation will be correct for all the cases.
Cluster Size | Small | Medium | Large |
---|---|---|---|
Kubernetes CPU Request | 1 | 3 | 5 |
Kubernetes CPU Limit | 1 | 3 | 5 |
Kubernetes Memory Request | 1024 MB | 3072 MB | 6144 MB |
Kubernetes Memory Limit | 1024 MB | 3072 MB | 6144 MB |
Dragent Memory Watchdog | 512 MB | 1024 MB | 2048 MB |
Cointerface Memory Watchdog | 512 MB | 2048 MB | 4096 MB |
Note that the agent has its own memory watchdog to prevent runaway memory consumption on the host in case of memory leaks. The default values of the watchdog are specified in the following agent configuration file.
watchdog:
max_memory_usage_mb: 1024
max_memory_usage_subprocesses:
sdchecks: 128
sdjagent: 256
mountedfs_reader: 32
statsite_forwarder: 32
cointerface: 512
promscrape: 640
The recommended value for promscrape
depends on the amount of timeseries and label data that required to be scraped on a particular node. The cluster size does not have an effect on promscrape
memory usage.
max_memory_usage_mb
corresponds to the dragent process in the agent.
All the values are given in MiB.
For example, use the following agent configuration to match the agent watchdog settings with large values:
watchdog:
max_memory_usage_mb: 2048
max_memory_usage_subprocesses:
sdchecks: 128
sdjagent: 256
mountedfs_reader: 32
statsite_forwarder: 32
cointerface: 4096
promscrape: 640
The serverless environment: As cloud platforms have evolved, both the convenience and the abstraction levels have increased simultaneously and new agent models are required.
For example, with Amazon’s ECS and EKS, users remain in charge of managing the underlying virtual host machines. In environments like Fargate, however, the hosts are implicitly allocated by the cloud provider and users simply run their containers without allocating, configuring, or having any knowledge of the underlying compute infrastructure.
While this “container as a service” model is convenient, it can introduce risk, as many users leave the containers unattended and don’t monitor for security events inside them that can exfiltrate secrets, compromise business data, and increase their AWS/cloud provider costs. In addition, it is not possible to install a standard agent in an environment where you do not have access to a host.
For these reasons, Sysdig has introduced a new “serverless agent” that can be deployed in such container-based cloud environments.
Additional platforms coming soon
Check the Overview for an explanation of when and why to use serverless agents in “container-as-a-service” cloud environments.
The Sysdig serverless agent provides runtime detection through policy enforcement with Falco. At this time, the serverless agent is available for AWS Fargate on ECS. It is comprised of an orchestrator agent and (potentially multiple) workload agents.
The Sysdig serverless orchestrator agent is a collection point installed on each VPC to collect data from the serverless workload agent(s) and to forward them to the Sysdig backend. It also syncs the Falco runtime policies and rules to the workload agent(s) from the Sysdig backend.
The Sysdig serverless workload agent is installed in each task and requires network access to communicate with the orchestrator agent.
The prerequisites, upgrade instructions, and downloaded YAML files differ between the two versions.
This option presumes your services use an existing CFT and you will install the Workload Agent using an automated process which will instrument all the Fargate Task Definitions in your CFT.
Before starting, ensure you have the following.
Deploy the serverless-instrumentation.yaml
for each desired VPC using CloudFormation:
Log in to the AWS Console, select CloudFormation, Create Stack with new resources, and specify the serverless-instrumentation.yaml
as the Template source.
Specify the stack details to deploy the Orchestrator Agent on the same VPC where your service is running. For standard deployments, provide the parameters highlighted in the figure below:
Click Next, complete the stack creation, and wait for the deployment to complete (usually a few minutes).
Once the stack is deployed, the Output tab provides the Transformation Instruction as shown in the figure below. Copy and paste the Transformation Instruction at the root level of your CFT.
All new deployments of your CFT will be instrumented.
When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.
Up through version 2.3.0, the installation process deployed two stacks (using Installation Option 2):
From version 3.0.0, you will deploy the “Instrumentation & Orchestration” stack only, using option 1.
To upgrade to version 3.0.0:
Transform: MyV2SysdigMacro
).
You now have two versions of the serverless agent components. When you are ready to switch from the earlier version, proceed with the next step../installer-linux-amd64 cfn-macro delete MyEarlierSysdigMacro
Transform: MyV2SysdigMacro
).In this scenario, the two components of the serverless agent are installed separately.
For the orchestrator agent, Sysdig provides a yaml to use as a CloudFormation Template which you can deploy through the AWS Console. You need one orchestrator deployment per VPC in your environment which your organization wants to secure.
For the workload agents, you need one workload agent per Fargate task definition. (If you have ten services and ten task definitions, each needs to be instrumented.)
We assume your services use an existing CFT and you will install the workload agent using an automated process which will instrument all the task definitions in your CFT.
On the AWS side:
AWS CLI configured and permissions to create and use an S3 bucket
Permissions to upload images to repos, deploy CloudFormation Templates (CFTs), and create task definitions for Fargate
The Fargate tasks you want to instrument with the Sysdig serverless agent
Two subnets that can connect with the internet. (Your service on Fargate must reach the orchestrator agent, and the orchestrator agent must reach the internet to communicate with Sysdig’s back end.)
A NAT gateway or an AWS Internet Gateway so the orchestrator agent can reach to Sysdig.
On the Sysdig side:
Sysdig Secure
Sysdig agent key
Endpoint for the Sysdig collector for your region; check SaaS Regions and IP Ranges for the endpoint to use.
Obtain the Sysdig Orchestrator Agent yaml to be used as the CloudFormation Template source.
For more information on CloudFormation (CFN), see AWS documentation.
Deploy the orchestrator agent for each desired VPC, using CloudFormation.
The steps below are an outline of the important Sysdig-related parts.
Log in to the AWS Console. Select CloudFormation and
Create Stack with new resources and specify the
orchestrator-agent.yaml
as the Template source.
Specify the stack details to deploy the orchestrator agent on the same VPC where your service is running.
Stack name: self-defined
Sysdig Settings
Sysdig Access Key: Use the agent key for your Sysdig platform.
Sysdig Collector Host: collector.sysdigcloud.com
(default);
region-dependent in Sysdig SaaS; custom in Sysdig on-prem.
Sysdig Collector Port: 6443
(default), or could be
custom for on-prem installations.
Network Settings
VPC Id: Choose your VPC.
VPC Gateway: Choose the type of Gateway in your VPC to balance the load of the orchestrator service.
Subnet A & B: These depend on the VPC you choose; select from the drop-down menu.
Advanced Settings
Sysdig Agent Tags: Enter a comma-separated list of tags
(eg. role:webserver,location:europe
) Note: tags will also
be created automatically from your infrastructure’s
metadata, including AWS, Docker, etc.
Sysdig Orchestrator Agent Image:
quay.io/orchestrator-agent:latest
(default)
Check Collector SSL Certificate: Default: true
.
False
means no validation will be done on the SSL
certificate received from the collector, used for dev
purposes only.
Sysdig Orchestrator Agent Port: 6667
(default).
Port where the orchestrator service will be listening for instrumented task connections.
Click Next, complete the stack creation, and wait for the deployment to complete (usually less than 10 minutes.)
In Output, take note of the
OrchestratorHost
and OrchestratorPort
values.
As of Serverless Agent Version 2.2.0, the orchestrator port can be configured via either SYSDIG_ORCHESTRATOR_PORT
(default 6667, which is hardcoded into other parts of the CFT) or the SysdigOrchestratorAgentPort
(alternate) parameter in the CloudFormation template.
If AWS Internet Gateway is used (as opposed to a NAT Gateway), uncomment
the line AssignPublicIp: ENABLED
in the orchestrator.yaml.
Choose one of the four options below (A, B, C1, C2).
Prerequisite: Have the orchestrator agent deployed in the appropriate VPC and have the Orchestrator Host and Port information handy.
Download the appropriate installer for your OS.
These set up Kilt, an open-source library mechanism for injection into Fargate containers.
Create a macro for the serverless worker agents, using the installer. Any service tagged with this macro will have the serverless worker agent(s) added and Fargate data will be collected.
Log in to AWS CLI.
Create a CFN macro that applies instrumentation. You will need the outputs from previous task. Example:
./installer-linux-amd64 cfn-macro install -r us-east-1 MySysdigMacro $OrchestratorHost $OrchestratorPort
Add the macro you created to the CFT that you use for your own service at the root.
Use: Transform: MySysdigMacro
.
All new deployments of that template will be instrumented.
Defining Entrypoint and Command in CFT:
In this step, the macro will go over the original CloudFormation
Template looking for ContainerDefinitions
to instrument. This
includes replacing the original entry point for the Sysdig entry
point, so ideally the CFT should explicitly describe Entrypoint
and Command.
Otherwise, the macro will try to pull the image to
retrieve a default Entrypoint
and Command
, which only works if
the image is publicly available. Otherwise the pull will fail and
the instrumentation will not be completed.
Complete!
When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.
Alternatively, you can use the Sysdig Provider on the Terraform registry to deploy the workload agents, as described below.
Prerequisite: Have the orchestrator agent deployed in the appropriate VPC and have the Orchestrator Host and Port information handy.
Set up the Sysdig Terraform provider:
terraform {
required_providers {
sysdig = {
source = "sysdiglabs/sysdig"
version = ">= 0.4.0"
}
}
}
provider "sysdig" {
sysdig_secure_api_token = var.sysdig_access_key
}
Pass the container-definition, orchestrator host, and orchestrator port to the sysdig_fargate_workload_agent
data source.
Notes:
The input container definitions must be in JSON format, following CloudFormation naming conventions.
When deploying the orchestrator agent with CFT, the port is always 6440.
When deploying the orchestrator agent with CFT, the host is the collector endpoint for your Sysdig region; e.g.
"collector-static.sysdigcloud.com"
for US East. For other regions, see SaaS Regions and IP Ranges.
data "sysdig_fargate_workload_agent" "instrumented" {
container_definitions = file("container-definitions/hello.json")
sysdig_access_key = var.sysdig_access_key
workload_agent_image = "quay.io/sysdig/workload-agent:latest"
orchestrator_host = module.sysdig_orchestrator_agent.orchestrator_host
orchestrator_port = module.sysdig_orchestrator_agent.orchestrator_port
}
Include the instrumented JSON in your Fargate task definition:
resource "aws_ecs_task_definition" "fargate_task" {
...
network_mode = "awsvpc"
requires_compatibilities = ["FARGATE"]
container_definitions = "${data.sysdig_fargate_workload_agent.instrumented.output_container_definitions}"
}
Complete!
When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.
/en/docs/installation/serverless-agents/aws-fargate-serverless-agents/manage-serverless-agent-logs)
The most commonly used variables include:
SYSDIG_ORCHESTRATOR
: (required) use the output OrchestratorHost
valueSYSDIG_ORCHESTRATOR_PORT
: (required) use the output from OrchestratorPort
SYSDIG_EXTRA_ARGS
: use to pass any extra arguments needed to the instrumentationSYSDIG_EXTRA_CONF
: use to add any extra configuration needed to the instrumentationSYSDIG_LOGGING
: add this variable and set to debug
if needed. Any other value will print minimal logs at startup. See also: Manage Serverless Agent LogsIn some cases, you may prefer not to use the Kilt or Terraform installers and instead to either:
Review the installation overview and prerequisites.
Install the Orchestrator Agent, as described above. Take note of the
OrchestratorHost
and OrchestratorPort
values.
Instrument the CloudFormation task definition to deploy the workload agent manually.
Add a new container to your existing CloudFormation task definition.
SysdigInstrumentation
for the image name.quay.io/sysdig/workload-agent:latest
Edit the containers you want to instrument.
Mount volumes from SysdigInstrumentation
Add SYS_PTRACE
capability to your container. See AWS Documentation for detail if needed.
Set /opt/draios/bin/instrument
as the entry point of your container.
Note: if you are using the entry point in your image, then prepend it: /opt/draios/bin/instrument,your,original,entry,point
Configure the Sysdig instrumentation through environment variables.
TestApp:
Type: AWS::ECS::TaskDefinition
Properties:
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 2048
Memory: 8GB
ExecutionRoleArn: !Ref TestAppExecutionRole
TaskRoleArn: !Ref TestAppTaskRole
ContainerDefinitions:
- Name: App
Image: !Ref TestAppImage
EntryPoint:
- /bin/ping
- google.com
Tags:
- Key: application
Value: TestApp
TestApp:
Type: AWS::ECS::TaskDefinition
Properties:
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
Cpu: 2048
Memory: 8GB
ExecutionRoleArn: !Ref TestAppExecutionRole
TaskRoleArn: !Ref TestAppTaskRole
ContainerDefinitions:
- Name: App
Image: !Ref TestAppImage
EntryPoint:
### Added for manual instrumentation
- /opt/draios/bin/instrument
### End added propertis
- /bin/ping
- google.com
### Added for manual instrumentation
LinuxParameters:
Capabilities:
Add:
- SYS_PTRACE
VolumesFrom:
- SourceContainer: SysdigInstrumentation
ReadOnly: true
Environment:
- Name: SYSDIG_ORCHESTRATOR # Required
Value: <host orchestrator output, region dependent>
- Name: SYSDIG_ORCHESTRATOR_PORT # Required
Value: "6667"
- Name: SYSDIG_ACCESS_KEY
Value: ""
- Name: SYSDIG_COLLECTOR
Value: ""
- Name: SYSDIG_COLLECTOR_PORT
Value: ""
- Name: SYSDIG_LOGGING
Value: ""
- Name: SysdigInstrumentation
Image: !Ref WorkloadAgentImage
### End of added properties
Tags:
- Key: application
Value: TestApp
Alternatively, you can include the workload agent in your container at build time. To do so, update your Dockerfile to copy the required files:
ARG sysdig_agent_version=latest
FROM quay.io/sysdig/workload-agent:$sysdig_agent_version AS workload-agent
FROM my-original-base
COPY --from=workload-agent /opt/draios /opt/draios
Prepend the /opt/draios/bin/instrument
command to the entrypoint of your container:
ENTRYPOINT ["/opt/draios/bin/instrument", "my", "original", "entry", "point"]
To separate instrumentation logs from your workload logs, the instrument
entrypoint will attempt to send logs to localhost:32000
, where they will be mixed with the workload logs.
SYSDIG_LOGGING
to silent
.See also: Manage Serverless Agent Logs
Note: In most cases, it is advised to upgrade directly to v3.0.0+, using the Option 1 upgrade path. These instructions are kept for special cases.
Obtain the Sysdig Orchestrator Agent yaml to be used as the CloudFormation Template source.
At this time, the yaml metadata does not specify the agent version, but this link always downloads the latest file.
Perform the steps to Install the Orchestrator Agent.
Note that in step 2.4, the OrchestratorHost
and OrchestratorPort
values will be unique.
Perform the steps to Install the Workload
Agents (if using Kilt. Workload agents installed with Terraform are automatically updated to _latest
.)Manual instrumentation requires redoing the manual process.)
In step 4, assign a unique name to your macro
(Transform: MyV2SysdigMacro
).
You now have two versions of the serverless agent components. When you are ready to switch from the earlier version, proceed with the next step.
Stop all running tasks and use CloudFormataion to delete the
earlier stack. Redeploy the new stack with the updated CFT.
(Transform: MyV2SysdigMacro
)
Clean up macros: Delete the previous kilt macro using the installer:
./installer-linux-amd64 cfn-macro delete MySysdigMacro
As of serverless agent version 2.2.0, task logs and instrumentation logs are handled separately.
Task logs continue to be with whatever log setup you have on your task container.
Instrumentation logs go to a separate log group created by the macro installer:
<macro_name>-SysdigServerlessLogGroup-<uid>
.
At this time, the log group name cannot be edited. By default, the logging level is set to info
.
SYSDIG_LOGGING
is used for instrumentation log level. Default = info
.
The full list of options are: none | critical| error | warning |notice | info | debug | trace
SYSDIG_ENABLE_LOG_FORWARD
Default = true
SYSDIG_LOG_LISTEN_PORT
Default = 32000
SYSDIG_LOG_FORWARD_ADDR
Default = localhost:32000
. Where the logwriter of the workload agent listens. Can be set to any other TCP endpoint.Advanced Configurations
SYSDIG_BUFFER_SIZE
: Receiving buffer size in bytes for the TCP listener, which can be increased. Default 1024
.SYSDIG_DEADLINE_SECONDS
: Connection deadline, which can be increased if clients are taking longer to connect. Default 3
.See other serverless agent environment variables here.
You can collect Prometheus metrics from environments where the Sysdig
agent is not available. Sysdig uses the remote_write
capabilities to
help you do so.
In Sysdig terminology, the remote endpoints that can read Prometheus metrics are known as Prometheus Remote Write. Prometheus Remote Write does not require the Sysdig agent to be installed in the Prometheus environment. This facility expands your monitoring capabilities beyond Kubernetes and regular Linux kernels to environments where the Sysdig agent cannot be installed.
Prometheus Remote Write can collect metrics from:
An existing Prometheus server
Additional environments:
Windows
Managed Cloud Environments, such as AWS and IBM
Fargate
IoT
Use Sysdig agent in environments where an agent can be installed. However, use the Prometheus Remote Write to collect metrics from ephemeral or batch jobs that may not exist long enough to be scraped by the agent.
With the Prometheus Remote Write, you can either monitor metrics through the Monitor UI or you can use PromQL to query the data by using the standard Prometheus query language.
Prometheus Remote Write resides in the ingest endpoints for each region
under /prometheus/remote/write
. The public Prometheus Remote Write
endpoints for each region are listed below:
Region | Endpoints |
---|---|
US East | https://api.sysdigcloud.com/prometheus/remote/write |
US West | https://us2.app.sysdig.com/prometheus/remote/write |
European Union | https://eu1.app.sysdig.com/prometheus/remote/write |
Asia Pacific (Sydney) | https://app.au1.sysdig.com/prometheus/remote/write |
You need to configure remote_write
in your Prometheus server in order
to send metrics to Sysdig Prometheus Remote Write.
The configuration of your Prometheus server depends on your
installation. In general, you configure the remote_write
section in
the prometheus.yml
configuration file:
global:
external_labels:
[ <labelname>: <labelvalue> ... ]
remote_write:
- url: "https://<region-url>/prometheus/remote/write"
bearer_token: "<your API Token>"
The communication between your Prometheus server and Prometheus Remote Write should use the authorization header with the Sysdig API key (not the agent access key) as the bearer token.
Alternatively, you can also use the bearer_token_file
entry to refer
to a file instead of directly including the API token.
Prometheus does not reveal the bearer_token
value on the UI.
To enable customization, Sysdig provides additional options to control which metrics you want to send to Prometheus Remote Write.
Prometheus Remote Write by default sends all the metrics to Sysdig
Prometheus Remote Write. These metrics are sent with a
remote_write: true
label appended to it so you can easily identify
them.
You can specify custom label-value pairs and send them with each time
series to the Prometheus Remote Write. Use the external_labels
block
in the global
section in the Prometheus configuration file. This is
similar to setting an agent tag and allowing you to filter or scope the
metrics in Sysdig Monitor.
For example, if you have two Prometheus servers configured to remote write to Prometheus Remote Write, you can include an external label to identify them easily:
Prometheus 1
global:
external_labels:
provider: prometheus1
remote_write:
- url: ...
Prometheus 2
global:
external_labels:
provider: prometheus2
remote_write:
- url: ...
With the general configuration, all the metrics are by default remotely
written to Prometheus Remote Write. You can control the metrics that you
collect and send to Sysdig. To select which series and labels to
collect, drop, or replace, and reduce the number of active series that
are sent to Sysdig, you can set up relabel configurations by using
the write_relabel_configs
block within your remote_write
section.
For example, you can send metrics from one specific namespace called
myapp-ns
as give below:
remote_write:
- url: https://<region-url>/prometheus/remote/write
bearer_token_file: /etc/secrets/sysdig-api-token
write_relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
regex: 'myapp-ns'
action: keep
The default limits are configured set for each user and can be raised as required. The defaults are good for most users, and the limits help protect against any misconfigurations.
Feature | Limit |
---|---|
Parallel writes | 100 concurrent requests. This doesn't necessarily mean 100 Prometheus servers because the time at which the data is written is distributed. |
Data points per minute | One million. The number of data points sent depends on how often metrics are submitted to Sysdig. A scrape interval of 10s will submit more DPM than an interval of 60s. |
Number of writes per minute | 10,000 |
It is possible to scope a Sysdig Team to only access metrics matching certain labels sent via Prometheus remote write. See Manage Teams and Roles
Metrics sent to Prometheus Remote Write can be accessed in Explore, but they are not compatible with the scope tree.
Label enrichment is unavailable at this point. Only labels collected at the source can be used. You should add additional labels to perform further scoping or pivoting in Sysdig.
Currently, Sysdig Dashboards do not support mixing metrics with different sampling. For example, 10 seconds and 1-minute samples. For optimal experience, configure the scrape interval to be 10s to combine remote write metrics with agent metrics.
Remote write functionality does not support sending metric metadata. Upstream Prometheus recently added support for propagation of metadata (metric type, unit, description, info) and this functionality will be supported in a future update to Prometheus Remote Write.
Suffix the metric name with _total
, _sum
, or _count
to
store them as a counter. Otherwise, the metrics will be handled
as a gauge.
Units can be set in Dashboards manually.
Sysdig Secure for cloud is the software that connects Sysdig Secure features to your cloud environments to provide unified threat detection, compliance, forensics, and analysis.
Because modern cloud applications are no longer just virtualized compute resources, but a superset of cloud services on which businesses depend, controlling the security of your cloud accounts is essential. Errors can expose an organization to risks that could bring resources down, infiltrate workloads, exfiltrate secrets, create unseen assets, or otherwise compromise the business or reputation. As the number of cloud services and configurations available grows exponentially, using a cloud security platform protects against having an unseen misconfiguration turn into a serious security issue.
Check Sysdig Secure - Secure for cloud to review the features provided per cloud.
Sysdig Secure for cloud is available on a variety of cloud providers, with simplified installation instructions available from the Get Started screen.
Supported cloud providers at this time are:
Native template deployment for
Core component for Sysdig Secure for Cloud, is called cloud-connector, which is also available through following methods:
Note: Installing this component will only allow you threat detection and image scaning features, although together with SysdigCompliance IAM role, it can handle Compliance and Identity and Access Posture too.
If needed, review the offering description on Sysdig Secure for cloud - AWS
All following options provide all four cloud features: threat detection, CSPM benchmarks, and image and container registry scanning
Terraform-based install instructions differ depending on what type of AWS account you are using.
At this time, the options include:
The default code provided in the Get Started page of Sysdig Secure is pre-populated with your Secure API token and will automatically install threat detection with CloudTrail, AWS benchmarks, and container registry and image scanning.
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
AWS
tab.
Copy the code snippet under Single Account and paste it in the terminal of your local machine. It should be pre-configured with your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them, then make the changes.
Confirm the Services are Working
Check Troubleshooting in case of permissions or account conflict errors.
For organizational accounts, the default code provided in the Get Started page of Sysdig Secure is pre-populated with your Secure API token and will automatically install threat detection with CloudTrail (only).
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
AWS
tab.
Copy the code snippet under Organizational Account and paste it in the terminal of your local machine. It should be pre-configured with your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them, then make the changes.
Confirm the Services are Working
Check Troubleshooting in case of permissions or account conflict errors.
Soon, this option will be expanded to include all the features currently in the single account option, as well as the ability to easily add multiple member accounts.
Both the Single Account and Organizational Account code examples are configured with sensible defaults for the underlying inputs. But if desired, you can edit the region, module enablement, and other Inputs. See details for:
Image Scanner feature is disabled by default. If you want to enable it, just use the deploy_scanning
input variable on your snippet such as:
module "secure-for-cloud_example"{
...
deploy_image_scanning_ecs = true
deploy_image_scanning_ecr = true
}
Check full list of created resources
Benchmark
aws_iam_role
aws_iam_role_policy_attachment
sysdig_secure_benchmark_task
sysdig_secure_cloud_account
General; Threat detection / CSPM / CIEM
aws_cloudwatch_log_stream
aws_ecs_service
aws_ecs_task_definition
aws_iam_role
aws_iam_role_policy
aws_s3_bucket
aws_s3_bucket_object
aws_s3_bucket_public_access_block
aws_security_group
aws_sns_topic_subscription
aws_sqs_queue
aws_sqs_queue_policy
Image Scanning
aws_cloudwatch_log_group
aws_cloudwatch_log_stream
aws_ecs_service
aws_ecs_task_definition
aws_iam_role
aws_iam_role_policy
aws_security_group
aws_sns_topic_subscription
aws_sqs_queue
aws_sqs_queue_policy
If cloud-connector or cloud-scanning is installed, these additional modules will be installed:
resource-group
cloudtrail
aws_cloudtrail
aws_kms_alias
aws_kms_key
aws_s3_bucket
aws_s3_bucket_policy
aws_s3_bucket_public_access_block
aws_sns_topic
aws_sns_topic_policy
ssm
ecs-fargate-cluster
If cloud-scanning is installed, these additional modules will be installed:
codebuild
aws_cloudwatch_log_group
aws_codebuild_project
aws_iam_role
aws_iam_role_policy
Find more troubleshooting options on the module source repository
This error may occur if the specified cloud account has already been onboarded to Sysdig.
Solution:
The cloud account can be imported into Terraform by running:
terraform import module.cloud_bench.sysdig_secure_cloud_account.cloud_account CLOUD_ACCOUNT_ID
This error may occur if your current AWS authentication session does not have the required permissions to create certain resources.
Solution:
Ensure you are authenticated to AWS using a user or role with the required permissions.Onboarding a Single Account using a CFT
Each of the features can be enabled from a single CloudFormation Template (CFT) from the AWS Console. Two options are available:
Secure For Cloud stack, deployed on ECS compute workload. Available in all regions
Secure For Cloud stack, deployed on AppRunner compute workload. A less resource-demanding deployment but not available in all regions; accepting ‘us-east-1’, ‘us-east-2’, ‘us-west-2’, ‘ap-northeast-1’ and ’eu-west-1’
A Sysdig Secure SaaS account
An AWS account and AWS services you would like to connect to Sysdig, with appropriate permissions to deploy a CFT.
Log in to your AWS Console and confirm that you are in the account and AWS region that you want to secure using Sysdig Secure for cloud.
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
AWS
tab.
Select between Install Secure For Cloud stack, deployed on ECS compute workload
or
Install Secure For Cloud stack, deployed on AppRunner compute workload
link.
The Connect Account dialog is displayed.
Enter:
The AWS account number with which you want to connect
An IAM Role name to be created for Sysdig Secure for cloud in AWS. This role name must not yet exist in your account.
The role provides read-only access to your resources to allow Sysdig to monitor and secure your cloud account. Access is scoped to the managed SecurityAudit policy.
Click Launch Stack
.
The AWS Console opens, at the
CloudFormation > Stacks > Quick Create
page. The Sysdig
CloudFormation template is pre-loaded.
Confirm that you are logged in the AWS account and region where you want to deploy the Sysdig Template.
Provide a Stack name
or accept the default.
Fill in the Parameters:
Sysdig Settings
Sysdig Secure Endpoint
: Default (US-East): https://secure.sysdig.com
.
If your Sysdig Secure platform is installed in another region, use that endpoint.
https://us2.app.sysdig.com
https://eu1.app.sysdig.com
Sysdig Secure API Token
: See Retrieve the Sysdig API Token to find yours.
Sysdig Role Name
: As specified in Step 3; IAM role name to be created for Sysdig to access your AWS account
Sysdig External ID
: Not to be modified. It’s the ExternalID to identify Sysdig on AWS, for the Trusted Identity
Sysdig Trusted Identity
: Not to be modified. It’s the ARN of the Trusted Identity of Sysdig on AWS, to be able to run CSPM benchmarks.
Modules to Deploy: Choose any or all.CSPM/Compliance
and Threat detection using CloudTrail
capabilities will be always deployed.
ECR Image Registry Scanning:
Integrates container registry
scanning for AWS ECR.
Fargate Image Scanning:
Integrates image scanning on any
container image deployed on a serverless Fargate task (in ECS).
Existing Infrastructure: Leave all this fields blank for resources to be created.
If you want to use existing components of your infrastructure, you can provide:
Network: Only available if stack is deployed on ECS. If provided, MUST specify ALL field values:
Cloudstrail SNS Topic: Specify the URL of the SNS Topic
Confirm the Capabilities required to deploy:
Check “I acknowledge that AWS CloudFormation might create IAM resources with custom names.”
Check “I acknowledge that AWS CloudFormation might require the following capability: CAPABILITY_AUTO_EXPAND”
Click Create Stack.
In the AWS Console, the main stack and associated substacks will show “CREATE_IN_PROGRESS”. Refresh the status to see “CREATE_COMPLETE” for all. There is a delay of 5-10 minutes for events to be sent from CloudTrail, but no event is lost.
A success message also appears in the Sysdig Secure Get Started page.
Log in to Sysdig Secure and check that each module you deployed is functioning. It may take 10 minutes or so for events to be collected and displayed.
Data Sources:
Select Select Integrations > Inbound | Cloud Accounts
to see all connected
cloud accounts.
Subscription: Select Settings > Subscription
to see an
overview of your account activity, including cloud accounts.
Insights: Check that Insights have been added to your navigation bar. View activity on the Cloud Account, Cloud User, or Composite insight views.
Policies and Rules: Check Policies > Runtime Policies
and confirm that
the AWS Best Practices
policy is enabled. This consists of the
most-frequently-recommended rules for AWS and CloudTrail. You can
customize it by creating a new policy of the AWS
CloudTrail
type.
Events: In the Events
feed, search ‘cloud’ to show events from
AWS CloudTrail.
Force an event: In case you want to manually create an event, choose one of the rules contained in the AWS Best Practices
policy and
execute it in your AWS account.
ex.: Create a S3 Bucket with Public Access Blocked. Make it public to prompt the event.
Remember that in case you add new rules to the policy you need to give it time to propagate the changes.
Compliance: Select Compliance
and see that
AWS Foundations Benchmark
is installed.
Review the benchmark results and confirm the account, region and date added.
Scan Results: CheckImage Scanning > Scan Results
and choose
the Origins
drop-down.
Confirm that AWS Registry
and/or AWS Fargate
are listed.
Filter by the desired origin and review scan results.
Force an event: Upload an image to a repository on your Elastic Container Registry (ECR). Follow View push commands
button guide provided by AWS.
Select Posture > Permissions and Entitlements
and check if the following are showing up in the Unused Permissions graphs:
Follow the instructions to remediate overly permissive entitlements and reduce security risks. ) to remediate overly permissive entitlements and reduce security risks.
If needed, review the offering description on Sysdig Secure for cloud - GCP.
Deployments on GCP use a Terraform file.
Terraform-based install instructions differ depending on what type of account you are using.
At this time, the options include:
The default code provided in the Get Started page of Sysdig Secure is pre-populated with your Secure API token and will automatically install threat detection, benchmarks, and container registry and image scanning.
Log in to Sysdig Secure as Admin and select
Get Started > Connect your Cloud account
. Choose the
GCP
tab.
Copy the code snippet under Single Account or Organizational Account and paste it in the terminal of your local machine. It should be pre-configured with your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them, then make the changes.
Confirm the Services are Working
Check Troubleshooting in case of permissions or account conflict errors.
Both the Single Account and Organizational Account code examples are configured with sensible defaults for the underlying inputs. But if desired, you can edit the region, module enablement, and other Inputs. See details for:
Image Scanner feature is disabled by default. If you want to enable it, just use the deploy_scanning
input variable on your snippet such as:
module "secure-for-cloud_example"{
...
deploy_scanning = true
}
Check full list of created resources
google_iam_workload_identity_pool
google_iam_workload_identity_pool_provider
google_project_iam_custom_role
google_project_iam_member
google_service_account
google_service_account_iam_binding
sysdig_secure_benchmark_task
sysdig_secure_cloud_account
google_cloud_run_service
google_eventarc_trigger
google_project_iam_member
google_storage_bucket
google_storage_bucket_iam_member
google_storage_bucket_object
google_cloud_run_service
google_cloud_run_service_iam_member
google_eventarc_trigger
google_project_iam_member
google_pubsub_topic
If Cloud-connector is installed in organizational mode, this additional module will be installed:
google_logging_organization_sink
google_pubsub_topic
google_pubsub_topic_iam_member
If Cloud-connector is installed in single-project mode, this additional module will be installed:
google_logging_project_sink
google_pubsub_topic
google_pubsub_topic_iam_member
If Cloud-scanning is installed, this additional module will be installed:
google_secret_manager_secret
google_secret_manager_secret_iam_member
Find more troubleshooting options on the module source repository
This error may occur if your current GCP authentication session does not have the required permissions to access the specified project.
Solution: Ensure you are authenticated to GCP using a user or service account with the required permissions.
This error may occur if your current GCP authentication session does not have the required permissions to create certain resources.
Solution: Ensure you are authenticated to GCP using a user or service account with the required permissions.
If you have sufficient permissions but still get this kind of error, try to authenticate gcloud
using:
$ gcloud auth application-default login
$ gcloud auth application-default login set-quota-project your_project_id
This error may occur if the specified GCP project has already been onboarded to Sysdig.
Solution: The cloud account can be imported into terraform by running
terraform import module.single account.module.cloud_bench.sysdig_secure_cloud_account.cloud_account PROJECT_ID
, where PROJECT_ID
is the numerical ID of the project (not the project name).
This error may occur if a Workload Identity Federation Pool or Pool Provider has previously been created, and then deleted, either via the GCP console or with terraform destroy
. When a delete request for these resources is sent to GCP, they are not completely deleted, but marked as “deleted”, and remain for 30 days. These “deleted” resources will block creation of a new resource with the same name.
Solution:
The “deleted” pools must be restored using the GCP console, and then imported into terraform, using terraform import module.single-account.module.cloud_bench.google_iam_workload_identity_pool.pool POOL_ID
and module.single-account.module.cloud_bench.google_iam_workload_identity_pool_provider.pool_provider POOL_ID/PROVIDER_ID
Email contains error codes such as:
Error Code: topic_not_found
or
Error Code: topic_permission_denied
Cause: The resources Sysdig deployed with Terraform will eventually be consistent, but it could happen that some pre-required resources are created but not ready yet.
Solution: This is a known issue that will only take place within first minutes of the deployment. Eventually, resource health checks will pass and modules will work as expected.
Log in to Sysdig Secure and check that each module you deployed is functioning. It may take 10 minutes or so for events to be collected and displayed.
Data Sources:
Select Integrations > Inbound | Cloud Accounts
to see all connected
cloud accounts.
Subscription: Select Settings > Subscription
to see an
overview of your account activity, including cloud accounts.
Insights: Check that Insights have been added to your navigation bar. View activity on the Cloud Account, Cloud User, or Composite insight views.
Policies and Rules: Check Policies > Runtime Policies
and confirm that
the GCP Best Practices
policy is enabled. This consists of the
most-frequently-recommended rules for GCP.
Events: In the Events
feed, search ‘cloud’ to show events from
GCP.
Compliance > Benchmarks > Tasks
and confirm a task with the name Sysdig Secure for Cloud (GCP)
exists.Sysdig Secure for Cloud (GCP)
task. Note that results may take up to 15 minutes to appear.Scan Results: CheckImage Scanning > Scan Results
and choose
the Origins
drop-down.
Confirm that GCP
is listed.
Filter by the desired origin and review scan results.
Force an event: Upload an image to a new Repository in a Container Registry. Follow repository Setup Instructions
provided by GCP.
If needed, review the offering description on Sysdig Secure for cloud - Azure.
Deployments on Azure use a Terraform file.
Terraform-based install instructions differ depending on what type of account you are using.
At this time, the options include:
The default code provided in the Get Started page of Sysdig Secure is pre-populated with your Secure API token and will automatically install threat detection, benchmarks, and container registry and image scanning.
Log in to Sysdig Secure as Admin
and select
Get Started > Connect your Cloud account
. Choose the
Azure
tab.
Copy the code snippet under Single Subscription or Tenant Subscriptions and paste it in the terminal of your local machine. It should be pre-configured with your Sysdig API token.
Then run:
$ terraform init
When complete, run:
$ terraform apply
which will present the changes to be made, ask you to confirm them, then make the changes.
Confirm the Services are Working
Check Troubleshooting in case of permissions or account conflict errors.
Both the Single Account and Organizational Account code examples are configured with sensible defaults for the underlying inputs. But if desired, you can edit the region, module enablement, and other Inputs. See details for:
Image Scanner feature is disabled by default. If you want to enable it, just use the deploy_scanning
input variable on your snippet such as:
module "secure-for-cloud_example"{
...
deploy_scanning = true
}
Check full list of created resources
azurerm_lighthouse_assignment
azurerm_lighthouse_definition
azurerm_role_definition
azurerm_subscription
sysdig_secure_cloud_account
sysdig_secure_benchmark_task
azurerm_container_group
azurerm_network_profile
azurerm_storage_account
azurerm_storage_blob
azurerm_storage_container
azurerm_subnet
azurerm_virtual_network
If Cloud-connector is installed, these additional modules will also be installed:
azurerm_container_registry
azurerm_eventgrid_event_subscription
azuread_application
azuread_application_password
azuread_service_principal
azuread_service_principal_password
azurerm_role_assignment
azurerm_role_definition
azurerm_eventhub
azurerm_eventhub_authorization_rule
azurerm_eventhub_namespace
azurerm_eventhub_namespace_authorization_rule
azurerm_monitor_diagnostic_setting
azurerm_resource_group
Find more troubleshooting options on the module source repository
This error may occur if your current Azure authentication session does not have the required permissions to create resources in the specified subscription.
Solution: Ensure you are authenticated to Azure using a Non-Guest user with the Contributor or Owner role on the target subscription.
Error: Error Creating/Updating Lighthouse Definition "dd9be15b-0ee9-7daf-b942-5e173dae13fb" (Scope "/subscriptions/***"): managedservices.RegistrationDefinitionsClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="InsufficientPrivilegesForManagedServiceResource" Message="The requested user doesn't have sufficient privileges to perform the operation."
with module.cloudvision_example_existing_resource_group.module.cloud_bench.module.trust_relationship["***"].azurerm_lighthouse_definition.lighthouse_definition,
on ../../../modules/services/cloud-bench/trust_relationship/main.tf line 28, in resource "azurerm_lighthouse_definition" "lighthouse_definition":
28: resource "azurerm_lighthouse_definition" "lighthouse_definition" {
This error may occur if the specified Azure Subscription has already been onboarded to Sysdig
Solution:
The resource can be imported into Terraform by using the terraform import
command. This will bring the resource under management in the current Terraform workspace.
Error: A resource with the ID "/subscriptions/***/resourceGroups/sfc-resourcegroup" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_resource_group" for more information.
with module.cloudvision_example_existing_resource_group.module.infrastructure_eventhub.azurerm_resource_group.rg[0],
on ../../../modules/infrastructure/eventhub/main.tf line 6, in resource "azurerm_resource_group" "rg":
6: resource "azurerm_resource_group" "rg" {
Log in to Sysdig Secure and check that each module you deployed is functioning. It may take 10 minutes or so for events to be collected and displayed.
Data Sources:
Select Integrations > Inbound | Cloud Accounts
to see all connected
cloud accounts.
Subscription: Select Settings > Subscription
to see an
overview of your account activity, including cloud accounts.
Insights: Check that Insights have been added to your navigation bar. View activity on the Cloud Account, Cloud User, or Composite insight views.
Policies: Check Policies > Runtime Policies
and confirm that
the Azure Best Practices
policy is enabled. This consists of the
most-frequently-recommended rules for Azure DevOps.
Events: In the Events
feed, search ‘cloud’ to show events from
Azure.
Compliance > Benchmarks > Tasks
and confirm a task with the name Sysdig Secure for Cloud (Azure)
exists.Sysdig Secure for Cloud (Azure)
task. Note that results may take up to 15 minutes to appear.Scan Results: CheckImage Scanning > Scan Results
and choose
the Origins
drop-down.
Confirm that Azure
is listed.
Filter by the desired origin and review scan results.
With Rapid Response, Sysdig has introduced a way to grant designated Advanced Users in Sysdig Secure the ability to remote connect into a host directly from the Event stream and execute desired commands there.
See also: Rapid Response.
Sysdig Secure On-Premises 4.0+ or Sysdig Secure SaaS
Have on hand:
Your Sysdig agent access key
Your Sysdig API endpoint
On-prem: custom, depending on your on-prem installation
SaaS: Region-dependent (use the “endpoint” entries, e.g. https://us2.app.sysdig.com
for US West)
A passphrase used to encrypt all traffic between the user and host.
NOTE: Sysdig cannot recover this passphrase. If lost, a user will not be able to start a session, nor will any session logs be recoverable.
Optionally, these can be added to the environment variables:
export API_ENDPOINT=https://secure-staging.mycompany.com
export ACCESS_KEY=$YOUR_SYSDIG_AGENT_INSTALLATION_KEY
export PASSPHRASE=$ENCRYPTION_PASSPHRASE
export API_TLS_SKIP_CHECK=false
The Rapid Response agent can be installed as a Docker container or as a Kubernetes DaemonSet.
Mount the host directories and binaries to gain access to the host.
docker run --hostname $HOST_NAME -d quay.io/sysdig/rapid-response-host-component:latest --endpoint $API_ENDPOINT --access-key $ACCESS_KEY --password $PASSPHRASE
Customize the Docker image.
The container is simply bash shell. To add custom scripts without needing to mount the underlying host filesystem, you can bake this into the Docker container, e.g. by installing kubectl, gcloud, netstat, or another command-line utility.
FROM quay.io/sysdig/rapid-response-host-component:latest AS base-image
FROM alpine:3.13
COPY --from=base-image /usr/bin/host /usr/bin/host
# add custom scripts and other directives
ENTRYPOINT ["host"]
Create a namespace and secrets for the Rapid Response agent:
kubectl create ns rapid-response
kubectl create secret generic sysdig-rapid-response-host-component-access-key --from-literal=access-key=$ACCESS_KEY -n rapid-response
kubectl create secret generic sysdig-rapid-response-host-component-passphrase --from-literal=passphrase=$PASSPHRASE -n rapid-response
Create the configmap and change the API-ENDPOINT parameter:
echo "apiVersion: v1
kind: ConfigMap
metadata:
name: sysdig-rapid-response-host-component
data:
api-endpoint: ${API_ENDPOINT}
api-tls-skip-check: 'false'" | kubectl apply -n rapid-response -f -
Deploy the DaemonSet.
Note: The agent does not automatically have access to the host filesystem; there are several mounts commented-out in the manifest that must be uncommented to investigate the host.
echo "# apiVersion: extensions/v1beta1 # If you are in Kubernetes version 1.8 or less please use this line instead of the following one
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: sysdig-rapid-response-host-component
labels:
app: sysdig-rapid-response-host-component
spec:
selector:
matchLabels:
app: sysdig-rapid-response-host-component
updateStrategy:
type: RollingUpdate
template:
metadata:
labels:
app: sysdig-rapid-response-host-component
spec:
hostNetwork: true
volumes:
# Add custom volume here
# Uncomment these lines if you'd like to map /root/ from the
# host into the container.
#- hostPath:
# path: /
# name: host-root-vol
- name: sysdig-rapid-response-host-component-config
configMap:
name: sysdig-rapid-response-host-component
optional: true
tolerations:
- effect: NoSchedule
key: node-role.kubernetes.io/master
containers:
- name: sysdig-rapid-response-host-component
image: quay.io/sysdig/rapid-response-host-component
#securityContext:
# The privileged flag is necessary for OCP 4.x and other Kubernetes setups that deny host filesystem access to
# running containers by default regardless of volume mounts. In those cases, access to the CRI socket would fail.
# privileged: true
imagePullPolicy: Always
resources:
limits:
cpu: 500m
memory: 500Mi
requests:
cpu: 250m
memory: 250Mi
# Add custom volume mount here
# Uncomment these lines if you'd like to map /root/ from the
# host into the container.
#volumeMounts:
#- mountPath: /host
# name: host-root-vol
env:
- name: API_ENDPOINT
valueFrom:
configMapKeyRef:
name: sysdig-rapid-response-host-component
key: api-endpoint
- name: API_TLS_SKIP_CHECK
valueFrom:
configMapKeyRef:
name: sysdig-rapid-response-host-component
key: api-tls-skip-check
- name: ACCESS_KEY
valueFrom:
secretKeyRef:
name: sysdig-rapid-response-host-component-access-key
key: access-key
- name: PASSWORD
valueFrom:
secretKeyRef:
name: sysdig-rapid-response-host-component-passphrase
key: passphrase" | kubectl apply -n rapid-response -f -
After installation/upgrade, complete the following steps:
Request enablement of the feature from Sysdig Support.
Configure an S3 bucket for Rapid Response logs: If you are using the default Cassandra storage for Capture files, you will need to configure an AWS or custom S3 bucket to store Rapid Response log files after a session. If you have already configured an S3 bucket for Captures, then Rapid Response logs will be routed there automatically, into their own folder.
Manage the following port/firewall considerations:
Ensure the host component is able to reach the endpoint defined
in API_ENDPOINT
Ensure there are no intermediate proxies that could enforce maximum time to live (since sessions could potentially have long durations)
Ensure that the host component can reach the object storage (S3 bucket) when configured.
Configure and use Rapid Response in the Sysdig Secure UI: See Rapid Response.
The Node Analyzer (NA) provides a method for deploying the components for three different Sysdig Secure features:
(Node) Image Analyzer: an existing tool that can now be installed and/or upgraded in a new way, alongside the other two components.
Benchmarks: Installs a new component (called a benchmark runner) which is required to use Benchmarks, including an updated interface and new improved features. The legacy Benchmark tool can still be accessed.
Host Scanning: a new tool for scanning not just the images/containers on a host, but the host itself.
All the Node Analyzer components, along with the Sysdig agent, are deployed per node or host. You can deploy them using various methods:
If you are installing Sysdig Secure for the first time and have not yet deployed any agents, you can use a single-line install to deploy both the Sysdig agent and the Node Analyzer (NA) tools. The script will make changes to each node or host within a cluster.
curl -s
https://download.sysdig.com/stable/install-agent-kubernetes | sudo bash -s
-- --access_key ACCESS_KEY --collector COLLECTOR_URL --collector_port 6443 --nodeanalyzer --api_endpoint API_ENDPOINT
For SaaS, see also the Get Started page in Sysdig Secure. Under “Connect Your Data Sources,” the script is generated with your endpoints automatically inserted.
On-Premises with Self-Signed Cert:
If you want the Node Analyzer to report to an On-Prem Sysdig backend
that uses a self-signed certificate, then: Add -cc false
to the
command line so the node analyzer will accept it.
To find the values yourself:
access_key:
This is the agent access key. You can
retrieve
this from Settings > Agent Installation
in the Sysdig Secure UI.
collector_url:
This value is
region-dependent in
SaaS and is auto-completed on the Get Started page in the UI. (It is
a custom value in on-prem installations.)
api_endpoint:
This is the base URL (
region-dependent) for
Sysdig Secure and is auto-completed on the Get Started page. E.g.
secure.sysdig.com
, us2.app.sysdig.com
, eu1.app.sysdig.com
.
When finished, you can Access the Node Analyzer Features.
Use this script in the following conditions:
Agent is already installed, you just want the NA tools
Node Image Analyzer already installed; you want to upgrade it to v2
You want to add Benchmarks v2 and Host Scanning features to your existing Sysdig Secure environment, as well as upgrade or install the Image Analyzer.
Note that if you already have the Node Image Analyzer (v1) installed, this script will upgrade that component automatically. An agent MUST already be installed. The script will make changes to every node in the cluster.
curl -s https://download.sysdig.com/stable/install-node-analyzer | sudo bash -s -- --api_endpoint API_ENDPOINT
When finished, you can Access the Node Analyzer Features.
To deploy the Node Analyzer using Kubernetes daemonsets, download the following configuration files, edit them as annotated within the files, and deploy them.
To deploy the Node Analyzer concurrently with the Sysdig agent, you
would also download the sysdig-agent-clusterrole.yaml
,
sysdig-agent-daemonset-v2.yaml
, and sysdig-agent-configmap.yaml
and
deploy them as described in Agent Install:
Kubernetes.
You need to deploy these YAMLs after installing the Sysdig agent in the
same nodes, and also in the same namespace (sysdig-agent
by default).
When finished, you can Access the Node Analyzer Features.
Use the “Sysdig” Helm chart, which installs the Sysdig agent and the Node Analyzer, with the following commands:
helm repo add sysdig https://charts.sysdig.com
helm repo update
helm install sysdig-agent --set sysdig.accessKey=ACCESS_KEY --set sysdig.settings.collector=COLLECTOR_URL --set sysdig.settings.collector_port=6443 sysdig/sysdig --set nodeAnalyzer.apiEndpoint=API_ENDPOINT
To find the values:
access_key:
This is the agent access key. You can
retrieve
this from Settings > Agent Installation
in the Sysdig Secure UI.
collector_url:
This value is
region-dependent in
SaaS and is auto-completed on the Get Started page in the UI. (It is
a custom value in on-prem installations.)
api_endpoint:
This is the base URL (
region-dependent) for
Sysdig Secure and is auto-completed on the Get Started page. E.g.
secure.sysdig.com
, us2.app.sysdig.com
, eu1.app.sysdig.com
.
Log in to Sysdig Secure and check that the features are working as expected.
Select Scanning > Image Results
.
Check for scanned container image results that originate with the
Sysdig Node Image Analyzer.
Check vulnerabilities in hosts or nodes, both for operation system
packages (e.g. rpm
, dpkg
) and non-operating system packages (e.g.
Java packages, Ruby gems).
Select Scanning > Hosts
.
Review the Host vulnerabilities listed.
Your active team scope is applied when loading host scanning results. Log in with the broadest team and user credentials to see the full report.
Select Benchmarks |Tasks
.
Either configure a new task or review your upgraded tasks. Click a line item to see the associated benchmark report.
Your active team scope is applied when loading benchmarks results. Log in with the broadest team and user credentials to see the full report.
The installation options above should be sufficient for the majority of users; the options below allow for customizations and special cases.
Depending on your organization’s network design, you may require the HTTP requests from Node Analyzer features to pass through a proxy in order to reach the Sysdig Secure backend. To do so, you must edit all three configmaps:
These are in the sysdig-agent
namespace by default.
Configure the following variables:
http_proxy/https_proxy
Use with the relevant proxy URL, e.g.
http://my_proxy_address:8080
.
In most cases, it is enough to specify http_proxy
. as it applies
to HTTPS connections as well.
no_proxy
Use this parameter to exclude certain subnets from using
the proxy, adding a comma-separated exclusion list, e.g.
127.0.0.1,localhost,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8
If the proxy server requires authentication it is possible to specify
credentials in the URL, e.g. http://username:password@my_proxy:8080
.
This is handled per-component.
It is possible to deploy the benchmark runner as a single Docker container:
docker run -d \
-v /:/host:ro \
-v /tmp:/host/tmp \
--privileged \
--network host \
--pid host \
-e BACKEND_ENDPOINT=https://<sysdig_backend_endpoint> \
-e ACCESS_KEY=<Sysdig agent access key> \
-e BACKEND_VERIFY_TLS=false \
-e TAGS=<custom_tags> \
quay.io/sysdig/compliance-benchmark-runner:latest
Note: If you don’t want to pass the access key directly via the command line, consider using an alternative method of passing environment variables, such as docker-compose.
The BACKEND_ENDPOINT
is only required if for Sysdig on-prem or
when using a Sysdig SaaS region other than US-EAST.
For example, for the EU SaaS endpoint would be:
https://eu1.app.sysdig.com
.
See also: SaaS Regions and IP Ranges.
BACKEND_VERIFY_TLS=false
is only needed if you are using an
on-prem backend with a self-signed certificate.
TAGS:
The list of tags for the host where the agent is installed.
For example: “role:webserver, location:europe
”, “role:webserver
”
or “webserver
”.
It is also possible to run the image analyzer as a single Docker container:
docker run -d \
-v /var/run:/var/run \
--privileged \
--network host \
-e AM_COLLECTOR_ENDPOINT=https://<sysdig_backend_endpoint>/internal/scanning/scanning-analysis-collector \
-e ACCESS_KEY=<Sysdig agent access key> \
-e VERIFY_CERTIFICATE=false \
quay.io/sysdig/node-image-analyzer:latest
Note: If you don’t want to pass the access key directly via the command line, consider using an alternative method of passing environment variables, such as docker-compose.
The AM_COLLECTOR_ENDPOINT
is only required if for Sysdig on-prem
or when using a Sysdig SaaS region other than US-EAST.
For example, for the EU SaaS endpoint would be:
https://eu1.app.sysdig.com/internal/scanning/scanning-analysis-collector
.
See also: SaaS Regions and IP Ranges.
VERIFY_CERTIFICATE=false
is only needed if you are using an
on-prem backend with a self-signed certificate.
To install the Host Scanning component in a non-Kubernetes environment, you can use:
docker run -d \
-v /:/host:ro \
--privileged \
-e HOST_BASE=/host
-e AM_COLLECTOR_ENDPOINT=https://<sysdig_backend_endpoint>/internal/scanning/scanning-analysis-collector \
-e ACCESS_KEY=<Sysdig agent access key> \
-e VERIFY_CERTIFICATE=false \
-e SCHEDULE=@dailydefault \
quay.io/sysdig/host-analyzer:latest
Note: If you don’t want to pass the access key directly via the command line, consider using an alternative method of passing environment variables, such as docker-compose.
The BACKEND_ENDPOINT
is only required if for Sysdig on-prem or
when using a Sysdig SaaS region other than US-EAST.
For example, for the EU SaaS endpoint would be:
https://eu1.app.sysdig.com
.
See also: SaaS Regions and IP Ranges.
BACKEND_VERIFY_TLS=false
is only needed if you are using an
on-prem backend with a self-signed certificate.
TAGS:
The list of tags for the host where the agent is installed.
For example: “role:webserver, location:europe
”, “role:webserver
”
or “webserver
”.
These cases affect only the Image Analyzer component of the Node Analyzer installation.
It is still possible to install the image analyzer component without benchmarks or host scanning. This option normally would apply only to previous users of the former node image analyzer who want to upgrade just that component, for whatever reason.
This can be done by downloading the sysdig-image-analyzer-daemonset.yaml and sysdig-image-analyzer-configmap.yaml and deploying.
You need to deploy these YAMLs after installing the Sysdig agent in the
same nodes, and also in the same namespace (sysdig-agent
by default).
By default, the image analyzer will automatically detect the socket to mount from:
Docker socket from /var/run/docker/docker.sock
CRI-O socket from/var/run/crio/crio.sock
CRI-containerd socket from/var/run/containerd/containerd.sock
Some setups require the analyzer to use custom socket paths.
If the socket is located outside /var/run
, the corresponding volume
must be mounted as well. You can configure it via the single line
installer script or by manually editing the daemonset and configmap
variables.
When using the installer, use the-cv
option to mount an additional
volume and add -ds -cs
or -cd
to specify a Docker, CRI, or
CRI-containerd socket respectively.
See the script -help
command for additional information.
Examples:
For K3S, which uses containerd, add:
-cd unix:///run/k3s/containerd/containerd.sock -cv /run/k3s/containerd
For Pivotal, which uses a custom path for the Docker socket, use:
-ds unix:///var/vcap/data/sys/run/docker/docker.sock -cv /var/vcap/data/sys/run/docker
During its regular operation, the Image Analyzer uses much less memory than the limit specified in the daemonset configuration. However, in some cases, processing an image may require more memory, depending, for example, on image size, content or package types.
This issue can be detected by looking for abnormal spikes in the memory usage of the Image Analyzer pods which are also showing analysis errors. In such cases we recommend trying to increase the analyzer memory usage up to three times the size of the unprocessed images, if the cluster available memory allows.
For special cases, the image analyzer can be configured by editing the
sysdig-image-analyzer
configmap in the sysdig-agent
namespace with
the following options:
Option | Description |
---|---|
| The Docker socket path, defaulting to If a custom path is specified, ensure it is correctly mounted from the host inside the container. |
| The socket path to a CRI compatible runtime, such as CRI-O, defaulting to If a custom path is specified, ensure it is correctly mounted from the host inside the container. |
| The socket path to a CRI-Containerd daemon, defaulting to If a custom path is specified, ensure it is correctly mounted from the host inside the container. |
| The endpoint to the Scanning Analysis collector, specified in the following format: |
| Can be set to |
| Can be set to |
| Proxy configuration variables. |
| |
|
The analyzer component of the Host
Scanning feature can be
configured by editing the sysdig-host-analyzer
configmap in
thesysdig-agent
namespace with the following options:
Option | Description |
---|---|
schedule | The scanning schedule specification for the host analyzer expressed as a crontab string such as “5 4 * * *” (more examples). The default value of @dailydefault instructs the analyzer to automatically pick a schedule that will start shortly after it is deployed and will perform a scan every 24 hours. |
dirs_to_scan | The list of directories to inspect during the scan, expressed as a comma separated list such as /etc,/var/lib/dpkg,/usr/local,/usr/lib/sysimage/rpm,/var/lib/rpm,/lib/apk/db |
collector_endpoint | The endpoint to the Scanning Analysis collector, specified in the following format: https://<API_ENDPOINT>/internal/scanning/scanning-analysis-collector |
max_send_attempts | The number of times the analysis collector is allowed to retry sending results if backend communication fails |
ssl_verify_certificate | Can be set to "false" to allow insecure connections to the Sysdig backend, such as for on-premise installs that use self-signed certificates. By default, certificates are always verified. |
debug | Can be set to "true" to show debug logging, useful for troubleshooting. |
http_proxy | Proxy configuration variables. |
https_proxy | |
no_proxy |
The benchmark runner component can be
configured by editing the sysdig-benchmark-runner
configmap in
the sysdig-agent
namespace with the following options:
Option | Description |
---|---|
collector_endpoint | The Secure API endpoint, specified in the following format: https://<API_ENDPOINT> |
ssl_verify_certificate | Can be set to "false" to allow insecure connections to the Sysdig backend, such as for on-premise installs that use self-signed certificates. By default, certificates are always verified. |
debug | Can be set to "true" to show debug logging, useful for troubleshooting. |
http_proxy | Proxy configuration variables. |
https_proxy | |
no_proxy |
If you have installed the CLI-based version of the Admission Controller, the UI-based version is not backwards-compatible. You will need to uninstall the old version and install the UI-based version instead.
To understand and use the Admission Controller after installing it, see Admission Controller.
For a more technical documentation see Chart Documentation.
The component must be installed on each cluster where you want to use it.
Make sure kubectl
is pointing to the target cluster where the
Admission Controller will be installed.
Add and synchronize the Helm repository:
helm repo add sysdig https://charts.sysdig.com
helm repo update
Install the Admission Controller on the target cluster with full capabilities , e.g.:
helm install sysdig-admission-controller sysdig/admission-controller \
--create-namespace -n sysdig-admission-controller \
--set sysdig.secureAPIToken=$SYSDIG_API_TOKEN \
--set clusterName=$CLUSTER_NAME \
--set sysdig.url=https://$SYSDIG_SECURE_ENDPOINT \
--set features.k8sAuditDetections=true
Check that installation was successful in the Sysdig UI.
Log in to Sysdig Secure and select Image Scanning>Admission Controller|Policy Assignments
.
Admission Controller will be disabled by default in your cluster, to avoid accidentally blocking deployment.
Cluster will be displayed in the Connected list, as healthy, but Disabled (gray colored dot).
You have to manually enable it by toggling the Enabled flag and status should change to accordingly (green colored dot):
Following parameters are the most common ones, but find the full list of available parmeters or specific use-cases
--create-namespace
: If supplied, will create a namespace--namespace
: Desired namespace where the Admission Controller will be installed--set sysdig.secureAPIToken
: Sysdig Secure API token as found in the
Sysdig UI under Settings/User Profile. Note that this
user must have administrator rights--set clusterName
: User-defined name for this cluster that will appear
in the admission controller interface in Sysdig’s backend. The
cluster name needs to match the agent cluster name.--set sysdig.url
: Sysdig endpoint. Default
https://secure.sysdig.com
is for the us-east
region.us-west
use https://us2.app.sysdig.com
https://eu1.app.sysdig.com
https://app.au1.sysdig.com
https://app.us4.sysdig.com/
--set features.k8sAuditDetections
: (true/false) Set true
to enable
Kubernetes audit logging via the Admission Controller. See also:
Kubernetes Audit Logging
(legacy installation) and Select the Policy Type
(Kubernetes Audit Policies)--set verifySSL
: (true/false) Sets the verification of the Sysdig Secure API; default: true (we recommend only changing this to false when doing initial testing / evaluation of an on-premises installation)--set scanner.verifyRegistryTLS
: (true/false) Verify TLS from registries on image pull; default: true (we recommend only changing this to false when doing initial testing / evaluation)--set scanner.psp.create
: (true/false) Whether to create a psp policy and role / role-binding; default: falseLog in to Sysdig Secure as administrator and select
Settings|User Profile
.
Under Sysdig Labs, enable the Admission Controller feature and click
Save
.
The links to the Admission Controller pages will appear under Image Scanning in the left-hand navigation.
If you already have the Sysdig Admission Controller installed and want to upgrade:
helm upgrade sysdig-admission-controller sysdig/admission-controller \
-n sysdig-admission-controller \
--set features.k8sAuditDetections=true \
--reuse-values
For those customers who already have the Admission Controller AND
already enabled Kubernetes audit logging via the legacy
method, you can still
install/upgrade to the new Admission Controller. Just be sure to set
features.k8sAuditDetections=false
to avoid collecting and displaying
duplicate events.
If you have installed the CLI-based version of the Admission Controller, the UI-based version is not backwards-compatible. You will need to uninstall the old version and install the UI-based version instead.
Deploy the following:
helm uninstall -n sysdig-admission-controller sysdig-admission-controller
Refer to Chart Documentation - Troubleshooting.
The topics in this section are designed for platform administrators who may install Sysdig Monitor and Secure on-premises, deploy and configure the agents, and configure, administer, and troubleshoot the environment.
This section helps you navigate to the topics of administering the Sysdig user and notification management.
Feature Documentation | Description |
---|---|
Super Admin Management | Locate the super admin, super user, and login tokens. |
User and Team Administration | Understand Sysdig’s users, teams, and role permissions. |
Notifications Management | Add, edit, or delete a variety of notification channel types, and disable or delete notifications when they are not needed. |
Find Your Customer ID and Name | Find customer ID for SaaS deployments. |
Authentication and Authorization (SaaS) | Set up secure access control for SaaS deployments. |
This section helps you navigate to the topics on License Management.
Feature Documentation | Description |
---|---|
Subscription | Review your account status regarding payment tier and licensed numbers of agents, serverless agents, cloud accounts. |
This section helps you troubleshoot on-premises and agent installation.
Feature Documentation | Description |
---|---|
Troubleshoot On-Prem Installations | Review general issues and troubleshooting tips for on-prem installations. Troubleshoot On-Prem Installations. |
Troubleshoot Sysdig Agent | Browse troubleshooting tips for Sysdig agents. |
Contact Sysdig Support | Get help from Sysdig Support. |
This section provides guidelines for deploying a Sysdig Platform on-premises.
Feature Documentation | Description |
---|---|
System Architecture | Understand the Sysdig Platform components and their relationships to each other and the environment. |
System Requirements | Review the hardware components and software resources required to host the Sysdig Platform. |
When installing Sysdig Platform on-premises, follow the instructions specific to your environment. Where available, the Installer tool is the recommended option.
For release 3.6.0 and higher, this material has moved to version-specific folders in GitHub.
For legacy installation instructions, see On-Premises Installation.
For information on upgrading environments up to version 3.5.1, see On-Premises Upgrades. For upgrading beyond 3.6.0, there will be version-specific folders in GitHub.
This section helps you navigate to the topics on securing the Sysdig Platform components.
Feature Documentation | Description |
---|---|
Manage User Profile and Password | Access the current user’s login credentials, team, and role, and retrieve the API token to use with custom scripts or applications. |
Securing User Passwords | Secure user credentials for Sysdig Platform components. |
Authentication and Authorization (On-Prem Options) | Set up secure access control for the on-prem deployments. |
The Settings panel can be accessed from both Sysdig Monitor and Sysdig Secure UIs, and by both administrator and non-admin users.
Log in to Sysdig Monitor or Sysdig Secure.
Hover the mouse over the User menu in the lower left corner of the navigation bar.
The two ways to access the Settings links are displayed:
Access the User Profile page to review and perform necessary actions:
Log in to Sysdig Monitor or Sysdig Secure and select Settings
from the user menu.
Select User Profile
.
Review settings and perform actions below.
The current user’s login email address, current team, and role on that team are listed in the User Profile section.
This option is visible to admins only.
If logged on as Administrator, you can access Admin Settings on this page which apply globally.
Hide Agent Install: Toggle this slider to hide the Agent Installation link in the Settings menu from non-admin users.
See Navigate the Settings Panel: Admin vs User and Agent Installation: Overview and Key for more information.
When using the Sysdig API with custom scripts or applications, an API security token (specific to each team) must be supplied.
Log in to Sysdig Monitor or Sysdig Secure and select Settings
from the user menu.
Select User Profile
.
The Sysdig Monitor or Sysdig Secure API token is displayed (depending on which interface and team you logged in to).
You can Copy
the token for use, or click the Reset Token
button
to generate a new one.
When reset, the previous token issued will immediately become invalid and you will need to make appropriate changes to your programs or scripts.
Use the Password Management fields to change this user’s password.
Required: Minimum 8 characters, not the last password used.
Recommended: We advise following NIST’s most up-to-date recommendations, with an emphasis on length and uniqueness.
Toggle the feature settings listed under Sysdig Labs to enable/disable specific beta functionalities to your installation. Data that has already been stored will not be affected by beta toggles.
(If there are no beta features, Sysdig Labs will not be displayed.)
When using the Sysdig API with custom scripts or applications, an API security token (specific to each user+team) must be supplied.
Log in to Sysdig Monitor or Sysdig Secure and select Settings
from the user menu.
Select User Profile
.
The Sysdig Monitor or Sysdig Secure API token is displayed (depending on which interface and team you logged in to).
You can Copy
the token for use, or click the Reset Token
button
to generate a new one.
When reset, the previous token issued will immediately become invalid and you will need to make appropriate changes to your programs or scripts.
This page describes the concepts behind Sysdig’s users, teams, and role permissions.
Users in Sysdig are identified by user name, email address, and password or third-party authentication option.
Users are either:
Invited manually by an Administrator via the Sysdig UI
Authenticated through a third-party system
Entered directly in the Sysdig database through the Admin API, which can bypass the invitation process if needed.
When invited, the new user is created in the Sysdig database upon the user’s first successful login to the Sysdig UI. Before the user accepts the invitation, enters a password, and logs in, they have a “pending” status.
From the outset, users in the Sysdig environment have one of three types of system privileges
(Super) Admin: This is the administrator whose email address is associated with the Sysdig billing account. This user has administrator access to everything. Most relevant in on-prem installations.
Administrator: Any administrator can grant Admin system privileges to any user. Administrators are automatically members of all teams.
Administrators can create/delete users; create/configure/delete teams; create/delete notification channels; manage licenses; and configure Agents from links in the Settings menu that are hidden from non-admins.
User (non-admin): By default, new users have read/write privileges to create, delete, and edit content in the Sysdig interface. They do not see options in the Settings menu that are restricted to Administrators.
User rights are further refined based on team and team role assignments, as described below.
When a user is created, it is automatically assigned to a default team (described below).
Notice that this default workflow grants all new users Edit access.
Teams can be thought of as service-based access control. Teams are created and assigned separately in Sysdig Monitor and Sysdig Secure.
Organizing users into teams enables enforcing data-access security policies while improving users’ workflows. There are different team roles, each of which has read/write access to different aspects of the app.
This limits the exposure of data to those who actually need it, and also makes users more productive by focusing them on data that is relevant to them.
The following are some potential use cases for Teams.
“Dev” vs “Prod”: Many organizations prefer to limit access to production data. Permits isolating physical infrastructure and the applications on top.
Microservices: Scoping data for individual dev teams to see their own dashboards and field their own alerts. Permits team creation based on logical isolation using orchestration or config management metadata in Sysdig Monitor.
Platform as a Service: Where Ops teams need to see the entire platform. Enabling certain people to see all data for all services as well as the underlying hardware. This is perfect for managed service providers who are managing a multi-tenant environment, or devops teams using a similar model within their own organization.
Restricted environments: Limiting data access for security and compliance. Certain services, such as authentication and billing, may have a very specific set of individuals authorized to access them.
Organizations that need to segment monitoring for efficiency: Wide-ranging use case from very large organizations forming teams to simplify access, to smaller orgs creating ephemeral troubleshooting teams, to teams formed to optimize QA and Support access to system data.
Out of the box, the Sysdig Platform has one immutable team for each product. Depending on licensing, an organization may use one or both:
Monitor Operations team
Secure Operations team
Key traits of the immutable Operations teams:
The teams cannot be deleted
Users in Operations teams have full visibility to all resources in that product
Administrators must switch to the Operations team before changing configuration settings for any team
Administrators create additional teams and can designate any team to become the default team for that product. The number of teams allowed in an environment is determined by licensing.
Users entered in the Sysdig Monitor UI are auto-assigned to the Monitor default team; users entered in the Sysdig Secure UI are auto-assigned to the Secure default team.
If the Essentials tier is licensed, only the default teams and roles are enabled. See Subscription for more details.
If upgrading from Essentials to Enterprise,
Capture functionality will
become available. Users must go to Settings>Teams><Your Team>
and
check the Enable Captures
box. They must then log out and log in
again.
Users can be assigned roles that expand or limit their basic system privileges on a per-team basis.
System Role | Team Role | |||
---|---|---|---|---|
Admin | Member of every team, with full permissions regardless of team assignment. Can create/delete/configure all users. Can create/delete/configure all teams. | |||
Team Manager (Monitor) | Advanced User (Monitor) | Standard User (Monitor) | ||
Non-Admin (Sysdig Monitor) | Can create/edit/delete dashboards, alerts, or other content + ability to add/delete team members or change team member permissions. NOTE: Team Managers only have user administration rights within the specific team(s) for which they are designated Managers, however, Team Manager users will see a list of users and teams they are assigned to, regardless of the team they have logged in to. | Can create/edit/delete dashboards, alerts, or other content. | Equivalent to an Advanced User with no access to the Explore page (e.g. for developers who are not interested in Monitoring information). | |
Team Manager (Secure) | Advanced User (Secure) | Service Manager (Secure) | Standard User (Secure) | |
Non-Admin (Sysdig Secure) | Same permissions as the Advanced User + ability to add/delete team members or change team member permissions. NOTE: Team Managers only have user administration rights within the specific team(s) for which they are designated Managers, however, Team Manager users will see a list of users and teams they are assigned to, regardless of the team they have logged in to. | Can access every Secure feature within the team scope in read and write mode. Advanced Users can create, delete, or update runtime policies, image scanning policies or any other content. The Advanced User cannot manage users. Free Tier users are automatically assigned to Advanced User role. | Same as Standard User, plus ability to invite existing users to the team and manage the notifications channels assigned to the team. | Can push container images to the scanning queue, view image scanning results, and display the runtime security events within the team scope. Standard Users cannot access Benchmarks, Activity Audit, Policy definitions, or certain write functions within other Secure features. |
See How Team Membership Affects Users' Experience of the UI for more detail. |
Team membership affects user experience of the Sysdig Monitor or Sysdig Secure UIs in various ways.
At the highest level, the dashboards, alerts, and policy events you see are limited by the settings of the team you are switched to.
In more detail, team settings affect the following:
Default landing page: The UI entry point is set on a per-team basis.
Explore tab and dashboards: These are set per-team, per-user and can be shared with the team.
On first login, all team members see the same Dashboards Assigned to Me view. If a user changes those dashboards, only that user will see the changes.
Dashboards created while part of a team are only visible to the user when logged in to that team, and if shared, are only visible to other team members.
Visible data: A team’s scope settings limit the data visible to team members while they are switched to that team, even if a user belongs to other teams with different settings that reveal additional data. In Sysdig Secure, for example, only the policy events that fired within your scope will be visible.
Alert and Event: These settings are team-wide. Any member of a team can change the team’s alert settings, and any additions or edits are visible to all members of the team.
Captures: Can only be taken on hosts/containers visible to team members, and members see only the list of captures initiated by other members who were switched to the current team.
API Token: Note that the Sysdig Monitor API Token found under Settings > User Profile is unique per-user, per-team. (See User Profile and Password. This is necessary to enable the generation of Custom Events via the API to target a specific team.
Users can switch between all teams to which they’ve been assigned, and Administrators can switch between all teams that have been created.
To do so:
Click the user menu in the lower-left corner of the navigation bar.
The assigned teams for this user are listed under Switch Teams.
NOTE: With version 3.6.0, you can also search for the teams in the user menu.
Click another team name.
A popup window gives an overview of the new team-based view of the environment. The UI changes according to the team settings.
Plan teams and roles strategically to isolate access to data, customize interfaces, and streamline workflows.
In general, administrators should:
Create teams, invite users, and set roles in a planned manner
Start with some dashboards and alerts for given teams to get started with
When a user logs in to a team for first time, they will see a wizard introducing dashboards, alerts, etc. specific to that team.
By default, new users (added manually or through a third-party authenticator) are assigned Advanced User rights. If a administrator wants to limit new users’ rights further, there are several ways to do so.
Between sending the invitation and the user’s first log in, change the user’s Role in the default Monitor team to Read User.
Note that there could theoretically be a lag in which the user would briefly have had Edit status.
Integrate users into Sysdig via the Admin API and define read-only permissions upon import.
Create a default team, in either Sysdig Monitor or Sysdig Secure, with very limited scope and visibility. Manually assign users to additional teams with broader permissions as needed.
If Team-Based Roles and Privileges don’t meet the specific needs of your organization, you can create your own custom roles. See .
If you are working with Sysdig Support Engineers to provision users and
teams via the Sysdig API, note how the user and team role names within
the UI map to the API ROLE
names.
Regular (non-admin) = ROLE_USER
Admin = ROLE_CUSTOMER
Advanced user = ROLE_TEAM_EDIT
Standard user = ROLE_TEAM_STANDARD
View-only user = ROLE_TEAM_READ
Team manager = ROLE_TEAM_MANAGER
Service manager (Sysdig Secure only) = ROLE_TEAM_SERVICE_MANAGER
This page describes how to add, delete, and configure user information from within the Sysdig Monitor or Sysdig Secure UI.
Users added in Sysdig Monitor will appear in the full list of users for both Sysdig Monitor and Sysdig Secure, if both products are in use. However, users will not have log in access to Sysdig Secure until they are added to a Sysdig Secure team.
Only Admin users can configure user account information.
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu.
Select Users.
Click the Add User link.
Enter the user’s email address, first name and last name:
Click Save to send the user invite, or click Cancel to discard the user.
For on-premises environments, you may need to have pre-configured your SMTP parameters in your Replicated or Kubernetes installation configmap.
The new user will be added to the User Management
table. Their status
will be listed as Pending
until the invitation is accepted.
Admin privileges cannot be assigned until the invitation has been accepted, and the user has logged into the interface for the first time. They can, however, be added to additional teams or have team-based roles assigned. For more information on configuring teams roles, refer to the Manage Teams and Roles documentation.
To edit an existing user:
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu.
Select Users.
Select the user from the User Management table.
Optional: Edit the first name / last name.
Optional: Toggle the Admin switch to enable/disable administrator privileges.
Click Save to save the changes, or Cancel to revert the unsaved changes.
User emails are read-only, and cannot be changed.
To delete an existing user:
Deleting a user cannot be undone. Any dashboards or explore groupings that the user created for any team will be permanently deleted.
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu. `
Select Users.
Select the user from the User Management table.
Click Delete User.
Click Yes, delete to confirm the change.
You can optionally delete the dashboards and artifacts that the user have created.
A custom role is a admin-defined role which allows Sysdig administrators to bundle a set of permissions and allocate it to one or more users or teams. This page describes how to create and use custom roles.
Custom Roles is supported only on SaaS regions. The feature is not currently available for on-prem environments.
Custom roles gives you the ability to provide granular access to users according to a selected list of permissions. If the Sysdig Roles don’t meet the specific needs of your organization, you can create your own custom roles. Select the permissions you want them to have based on the resource they should have the access to and bundle it together. Just like built-in Sysdig roles, you can assign custom roles to users and teams. Custom roles ensures that the users have only the permission they need and help prevent unwanted access to other resources.
Custom roles operate on concepts similar to roles-based access control system (RBAC).
Allow you to give access to a specific set of predefined dashboards to a group of users, who should not be able to view any additional data, nor change or share these dashboards.
Allow you to create a service account for Sysdig Secure that is not tied to a particular user but can be used to automate your CI/CD pipeline.
Allow you to identify the owner of a particular image so the security issue can be assigned to the actual team who owns the issue.
Create a team role that can only invite users but not actually manage the team.
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings.
Select Roles.
Click New Role.
The New Role page is displayed.
Specify the following:
Select the features and do one of the following:
Click Save New Role.
You can set up a custom role as the default user role for teams. To do so:
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings.
Select Teams.
Do one of the following:
From the Default User Role drop-down, select one of the custom role you have created.
Complete creating or editing the team as given in Manage Teams and Role.
Click Save.
Click Customize to view and select granular permissions for each product features. Alternatively, use the drop-down to grant read access or full access to all the privileges simultaneously.
Category | Item | Permission | Description |
---|---|---|---|
Overview/Insights | Overview/Insights | ||
Read | Access Overview/Advisor | ||
Dashboards | Dashboard | ||
Read | Access dashboards in scope of a team | ||
Edit | Modify dashboards in scope of a team | ||
Dashboard Metrics Data | |||
Read | N/A | ||
Explore/Metrics | Agent Console | ||
View | Use Agent Console commands | ||
Agent Console - Agent Status | |||
Read | Use Agent Console commands which access agent status | ||
Agent Console - Configuration | |||
View | Use Agent Console commands to view the configuration of the agent which does not contain sensitive information like passwords | ||
Agent Console - Diagnostics | |||
Read | Use Agent Console commands which access internal diagnostics of the agent | ||
Agent Console - Network Calls | |||
Exec | Use Agent Console commands which make network calls to remote pods and endpoints | ||
Agent Console - Sensitive Configuration | |||
View | Use Agent Console commands to view the configuration of the agent which does contain sensitive information like passwords. There are currently zero commands that implement this permission | ||
Explore | |||
Read | Metric querying with Explore | ||
Edit | N/A | ||
LiveLogs | |||
View | Access LiveLogs feature | ||
Shared Groupings with Team | |||
Toggle | Share metrics grouping with the team | ||
Alerts | Alert Events | ||
Read | Access the events generated by triggered alerts in scope of a team | ||
Edit | Acknowledge an event triggerred by an alert in the events feed in scope of a team | ||
Alerts | |||
Read | Access the alerts in scope of a team | ||
Edit | Modify alerts in scope of a team | ||
Events | Custom Events | ||
Read | Access the infrastructure & other events created by Sysdig Agent or Sysdig API | ||
Edit | Acknowledge the infrastructure and other events created by Sysdig Agent or Sysdig API | ||
Captures / Investigate | Captures | ||
View | View captures in the UI | ||
Read | Access captures | ||
Edit | Modify captures | ||
Settings | API Access Token | ||
View | View your API token | ||
Read | Access users API token in scope of a team | ||
Edit | Reset users API token in scope of a team | ||
AWS Settings | |||
Read | Access AWS settings | ||
Agent Installation | |||
Read | Get agent access key (required for agent installation) | ||
Alert Downtimes | |||
Read | List alert downtimes for the customer | ||
Global Notification Channels | |||
Read | Access global notification channels | ||
Notification Channels | |||
Read | Access notification channels in scope of a team | ||
Edit | Modify notification channels in scope of a team | ||
Service Accounts | |||
Read | Access service accounts in scope of a team | ||
Edit | Modify service accounts in scope of a team | ||
Subscriptions | |||
Read | Access customer subscription details | ||
Sysdig Storage | |||
Read | View Sysdig storage configuration | ||
Team Agent Console Access Toggle | |||
Read | See the agent console access settings for a team | ||
Edit | Toggle access to agent console for a team | ||
Team Captures Access Toggle | |||
Read | See the capture settings for a team | ||
Edit | Toggle access to captures for a team | ||
Team Membership | |||
Read | Access team members | ||
Edit | Modify team members | ||
Teams | |||
Manage | Modify team settings without the ability to modify team membership for users | ||
Users | |||
Read | Access existing users data | ||
Create | Invite new users | ||
Users List | |||
Read | See the list of users for a customer | ||
Integrations | Custom Integrations | ||
Read | Access custom integrations in spotlight | ||
Edit | Modify custom integrations in spotlight | ||
Infrastructure | |||
Read | View discovered infrastructure | ||
Integrations | |||
Read | View discovered workload integrations | ||
Monitoring Integrations | |||
Validate | Change monitoring integration status to Pending Metrics | ||
Edit | Change monitoring integration type or status | ||
Providers | |||
Read | N/A | ||
Spotlight | |||
Read | Access spotlight | ||
Data Access Settings | Datastream | ||
Read | Access data stream configuration | ||
Groupings | |||
Read | Access default and custom groupings | ||
Edit | Create and edit custom groupings | ||
Metadata | |||
Read | N/A | ||
Metrics Data | |||
Read | Access metrics data | ||
Metrics Descriptors | |||
Read | Access metrics descriptors | ||
PromQL Metadata | |||
Read | Access Prometheus metrics and labels |
Category | Item | Permission | Description |
---|---|---|---|
Scanning | Image Import | ||
Edit | Import scanning images | ||
Scanning | |||
Write | Modify scanning alerts and registry credentials | ||
Read | Access scan results - Only found references in UI code | ||
Exec | Execute backend scanning (Scan button in UI)? - Couldn’t find any reference in code | ||
Scanning Alerts | |||
Read | Access scanning alerts | ||
Edit | Modify scanning alerts | ||
Scanning Image Results | |||
Read | List scanning images | ||
Create | Create scanning events | ||
Scanning Policies | |||
Read | Access security policies | ||
Edit | Modify security policies | ||
Scanning Policy Assignments | |||
Read | Access policy mappings | ||
Edit | Create and modify policy mappings | ||
Scanning Registry Credentials | |||
Read | List container registries | ||
Edit | Create and modify container registries configuration | ||
Scanning Runtime | |||
Edit | Query runtime containers API (API only, not enforced in UI) | ||
Scanning Scheduled Reports | |||
Read | View and download existing reports | ||
Edit | Create and modify reports | ||
Scanning Trusted Images | |||
Read | Access the trusted images list | ||
Edit | Modify the trusted images list | ||
Scanning Untrusted Images | |||
Read | Access the untrusted images list | ||
Edit | Modify the untrusted images list | ||
Scanning Vulnerability Exceptions | |||
Read | Access vulnerability exceptions | ||
Edit | Edit vulnerability exceptions | ||
Posture | Benchmark Tasks | ||
Read | Access scheduled benchmark taks | ||
Edit | Create and modify scheduled benchmark adn compliance tasks | ||
Benchmarks | |||
Read | Access benchmark results | ||
Compliance | |||
Read | Access Compliance tasks and reports | ||
Policies | Image profiling | ||
Write | Write image profiles | ||
Read | View existing image profiles | ||
Exec | Execute image profiling | ||
Policies | |||
Read | Access policies | ||
Edit | Modify policies | ||
Policy Advisor | |||
Write | Create PSP advisor simulation | ||
Read | Read PSP advisor simulations | ||
Exec | Execute PSP advisor simulation | ||
Network Security | Network Security | ||
Read | Access Kubernetes Network Security policy advisor | ||
Integrations | Providers | ||
Read | N/A | ||
Settings | API Access Token | ||
View | View your API token | ||
Read | Access users API token in scope of a team | ||
Edit | Reset users API token in scope of a team | ||
AWS Settings | |||
Read | Access AWS settings | ||
Agent Installation | |||
Read | Get agent access key (required for agent installation) | ||
Cloud Accounts | |||
Read | Access cloud accounts | ||
Edit | Edit cloud accounts | ||
Events Forwarder | |||
Read | Access event forwarding configuration | ||
Global Notification Channels | |||
Read | Access global notification channels | ||
Notification Channels | |||
Read | Access notification channels in scope of a team | ||
Edit | Modify notification channels in scope of a team | ||
Service Accounts | |||
Read | Access service accounts in scope of a team | ||
Edit | Modify service accounts in scope of a team | ||
Subscriptions | |||
Read | Access customer subscription details | ||
Sysdig Secure Settings | |||
Edit | Modify Sysdig Secure configuration | ||
Sysdig Storage | |||
Read | View Sysdig storage configuration | ||
Team Agent Console Access Toggle | |||
Read | See the agent console access settings for a team | ||
Edit | Toggle access to agent console for a team | ||
Team Captures Access Toggle | |||
Read | See the capture settings for a team | ||
Edit | Toggle access to captures for a team | ||
Team Membership | |||
Read | Access team members | ||
Edit | Modify team members | ||
Teams | |||
Manage | Modify team settings without the ability to modify team membership for users | ||
Users | |||
Read | Access existing users data | ||
Create | Invite new users | ||
Users List | |||
Read | See the list of users for a customer | ||
Captures / Investigate | Activity Audit Commands | ||
Read | Access activity audit commands | ||
Captures | |||
View | View captures in the UI | ||
Read | Access captures | ||
Edit | Modify captures | ||
Rapid Response | |||
Exec | Use rapid response | ||
Data Access Settings | Groupings | ||
Read | Access default and custom groupings | ||
Metrics Data | |||
Read | Access metrics data | ||
Metrics Descriptors | |||
Read | Access metrics descriptors | ||
Events | Policy Events | ||
Read | Access policy events |
The use of teams provides a strategic way to organize groups, streamline workflows, or protect data, as needed by an organization. Administrators who design and implement teams should have in-depth knowledge of organizational infrastructure and goals.
For more information, including foundational concepts, see User and Team Administration.
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings from the user menu.
Select Teams.
Click Add Team.
Configure the team options and click Save.
For more information on each configuration option, refer to Team Settings.
Ensure that the team names are unique in both Monitor and Secure products. For example, if you attempt at creating a team in Secure with the same name as one created in Monitor, you will see an error message stating that a team with the same name already exists and you will be prevented from creating the team.
Setting | Req'd | Description |
Color | Yes | Assigns a color to the team to make them easier to identify quickly in a list. |
Name | Yes | The name of the team as it will appear in the "Switch to" drop-down selector and other menus. |
Description | No | Longer description for the team. |
Default Team | No | If users are not assigned to any team, they will automatically be a part of this team if it's turned on. |
Default User Role | No | You can choose either [Custom Roles]((en/docs/administration/administration-settings/user-and-team-administration/manage-custom-role/#manage-custom-role) or [Sysig Team-Based Roles](en/docs/administration/administration-settings/user-and-team-administration/#team-based-roles-and-privileges). If no specific choice is made, Advanced User will be automatically selected. Choose a different role from the drop-down menu to set a different default user role for this team. . |
Default Entry Point | Yes | Defaults to the Explore page; choose an alternate entry if needed. |
Team Scope | Yes | Determines the highest level the data to which team members will have visibility. Agent Metrics: If set to Host, Team members can see all Host-level and Container-level information. If set for Container, Team members can see only Container-level information. Prometheus Remote Write Metrics: Visible if Prometheus Remote Write is enabled for your account. Use this option to determine what level of Prometheus Remote Write data that your Team members can view. You can further limit what data team members can see by specifying tag/value expressions for metrics for each data source. The drop-down menu defaults to "is", but can be changed to "is not", "in", "contains", and etc. Complex policies can be created by clicking Add another to create AND chains of several expressions. Note that making changes to the Team Scope settings can have a dramatic impact on what's visualized in the Team's Dashboards that are already configured, so you may want to carefully review these before/after your change. |
Additional Permissions | Sysdig Capture: Enable this option to allow this team to take Sysdig Captures. Captures will only be visible to members of this team. WARNING: Captures will include detailed information from every container on a host, regardless of the team's Scope. Infrastructure Events: Enable this option to allow this team to view ALL Infrastructure and Custom Events from every user and agent. Otherwise, this team will only see infrastructure events sent specifically to this team. AWS Data: Enable this option to give this team access to AWS metrics and tags. All AWS data is made available, regardless of the team's Scope. Agent CLI: Enable this option to give this team access to Agent Console. Infrastructure Event: Enable this option to give this team access to infrastructure events. | |
Team Users | No | Click to select any non-Admin users to be immediately added to this Team. Admins are filtered out by default, since they are members of every team automatically. |
Some Sysdig Monitor teams benefit from using a default entry point other than the usual Explore page, as users who don’t need in-depth monitoring information can onboard and navigate Sysdig Monitor more efficiently.
Use the Default Entry Point setting on the Team page, as shown in Create a Team.
Note: If selecting a dashboard, open the secondary Dashboard drop-down menu, or type the name of the dashboard to select it.
The dropdown is only populated with shared dashboards accessible by anyone on the team.
Users can be assigned to multiple teams. Team assignment is made from the Team page (not the User page), and must be done by an Administrator or Team Manager.
Users added in Sysdig Monitor will appear in the full list of users for both Sysdig Monitor and Sysdig Secure, if both products are in use. However, users will not have log in access to Sysdig Secure until they are added to a Sysdig Secure team.
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings from the user menu.
Select Teams.
Select the relevant team from the list, or search for it with the search box, and then select the relevant team.
In the Team Users section, click the Assign User button.
Select the user from the drop-down list, or search for it and then select it.
Click the Role drop-down menu to select the user role:
Optional: Repeat steps 3 to 5 for each additional user.
Click Save.
Review Team-Based Roles and Privileges for an overview.
Note that the Advanced User permission can be further refined into either a View-only user or a Team Manager.
Managers can add or delete members from a team, or toggle members' rights between Edit, Read, or Manager.
Note that Admins have universal rights and are not designated as Team Managers, Advanced Users, View-Only users, or Standard users.
Manager or Advanced User permissions can be assigned even to Pending users; administrators do not have to wait for the user’s first login to set these roles.
To assign a role to a user on a team:
Log in to Sysdig Monitor or Sysdig Secure as Administrator and either create a team or select a team to edit.
Add a user or select a user from the list of team members.
Select the appropriate role from the drop-down menu.
Reminder of the role privileges:
Admin: Member of every team with full permissions. Can create/delete/configure all users and teams.
Team Manager: Advanced User privileges + ability to add/delete team members or change team member permissions.
Advanced User:
In Sysdig Monitor: Read/write access to the components of the application available to the team. Can create/edit/delete dashboards, alerts, or other content.
In Sysdig Secure: Read/write access to the components of the application available to the team. Can create, delete, or update runtime policies, image scanning policies or any other content.
View-Only:
In Sysdig Monitor: Read access to the environment within team scope, but cannot create, edit, or delete dashboards, alerts, or other content.
In Sysdig Secure: Read access to every Secure feature in the team scope, but cannot modify runtime policies, image scanning policies or any other content.
Standard User:
In Sysdig Monitor: An Advanced User with no access to the Explore page (e.g. for developers who are not interested in Monitoring information).
In Sysdig Secure: Can send container images to the scanning queue, view image scanning results, and display the runtime security events within the team scope. Standard Users cannot access Benchmarks, Activity Audit, Policy definitions, or certain write functions within other Secure features.
Service Manager: Sysdig Secure only. Same as Standard User, plus ability to invite existing users to the team and manage the notifications channels assigned to the team.
Save edits.
To configure an existing team:
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings from the user menu.
Select Teams.
Select the relevant team from the list, or search for it with the search box, and then select the relevant team.
Edit as needed, and click Save.
For more information regarding the configuration options, see Table 1: Team Settings.
When a team is deleted, some users may become “orphans”, as they are no longer a part of any team. These users will be moved to the default team.
The default team cannot be deleted. A new default team must be selected before the old default team can be deleted.
To delete a created team:
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings from the user menu.
Select Teams.
Select the relevant team from the list, or search for it with the search box, and then select the relevant team.
Click Delete Team, then Yes, delete to confirm the change.
Alerts are used in Sysdig Monitor when Event thresholds have been crossed, and in Sysdig Secure when Policy violations have occurred. Alerts can be sent over a variety of supported notification channels.
Notification Management describes how to add, edit, or delete a variety of notification channel types, and how to disable or delete notifications when they are not needed, for example, during scheduled downtime.
Alerts are used in Sysdig Monitor when Event thresholds have been crossed, and in Sysdig Secure when Policy violations have occurred. Alerts can be sent over a variety of supported notification channels.
In the Settings panel of either Sysdig Monitor or Sysdig Secure, set up the notification channels to be used for alerting.
Notification channel management can be finessed by role-based access as follows:
Notification channels can now be “global” or limited to a particular team
Global channels can be managed by admins and can be viewed/used by other roles, while team-limited channels are available only to team members
Team Manager , Advanced User, and Service Manager (Secure) roles can create/update/delete team-scoped notification channels, they can also read and use the global ones
Standard and View Only roles can read team-limited and global notification channels
Admins will be able to create global notification channels and migrate channels from “global” to “team-limited”, and also from one team to another.
To add a new notification channel:
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu.
Select Notification Channels.
The Notifications main page is displayed:
Click Add Notification Channel +, and select the desired notification channel.
Follow the channel-specific steps to complete the configuration process:
After you have set up a notification channel, it will appear as an available option to be assigned when you Add an Alert .
To edit a notification channel:
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings.
Select Notification Channels
.
Locate the target channel and click the Edit
button.
Make the edits and click Done Editing
to save the changes.
To test a notification channel:
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings.
Select Notification Channels
.
Select the three dots next to a created Notification Channel and
click Test Channel
.
If a notification is not received within 10 minutes, the notification channel is not working, and the configuration should be reviewed.
Sysdig Monitor integrates easily with AWS Simple Notification Service (SNS).
On the AWS side:
To automatically push Sysdig Monitor alerts to the SNS topic of your choice:
From the AWS console, open the SNS management console
In the Create topic section, Create a new topic (if needed).
The topic’s Name, ARN, (optional) Display name, and Topic owner’s AWS account ID are displayed in the Details section.
Select the topic on the list.
Expand Access policy - optional.
Select Basic (By default).
Under Define who can publish messages to the topic, select Only the specified AWS accounts and enter the Sysdig Monitor account ID: 273107874544 (US-East Only).
For account IDs corresponding to other regions, see SaaS Regions and IP Ranges.
Click Create topic.
Ensure that you subscribe to the created topic.
On the navigation panel, choose Subscriptions.
On the Create subscription page, enter the Topic ARN of the topic you created earlier.
Specify other details and click Create subscription.
For further information about AWS SNS, refer to the AWS documentation.
For SNS notification, you can click the ‘help’ button for tips on setting up your SNS topic.
You will need to allow publishing rights to the Sysdig Monitor account ID corresponding to your region.
This can be done by creating a new policy on your SNS topic in AWS Console as shown in the below images:
Select “Edit topic policy” as shown below from “Other topic actions.”
In the “Basic view” tab of the “Edit topic policy” dialog, select “Only these AWS users” from the publisher’s list and enter the Sysdig ID.
In the Sysdig Monitor UI:
Complete steps 1-3 in Set Up a Notification
Channel to log in to
the Sysdig UI and select Amazon SNS Topic
.
Enter the Topic created on the AWS side, along with a Channel Name, Enablement, and Notification toggles as appropriate.
From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Click Save
.
To send an alert notification via email, you must first set up the email notification channel.
To do so, complete steps 1-3 in Set Up a Notification Channel, then:
Select Email
.
Enter the relevant details for the email notification:
From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Click Save
.
If you enabled Test notification, a test email will be sent.
You can now configure an alert to use email notifications.
For on-premises environments, you may need to have pre-configured your SMTP parameters in your Replicated or Kubernetes installation configmap.
You can notify all the users of a team when an alert is triggered. Sysdig allows you to create a new notification channel where you can select a team from a list of existing ones as the target of the channel.
To send an alert notification via email to a team, you must first set up the team email notification channel. To do so, complete steps 1-3 in Set Up a Notification Channel, then:
Select Team Email.
Enter the relevant details for the email notification:
Select the Team Name and specify a Channel Name to identify the channel you are creating.
From Shared With drop-down, choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Click Save.
If you enabled Test notification, a test email will be sent.
You can now configure an alert to use email notifications.
To send an alert notification via PagerDuty, you must first set up the PagerDuty notification channel.
Have an account configured at PagerDuty.com.
Have your PagerDuty credentials available (account, password, and service).
Have the base user role of Manager. With a PagerDuty base user role of Manager, you can auto-fetch the service information during the Sysdig/PagerDuty integration process.
Check your team and base user permissions. If your PagerDuty team permissions are Manager but base user permissions are Responder or lower, you can enter the necessary data in the Sysdig UI manually.
Base user roles in the PagerDuty UI.
To launch the process from the Sysdig UI, complete steps 1-3 in Set
Up Notification
Channels and select
PagerDuty
.
Do one of the following:
Select Auto-fetch
when prompted. Ensure that you have the base user role of
Manager or higher in PagerDuty.
Select Manual
and enter the necessary configuration parameter. Skip to Step 5 for details.
Once you complete the pre-configuration, the PagerDuty Integration screen is displayed.
Do one of the following:
Enter the email
and password
associated with your PagerDuty
account, and click Sign In.
Enter the appropriate PagerDuty subdomain for single sign-on and click
Sign In Using Your Identity Provider.
A PagerDuty service selection screen is displayed.
Option 1: If you have never integrated before, you are prompted
to choose a PagerDuty Servicename
.
Option 2: If at least one service has already been integrated, you can select that one.
Click Connect
.
Once integration is authorized, the Sysdig page for a new PagerDuty notification channel is displayed, with the information auto-filled.
From Shared With, choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Do one of the following:
Confirm the auto-populated information and click Save
.
If you chose Manual entry in Step 2, then type the information and
click Save
.
You can now Add an Alert to use PagerDuty notifications.
There is a known issue whereby changing a notification from “Acknowledged” to “Unacknowledged” does not update correctly in PagerDuty.
What occurs:
Event has triggered Notification, Notification is sent to PagerDuty.
Open Event and click on “Acknowledge” button in Sysdig.
Notification is sent to PagerDuty, and status is changed to “Acknowledged.”
Open Event and click on “UnAcknowledge” button in Sysdig.
Status is not changed in PagerDuty. It remains “Acknowledged” rather than changing to “Triggered” in PagerDuty.
To send an alert notification via Slack you must first set up the Slack notification channel.
To do so:
Prerequisite:
Have a Slack account configured at Slack.com and know which notification channel to use for notifications.
To launch the process from the Sysdig UI, complete steps 1-3 in Set Up Notification Channels and select Slack.
You will be prompted to log in to your Slack account.
Select a Slack channel from the drop-down list to be used for notifications and click Authorize.
From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Complete configuration as desired and click Done.
Click Test to check the new functionality.
You can now configure an alert to use Slack notifications.
To integrate with your VictorOps
Log in to VictorOps.
Go to Settings > Alert Behavior > Integrations in the VictorOps interface.
Select REST from the list of Featured Integrations.
Complete steps 1-3 in Set Up a Notification
Channel to log in to
the Sysdig UI and select VictorOps
.
Enter the VictorOps parameters in the Sysdig Notification Channel fields, as follows:
API Key: everything between "/alert/" and “/$routing_key” in the REST URL
Routing Key: A VictoOps way of routing alerts to appropriate teams. See their Routing Keys documentation for details, if needed.
Channel Name: Choose a meaningful name like “VictorOps”.
Enable the channel and desired notification types.
From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Click Save
.
Go directly to the OpsGenie Integrations Page to configure the integration on the OpsGenie side.
OpsGenie maintains documentation on how to integrate with Sysdig products (formerly called Sysdig Cloud) here.
Complete steps 1-3 in Set Up a Notification
Channel to log in to
the Sysdig UI and select OpsGenie.
Copy/paste your OpsGenie integration API key and add a Channel Name, Enablement, and Notification toggles as appropriate.
From Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Click Save
.
Sysdig Monitor supports sending an alert notification to Microsoft teams. Microsoft Teams has different types of integrations for third-party applications, of which Sysdig supports Incoming Webhooks.
Incoming Webhooks are a type of Connector in Teams that provide a simple way for an external app to share content in team channels. They are often used as tracking and notification tools. Microsoft Teams provides a unique URL to which you can send a JSON payload with the message that you want to POST, typically in a card format. Cards are UI containers that contain content and actions related to a single topic and are a way to present message data in a consistent way.
You will need to enter the URL that you copied from the Connector. Sysdig will format a message by using a custom card template and send it to the channel. The message will show up as a new notification in the Microsoft application.
Have the destination URL handy. You can copy it from the Connectors > Incoming Webhook window on the Microsoft Teams UI. For more information, see Add an incoming webhook to a Teams channel.
Webhooks via HTTPS work only if a signed or valid certificate is in use.
Complete steps 1-3 in Set Up a Notification Channel and choose Microsoft Teams.
Enter the configuration options:
URL: The destination URL you have copied from Microsoft Teams UI.
Channel Name: Add a meaningful name for your Microsoft Teams channel.
Enabled: Toggle on or off.
Notification options: Toggle for notifications when alerts are resolved or acknowledged.
Test notification: Toggle to be notified that the configured URL is working.
Shared With: Choose whether to apply this channel globally. All Teams or to a specific team from the drop-down.
Click Save.
Sysdig Monitor and Sysdig Secure support sending an alert notification to a destination, such as a website, custom application, and so on for which Sysdig does not have a native integration. Do this using a custom Webhook channel.
Webhooks via HTTPS only work if a signed/valid certificate is in use.
Have your desired destination URL on hand.
Complete steps 1-3 in Set Up a Notification
Channel and choose
Webhook
.
Enter the webhook channel configuration options:
URL: The destination URL to which notifications will be sent.
Channel Name: Add a meaningful name, such as Ansible, PagerDuty, OpsGenie, and so on.
Enabled: Toggle on and off notifications.
Notification options: Toggle for notifications when alerts are resolved or acknowledged.
Test notification: Toggle to be notified that the configured URL is working.
Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Allow insecure connections: Enable if you want to skip the TLS verification.
Custom headers: Add custom headers to your alert notification.
If your webhook integrations require additional headers you specify them by using a custom header.
For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default header, but you can add it using a custom header.
Alternatively, you can choose to add custom headers programmatically as described in Configure Custom Headers and Custom Data Programmatically.
Custom Data: Specify the custom data you want to attach to the alert notification. The data must be a valid JSON document. This information will be included in the request body of the HTTP call. Systems that receive these webhook alerts can parse the data and take action based on the contents.
Click Save
.
When the channel is created, you can use it on any alerts you create.
Then, when the alert fires, the notification will be sent as a POST in JSON format to your webhook endpoint. (See Alert Output, below.)
For testing purposes, you can use a third-party site to create a temporary endpoint to see exactly what a Sysdig alert will send in any specific notification.
By default, alert notifications follow a standard format (see Description of POST Data, below).
However, some integrations require additional headers and/or data, which you can append to the alert format using a custom header or custom data entry.
For example, Ansible uses token-based authentication, which requires an entry for the bearer token. This entry is not included in the default alert template built into Sysdig, but you can add it using a custom header.
In addition to the Webhook UI option, you can do this from the command line, as described below.
additionalHeaders
is usually used for authentication
customData
is used to add values to the alert
After it has been created via the API, any manipulation will mangle the notification channel. Use with care.
This example adds two custom headers and defines additional custom data, as well as the format for that data.
Use the curl command to retrieve all configured notification channels:
curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'Authorization: Bearer API-KEY'
Add the custom headers and execute the request:
curl -X PUT https://app.sysdigcloud.com/api/notificationChannels/1 -H 'Authorization: Bearer API-KEY' -H 'Content-Type: application/json' -d '{
"notificationChannel": {
"id": 1,
"version": 1,
"type": "WEBHOOK",
"enabled": true,
"name": "Test-Sysdig",
"options": {
"notifyOnOk": true,
"url": "https://hookb.in/v95r78No",
"notifyOnResolve": true,
"customData": {
"String-key": "String-value",
"Double-key": 2.3,
"Int-key": 23,
"Null-key": null,
"Boolean-key": true
},
"additionalHeaders": {
"Header-1": "Header-Value-1",
"Header-2": "Header-Value-2"
}
}
}
}'
Alerts that use a custom webhook for notification send a JSON-format with the following data.
{
"timestamp": 1620222000000000, // Time when the alert triggered in microseconds
"timespan": 60000000, // duration of the alert in microseconds (how long the value should be true before triggering)
"alert": {
"severity": 2, // severity from 0 to 7, use severityLabel for a human readable version
"editUrl": "https://app-staging.sysdigcloud.com/#/alerts/21998727", // alert edit URL
"severityLabel": "Medium", // human readable version of severity
"subject": "CPU temp is High on homebridge:9100 is Triggered", // Alert subject
"scope": null, // scope of the alert if set from the UI
"name": "CPU temp is High", // name of the alert
"description": null, // description, not used ATM
"id": 21998727, // alert id
"body": "CPU temp is High on homebridge:9100 is Triggered\n\n\nEvent Generated:\n\nSeverity: Medium\n Metric:\n node_hwmon_temp_celsius = 65.8121\nSegment:\n instance = 'homebridge:9100'\nScope:\n Everywhere\n\nTime: 05/05/2021 01:40 PM UTC\nState: Triggered\nNotification URL: https://app-staging.sysdigcloud.com/#/events/notifications/l:2419200/14918845/details\n\n------\n\nTriggered by Alert:\n\nName: CPU temp is High\nTeam: Monitor Operations\nScope:\n Everywhere\nSegment by: instance\nWhen: avg(avg(node_hwmon_temp_celsius)) > 40\nFor at least: 1 m\nAlert URL: https://app-staging.sysdigcloud.com/#/alerts/21998727\n\n\n"
},
"event": {
"id": 14918845, // id of the generated event
"url": "https://app-staging.sysdigcloud.com/#/events/notifications/l:604800/14918845/details" // url of the event in the feed
},
"state": "ACTIVE", // status of the alert, can be ACTIVE or OK
"resolved": true,
"entities": [ // list of entities that triggered the alert, at the moment we send a notification per entity, so this array will always contain a single object
{
"entity": "instance = 'homebridge:9100'", // segment that triggered
"metricValues": [ // value of the metric at the time of triggering
{
"metric": "node_hwmon_temp_celsius",
"aggregation": "avg",
"groupAggregation": "avg",
"value": 65.812167
}
]
}
],
"endEntities": [ // list of entities when the alert was resolved (same as "entities")
{
"entity": "instance = 'homebridge:9100'",
"metricValues": [
{
"metric": "node_hwmon_temp_celsius",
"aggregation": "avg",
"groupAggregation": "avg",
"value": 39.812167
}
]
}
],
"condition": "avg(avg(node_hwmon_temp_celsius)) > 40", // alert condition in string form
"source": "Sysdig Cloud", // source of the event
"labels": { // list of labels associated to this event (they strongly depend on the segmentation and scope of the alert)
"instance": "homebridge:9100"
}
}
$ curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'authorization: Bearer dc1a42cc-2a5a-4661-b4d9-4ba835fxxxxx'
{"timestamp":1543419336542,"status":401,"error":"Unauthorized","message":"Bad credentials","path":"/api/notificationChannels"}
$ curl -X GET https://app.sysdigcloud.com/api/notificationChannels -H 'Authorization: Bearer dc1a42cc-2a5a-4661-b4d9-4ba835fxxxxx'
{"notificationChannels":[{"id":18968,"version":2,"createdOn":1543418691000,"modifiedOn":1543419020000,"type":"WEBHOOK","enabled":true,"sendTestNotification":false,"name":"robin-webhook-test","options":{"notifyOnOk":true,"url":"https://postb.in/6dtwzz7l","notifyOnResolve":true}}]}
$
The webhook feature is used to integrate the following channels:
Sysdig can be integrated with ServiceNow using a custom webhook.
Have a ServiceNow account set up and working.
If necessary, refer to ServiceNow developer documentation here.
Login to ServiceNow (developer entry) and create a Scripted REST API:
Click New
and submit the form with the following:
Name: SysdigAlert API ID: sysdigalert
Return to the Scripted REST APIs
and open the resource just
created.
Scroll down to the related list area, select Resources
, and
click New
. This will create a new Scripted REST API resource.
Fill in the Name
field e.g. Demo.
Scroll down to Security
and clear the checkbox that requires
authentication.
Change the HTTP method
from GET to POST.
The resource is created.
Now give the resource the code to execute.
The default objects to work with in a Scripted REST API Resource are
response
and request
.
For more details on request and response see Scripted_REST_Request_API and Scripted_REST_Response_API
The created resource will already have some example code:
(function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
// implement resource here
})(request, response);
Change this default code to:
(function process(/*RESTAPIRequest*/ request, /*RESTAPIResponse*/ response) {
gs.info(request.body.dataString);
})(request, response);
Note the following resource path to this newly created resource is now visible: /api/snc/sysdigalert.
The url to this resource would be https://yourInstance.service-now.com/<resource_Path or https://yourInstance.service-now.com/api/snc/sysdigalert ``
Click Submit/Update on this resource.
Now that the custom API endpoint in ServiceNow is created, you can configure Sysdig alerts to use a custom webhook to trigger the ServiceNow integration.
API URL: your instance name URL
Name: ServiceNow (or whatever name you’d like for this Sysdig alert webhook)
Notify when OK: Optional
Notify when Resolved: Optional
Test Notification: Use this toggle and/or set up a test alert as described in the following section.
To test if this ServiceNow integration is set up and working correctly, you can set up a test alert to trigger.
For example, you could create an alert for CPU usage:
In ServiceNow, navigate to System Log >
All
to see a sample
triggered webhook.
Sysdig supports automatically sending alert notifications to an IBM Cloud Functions Channel. You generally use it for the following use cases.
Configure an IBM Functions as a new notification channel in Sysdig Monitor.
Pass extra parameters to IBM Functions.
Modify an IBM Functions.
Delete an IBM Functions.
The following notification channel types are supported:
Public (with or without X-Require-Whisk-Auth header)
Private (using IAM token)
To configure IBM Cloud Functions Channel:
Log in to the Sysdig UI and select IBM Cloud Functions Channel by completing steps 1-3 as described in Set Up a Notification Channel.
Specify the channel URL.
You can specify one of the following channel types.
Public: The public channels are of type IBM Web Action. They require no authentication and can be used to implement HTTP handlers that respond with headers, status code, and body content of different types.
URL: An example URL is https://eu-gb.functions.cloud.ibm.com/api/v1/namespaces/13eeeeee-623b-4776-ba35-4065bcbfee7b/actions/hello-world/myaction
Private: The private channels are of type IAM Secured action. These actions require IAM token-based authentication.
URL: An example URL is https://eu-gb.functions.cloud.ibm.com/api/v1/web/namespaces/13eeeeee-623b-4776-ba35-4065bcbfee7b/actions/hello-world/helloworld?param=true
Continue with one of the following:
Specify the following:
IAM API Key:
Channel Name: A unique name to identify the channel.
Enable the channel and desired notification types:
Enabled: The toggle button to enable or disable the IBM channel.
Notify when Resolved: Send a new notification when the alert condition is no longer triggered. Enable or disable the notification toggle as appropriate.
Notify when Acknowledged: Send a new notification when the alert is manually acknowledged by a user. Enable or disable the notification toggle as appropriate.
Test notification: Send a notification when the changes are saved. Enable or disable the notification toggle as appropriate.
Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Additional Parameters: Specify optional parameters to pass to
the function. For example, name: jane
is passed to the action as
{name: "Jane"}
.
Specify the following:
Whisk Auth Token (optional): Optionally provide the Whisk authentication token. If you specify one, the public channel (web action) can only be invoked by requests that provide appropriate authentication credentials. See Securing web actions for more details.
Channel Name: A unique name to identify the channel.
Enable the channel and desired notification types:
Enabled: The toggle button to enable or disable the IBM channel.
Notify when Resolved: Send a new notification when the alert condition is no longer triggered. Enable or disable the notification toggle as appropriate.
Notify when Acknowledged: Send a new notification when the alert is manually acknowledged by a user. Enable or disable the notification toggle as appropriate.
Test notification: Send a notification when the changes are saved. Enable or disable the notification toggle as appropriate.
Shared With: Choose whether to apply this channel globally (All Teams) or to a specific team from the drop-down.
Additional Parameters: Specify optional parameters to pass to
the function. For example, hostname: BLR
is passed to the action
as {hostname: "BLR"}
. The URL would be
/demo/hello.http?hostname=BLR
.
To temporarily disable a notification channel:
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings.
Select Notification Channels.
Toggle the Enabled
slider to off.
Administrators can choose to turn off all alert events and notifications if desired, for example, during a scheduled system downtime.
Muting notifications affects all channels globally. When muting is switched on, no notifications will be sent through any configured channel. You can choose whether to notify specific channels that notifications are temporarily disabled. Muting and re-enabling notifications is a MANUAL process.
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings.
Select Notification Channels.
Select the Downtime
toggle.
Optional: check the Yes
box to Notify Channels
when
prompted, and select the desired channels.
At this time, only Email and Slack channels can be notified when downtime is started/stopped.
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings.
Select Notification Channels
.
Select the three dots next to a created channel and click
Delete Channel
.
Sysdig alert jobs begin immediately at start-up. However, in instances where Sysdig goes down unexpectedly, or without proper shutdown/startup procedures implemented, data can be missing, triggering alert notifications.
A start-up delay in alert jobs can be configured in on-premises
environments, by setting the draios.alerts.startupDelay
parameter. The
parameter requires a duration value; the example below shows a duration
of 10 minutes:
draios.alerts.startupDelay=10m
This parameter can be configured for either Replicated or Kubernetes environments:
For Replicated environments, add the parameter to the Sysdig application JVM options list. For more information, refer to the Sysdig Install with Replicated documentation.
For Kubernetes environments, add the parameter to the
**sysdigcloud.jvm.worker.options
**parameter in the configmap
.
For more information on editing the configmap
refer to the
On-Premises
Installation
documentation.
When the Sysdig agent is installed in an AWS environment, the Sysdig Platform can collect both general metadata and various types of CloudWatch metrics.
There are three ways to integrate an AWS account into Sysdig:
By manually entering an AWS access key and secret key, and manually managing/rotating them as needed
By passing a parameter that allows Sysdig to autodetect an AWS ECS role and its permissions, passing an “implicit key” (On-Prem only).
The implicit option requires no manual key rotation as AWS handles those permissions behind the scenes.
Using AWS Role delegation. Role delegation is an alternative to the existing integration methods using the access keys. This method is considered secure as sharing developer access keys with third-parties is not recommended by Amazon.
The Sysdig Monitor UI includes links to help easily integrate CloudWatch metrics into Sysdig Monitor, as described below.
After integrating with an AWS account, data will become visible in the Sysdig UI after a 10-15 minute delay.
The Sysdig interface prompts you to perform this integration from the administrator’s Settings menu.
Once an agent has been installed, log in to Sysdig Monitor or Sysdig Secure as administrator to perform integration steps or review/modify existing AWS settings.
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings from the user menu.
Choose AWS Accounts.
A page showing manual key integration, with access key and secret key fields displayed.
NOTE: If there is no AWS integration yet then click on ADD and provide the access key and secret key.
Have your AWS EC2 account details available. Integration begins on the AWS side and is completed in the Sysdig Monitor UI.
You could use the existing IAMReadOnly policy instead, but creating a Sysdig-specific policy provides more granular access control, the activity can be easily distinguished in CloudTrail, and it is considered best practice.
In AWS, select IAM and create a policy to be used for Sysdig. (Sample policy name: SysdigMonitorPolicy.)
Using the JSON editor view, copy/paste the Sysdig-specific policy code into the new policy and save it.
You can review the policy in the Visual Editor.
When reviewing the completed policy in the Visual editor, you should see something like:
Use an existing IAM user, or (best practice) create a specific IAM user for the Sysdig Backend to programmatically access CloudWatch and use its data.
In the IAM Console, add a User.
Select AWS Access Type: Programmatic Access.
Select ‘Attach existing policies directly’, search for and then select the newly created policy (Sample policy name: SysdigMonitorPolicy.)
Select ‘Create User’ option.
Copy and save the resulting access key and secret key (Note: the Secret is only displayed once, so make sure to download the credentials file or store the key securely that you can reference again.)
Log in to Sysdig Monitor or Sysdig Secure as the administrator and select Settings from the user menu.
Select AWS.
Add an account by entering the User Access Key and Secret Key, then clicking Save.
The credentials will be listed with a Status of OK checked.
Should an Error occur, double-check the credentials entered. Mis-typing is the most common cause of errors.
Navigate to the AWS page in the Sysdig Monitor UI, if you are not already there.
Toggle the CloudWatch Integration Status to Enabled.
Sysdig Monitor will poll the CloudWatch API every five minutes. Note that this incurs additional charges from AWS.
After integrating with an AWS account, data will become visible in the Sysdig UI after a 10-15 minute delay.
If the integrated AWS account changes on the AWS side, an Error will be listed in the Credentials Status on the Settings > AWS page.
Use the Refetch Now
button to re-establish the integration.
If Sysdig is installed in an EC2 instance, you can take advantage of the existing EC2 IAM role of that instance. This can simplify administration, as you do not have to manually rotate public and private keys provided to the Sysdig backend.
Have your on-premises Sysdig platform installed in an AWS EC2 instance that has a proper IAM role.
For this option, you cannot use the AWS Integration step in the Welcome Wizard.
To enable implicit key, you must set the following parameter:
-Ddraios.providers.aws.implicitProvider=true
Use the parameter either during initial installation, or, if you already entered keys manually, to switch to an implicit key.
If switching, you must then restart the api, worker, and collector components in the backend.
In the Settings > AWS page, the former credentials will be overwritten it will show implicit key.
Enablement steps depend on whether you are using Kubernetes or Replicated as your orchestrator.
Edit the config.yaml
to add to the following entries (in the
Data
section of config.yaml
):
sysdigcloud.jvm.api.options:
sysdigcloud.jvm.worker.options:
sysdigcloud.jvm.collector.options:
If you are switching from manual to implicit keys, you must also restart the API, worker, and collector components.
See To Make Configuration Changes for details.
Enable Cloudwatch integration in the Sysdig UI.
Sysdig is designed to collect metadata for particular AWS services, which are reflected in the IAM policy code.
The services are:
DynamoDB
EC2 hosts
ECS
Elasticache
RDS
SQS
When you implement the code and integration steps as described above, it will trigger two types of collection: first the metadata for each service is collected, and then Sysdig will poll for the metrics about the metadata returned. So, if the service is not enabled in your environment, no metadata (and no metrics) are collected about it. If it is enabled, but you do not want to poll metrics, then delete the lines of code related to that service from the IAM policy. This will avoid potential unwanted AWS API requests and potential AWS charges.
If you have an on-premises Sysdig Backend, and have restricted outbound security groups, you may need to allow HTTPS & DNS access in order for the Sysdig Backend components to make connection to the Amazon APIs. As Amazon API endpoints are referenced by name and have a large number of IP’s, this may need to be full 0.0.0.0/0 outbound access for HTTPS & DNS.
If you need to filter just to Amazon IP ranges, you can use the following as a guide: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html
To enable metrics collection from only certain AWS regions in your environment, it is necessary to open a ticket with Sysdig Support. See Contact Support for details.
For information on the resulting AWS services visible in Sysdig Monitor, see the AWS-related information in the Metrics Dictionary (also available from within the Sysdig Monitor UI).
For information on how licensing affects AWS service views, see About AWS Cloudwatch Licensing.
Best Practice: Create a Sysdig-specific IAM policy to be used for granting programmatic access to Sysdig. Copy/paste the code snippet below into this policy. It enables Sysdig to collect metadata and CloudWatch metrics from the following services, as applicable to your environment:
Dynamodb
EC2 hosts
ECS
Elasticache
RDS
SQS
If you want to use your own AWS S3 bucket to store Sysdig capture files, you can append those code snippets to this IAM Policy as well. See Storage: Configure AWS Capture File Storage (Optional) for details.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"autoscaling:Describe*",
"cloudwatch:Describe*",
"cloudwatch:Get*",
"cloudwatch:List*",
"dynamodb:ListTables",
"dynamodb:Describe*",
"ec2:Describe*",
"ecs:Describe*",
"ecs:List*",
"elasticache:DescribeCacheClusters",
"elasticache:ListTagsForResource",
"elasticloadbalancing:Describe*",
"rds:Describe*",
"rds:ListTagsForResource",
"sqs:ListQueues",
"sqs:GetQueueAttributes",
"sqs:ReceiveMessage"
],
"Effect": "Allow",
"Resource": "*"
}
]
}
See Changing the AWS Services that are Polled for more detail.
This section describes how to configure Sysdig Monitor to utilize the Amazon Web Service (AWS) AssumeRole functionality and authorize Sysdig Monitor to discover cloud assets, grab CloudWatch metrics from your AWS account, and utilize custom S3 bucket for storing captures. Upon integrating with an AWS role, you can delegate access to AWS resources that are not associated with your Sysdig AWS account.
Setting up cross-account access through roles eliminates the need to create individual IAM users in each account. In addition, users don’t have to sign out of one account and sign in to another in order to access resources in different AWS accounts.
Role delegation is an alternative to the existing integration method using the access keys. This method is considered secure as sharing developer access keys with third-parties is not recommended by Amazon.
This topic assumes that you have the following ready and you are familiar with AWS.
Sysdig Monitor API Token
API endpoint. In this topic, it is referred to as {{host}}
SaaS: See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, in US East, the endpoints are:
Monitor: https://app.sysdigcloud.com
Secure: https://secure.sysdig.com
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
On-Prem: Depends on the on-prem deployment.
Administrator privileges to configure AWS integration
API client. Examples in this topic use curl
AWS account ID
SaaS: The default AWS account ID is 273107874544
(US East
region). For other regions, check AWS account
IDs .
On-Prem: Customer-specific.
This section describes how to enable AWS role delegation using an API.
Retrieve your external ID as follows:
curl -k --request GET \
--url host/api/users/me \
--header 'authorization: Bearer e71d7c0f-501e-47d4-a159-39da8b716f44' | jq '.[] | .customer | .externalId'
An example of External ID from the response will be
04acdd59-4c98-4d11-8ee5-424326248161
.
Integrating the Sysdig Platform with Amazon Web Services requires configuring role delegation using AWS IAM.
Create a new role in the AWS IAM Console:
For the role type, select Another AWS account.
(SaaS) Enter the Sysdig account ID for Account ID.
This means that you are granting read-only access to your AWS data.
Select Require external ID and enter the one you retrieved in the previous step. Leave MFA disabled.
Click Next: Permissions.
Create the following policies:
sysdig_cloudwatch
: Gives access to the list and describe
supported AWS resources and get CloudWatch metrics for them.
sysdig_s3
: Defines the bucket name where we wish to store the
captures
For more information on policies, see IAM Policy Code to Use.
For detailed instructions on how to create a policy, see Integrate AWS Account Manually.
If a policy has already been created, search for it on this page and select it, then skip to step. Otherwise, click Create Policy, which opens in a new window.
Click Review policy.
Name the policy and provide an apt description. For example,
sysdig_cloudwatch
.
Click Create Policy.
You can now close this window.
In the Create role window, refresh the list of policies and select the policies you just created.
Click Next: Review.
Give the role a name and an apt description. For example,
sysdig_role
.
Click Create Role.
Select Roles > sysdig-role.
Copy Role ARN.
Using the role that you have created, add an AWS account on the Sysdig Monitor side. Use the following API call:
curl --request POST \
--url {{host}}/api/providers \
--header 'authorization: Bearer e71d7c0f-501e-47d4-a159-39da8b716f44' \
--header 'content-type: application/json' \
--data '{"name": "aws","credentials": {"role": "<Role_ARN>"},"alias": "role_delegation"}'
Replace <Role_ARN>
with the one that you have copied in the previous
section.
The response lists all the providers. An example response is given below:
{
"provider": {
"id": 7,
"name": "aws",
"credentials": {
"id": "role_delegation",
"role": "arn:aws:iam::485365068658:role/sysdig-access3"
},
"tags": [],
"status": {
"status": "configured",
"lastUpdate": null,
"percentage": 0,
"lastProviderMessages": []
},
"alias": "role_delegation"
}
}
Verify the role delegation has been created.
Log in to Sysdig Monitor or Sysdig Secure as an administrator.
Do one of the following:
Select Integrations > AWS CloudWatch.
From the user menu, select Settings > AWS.
The role that you have been created will be added to the list of AWS Accounts.
Proceed to enable CloudWatch and AWS S3 bucket.
See AWS: Integrate AWS Account and CloudWatch Metrics (Optional) for more information.
Create an AWS user that will be used to fetch temporary credentials.
Assign a policy to the user to allow AssumeRole. For example:
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "arn:aws:iam::{ACCOUNT-ID}:role/{ROLE_NAME}*"
}
}
Make the access keys available to users from one of the sources:
Environment variables
Java system properties
Instance profile credentials delivered through the Amazon EC2 metadata service.
EC2 metadata service is recommended if the installation is on AWS.
Create Secret:
apiVersion: v1
kind: Secret
metadata:
name: aws-credentials
type: Opaque
data:
aws.accessKey: {{BASE64_ENCODED_ACCESS_KEY_ID}}
aws.secretKey: {{BASE64_ENCODED_ACCESS_KEY_SECRET}}
Expose variables in deployment descriptors (sysdigcloud-collector
,
sysdigcloud-worker
, sysdigcloud-api
) and reference values in the
newly created secret:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
key: aws.accessKey
name: aws-credentials
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
key: aws.secretKey
name: aws-credentials
Add variables to descriptors on each platform update until new variables are part of the installer.
The supported AWS are EC2, RDS, Elastic Load Balancer (ELB), ElastiCache, SQS, DynamoDB, and Application Load Balancer (ALB).
By default, all the resources are fetched for all regions supported by AWS. You can avoid this by whitelisting regions when creating a provider key via the API. Example body of the provider key request when whitelisting regions:
{
"name": "aws",
"credentials": {
"role": "arn:aws:iam::676966947806:role/test-assume-role"
},
"additionalOptions": "{\"regions\":[\"US_EAST_1\",\"US_EAST_2\"]}"
}
Use the AWS option in the Settings menu to configure AWS role delegation.
Log in to the Sysdig Monitor as an administrator
Do one of the following:
Select Integrations > AWS CloudWatch.
From the user menu, select Settings > AWS.
The AWS Account page is displayed.
Click Add Accounts.
The Identity Authentication page opens to the Role Delegation tab.
Specify the following:
Role ARN: The Role ARN associated with the role you have created for role delegation. The ID is available on the summary page of the role on the AWS console. For more information, see Integrate with AWS Role Delegation.
AWS External ID: Ensure that AWS External ID is displayed on the page.
SaaS: For account IDs corresponding to your region, see SaaS Regions and IP Ranges
On-Prem: Customer-specific.
Click Save.
The Sysdig Capture feature allows you to record detailed system trace data via remote connection from any of your agent-installed hosts. In SaaS installations, by default, this data will be stored on Sysdig’s secure Amazon S3 storage location. This location will have a separate partition for your account. In on-premises installations, by default, the data will be stored in the Cassandra database.
This page describes two custom alternatives: using an AWS S3 bucket (available for SaaS and on-prem) and using custom S3 storage.
Storage Options | SaaS | On-Prem |
---|---|---|
Sysdig Provided Storage | Sysdig provided | Installation provided
|
AWS S3 |
| |
S3 Compatible |
if Google Cloud Storage is used as the S3 compatible storage, you will not be able to bulk delete captures due to compatibility issues with Google’s S3 API implementation. You can delete captures one by one or delete them directly from the Google console.
To configure this option, use the fields provided by Sysdig Settings UI and then append some code to the IAM Policy you created in AWS for Sysdig integration.
Your AWS account must be integrated with Sysdig, but the CloudWatch feature is not required to be enabled.
See AWS: Integrate AWS Account and CloudWatch Metrics (Optional)
Ensure that your S3 bucket name is available.
To use your own AWS S3 bucket to store Sysdig capture files, append the following code snippets to the **AWS Identity and Access Management (IAM) **page.
{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:Put*",
"s3:List*",
"s3:Delete*",
"s3:Get*"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::BUCKET_NAME",
"arn:aws:s3:::BUCKET_NAME/*"
]
}
]
}
If you are using AWS KMS for AWS S3 encryption, ensure that necessary privileges are given to the Sysdig Account or Role to use the custom key.
Use the Key users option to do so:
Log in as an Administrator to Sysdig Monitor or Sysdig Secure.
From the user menu in the lower-left navigation, do one of the following:
Enable the **Use a custom S3 bucket
**toggle and enter your AWS
S3 bucket name.
When enabled, you will have the option to select between “Sysdig Monitor Storage” or your own storage bucket when configuring a file capture. See Create a Sysdig Capture File.
You can set up a custom Amazon-S3-compatible storage, such as Minio or IBM Cloud Object Storage, for storing Captures in a Sysdig SaaS deployment. The capture storage location can be used for both Sysdig Monitor and Sysdig Secure. This is an API-only functionality and currently, no UI support is available.
The following APIs are supported for this functionality:
List existing AWS integrations
Create a new AWS integration
Update an existing AWS integration
Configure storage configuration
Ensure that the feature is enabled for your account.
Use the access key and secret as the credentials.
Configure a new AWS integration. Set the skipFetch
field to
true. This will cause the AWS integration to ignore fetching data
from this account. Therefore, both the AWS metadata and AWS
CloudWatch will not be fetched and you can use this storage
exclusively for Sysdig Capture.
Configure the storage interface with the new account, by specifying the AWS integration ID to use to authenticate the endpoint, bucket name, and the path specified in the bucket.
The AWS account ID is currently shown as null on the UI.
The API returns the list of configured AWS integrations.
GET {{host}}/api/providers
Authorization: Bearer {{API_Token}}
Field | Response |
---|---|
| String The unique identifier of the integration. |
| String The name of the integration and by default is set to |
| String The dictionary containing the information about how Sysdig authenticates to AWS:
|
| Boolean
|
| String Status denotes the status of the integration. |
| String The unique identifier of the AWS account. The value will be |
| Ignore this deprecated field. |
{
"providers": [
{
"id": 2398,
"name": "aws",
"credentials": {
"id": "AKIA4JRXW5ZVZU6MHNPE",
"role": null
},
"skipFetch" : false,
"status": {
"status": "done",
"lastUpdate": 1617274193293,
"percentage": 100,
"lastProviderMessages": []
},
"alias": null,
"accountId": "845151661675"
}
]
}
POST {{host}}/api/providers
content-type: application/json
Authorization: Bearer {{API_Token}}
{
"name":"aws",
"skipFetch": false,
"credentials": {
"id":"<AWS_Access_Key_ID>",
"role":null,
"key":"<AWS_SecretKey>"
}
}
Field | Description |
---|---|
| String The name of the integration and by default is set to |
| Boolean
|
| The dictionary containing the information about how Sysdig authenticates to AWS:
|
To update existing storage settings, perform a PUT HTTP call to the endpoint as follows:
PUT {{host}}/api/sysdig/settings
content-type: application/json
Authorization: Bearer {{API_Token}}
{
"enabled":true,
"buckets":[
{
"folder":"/folder1/folder2",
"name":"bucketName",
"providerKeyId": 3,
"endpoint": "http://127.0.0.1:9009"
}
]
}
Field | Description |
---|---|
| Boolean Indicates whether the custom storage is configured. If the value is |
| Returns the list of buckets that you can set. Currently, only one bucket is supported.
|
You can set up a custom Amazon-S3-compatible storage, such as Minio or IBM Cloud Object Storage, for storing Captures in a Sysdig on-premises deployment. The capture storage location can be used for both Sysdig Monitor and Sysdig Secure. This is an API-only functionality and currently, no UI support is available.
You must configure values.yaml
corresponding to your Sysdig
installation in order for this configuration to work.
Your on-premise installation is Installer-based. If you have installed Sysdig Platform manually and you want to configure custom S3 buckets to store your capture files, contact your Sysdig representative.
Ensure that AWS-client compatible credentials used for authentication are present in the environment.
Ensure that the list, get, and put operations are functional on the S3 bucket that you wish to use. Confirm this by using the S3 native tools, for example, as described in AWS CLI for IBM Cloud.
Configure the following parameters in the values.yaml
file so that
collectors, workers, and the API server are aware of the custom endpoint
configuration.
sysdig.s3.enabled
Required: true
Description: Specifies if storing Sysdig Captures in S3 or S3-compatible storage is enabled or not.
Options:true|false
Default:false
For example:
sysdig:
s3:
enabled: true
sysdig.s3.endpoint
Required: true
Description: The S3 or S3-compatible endpoint for the bucket. This option is ignored if sysdig.s3.enabled is not configured.
For example:
sysdig:
s3:
endpoint: <your S3-Compatible custom bucket>
sysdig.s3.capturesFolder
Required: false
Description: Name of the folder in S3 bucket to be used for storing captures, this option is ignored if sysdig.s3.enabled is not configured.
For example:
sysdig:
s3:
capturesFolder: my_captures_folder
The path to the capture folder in the S3 bucket will be {customerId}/{my_captures_folder}
. For on-prem deployments, the customerID is 1. If finance is your capture folder, the path to the folder in the S3 bucket will be 1/finance
.
sysdig.s3.bucketName
Required: true
Description: The name of the S3 or S3-compatible bucket to be used for captures. This option is ignored if sysdig.s3.enabled is not configured
For example:
sysdig:
s3:
bucketName: <Name of the S3-compatible bucket to be used for captures>
sysdig.accessKey
Required: true
Description: The AWS or AWS-compatible access key to be used by Sysdig components to write captures in the S3 bucket.
For example:
sysdig:
accessKey: <AWS-compatible access key>
sysdig.secretKey
Required: true
Description: The AWS or AWS-compatible secret key to be used by Sysdig components to write captures in the s3 bucket.
For example:
sysdig:
secretKey: <AWS-compatible secret key>
For example, the following AWS CLI command uploads a Sysdig Capture file to a Minio bucket:
aws --profile minio --endpoint http://10.101.140.1:9000 s3 cp <Sysdig Capture filename> s3://test/
In this example, the endpoint is http://10.101.140.1:9000/
and the
name of the bucket is test
.
When you finish the S3 configuration, continue with the instructions on on-premise installation by using the Installer.
SaaS customers of Sysdig can be identified by a unique customer number and name, provided by email when the Sysdig environment is first provisioned. While it is generally unnecessary to know the customer number and it is not prominently displayed in the user interface, some configuration settings may require it.
For on-premises environments, the customer ID will typically be 1
.
To retrieve the customer ID:
Log into the Sysdig interface.
Select one of the following from the user menu:
The Authentication section lists the Customer ID and Customer Name associated with your account.
To retrieve the customer ID:
Log into the Sysdig interface.
Navigate to the URL endpoint /api/user/me?_product=SDC
.
E.g. https://app.sysdigcloud.com/api/user/me?_product=SDC
The JSON file contents are displayed in the browser.
Find the customer:id
portion of the JSON to determine the customer
ID and name:
The Agent Installation page provides a shortcut for copy/pasting the necessary code lines for different flavors of agent installation.
You can also retrieve the agent access key (copy/paste).
This page can be hidden from non-admins if administrators choose. See also Change Admin Settings in the User Profile page.
To retrieve the key or use the agent install code snippets:
Log in to Sysdig Monitor or Sysdig Secure as an administrator.
Select one of the following:
Choose Agent Installation
.
Optional: Use the Copy button to copy the access key at the top of the page.
Optional: Review and use the sample code to install an agent, as listed.
See also: Getting Started with Sysdig Monitor and Getting Started with Sysdig Secure.Getting Started with Sysdig Monitor
With Free Tier, use Sysdig Secure for cloud functions free forever:
Test all the features of Sysdig Monitor and/or Sysdig Secure with the
free 30-day trial.
After 30 days, your account will be disabled and you can contact Sysdig
sales to upgrade to an Enterprise license.
For Sysdig Secure for cloud , the default config can audit a maximum of 50 cloud accounts. If this limit needs to be increased, please contact your account team. If you exceed the license purchased, Sysdig will not block cloud connection or stop the service and the account team will reach out to you.
You can license Sysdig Secure, Sysdig Monitor, or both (Sysdig Platform). For details, see https://sysdig.com/pricing/.
Log on as Administrator to Sysdig Monitor or Sysdig Secure.
From the Selector button in the left-hand navigation, choose
Settings > Subscription
.
Your current plan details are displayed.
You can license each of these elements independently:
Sysdig agents (host agents)
Cloud accounts (Secure only, see also: Data Sources)
Fargate tasks/serverless agents (Secure only, see also Serverless Agents)
The number of licenses purchased has the following effects on how Sysdig is used:
The agent count defines the maximum number of connected host agents you can deploy. E.g if you purchase 100 licenses, you can install 100 agents.
Fargate Tasks using Sysdig Serverless Agents: Defines the number of serverless agents connected to Sysdig backend.
Cloud accounts- licensed number: Number of cloud accounts you can connect to Sysdig backend.
The distinction between reserved and on-demand agents is financial, not technical; when on-demand agents are used they perform exactly like reserved agents.
Reserved agents are dedicated agents that are provisioned for a user regardless of usage. You can purchase reserved agents on a monthly or annual basis. As a Sysdig SaaS account administrator, you can increase your reserved agents at any time from within the Sysdig application.
On-demand agents are for short-term use and you pay only for what you use at an hourly rate. You have the ability to add and control on-demand agents. For example, an organization might schedule scale testing for two days and license an extra 500 on-demand agents for that time frame.
In the Sysdig application, use the Customize Your Plan > Enable On-demand Agents option on the Subscription page to add or remove agents. There is a hard limit of 500 agents for any account. If the total of reserved and on-demand exceeds this limit, you will not be able to purchase additional agents. On-demand agents are available only in Sysdig SaaS applications.
The Sysdig platform uses a concurrent licensing model in determining when to allow an installed agent to connect to the back-end servers and report on host metrics. This means you can install Sysdig agents onto any number of instances. However, only the licensed number of agents will be allowed to connect and send metrics for recording and reporting.
Agents connect on a “first-come, first-served” basis and in the event of an over-subscription (more agents wanting to communicate than are licensed) they will attempt to reconnect on a periodic basis. Once an existing communicating instance goes down and disconnects, the next agent attempting to connect will be allowed in.
To avoid having agents refused connection due to over-subscription, monitor the number of established and allowed connections. To see how many licenses are in use, see the Settings > Subscription page. Use this information to either purchase additional license capacity from the UI, or to shut down lower-priority agents via normal orchestration and system administration means.
Multiple Installs: An agent is essentially an “install” of the software. If your system changes external IP addresses, or if you shut down a VM image and bring it back up elsewhere, this will remain the same agent connection. However, identical installs that are simultaneously sending data (usually an accident) will be considered two connections. A MAC address is used to identify a host for licensing purposes.
Time Lag for License Release: When shutting down a host for any reason, the agent’s license will not be immediately released. This permits the agent to retain its licensing slot for short outages or a reboot. The time-out interval can take up to 20 minutes, and if the connection has not been re-established within the interval the license will be released for use by the next host waiting to connect.
Sysdig Monitor allows you to consume custom metrics through a flexible and cost-effective Time Series Billing model aligned with your usage. With the enhanced billing experience, you can view your time-series consumption at a glance, analyze trends, and change subscription plans if need be.
Use the Sysdig Subscription page to control your licensing, and thereby usage and spending. Based on your current subscription tier, time-series usage, and the number of active agents, you can estimate the bill and take further actions.
Time Series Billing works only in SaaS environments and is not currently available in on-prem environments.
You consume more than the per-agent limit because Time Series Billing accounts for the following:
Time series through Monitoring Integrations
For example, Nginx and Redis.
Time series through Prometheus Remote Write
Time series through optional Pay as you Go (PAYG) and metric packs.
See Use Cases for more details.
Previously, the technical limit was 10K, no PAYG and metrics packs mechanisms, no system in place to bill metrics collected outside agents.
Validate what you are being charged on, understand and control metric usage, and drop the data that is not required, either by metric or by the scope of the metric. See Control Time Series Ingestion.
Pay as you go and metric pack.
Time series consumption is calculated by using the reserved time series included in the subscription. The basic plan includes 2000 time series per agent, and you can purchase more by adding on-demand agents or metric packs.
Sysdig meters and bills only custom metrics.
Prometheus
JMX
StatsD
App checks
The number of time series included with the subscription. The value is
calculated as
(the number of reserved agents + the number of connected on-demand agents) * the number of time series per agent
.
Time series consumed beyond your subscription limit will be charged and
is aggregated across all agents running in your environment. What it
means is that you can consume 3000 metrics on an agent and 1000 on
another without incurring additional charges.
Contact Sales to purchase beyond your subscription limit.
Time Series Billing limit of 2000 is applicable only to custom metrics, while Sysdig and Sysdig KSM are included at no additional charges.
A metric pack includes 1000 time series and is charged per month.
To help you so, Sysdig provides an at-a-glance visualization of the following:
Time Series Usage
Reserved: See Reserved Time Series.
Overage: Time series ingested beyond Reserved time series is Overage.
Ingested: The time series that are collected, analyzed, processed for storage.
Time Series Usage Dashboard
Reserved and On-Demand Agents
Agent Usage Dashboard
Usage history in CSV format
On the Subscription page, under Subscription Details , click the three dots.
Click Edit Subscription.
The Subscription Plan page gives you the directions to change the subscription plan.
To help you identify the usage trends that are important to you, Sysdig provides the following metrics:
sysdig_ts_usage
: The metric reports the number of time series
ingested for a user in a 20-minutes interval. The dashboard reports
the 1-hour usage, which is the sum of the maximum of three
20-minute sysdig_ts_usage
measurements taken in an hour. This
metric can be segmented on metric categories as well.
sysdig_ts_usage_10s
: The metric reports the number of time series
ingested for a user in every 10-seconds window, per host (agent),
and per metric category.
You can download the usage report in a CSV file. On the Subscription page, under Subscription Details, click Download Usage to download a copy of the usage report. You can view the following:
User ID
Time
Number of Reserved Agents
Number of Connected On-Demand Agents
Time Series included per agent
Total used time series
The ratio of used and reserved time series
Sysdig provides a Time Series Usage Dashboard with insight into the usage data. You can view time series ingestion at a glance and discover and analyze trends. The dashboard shows the average number of active time series per host; current ingestion rate; churn percentage; and so on.
On the Subscription page, under Usage, click Time Series Dashboard. You can view the following:
Current 1 Hour Ingestion
Current Ingestion from Agents
Churn Percentage
The average number of time series per host
The number of time series ingested per category
Host-level ingestion
Time series usage is calculated by using the sysdig_ts_usage
metric.
The metric reports the number of time series ingested for a user in an
hour (sum of the maximum of three 20-minutes). For each hour, the number
of time series ingested is calculated, and then the value is deducted
from the number of reserved time series. This value is stored as the
usage record.
An hour period is considered in order to take the
churn
into account. Sysdig uses the sysdig_ts_usage_10s
metric to calculate
the spikes caused by
churns
and provides you the churn percentage in the dashboard.
Sysdig uses the 95th percentile to calculate the exceeding cost of usage. At the end of the month, the 95th percentile of the total number of active series ingested per hour is calculated. Calculating the 95th percentile reduces the chances of billing you for unexpected spikes causes by churns.
When a time series stops receiving new data points, it becomes inactive. This event is called time series churn. It occurs more often during an upgrade in a Kubernetes cluster or when containers are replaced by new ones. Restarts, redeploys, dynamic workloads also cause churn and generate additional time series.
In such cases, the container_id
label in a container metric changes,
and subsequently, all the existing time series are replaced by new time
series (with the new container_id
value).
The churn rate is the number of time series that churn over time. Sysdig
uses the sysdig_ts_usage_10s
metric to analyze these scenarios.
The Time Series Usage Dashboard provides a ratio of time series detected at 1-hour period and 10-seconds period. This ratio is known as the churn percentage and it is expressed as this PromQL query:
(sum(sysdig_ts_usage{metric_category!='PROMETHEUS_REMOTE_WRITE'}) - sum(sysdig_ts_usage_10s)) / sum(sysdig_ts_usage{metric_category!='PROMETHEUS_REMOTE_WRITE'}) * 100
The time series collected by Prometheus Remote Write are excluded from this ratio because they are not collected by the Sysdig agent.
The billing is calculated per month. A basic subscription will provide you 2000 time series per agent. This limit is applicable only to custom metrics, while you can continue consuming Sysdig and KSM metrics without incurring additional costs. Time series consumed beyond your subscription limit will be charged and is aggregated across all agents running in your environment.
For example, if you have three agents running with the following consumption:
Agent 1 collecting 2000 time series per hour
Agent 2 collecting 1000 time series per hour
Agent 3 collecting 4000 time series per hour
Time series billing is calculated as follows:
Total consumption = 7000
Allowed number of time series per hour: 3 * 2000 = 6000
Effectively, you are paying only for (7000 - 6000) = 1000
because the
cost is calculated on the aggregated time series consumed across all the
agents running in your environment.
For more information on controlling metric usage, see the following:
See the following example with the following configuration:
Two Prometheus Servers
Prometheus Server 1 generates 50,000
time series
Prometheus Server 2 generates 150,000
time series
A Sysdig agent that collects 1000
time series
A subscription capacity of 2000
time series
The billing for the month is calculated as follows:
Time series usage: Total usage - Subscription capacity
(50,000 + 150,00 + 1000) - 2000 = 199,000
If the base price is $7.5 for up to a unit of 1K time series per month, the total base cost is calculated as follows:
The number of units consumed = (199,000 / 1000) = 199
The total cost = $7.5 * 199 = $1592.50
See the following example with the following configuration:
Two Prometheus Servers
Prometheus Server 1 generates 50,000
time series
Prometheus Server 2 generates 150,000
time series
A Sysdig agent that collects 1000
time series
100
metric pack, which is equivalent to 100000
time series
The billing for the month is calculated as follows:
Total subscription capacity: Total usage - (subscription capacity + time series from metric pack)
201,000 - (100,000 + 2000) = 99,000
If the base price is $7.5 for up to a unit of 1K time series per month and $5 for a metric pack of 1K time series, the total base cost is calculated as follows:
The number of units consumed = (199,000 / 1000) = 199
The total cost = Cost of metric pack + cost of total usage
($7.5 * 199) + (100 * $5) = $2092.50
In the Explore
tab or Dashboards
of Sysdig Monitor, the number of
metrics displayed for each AWS service is limited by the number of agent
licenses purchased and/or used, by region.
The license count:
Includes Reserved agents plus On-Demand agents (even if not in use).
Is used to determine how many AWS resources are displayed for each service in each region.
Is not transferable between different AWS services.
For each AWS service type, services are displayed in the following priority:
EC2: Pick instances with agents installed, then instances belonging to ECS, instance is launched before another, alphabetically by instance ID, up to license count.
RDS: Pick by creation time, oldest instances first, up to license count.
ELB: Pick by number of balanced instances (larger ELBs 1st), then by creation time, oldest first, up to license count.
ElastiCache: Sort by name and display up to license count items.
SQS: Sort queues by name and pick up to license count number of queues to fetch. Data is shown only for queues that are reporting metrics.
DynamoDB: Sort by name and display up to license count items.
ALB: Sort by name and display up to license count items.
For more information on AWS metrics, see AWS in the Metrics Dictionary.
Suppose you have 200 AWS instances, have purchased 100 Sysdig agent licenses, and have actually installed 50 agents.
The following limits would apply to your views of AWS services, per region:
EC2: The 50 instances with agents installed would be shown first, then 50 more instances, first from EC2, then from ECS, then per uptime.
RDS: 100 RDS listings would be shown, oldest first.
ELB: 100 ELBs would be shown (largest first), then by creation time, oldest first.
ElastiCache: 100 ElastiCache objects would be shown, alphabetically by name.
SQS: 100 SQS queues that are reporting metrics would be shown.
DynamoDB: 100 DynamoDBs would be shown, alphabetically by name.
ALB: 100 ALBs would be shown, alphabetically by name.
To increase the limit of items in the AWS Services views, contact Sysdig Sales to enable additional resources depending on your license agreement.
Sysdig Monitor and Sysdig Secure are designed to work with several user authentication/authorization methods:
Type | Enabled by Default | Integration Steps Required |
---|---|---|
User email/password | Yes | No |
Google OAuth | No | No |
SAML | No | Yes |
OpenID Connect | No | Yes |
The user’s view:
The pages in this section describe the integration and enablement steps required for SAML or OpenID Connect, and the Identity Provider (IdP) services that support these protocols, such as Okta, OneLogin, Keycloak.
In the SaaS environment, Google login can be enabled with a simple drop-down selection; the integration has already been performed.
See SaaS Regions and IP Ranges before proceeding to configure authentication.
To integrate SAML or OpenID Connect with both Sysdig Monitor and Sysdig Secure, you must go through the integration steps twice, once for each Sysdig product.
With the new Authorization UI, the basic process of enabling a Single Sign-On (SSO) option is:
Determine which SSO option (GoogleOAuth, SAML, OpenID) your enterprise uses, and which IdP service (Okta, OneLogin, etc.) is used, if any.
Enter the required connection settings for the chosen SSO on the
appropriate Authentication
tab. (Note: for Google, the settings
are already entered.)
Configure any associated IdP settings on the IdP side.
Select the SSO option from the Enabled Single Sign-On
drop-down
and click Save
Authentication.
If enabling for both Sysdig Monitor and Sysdig Secure, repeat the process on the second application.
View of the Authentication page for Google OAuth in the SaaS environment.
This guide is specific to cloud-based (SaaS) Sysdig environments. If you are configuring an On-Premises Sysdig environment, refer to Google OAuth (On-Prem) instead.
In the SaaS environment, Google users have the option to log in via Google OAuth.
As the SaaS platform is preconfigured to permit such logins, environments that already use Google services (such as G Suite) may find this the most convenient approach for simplified login.
Since Google OAuth is pre-configured by Sysdig, the administrator needs only select it as the chosen Authentication option to enable it.
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings.
Select Authentication
.
(Select the Google OAuth
tab if you want to see the preconfigured
(un-editable) settings. )
Select Google OAuth
from the Enabled Single Sign-On
dropdown
and click Save Authentication
.
Repeat for Sysdig Monitor or Sysdig Secure, if you want to enable on both applications.
Note the following requirements for successful Google OAuth login:
The user must have already logged in successfully at least once to your environment (such as via email-based Invitation and having set an initial password)
The user’s login username in the Sysdig platform must precisely match the user’s Google email address (that is, it cannot be a shortened/altered Google email alias)
For such a user to log in via Google OAuth, click the
Log in with Google
button.
If the user’s browser has not already successfully authenticated via Google and/or has multiple Google profiles known by their browser, they will be presented a Google page to select a profile and enter a password (if necessary) before being redirected back to your Sysdig environment.
See also User and Team Administration for information on creating users.
This guide is specific to cloud-based (SaaS) Sysdig environments. If you are configuring an On-Premises Sysdig environment, refer to SAML (On-Prem) instead.
SAML support in the Sysdig platform allows authentication via your choice of Identity Provider (IdP).
The Sysdig platform ordinarily maintains its own user database to hold a username and password hash. SAML instead allows for redirection to your organization’s IdP to validate username/password and other policies necessary to grant access to Sysdig application(s). Upon successful authentication via SAML, a corresponding user record in the Sysdig platform’s user database is automatically created, though the password that was sent to the IdP is never seen nor stored by the Sysdig platform.
This section describes how to integrate and enable SAML with both Sysdig Monitor and Sysdig Secure.
For specific IdP integration information, refer to:
See also Caveats
Step | Options | Notes |
---|---|---|
1. Know which IdP your company uses and will be configuring. | These are the IdPs for which Sysdig has performed detailed interoperability testing and confirmed how to integrate using their standard docs. If your IDP is not listed, it may still work with the Sysdig platform. Contact Sysdig Support for help. | |
2.Decide the login flow you want users to experience (choose from three options): | Click SAML button and enter a company name | Open the domain URL corresponding to your Sysdig application and region and enter your company name. For example, domain URLs of Monitor and Secure for US East are app.sysdigcloud.com and secure.sysdig.com respectively. Contact the Sysdig Support to set your company name on the account. This is applicable to all supported IdPs. |
Type/bookmark a URL in browser | For example, the URLs for the US East are: Monitor: https://app.sysdigcloud.com/api/saml/COMPANY_NAME?redirectRoute=%2F&companyName=COMPANY_NAME For example, for the EU region: Monitor: https://eu1.app.sysdig.com/api/saml/COMPANY_NAME?redirectRoute=%2F&companyName=COMPANY_NAME> For URLs corresponding to other regions, see SaaS Regions and IP Ranges. | |
Log in from an IdP interface | The individual IdP integration pages describe how to add Sysdig to the IdP interface. You will need your Sysdig customer number on hand. | |
3. Perform the configuration steps in your IdP interface and collect the resulting config attributes. | Collect metadata URL (or XML) and test it. If you intend to configure IDP-initiated login flow, have your Sysdig customer number on hand. It will be referenced in later configuration steps as | |
4 a. Log in to Sysdig Monitor or Sysdig Secure 4 b. Repeat process for the other Sysdig product, if you are using both Monitor and Secure. | You will enter a separate redirect URL in your IdP for each product; otherwise the integration processes are the same. |
Select the appropriate IdP from the list below, and follow the instructions:
To enable baseline SAML functionality:
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings
from the User Profile button in the left
navigation.
Select Authentication
.
Select the SAML
tab.
Enter the relevant parameters (see table below) and click Save
.
It is strongly recommended that “Signed Assertion” and “Validate Signature” are enabled to ensure that the SAML SSO process is as secure as possible.
Connection Setting | Options | Description | Sample Entry |
---|---|---|---|
Metadata | URL | The URL provided at the end of the IdP configuration steps. | |
XML | An option that can be used for an IdP that doesn’t support extracting metadata XML via URL. | ||
Signed Assertion | off/on | Should Sysdig check for assertions signed in responses (to assist in validating correct IdP). | ON |
Email Parameter | Name of parameter in the SAML response for user email ID. Sysdig uses this to extract the user’s email from the response. | ||
Validate Signature | off/on | Sysdig backend should verify that the response is signed. | ON |
Verify Destination | off/on | Flag to control whether Sysdig should check the “destination” field in the SAMLResponse. Recommend ON, as a security measure. May be OFF in special cases, such as a proxy in front of the Sysdig back end. | ON |
Create user on login | off/on | Flag to control whether a user record should be created in the Sysdig database after first successful SAML log in. | |
Disable username and password login | off/on | Switch “on” to disallow user name and password log in. (Useful with SAML OpenID.) |
Select SAML
from the Enabled Single Sign-On
dropdown
Click Save Authentication
.
Repeat entire enablement process for Sysdig Monitor or Sysdig Secure, if you want to enable on both applications.
Sysdig supports SAML Single Logout (SLO).
SLO is a feature in federated authentication where Sysdig users can sign out of both their Sysdig session (Service Provider) and associated IdP (Identity Provider) simultaneously. SLO allows you to terminate all sessions established via SAML SSO by initiating a single logout process. Closing all user sessions prevents unauthorized users from gaining access to Sysdig resources.
When a user initiates a logout, Sysdig sends a digitally-signed logout request to the IdP. The IdP validates the request and terminates the current login session, then redirects the user back to the Sysdig login page.
SLO is currently supported only in US-West and EU-Central regions.
Sysdig does not support HTTP Post binding for single logout, and therefore, SLO with Okta is not functional at this point.
Configure logout URLs:
Monitor: <base_URL>/api/saml/slo/logout
Secure: <base_URL>/api/saml/slo/secureLogout
Choose HTTP Redirect as the binding method.
This option is an alternative to the HTTP POST method, which Sysdig does not support currently.
If your IdP mandates, upload the public key for Sysdig.
Contact Sysdig Support to retrieve the public key associated with your deployment.
Certain IDPs, such as Azure, don’t require uploading the public key.
Log in to Sysdig Monitor or Sysdig Secure as an administrator and select Settings.
For on-prem deployments, log in as the super admin.
Navigate to Settings > Authentication, and select SAML under Connection Settings.
Enter the SAML configuration.
Ensure that Enable SAML single logout is toggled on.
Click Save.
Ensure that you select SAML from the Enable Single Sign On drop-down.
As noted in the Basic Enablement Workflow above, you can offer users three ways to log in with a SAML configuration:
They can begin at the Sysdig SaaS URL and click the SAML button.
See SaaS Regions and IP Ranges and identify the correct Sysdig SaaS URL associated with your Sysdig application and region. For example, URLs of Monitor and Secure for US East are:
Monitor: app.sysdigcloud.com
Secure: secure.sysdig.com
They will be prompted to enter a Company Name, so the Sysdig platform can redirect the browser to your IdP for authentication.
Contact Sysdig Support to set your company name on the account.
You can provide an alternative URL to avoid the user having to enter a company name, in the format:
Sysdig Monitor: https://app.sysdigcloud.com/api/saml/
COMPANY_NAME
Sysdig Secure: https://secure.sysdig.com/api/saml/
COMPANY_NAME
?product=SDS
For other regions, the format is https://<region>.app.sysdig.com/api/saml/auth. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Secure in the EU, you use https://eu1.app.sysdig.com/api/saml/secureAuth.
You can configure an IdP-initiated login flow when configuring your IdP. The users then select the Sysdig application from your IDP’s app directory and do not browse directly to a Sysdig application URL at all.
Users that complete their first successful SAML login to Sysdig Secure may receive the error message “User doesn’t have permission to login in Sysdig Secure”. This is because only members of the Secure Operations team are permitted access to Sysdig Secure, and newly-created logins are not in this team by default. Such a user should contact an Administrator for the Sysdig environment to be added to the Secure Operations team.
Environments that wish to have all users access Secure by default could use this sample Python script to frequently “sync” the team memberships.
See Developer Documentation for tips on using the sample Python scripts provided by Sysdig.
See also User and Team Administration for information on creating users.
Review SAML (SaaS) before you begin.
Configure Sysdig Monitor and/or Sysdig Secure as a SAML application using Okta’s documentation for Setting Up a SAML Application in Okta. The notes below call out specific steps that require additional action.
IDP-initiated Login Flow
If you don’t intend to configure IDP-initiated login flow, check the boxes for “Do not display application icon to users” and “Do not display application icon in the Okta Mobile app”.
URL, URI and RelayState Values
Enter the values shown in the table below. If you wish to configure
IDP-initiated login flow, replace CUSTOMER-ID-NUMBER
with the
number retrieved as described in Find Your Customer
Number.
See SaaS Regions and IP Ranges and identify the correct URLs associated with your Sysdig application and region. For example, in US East, the endpoints are:
Setting | Value for Sysdig Monitor | Value for Sysdig Secure |
---|---|---|
Single sign on URL |
|
|
Audience URI (SP Entity ID) |
|
|
Default RelayState (optional - only configure if you intend to use IDP-initiated login flow) | #/&customer= | #/&customer= |
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com/api/saml/auth.
Email and Name Values
Instead of the values shown in the Okta example, add the values:
Name | Value |
---|---|
email | user.email |
first name | user.firstName |
last name | user.lastName |
Note that the attributes are case sensitive, so use caution when entering them.
Only email
is required. However, including first/last name is
recommended, since these values will now be included in the records
created in the Sysdig platform’s database when new users successfully
login via SAML for the first time.
SAML Configuration Metadata Value
Copy the URL and paste in the Metadata entry on the SAML Configuration page in the SAML connection settings.
To ensure the metadata URL you copy at the end of the IDP configuration procedure is correct, you can test it by directly accessing it via your browser.
When accessing the URL, your browser should immediately download an XML file that begins similarly to the example shown below. No entry of credentials or other security measures should be required to successfully download it. If this is not the case, revisit the IDP configuration steps.
<?xml version= "1.0" ?> <EntityDescriptor xmlns= "urn:oasis:names:tc:SAML:2.0:metadata" entityID= "https://app.onelogin.com/saml/metadata/680358" > `<IDPSSODescriptor xmlns:ds=` `"http://www.w3.org/2000/09/xmldsig#" ` `protocolSupportEnumeration=` `"urn:oasis:names:tc:SAML:2.0:protocol"` `>names:tc:SAML:` `2.0` `:metadata` `" entityID="` ` https://app.onelogin.com/saml/metadata/ ` `680358` `">` ... |
Review SAML (SaaS) before you begin.
Configure Sysdig Monitor and/or Sysdig Secure as a SAML application using OneLogin’s article titled Use the OneLogin SAML Test Connector. The notes below call out specific steps that require additional action.
At the step for “Adding the SAML Test Connector”, select SAML Test Connector (IdP w/ attr w/ sign response). If you don’t intend to configure IDP-initiated login flow, uncheck the slider so it will no longer be “Visible in portal”.
At the “Test Connector Configuration Page”, enter the values shown in
the table below. If you wish to configure IDP-initiated login flow,
replace CUSTOMER-ID-NUMBER
with the number retrieved as described
in the Find Your Customer
Number article.
See SaaS Regions and IP Ranges and identify the correct URLs associated with your Sysdig application and region. For example, given below are the URLs for the US East region.
Field | Value for Sysdig Monitor | Value for Sysdig Secure |
---|---|---|
RelayState (optional - only configure if you intend to use IDP-initiated login flow) | #/&customer= | #/&customer= |
Recipient |
|
|
ACS (Consumer) URL Validator |
|
|
ACS (Consumer) URL |
|
|
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com/api/saml/auth.
(Optional) If you want the user’s First Name and Last Name to be included in the records created in the Sysdig platform’s database when new users successfully login via SAML for the first time, click to the Parameters tab. Click Add parameter and create each of two New Fields, checking the box each time to Include in SAML assertion. Then click to Edit each field and select the Value shown from the drop-down menu before clicking Save.
Field Name | Value |
---|---|
first name | First Name |
last name | Last Name |
Note that the Field Names are case sensitive, so be careful to enter them as all lowercase.
The following shows an example of a correctly-configured field for First Name:
Click to the SSO tab, copy the Issuer URL, and paste in the Metadata entry on the SAML Configuration page in the SAML connection settings.
To ensure the metadata URL you copy at the end of the IDP configuration procedure is correct, you can test it by directly accessing it via your browser.
When accessing the URL, your browser should immediately download an XML file that begins similarly to the example shown below. No entry of credentials or other security measures should be required to successfully download it. If this is not the case, revisit the IDP configuration steps.
<?xml version= "1.0" ?> <EntityDescriptor xmlns= "urn:oasis:names:tc:SAML:2.0:metadata" entityID= "https://app.onelogin.com/saml/metadata/680358" > `<IDPSSODescriptor xmlns:ds=` `"http://www.w3.org/2000/09/xmldsig#" ` `protocolSupportEnumeration=` `"urn:oasis:names:tc:SAML:2.0:protocol"` `>names:tc:SAML:` `2.0` `:metadata` `" entityID="` ` https://app.onelogin.com/saml/metadata/ ` `680358` `">` ... |
Review SAML (SaaS) before you begin.
These instructions assume you already have a working, Internet-accessible ADFS ( Active Directory Federation Service) server. Interoperability testing has been performed specifically with ADFS on Windows Server 2012 R2.
Follow the instructions below to configure ADFS with the ADFS Management tool in the Windows Server Manager.
Right-click to Service > Edit Federation Service Properties.
Note the hostname in the Federation Service Identifier, as this will
be used in the metadata URL that you paste in the Metadata entry on
the SAML Configuration page in the Sysdig authentication settings.
Specifically, the metadata URL will be of the format
https://HOSTNAME/FederationMetadata/2007-06/FederationMetadata.xml
.
Also, so that the Sysdig platform can access this URL directly, this
host must resolve in DNS and have a valid (not self-signed)
SSL/TLS certificate.
Add a Relying Party Trust configuration for the Sysdig application.
Right-click to Relying Party Trusts > Add Relying Party Trust and click Start to begin the wizard.
In the Select Data Source step, click the button to Enter data about the relying party manually, then click Next
Enter a Display name of your choosing (e.g. “Sysdig Monitor” or “Sysdig Secure”), then click Next
Click Next to accept the default option to use AD FS profile
Click Next to skip the selection of an optional token encryption certificate (Sysdig does not support this option)
Check the box to Enable support for the SAML 2.0 Web SSO protocol, then enter one of the following values for Relying party SAML 2.0 SSO service URL:
If configuring Sysdig Monitor, enter:
https://app.sysdigcloud.com/api/saml/auth
If configuring Sysdig Secure, enter:
https://secure.sysdig.com/api/saml/secureAuth
Then click Next.
For the Relying party trust identifier, enter one of the following values:
If configuring Sysdig Monitor, enter:
https://app.sysdigcloud.com
If configuring Sysdig Secure, enter:
https://secure.sysdig.com
Then click Add, then click Next
Click Next to skip configuration of multi-factor authentication
Choose a policy for whether users will be permitted to login to the Sysdig application. The default to Permit all users to access the relying party will typically be acceptable. Click Next.
Review the summary and click Next to complete the configuration of the Relying Party Trust
The next step will involve adding Claim Rules, so you can leave the box checked to Open the Edit Claim Rules dialog and click the Close button to be brought immediately into the Claim Rules editor
Ensure that the SamlResponseSignature
option matches the Sysdig
authentication configuration.
Use the
Set-AdfsRelyingPartyTrust/Get-AdfsRelyingPartyTrust
cmdlets
via PowerShell to configure SamlResponseSignature
.
-SamlResponseSignature
Specifies the response signatures that the relying party expects. The acceptable values for this parameter are:
AssertionOnly
MessageAndAssertion
MessageOnly
For more information, see Set-AdfsRelyingPartyTrust.
Navigate to Settings > Authentication on the Sysdig
app and check the Sysdig authentication setting maps to the
SamlResponseSignature
:
For MessageAndAssertion
, enable both the options.
Next, use the Claim Rules to ensure that login data is sent as needed to the Sysdig platform. A user’s login to the Sysdig platform is based on an email address, and a default ADFS configuration would not send the email address as required. The following configuration ensures the correct field from Active Directory is delivered in the claim.
If not already in the Claim Rules editor from the previous step, navigate to it by right-clicking on the Relying Party Trust that was just created and selecting Edit Claim Rules
Click Add Rule. At the following screen, accept the default rule template to Send LDAP Attributes as Claims and click Next.
Enter a name for the rule, select Active Directory as the Attribute store, then use the pull-down selectors to pick E-Mail Address as both the LDAP Attribute and Outgoing Claim Type, then similarly make pull-down selections for Given Name and Surname. Once these selections are made, click Finish.
Now click Add Rule again, this time selecting the template for Transform an incoming claim
Enter a name for the rule, then use the pull-downs to select an Incoming claim type of E-Mail Address, an Outgoing claim type of Name ID, and an Outgoing name ID format of Email, then click Finish.
(Optional) If you want the user’s First Name and Last Name to be included in the records created in the Sysdig platform database when new users successfully login via SAML for the first time, additional Transform rules must also be created. Only the email-based username is strictly required and we already created a rule for this, so this step is optional.
If you wish to do this, click Add Rule and once again select the template for Transform an incoming claim. Enter a name for the rule, then use the pull-down to select an Incoming claim type of Given Name, and for the Outgoing claim type, directly type first name into the field. After clicking Finish, click Add Rule and create a similar rule to transform the Incoming claim type of Surname to the Outgoing claim type of last name.
Having clicked Finish after creating your last rule, you will see all rules now in the editor. Click Ok, and your ADFS configuration for your Sysdig application is complete.
(Optional) The steps above represent a Service-Provider-Initiated SAML configuration. If you would prefer an IdP-initiated SAML configuration, this is also possible with ADFS, but requires the additional steps described below.
The Sysdig platform requires a specific setting of RelayState in
order to accept IdP-initiated login flows. On the ADFS versions
tested, we’ve found this use of RelayState is disabled by default,
and a Microsoft
article
describes the topic in detail. To enable it, as described in a
Microsoft forum
thread,
on your ADFS host, edit
%systemroot%\ADFS\Microsoft.IdentityServer.Servicehost.exe.config
and add <useRelayStateForIdpInitiatedSignOn enabled="true" />
to the <microsoft.identityserver.web>
section. Once the
modification is saved, restart ADFS services for the change to take
effect.
You will need to retrieve your Sysdig customer number as described in the Find Your Customer Number article.
You will then need to generate an IdP-initiated login URL.
In addition to having the correct settings, it must be properly URL encoded. To ease this configuration, use this ADFS RelayState Generator tool. When launched, enter the values below, then hit the Generate URL button.
For the IDP URL String, enter
https://YOUR_ADFS_SERVER/adfs/ls/idpinitiatedsignon.aspx
For the Relying Party Identifier, enter one of the following values:
If configuring Sysdig Monitor, enter
https://app.sysdigcloud.com
If configuring Sysdig Secure, enter
https://secure.sysdig.com
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com/. See SaaS Regions and IP Ranges for more information.
For the Relay State/Target App, enter
#/&customer=CUSTOMER-ID-NUMBER
, substituting the
CUSTOMER-ID-NUMBER
you retrieved in the previous step
This Results URL will be used in the metadata URL that you paste in the Metadata entry in the SAML connection settings .
Use the Results URL from the tool to test your IdP-initiated
login. Note that per this Microsoft forum
thread,
it is apparently not possible to configure ADFS to use such a URL
when your users select the application from the pull-down menu at
https://YOUR_ADFS_SERVER/adfs/ls/idpinitiatedsignon.aspx
.
However, you may embed the URL into a custom portal or bookmarks
list.
Now you can test login using an Active Directory user that has an Email address configured.
To ensure the metadata URL you copy at the end of the IDP configuration procedure is correct, you can test it by directly accessing it via your browser.
When accessing the URL, your browser should immediately download an XML file that begins similarly to the example shown below. No entry of credentials or other security measures should be required to successfully download it. If this is not the case, revisit the IDP configuration steps.
<?xml version= "1.0" ?> <EntityDescriptor xmlns= "urn:oasis:names:tc:SAML:2.0:metadata" entityID= "https://app.onelogin.com/saml/metadata/680358" > `<IDPSSODescriptor xmlns:ds=` `"http://www.w3.org/2000/09/xmldsig#" ` `protocolSupportEnumeration=` `"urn:oasis:names:tc:SAML:2.0:protocol"` `>names:tc:SAML:` `2.0` `:metadata` `" entityID="` ` https://app.onelogin.com/saml/metadata/ ` `680358` `">` ... |
This topic explains how to configure SAML Single Sign On (SSO) with Azure Active Directory (AD) and helps you configure Sysdig to allow users to access Sysdig application by using SSO.
Administrator privileges on Sysdig and Azure.
Log in to the Azure AD portal.
Select Azure Active Directory, then click Enterprise Applications.
The Enterprise applications - All application screen is displayed.
Click New Application.
On the Add an Application screen, select Non-gallery application.
Give your application a name, and click Add at the bottom of the page.
On the menu, select Single sign-on.
Choose SAML as the sign-on method.
Edit the Basic SAML Configuration as follows:
In the configuration page, click the edit icon.
Specify the following:
Identifier (Entity ID): Uniquely identifies the Sysdig application. Azure AD sends the identifier to the Sysdig application as the audience parameter of the SAML token. Sysdig validates this as part of the SSO process.
For example, the identifier for Sysdig Monitor for the EU region is https://eu1.app.sysdig.com.
See SaaS Regions and IP Ranges for the complete list of entity IDs for different regions.
Reply URL: Specifies where Sysdig expects to receive the SAML token.
For example, the identifier for Sysdig Monitor for the EU region is https://eu1.app.sysdig.com/api/saml/auth.
See SaaS Regions and IP Ranges for the complete list of reply URLs for different regions.
Relay State: Specifies to the application where to redirect the user after authentication is completed. Typically the value is a valid URL for Sysdig. If you are configuring SSO for SaaS, change the relay state to reflect the correct customer number associated with your Sysdig application. For on-prem installations, the customer number is always 1.
The format is:
#/&customer=1234
Sign on URL: It is the sign-in page for the Sysdig application that will perform the service provider-initiated SSO. Leave it blank if you want to perform identity-provider-initiated SSO.
For more information on configuration parameters, see Configure SAML-based single sign-on to non-gallery applications.
Under SAML Signing Certificate, copy the App Federation Metadata URL.
Log in to your Sysdig instance as an admin.
For on-prem deployments, log in as the super admin.
Navigate to Settings > Authentication, and select SAML under Connection Settings.
Enter the following:
Metadata: Enter the App Federation Metadata URL you copied.
Email Parameter: Set the value to emailaddress.
Azure AD claims are:
saml = AD
givenname = user.givenname
surname = user.surname
emailaddress = user.mail
name = user.userprincipalname
Unique User Identifier = user.userprincipalname
In the Sysdig application, you need to set the email to
emailaddress
which is what Azure AD sends to Sysdig in the
SAML assertion. Alternatively, Azure AD can be modified to send
another attribute.
Click Save.
Select SAML from the Enable Single Sign On drop-down.
Log in to the Azure AD portal.
Click Azure Active Directory, and note down the domain name.
Select Azure Active Directory, then Users.
The Users - All Users screen is displayed.
Select New Users .
You can either create a new user or invite an existing AD.
Enter name, username, and other details, then click Create.
In the Profile page, add the Email and Alternate Email parameters. The values can match
Navigate to the Sysdig application.
Click Users and Group, then click the Add user button.
Select the Users and Groups checkbox, then choose the newly created user to add to the application.
Click Select, then Assign at the bottom of the screen.
Ensure that Flag to enable/disable create user on login is enabled. Typically this setting is enabled by default.
If you are using both Sysdig Monitor and Secure, ensure that the user accounts are created on both the products. A user that is created only on one Sysdig application will not be able to log in to another by using SAML SSO.
if you are on Sysdig Platform versions 2.4.1 or prior, contact Sysdig Support to help with user creation.
If Azure Active Directory does not allow you to create Sysdig as a Non- Gallery application, perform the following:
In Azure AD, click Enterprise Applications > New Application.
Select Application you’re developing.
You will be taken to the app registration page:
Select New Registration:
Provide a name for the application you are registering.
Enter the redirect URI.
For example, the redirect URI for Sysdig Monitor for the EU region is https://eu1.app.sysdig.com/api/saml/auth. See SaaS Regions and IP Ranges for the redirect URLs for other regions.
Click Register to complete the registration.
In the Overview tab click Add an Application ID URI:
Click Add a scope.
Add the application ID URI as follows:
https://<your_sysdig_url>:443
Replace <*your_sysdig_*url> with the URL appropriate to your application and region. See SaaS Regions and IP Ranges for more information.
In the Overview tab, click Endpoints, and copy the Federation Metadata URL.
Log in to Sysdig, navigate to SAML Authentication screen, and enter the Federation Metadata URL.
You will still need to ensure that the user creation on the login option is enabled.
Save the settings.
This guide is specific to cloud-based (SaaS) Sysdig environments. If you are configuring an On-Premises Sysdig environment, refer to OpenID Connect (On-Prem) instead.
OpenID support in the Sysdig platform allows authentication via your choice of Identity Provider (IdP). This section describes how to integrate and enable OpenID Connect with both Sysdig Monitor and Sysdig Secure.
The Sysdig platform ordinarily maintains its own user database to hold a username and password hash. OpenID instead allows for redirection to your organization’s IdP to validate username/password and other policies necessary to grant access to Sysdig application(s). Upon successful authentication via OpenID, a corresponding user record in the Sysdig platform’s user database is automatically created, though the password that was sent to the IdP is never seen nor stored by the Sysdig platform.
Step | Options | Notes |
---|---|---|
1. Know which IdP your company uses and will be configuring. | These are the OpenID Providers for which Sysdig has performed detailed interoperability testing and confirmed how to integrate using their standard docs. If your OpenID Provider is not listed (including ones that do not support OpenID Connect Discovery), it may still work with the Sysdig platform. Contact Sysdig Support for help. | |
2. Decide the login flow you want users to experience: 3 options | Click OpenID button and enter a company name | From app.sysdigcloud.com or secure.sysdig.com > page to enter company name. |
Type/bookmark a URL in a browser |
Contact Sysdig for the Company Name associated with your account. | |
Log in from an IdP interface | The individual IdP integration pages describe how to add Sysdig to the IdP interface. You will need your Company Name on hand. | |
3. Perform the configuration steps in your IdP interface and collect the resulting config attributes. | Collect metadata URL (or XML) and test it. If you intend to configure IDP-initiated login flow, you need the following:
| |
4 a. Log in to Sysdig Monitor or Sysdig Secure and configure authentication. 4 b. Repeat process for the other Sysdig product, if you are using both Monitor and Secure. |
You will enter a separate redirect URL in your IdP for each product; otherwise the integration processes are the same. |
Select the appropriate IdP link below, and follow the instructions:
To enable baseline OpenID functionality:
Log in to Sysdig Monitor or Sysdig Secure as administrator and
select Settings.
Select Authentication
.
Select the OpenID
tab.
Enter the relevant parameters (see table below) and click Save
.
Connection Setting | Description |
---|---|
Client ID | ID provided by your IdP |
Client Secret | Secret provided by your IdP |
Issuer URL | URL provided by your IdP. Example:https://YOUR-ONELOGIN-DOMAIN.onelogin.com/oidc |
Okta, OneLogin, and Keycloak support metadata auto-discovery, so these settings should be sufficient for those IdPs.
In some cases, an OpenID IdP may not support metadata auto-discovery, and additional configuration settings must be entered manually.
In this case:
On the OpenID tab, toggle the Metadata Discovery
button to OFF
to display additional entries on the page.
Enter the relevant parameters derived from your IdP (see table
below) and click Save
.
Connection Setting | Description |
---|---|
Base Issuer | Required. Often the same Issuer URL, but can be different for providers that have a separate general domain and user-specific domain (for example, general domain: https://openid-connect.onelogin.com/oidc, user-specific domain: https://sysdig-phil-dev.onelogin.com/oidc)f |
Authorization Endpoint | Required. Authorization request endpoint |
Token Endpoint | Required. Token exchange endpoint |
JSON Web Key Set Endpoint | Required. Endpoint that contains key credentials for token signature verification |
Token Auth Method | Authentication method. Supported values:
|
Select OpenID for SSO
Select OpenID
from the Enabled Single Sign-On
dropdown.
Click Save
Authentication.
Repeat entire enablement process for Sysdig Monitor or Sysdig Secure, if you want to enable on both applications.
As noted in the Basic Enablement Workflow above, you can offer users three ways to log in with an OpenID configuration:
They can begin at the Sysdig SaaS URL and click the OpenID button.
See SaaS Regions and IP Ranges and identify the correct SaaS URL associated with your Sysdig application and region. For example, URLs of Monitor and Secure for US East are:
Monitor: app.sysdigcloud.com
Secure: secure.sysdig.com
For other regions, the format is https://<region>.app.sysdig.com. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com.
They will be prompted to enter a Company Name, so the Sysdig platform can redirect the browser to your IdP for authentication.
=
You can provide an alternative URL to avoid the user having to enter a company name, in the format:
Monitor: https://app.sysdigcloud.com/api/oauth/openid/
CompanyName
Secure:
https://secure.sysdig.com/api/oauth/openid/
CompanyName
?product=SDS
You can configure an IdP-initiated login flow when configuring your IdP. The users then select the Sysdig application from your IDP’s app directory and do not browse directly to a Sysdig application URL at all.
See also User and Team Administration for information on creating users.
Review OpenID Connect (SaaS) before you begin.
The notes below describe minimal steps to be taken in Okta. You may need to adjust the steps based on the specifics of your environment.
Log in to your Okta organization as a user with administrative privileges and click to the Admin dashboard
Click on the Add Applications shortcut, then click the Create New App button
Select Web as the Platform type, then click OpenID Connect as the Sign-on method, then click Create
Create a new application:
Enter your choice of General Settings.
For Login redirect URIs, enter one of the following values:
See SaaS Regions and IP Ranges and identify the correct domain URL (redirect URL) associated with your Sysdig application and region. For example, domain URLs of Monitor and Secure for US East are:
Sysdig Monitor:
https://app.sysdigcloud.com/api/oauth/openid/auth
Sysdig Secure:
https://secure.sysdig.com/api/oauth/openid/secureAuth
For other regions, the format is https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com/api/oauth/openid/auth.
Click Save.
You should next be placed in a General tab. Take note of the Client ID and Client secret that are shown.
You will enter them on the OpenID Configuration page in the Sysdig authentication settings.
Click to the Sign On tab. Take note of the Issuer URL that is shown, as it will need to be sent to Sysdig Support.
You will enter it in the OpenID Configuration page in the OpenID settings.
Review OpenID Connect (SaaS) before you begin.
The notes below describe minimal steps to be taken in OneLogin. You may need to adjust the steps based on the specifics of your environment.
Log in to your OneLogin organization as a user with administrative privileges and click to Apps > Custom Connectors, then click the New Connector button.
Create a new Connector:
Enter your choice of connector name.
Select a Sign on Method of OpenID Connect.
For Redirect URI, enter one of the following values:
See SaaS Regions and IP Ranges and identify the correct domain URL (redirect URL) associated with your Sysdig application and region. For example, domain URLs of Monitor and Secure for US East are:
Sysdig Monitor:
https://app.sysdigcloud.com/api/oauth/openid/auth
Sysdig Secure:
https://secure.sysdig.com/api/oauth/openid/secureAuth
For other regions, the format is https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Secure you use https://eu1.sysdig.com/api/oauth/openid/secureAuth.
Click Save.
From the More Actions pull-down menu, select Add App to Connector
Click Save to add the app to your catalog. Once clicked, additional tabs will appear.
Click to the SSO tab. Change the setting in the Token Endpoint drop-down to POST, then click Save.
While still on the SSO tab, take note of the Client ID and Client Secret that are shown (click Show client secret to reveal it).
You will enter them in the OpenID settings.
Note that the Issuer URL will consist of
https://YOUR-ONELOGIN-DOMAIN.onelogin.com/oidc
You will enter them in the OpenID settings.
During testing, we’ve found OneLogin sometimes does not persist changes that are made in the OpenID Provider configuration. If you make changes to your OneLogin configuration and experience issues such as HTTP 400 Bad Request when attempting logins to your Sysdig application, you may need to delete your Custom Connector and App config in OneLogin and recreate it from scratch.
Review OpenID Connect (SaaS) before you begin.
The notes below describe minimal steps to be taken in Keycloak. You may need to adjust the steps based on the specifics of your environment.
Log in to your Keycloak server’s Administrative Console.
Select a realm or create a new one.
Click Clients
, then click the Create
button.
Enter the Client ID
of your choosing (e.g. “SysdigMonitor”) and
take note of it.
You will enter it in the OpenID Configuration page in the Sysdig Authentication Settings.
Make sure the Client Protocol
drop-down has openid-connect
selected. Click the Save
button.
Configure OpenID Connect client:
Click the toggle for Authorization Enabled
to ON
For Valid Redirect URI
, enter one of the following values:
See SaaS Regions and IP Ranges and identify the correct domain URL (Redirect URI) associated with your Sysdig application and region. For example, domain URLs of Monitor and Secure for US East are:
Sysdig Monitor:
https://app.sysdigcloud.com/api/oauth/openid/auth
Sysdig Secure:
https://secure.sysdig.com/api/oauth/openid/secureAuth
For other regions, the format is https://<region>.app.sysdig.com.
Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor you use https://eu1.app.sysdig.com/api/oauth/openid/auth.
Click Save .
Click to the Credentials
tab. Take note of the Secret
that is
shown.
You will enter it in the OpenID settings
Note that the Issuer URL
will consist of
https://KEYCLOAK_SERVER_ADDRESS/auth/realms/REALM_NAME,
where
KEYCLOAK_SERVER_ADDRESS
and REALM_NAME
are derived from your
environment where you just created the configuration. You will enter
it in the OpenID settings.
OpenID Connect is a security-token based extension of the OAuth 2.0 authorization protocol to do single sign-on. Azure Active Directory provides an implementation of OpenID Connect (OIDC) protocol and Sysdig supports it for single sign-on and API access to Sysdig application.
Enabling Azure OpenID Connect for single sign-on to Sysdig applications include configuration on the Microsoft Active Directory as well as on the Sysdig application.
Administrator privileges on Sysdig and Azure Active Directory (AD).
Log in to the Azure AD portal.
Search for Azure Active Directory and do one of the following:
Select your Active Directory service
Create a new one.
Click App registration > New registration.
In the Register an application page, specify the following:
Name: Display name to identify your Sysdig application. For example, Sysdig Secure.
Supported account types: For Sysdig SaaS, choose Accounts in this organizational directory only (Default Directory only - Single tenant). All user and guest accounts created in your active directory can use Sysdig application and API.
Redirect URI: Authenticated Sysdig users are redirected to this URI.
See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, domain URLs of Monitor and Secure for US East are:
For Sysdig Secure: https://secure.sysdig.com/api/oauth/openid/secureAuth
For Sysdig Monitor: https://app.sysdigcloud.com/api/oauth/openid/auth
For other regions, the format is:
https://<region>.app.sysdig.com
Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor you use https://eu1.app.sysdig.com/api/oauth/openid/auth.
For on-prem installations, the redirect URI will be deployment-specific.
You can add only a single redirect URI on this page. Use the Authentication page associated with your application to add additional redirect URIs.
Click Register.
Add additional redirect URIs.
Select your application from App registration.
Click Authentication from the left navigation.
Add the redirect URIs corresponding to Monitor and Secure.
Create a Secret for the Sysdig application.
It is a string that the Sysdig application uses to prove its identity when requesting a token.
Click Certificates & secrets.
Under Client Secrets, click New client secret.
Enter a description that identifies the secret and choose an expiration period.
Click Add.
Copy the client secret. You will need the client secret while configuring OpenID Connect SSO on the Sysdig application.
Copy the Client ID and OpenID Connect endpoints corresponding to the application that you have created.
Select your application from App registration.
Copy the Application (client) ID.
You will need the client ID while configuring OpenID Connect SSO on the Sysdig application.
Click Endpoints.
Copy the OpenID Connect metadata document and open it in a browser.
Copy the OpenID Connect URI (Issuer URI).
For example, https://login.microsoftonline.com/5a4b56fc-dceb-4a64-94ff-21e08e5892f5/v2.0
To enable Azure OpenID functionality on the Sysdig application, you need the following:
Client ID
Client Secret
Issuer URL.
See Enable OpenID in Settings to learn how to complete your configuration.
Sysdig Platform supports disabling password-based authentication on both SaaS and on-prem deployments. As an administrator (super administrator for on-prem), you can use either the Authentication option on the UI or the API to achieve it. This configuration is applicable to those who use single sign-on.
For On-Prem environments, see Disable Password Authentication.
You can use the UI to disable password authentication only for SAML and OpenID authentication methods. For Google Oauth, use the API method as given below.
As an administrator, perform the following:
As an administrator, perform the following:
Get the Sysdig Platform settings:
See SaaS Regions and IP Ranges and identify the correct domain URL associated with your Sysdig application and region. For example, for Sysdig Monitor on US East is:
GET https://app.sysdigcloud.com/api/auth/settings/
For other regions, the format is https://<region>.app.sysdig.com/api/auth/settings. Replace <region> with the region where your Sysidig application is hosted. For example, for Sysdig Monitor in the EU, you use https://eu1.app.sysdig.com/api/auth/settings.
Find the ID of the active SSO setup:
GET https://app.sysdigcloud.com/api/auth/settings/active
Retrieve the specific settings associated with the SSO setup:
GET https://app.sysdigcloud.com/api/auth/settings/{id}
The setting is displayed in a JSON file.
In the JSON file, change the following from false to true:
settings/forbidPasswordLogin: True
Update the setting with a request to the same URL with the same JSON, with the changed parameter. URL depends on the type of deployment.
PUT https://app.sysdigcloud.com/api/auth/settings/{id}
(For SaaS) When you want inactive sessions to deactivate after a time-out period, you can configure it on the Sysdig application. You can determine how long a user’s browser can be idle after which they will be automatically logged out from the session.
To do so
Log in to Sysdig Monitor or Sysdig Secure as administrator and select Settings.
Select Authentication.
Scroll down and locate the Session Expiration settings.
Specify the Session Expiration setting:
Enable session expiration by using the Terminate session after inactivity period (in minutes) of slider.
Specify the time-out period in minutes.
Click Save.
A theme specifies the visual appearance of the Sysdig applications. Theme Preferences allows you to change the look and feel of the Sysdig applications to match your visual and accessibility requirements. The list of available themes includes Light, Dark, Match OS Preferences.
To configure a theme:
Log in to Sysdig Secure or Sysdig Monitor.
Navigate to User Menu. Select Theme from the upper right corner of the User Menu, OR click the Settings gear and choose Users.
Select a desirable theme from the Theme Preferences drop-down.
Match OS Preferences: The theme will be aligned with that of your operating system. For example, if your Desktop theme is Dark, the app theme will also be set to Dark.
Your OS theme will override the application theme preferences. For example, changing the OS theme to Dark while your application theme preference is Light will automatically switch the application theme to Dark.
The Certificates Management module for Sysdig Secure provides a simple interface for administrators to create, upload, update, or delete the certificates that are used for content exported from the Sysdig environment.
Specifically, it:
At this time, the feature is for Sysdig Secure SaaS only, and is integrated with the appropriate event forwarding options:
(Note: Kafka authentication is handled through a different mechanism.)
Log in to Sysdig Secure as admin and navigate to Settings
from your user profile.
Select Certificates Management
.
Certificate creation requires several steps, defined below.
Thereafter, you can assign the certificate to the event forwarding integrations.
You must have a signed key and certificate from a Certificate Authority (CA), a step that your organization may already have done. If not, follow these steps:
Generate the CA key:
openssl genrsa -out ca.key 4096
Generate the CA certificate, setting the expiration to 10 years from now:
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -out ca.pem
You will be prompted to provide details to populate the certificate information. Be as thorough as possible. Save the resulting ca.pem
file.
The Certificates Management UI streamlines the process of obtaining a certificate-signing request (CSR).
Log in to Sysdig Secure as admin
and select Settings > Certificates Management
.
Select New CSR
and copy the text using the Copy and Next
button.
You will be prompted to leave the Sysdig UI to finish generating the certificate in an external tool.
Use a 3rd-party tool, such as OpenSSL or Digicert, to generate the .crt
file. Follow the instructions given with that tool, using the CSR you generated.
Return to the Certificates Management
page in the Sysdig Secure Settings
and either click Next
in the popup window or select Upload Certificate
.
Assign the certificate a meaningful name.
Click Upload and Create
.
The certificate will appear in the certificates list and can be applied as needed.
Log in to Sysdig Secure as admin and select Settings > Event Forwarding
.
Choose an existing or new integration for Splunk, Syslog, or Webhook.
Select the correct uploaded certificate from the Certificate
field and Save
.
Use this setting to create a customized login banner to help your organization comply with its security standards.
To create a custom banner:
Navigate to Settings > Login Message
from Sysdig Monitor or Sysdig Secure.
Enter your desired message text, using Markdown to format.
Click Preview
to review. Click Save
.
Log out and log back in to see the message displayed.
As of July 2020, the data retention policies for Sysdig Monitor and Sysdig Secure are as described below.
When a host or instance is no longer monitored (i.e. when an agent is uninstalled), the historical data continues to be retained for the stated times.
Essentials | Enterprise | |
---|---|---|
Metrics data | 10s : 4 hours 1min : 2 days 10 min : 2 weeks 1 hr: 2 months 1 day: 15 months | same as Essentials |
Events | Alert events: 30 days Custom events: 14 days OR 10M (per customer) | same as Essentials |
Captures | 90 days | same as Essentials |
Essentials | Enterprise | |
---|---|---|
Policy events | 1M events or 90 days | same as Essentials |
Activity audit | 5 days | 90 days |
Benchmarks | 30 days | 90 days |
Scan results The image eviction conditions above are applied simultaneously; the retention policy will trigger for the first one that matches. | Image data is kept for a maximum of 7 days. Sysdig Secure will retain a maximum of 3 tags per repository and a maximum of 3 different images per tag * Images used by a container that is monitored by a Sysdig agent (Runtime images) will always be kept, regardless of the limits above. | Image data is kept for a maximum of 90 days. Sysdig Secure will retain a maximum of 5 tags per repository and a maximum of 5 different images per tag * Images used by a container that is monitored by a Sysdig agent (Runtime images) will always be kept, regardless of the limits above. |
Vuln Management Reports | 14 days | same as Essentials |
Captures | 90 days | same as Essentials |
Sysdig provides a set of APIs for auditing and reporting on the use of the Sysdig platform itself. (This is in contrast to the Activity Audit or Kubernetes Audit Log features which audit activity on your target environments.)
The audit includes the following request methods against the Sysdig system:
PUT
POST
DELETE
PATCH
GET
The data retention for system audit info is 90 days.
Know your:
Command | Description |
---|---|
filter=source in ("auditTrail") | Informs the events feed API that you want to fetch auditTrail type of events |
{{host}} | Host of the region for which you want to fetch audit events e.g., https://app.sysdigcloud.com for AWS us-east |
{{from}} | (nanoseconds) Timestamp date range, e.g. from=1648477226000000000&to=164934122600000000 |
{{to}} | (nanoseconds) Timestamp date range, e.g. from=1648477226000000000&to=164934122600000000 |
{{limit}} | (integer) - upper bound is 999. Defines how many events you will receive, and is used in combination with offset. For example: offset=100&limit=100 (skip first 100 and show next 100) |
{{offset}} | (integer) Used when we implement paging; allows you to skip the first x events. For example, offset=100 will skip the first 100 events |
{{token}} | (string) - Sysdig Secure or Sysdig Monitor API token |
For Sysdig Secure
GET {{host}}/api/v1/secureEvents?filter=source in ("auditTrail")&from={{from}}&to={{to}}&limit={{limit}}
X-Sysdig-Product: SDS
Authorization: Bearer {{token}}
For Sysdig Monitor
GET {{host}}/api/v1/secureEvents?filter=source in ("auditTrail")&from={{from}}&to={{to}}&limit={{limit}}
X-Sysdig-Product: SDC
Authorization: Bearer {{token}}
auditTrail.entityType
is used if you want to list audit events only for a specific entity or list of entities. In this example, we want to fetch only auth
audit events.
X-Sysdig-Product:
= SDS
(Sysdig Secure) SDC
(Sysdig Monitor)
GET {{host}}/api/v1/secureEvents?filter=source in ("auditTrail") and auditTrail.entityType in ("auth")&from={{from}}&to={{to}}&limit={{limit}}
X-Sysdig-Product: SDS
Authorization: Bearer {{token}}
Some entities are also used in Sysdig Secure but are served from Monitor
ui_user_settings
user
policy
falco_rules_file
team
customer_settings
event
api_token
overview
datasource
secure_settings
inactivity_settings
plan_settings
role
application_user_settings
dashboard_template
dashboard
integration
auth
default_dashboard
downtime
alert
capture
alert_notification
agent
alert_template_group
auth_settings
auth_sso
aws_settings
falco_list
falco_macro
invoice
notification_channel
permission
provider
runtime_policy_rule
s3_settings
service_account
silencing_rule
subscription
usage_report
policy
falco
account
event
feature
health
networkSecurity
report
task
compliance
dataSource
forwarding_integration
framework
host
list
macro
networkTopology
policyTuner
provider
resource
rule
schema
user
Sysdig SaaS applications are deployed in five data center regions—US East (Virginia), US West AWS (Oregon), US West GCP (Dallas), AP Australia (Sydney) and the European Union (Frankfurt). At the data centers, Sysdig ensures the best security and compliance standards for your data. This page lists the current Sysdig SaaS domains and IP ranges for each region.
For code-based access: Note: The endpoints for Sysdig Monitor and Sysdig Secure are the same in the US West (AWS and GCP), AP Australia, and EU regions. When configuring code-based access to Sysdig Secure, use the endpoint rather than the website URL.
For Single Sign-On: Sysdig SaaS users require the website address to reach the Sysdig applications. Use the appropriate website URL while configuring a single sign-on.
Collector: Additionally, Sysdig agents in a SaaS-based deployment need to be able to reach the Sysdig collector. Depending on your network configuration, you might need to modify your firewall configuration to permit outbound connections from agents to the collector.
Inbound IP Addresses: The traffic originating from the Sysdig agent to the Sysdig backend is known as inbound traffic. Allow the agent to send communication outbound on TCP 6443 to the inbound IP ranges associated with your SaaS region.
Outbound IP Addresses: Also known as source IP addresses and all the traffic originating from the Sysdig backend hosted in each region flows through one of the corresponding source IP addresses.
Sysdig Application | Domain | IP Range |
---|---|---|
Sysdig Monitor | https://app.sysdigcloud.com | For US East, IPs are assigned dynamically as the system scales. Therefore, we cannot provide the source IP range of the originating traffic. |
Sysdig Secure | https://secure.sysdig.com | |
Sysdig Collector | collector.sysdigcloud.com | |
Node Image Analyzer | https://collector-static.sysdigcloud.com/internal/scanning/scanning-analysis-collector | |
Secure Feed (whitelist for the VM scanning engine) | https://secure-feeds-prod.s3.us-east-1.amazonaws.com |
Sysdig Application | Domain | IP Range |
---|---|---|
Sysdig Monitor | All the traffic originating from the US West datacenter will have one of the following source IP addresses:
The inbound IP addresses are:
| |
Sysdig Secure | Endpoint: https://us2.app.sysdig.com Website URL: https://us2.app.sysdig.com/secure/ | All the traffic originating from the US West datacenter will have one of the following source IP addresses:
The inbound IP addresses are:
|
Sysdig Collector |
| |
Node Image Analyzer | https://us2.app.sysdig.com/internal/scanning/scanning-analysis-collector | |
Secure Feed whitelist for the VM scanning engine | https://secure-feeds-production-us-west-2-263844535661.s3.us-west-2.amazonaws.com |
Sysdig Application | Domain | IP Range | |
---|---|---|---|
Sysdig Monitor | All traffic originating from the European Union (EU) datacenter will have one of the following source IP addresses:
The inbound IP addresses are:
| ||
Sysdig Secure | Endpoint: https://eu1.app.sysdig.com Website URL: https://eu1.app.sysdig.com/secure/ | All traffic originating from the European Union (EU) datacenter will have one of the following source IP addresses:
The inbound IP addresses are:
| |
Sysdig Collector |
| ||
Node Image Analyzer | https://eu1.app.sysdig.com/internal/scanning/scanning-analysis-collector | ||
Secure Feed (whitelist for the VM scanning engine) | https://secure-feeds-production-eu-central-1-263844535661.s3.eu-central-1.amazonaws.com |
Sysdig Application | Domain | IP Range |
---|---|---|
Sysdig Monitor | https://app.au1.sysdig.com | Outbound IPs: 13.236.248.84 13.236.151.38 13.54.145.96 The inbound IPs: 13.238.59.195 52.62.57.59 52.64.82.29 |
Sysdig Secure | Endpoint: https://app.au1.sysdig.com/ Website URL: https://app.au1.sysdig.com/secure/ | Outbound IPs: 13.236.248.84 13.236.151.38 13.54.145.96 The inbound IPs: 13.238.59.195 52.62.57.59 52.64.82.29 |
Sysdig Collector | ingest.au1.sysdig.com | |
Node Image Analyzer | https://app.au1.sysdig.com/internal/scanning/scanning-analysis-collector | |
Secure Feed (whitelist for the VM scanning engine) | https://secure-feeds-production-ap-southeast-2-263844535661.s3.ap-southeast-2.amazonaws.com |
Sysdig Application | Domain | IP Range |
---|---|---|
Sysdig Monitor | https://app.us4.sysdig.com | Outbound IP: 34.145.19.124 Inbound IP: 34.145.19.124 |
Sysdig Secure | Endpoint: https://app.us4.sysdig.com/ Website URL: https://app.us4.sysdig.com/secure/ | |
Sysdig Collector | ingest.us4.sysdig.com | Inbound IP: 34.145.123.253 |
Node Image Analyzer | https://app.us4.sysdig.com/internal/scanning/scanning-analysis-collector | |
Secure Feed (whitelist for the VM scanning engine) | https://storage.googleapis.com/us4-prod-usw1-e33c-us-west1-us-secure-feeds |
Sysdig Agent uses the following ports to communicate with the Sysdig Collector.
Regions | Port |
---|---|
US East |
|
US West |
|
EU |
|
Asia Pacific (Sydney) |
|
US West (GCP) |
|
Regions | AWS Account IDs |
---|---|
US East | 273107874544 |
US West | 263844535661 |
EU | 263844535661 |
Use the following Prometheus endpoints for Grafana integrations.
Region | Endpoint |
---|---|
US East | https://app.sysdigcloud.com/prometheus |
US West | https://us2.app.sysdig.com/prometheus |
US West (GCP) | https://app.us4.sysdig.com/prometheus |
EU Central | https://eu1.app.sysdig.com/prometheus |
Asia Pacific (Sydney) | https://app.au1.sysdig.com/prometheus |
The term “on-premises” (or “on-prem”) is both industry-standard and evolving, so it means different things to different people.
In the context of Sysdig, on-prem customers install and manage the Sysdig backend components as they see fit. This could be in a data center, or in an enterprise’s cloud-provider space, such as AWS or GKE.
Install and Upgrade information is now on GitHub.
With version 3.6.0, the Sysdig Platform can no longer be installed using Replicated.
As part of our continued focus on our customers, we are now offering oversight services for all on-premise installs and upgrades. Your Technical Account Manager (TAM), in conjunction with our support organization and Professional Services [where applicable], will work with you to:
Assess your environment to ensure it is configured correctly
Review your infrastructure to validate the appropriate storage capacities are available
Review and provide recommendations for backing up your Sysdig data
Work with you to ensure our teams are ready to assist you during the install and upgrade process
Provide the software for the install
Be available during the process to ensure a successful deployment
You can always review the process in the documentation on GitHub (v. 3.6.0+) or the standard docs site (for older versions).
If you are a new customer looking to explore Sysdig, please head over here to sign up for a trial on our SaaS Platform. Alternatively, you can contact us here.
Before installing an on-premises solution, review the Sysdig architecture, sizing tips, configuration options, and installation options.
Each on-premise release is tested on several platforms and Kubernetes orchestrators. You can find the the official matrix in the onprem-install-docs repository. Click the on-premise version and navigate to the release notes page to view the supported platforms.
The actual installation instructions can be found in the onprem-install-docs repository.
Review the diagram and component descriptions. When installing on-premises, you can decide where to deploy various components.
Sysdig will collect monitoring and security information from all the target entities. To achieve this, one Sysdig agent should be deployed in each host. These hosts can be:
The nodes that make up a Kubernetes or OpenShift cluster
Virtual machines or bare metal
Living in a cloud environment (i.e. AWS, Google Cloud, IBM Cloud, Azure, etc.) or on the customer’s premises
The Sysdig agent can be installed as a container itself using a Helm chart, Kubernetes operator, etc.
Once the agent is installed in the host it will automatically start collecting information from the running containers, container runtime, the orchestration API (Kubernetes, OpenShift, etc), metrics from defined Prometheus endpoints, auto-detected JMX sources, StatsD, and integrations as well as the host itself.
The Sysdig agent maintains a permanent communication channel with the Sysdig backend which is used to encapsulate messages containing the monitoring metrics, infrastructure metadata, and security events. The channel is protected using standard TLS encryption and transports data using binary messages. Using this channel, the agent can transmit data, but also receive additional configuration from the backend, such as security runtime policies or benchmarks.
The Sysdig backend is used directly in its SaaS version, thus being managed transparently by Sysdig Inc., or it can also be installed on the customer’s premises. This distinction does not affect the operation of the platform described below.
Once the agent messages are received in the backend, they are processed and extracted into data available to the platform - time series, infrastructure and security events, and infrastructure metadata.
The main components of the backend/platform include:
Extraction and post-processing of the metric data from the agent, so that full time-series, with all the necessary infrastructure metadata, is available to the user
Maintenance of the infrastructure metadata (most notably Kubernetes state), so that all events and time series can be enriched and correctly grouped
Storage of time-series and event data
Processing of time-series data to calculate alert triggers
Queuing the security events triggered by the agents to be shown on the event feed, notifying by the configured notification channels and alerts and forwarding via the Event Forwarder to external platforms like Splunk, Syslog or IBM MCM / Qradar
Aggregating and post-processing other security data such as container fingerprints that will be used to generate container profiles, or security benchmark results.
The Sysdig platform then stores this post-processed data in a set of internal databases that will be combined by the API service to create the data views, such as dashboards, event feeds, vulnerability reports, or security benchmarks.
The Sysdig platform provides several ways to consume and present its internal data. All APIs are RESTful, HTTP JSON-based, and secured using TLS. The same APIs are used to power the Sysdig front end, as well as any API clients (such as sdc-cli).
Monitor API
User and Team management API
Dashboard API
Events API
Alerts API
Data API (proprietary Sysdig API for querying time-series data)
Secure API
Image Scanning API
Security Events API
Activity Audit API
Secure Overview API
PromQL API: Prometheus compatible HTTP API for querying time -series data
These enable different use cases:
User access to the platform via the Sysdig user interface
Programmatic input and extraction of data, i.e.
Automatic user creation
Terraform scripts to save or recover configuration state
Inline scanning to push scanning results from the CI/CD pipeline
Instrumentation using the sdc-cli.
PromQL API interface that can be used to connect any PromQL-compatible solutions, such as Grafana.
A 64-bit Linux distribution with a minimum kernel version of 3.10, and support of docker-engine 1.7.1 or later, is required for each server instance.
Recommended Linux distributions: RedHat, Ubuntu, Amazon AMI, Amazon Linux 2.
For the Docker installation, running devicemapper in ’loopback mode' is not supported. It has known performance problems and a different storage driver should be used.
Please see this note from our Replicated infrastructure partner: devicemapper-installation-warning.
Installing the latest version of Docker is recommended.
Cassandra is used as the metrics store for Sysdig agents. It is the most dynamic component of the system, and requires additional attention to ensure that your system is performing well and highly responsive.
This component is stateful, and should be treated more carefully than stateless components. Cassandra sizing is based on a minimum replication factor as well as the number of agents writing data.
A minimum replication factor of 3 is recommended for the Sysdig application, which allows the cluster to survive the failure of 1 Cassandra instance.
Each agent consumes anywhere from 500MB to 2GB of Cassandra storage, with average sizing at 1.5GB/agent. Because of Sysdig’s data aggregation model, this storage should comfortably handle multi-year history. This needs to then be multiplied by the replication factor to determine the total disk space required. A rough calculation might be:
100 agents = 150GB raw, X replication factor of 3, = 450GB total
To be safe we recommend that you size some additional disk space as buffer (say 25-50%) on top of that.
The following firewall/security configurations are required for inbound and outbound traffic for the Sysdig platform:
Port | State | Direction | Description |
6666 | Open (optional) | Inbound | Agent communication (unencrypted) |
6443 | Open | Inbound | Agent Communication (TLS/encrypted) |
443 | Open | Inbound | Sysdig Monitor user-interface access inbound |
443* | Open | Outbound | *Optional, used if collecting AWS CloudWatch metrics. See also AWS: Integrate AWS Account and CloudWatch Metrics (Optional). |
443* | Open | Outbound | *Optional, needed if using Sysdig Secure Image Scanning to download vulnerability definitions. Must be open to Cloudflare IP ranges: https://www.cloudflare.com/ips/. |
8800 | Open | Inbound | Replicated Management Console access (for on-premises installations that don't use Kubernetes) |
Warning: Port 6666 should only be opened if agents will be communicating with the collectors without encryption.
Additional ports may need to be configured for the Replicated infrastructure manager. Refer to the Replicated port requirements documentation for more information.
All non-airgapped hosts require outbound HTTP/S internet access for:
License validation
Pulling Sysdig/Agent containers from the Docker hub repository
Release update checks
Note: Sysdig does not support HTTP/S proxies for Sysdig platform components.
In release #760 and newer of the Sysdig platform back-end, an option is available to configure outgoing HTTP/HTTPS connections to be made via proxy. This has been tested and supports outgoing web connections that are necessary to support the following features:
Notification Channels
PagerDuty
Slack
Amazon SNS
VictorOps
OpsGenie
WebHook
Gathering of AWS CloudWatch data
Capture storage to an AWS S3 bucket
Proxied web connectivity to support authentication mechanisms (SAML. OpenID Connect, OAuth) are not supported at this time.
The proxy settings are configured via the JVM options passed to the Sysdig software components. JVM options can be added/appended at any time (with a required restart).
In a Replicated on-premises install, use the Advanced Settings panel to enter JVM options in the Sysdig application JVM options field. (See “Define Advanced Settings” on Install Components (Replicated).)
If JVM settings have already been set, log in to the Replicated Management console and choose the Settings tab. At the bottom of the screen, check the box to Show Advanced Settings to reveal the configuration option.
In a Kubernetes-based on-premises install, set the sysdigcloud.jvm.options in the config.yaml used to set the ConfigMap:
# Optional: Sysdig Cloud application JVM options. For heavy load environments you'll need to tweak
# the memory or garbage collection settings
sysdigcloud.jvm.api.options: ""
sysdigcloud.jvm.worker.options: ""
sysdigcloud.jvm.collector.options: ""
Enter the proxy parameters, as in the example below.
This JVM options string will forward all HTTP and HTTPS traffic via outgoing port 8888 on a proxy at hostname proxy.example.com. The IP address may be specified instead of hostname.
-Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=8888 -Dhttps.proxyPort=8888 -Dhttps.proxyHost=proxy.example.com
# Optional: Sysdig Cloud application JVM options. For heavy load environments you'll need to tweak
# the memory or garbage collection settings
sysdigcloud.jvm.api.options: -Xms2048m -Xmx2048m -Dhttp.proxyHost=xxx.xxx.sysdig.com -Dhttp.proxyPort=80 -Dhttps.proxyHost=xxx.xxx.sysdig.com -Dhttps.proxyPort=80
Do not use local host or 127.0.0.1. By default, HTTP/HTTPS requests to localhost or 127.0.0.1 will not be directed by the back-end toward any configured proxy, which is necessary for the functioning of some web components internal to the Sysdig platform containers.
If you deploy the Sysdig platform in AWS, add an additional proxy parameter
-Dhttp.nonProxyHosts=169.254.169.254
Rational: This provides a work-around for the backend occasionally making HTTP requests to a special instance metadata address 169.254.169.254, which is undesirable when using a proxy.
This IP address will be excluded from proxying by default in a future release.
If you have additional proxy exclusions you wish to specify that are unique to your environment, these can also be added using the pipe separator.
For example, assume your deployment was in AWS and you also had a webhook target 192.168.1.2 that was not reachable via your proxy.To exclude both:
Replicated: your complete string to enter into the console for Sysdig application JVM options would be:
-Dhttp.proxyHost=proxy.example.com -Dhttp.proxyPort=8888 -Dhttps.proxyPort=8888 -Dhttps.proxyHost=proxy.example.com -Dhttp.nonProxyHosts=169.254.169.254|192.168.1.2
Kubernetes: when setting the sysdigcloud.jvm.api.options
and
sysdigcloud.jvm.worker.options
in the
config.yaml
for the ConfigMap, the pipe separator must be double-escaped, such
as:
-Dhttps.proxyPort=80 -Dhttps.proxyHost=xx.xx.sysdig.com -Dhttp.nonProxyHosts=169.123.169.123\\|127.0.0.1\\|localhost\\|.sysdig.com"
The Sysdig platform requires the system clocks to be closely synchronized between hosts. When provisioning hosts for installation, ensure the system clocks are synchronized.
Recommended: Install NTP to ensure all host clocks stay synchronized.
For MySQL, Redis, and the initial “super admin” user, a strong password is recommended, 16-20 characters, alphanumeric.
For Cassandra and MySQL, it is also possible to set up third-party authentication
For Redis, users can set up an SSH tunnel and Sysdig can connect over this tunnel.
When planning to install Sysdig products on-premises, enterprises should:
Organize resources for a test environment and the production environment
Understand the architecture, component requirements, and installation options in Architecture & System Requirements
Review the supported platforms and orchestrators. You can find the the official matrix in the onprem-install-docs repository. Click the on-premise version and navigate to the release notes page to view the supported platforms.
See the installation instructions in the onprem-install-docs repository.
Decide whether to install using Replicated orchestrator or Kubernetes
Consider the SSO options and plan accordingly. See Authentication and Authorization (On-Prem Options).
As part of our continued focus on our customers, we are now offering oversight services for all on-premise installs and upgrades. Your Technical Account Manager (TAM), in conjunction with our support organization and Professional Services [where applicable], will work with you to:
Assess your environment to ensure it is configured correctly
Review your infrastructure to validate the appropriate storage capacities are available
Review and provide recommendations for backing up your Sysdig data
Work with you to ensure our teams are ready to assist you during the install and upgrade process
Provide the software for the install
Be available during the process to ensure a successful deployment
You can always review the process in the documentation on GitHub (v. 3.6.0+) or the standard docs site (for older versions).
If you are a new customer looking to explore Sysdig, please head over here to sign up for a trial on our SaaS Platform. Alternatively, you can contact us here.
For v 3.6.0+, go to the GitHub repo. On-prem installation documentation is transitioning to GitHub.
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
The Sysdig Installer tool is a binary containing a collection of scripts that help automate the on-premises deployment of the Sysdig platform (Sysdig Monitor and/or Sysdig Secure), for environments using Kubernetes or OpenShift. Use the Installer to install or upgrade your Sysdig platform. It is recommended as a replacement for the earlier Kubernetes manual installation and upgrade procedures.
To install, you will download the installer binary and a values.yaml file, provide a few basic parameters, and launch the Installer. In a normal installation, the rest is automatically configured and deployed.
You can perform a quick install if your environment has access to the internet, or a partial or full airgapped installation, as needed. Each is described below.
See Frequently Used Installer Configurations to:
Customize or override settings
Use hostPath for static storage of Sysdig components
Use Kubernetes node labels and taints to run only Sysdig pods on selected nodes in a cluster
With Sysdig Platform 3.5.0, the installer has been simplified from
previous versions. Upgrade differs from Install in that you run an
installer diff
to discover the differences between the old and new
versions and then installer deploy
for the new version.
If you are installing the Sysdig Platform for the first time, ignore the For Upgrade Only step in the process.
If you are upgrading, check the Upgrade notes before you begin.
The installer must be run from a machine with kubectl/oc
configured
with access to the target cluster where the Sysdig platform will be
installed. Note that this cluster may be different than where the Sysdig
agent will be deployed.
Requirements for Installation Machine with Internet Access
Network access to Kubernetes cluster
Network access to quay.io
A domain name you are in control of.
Additional Requirements for Airgapped Environments
Edited values.yaml with airgap registry details updated
Network and authenticated access to the private registry
Access Requirements
Sysdig license key (Monitor and/or Secure)
Quay pull secret
You may use dynamic or static storage on a variety of platforms to store the Sysdig platform components (stateful sets). Different configuration parameters and values are used during the install, depending on which scenario you have.
Use Case 1: Default, undefined (AWS/GKE)
If you will use dynamic storage on AWS or GKE and haven’t configured any storage class there yet, then the Quick Install streamlines the process for you.
storageclassProvisioner:
Enter aws
or gke
. The installer will
create the appropriate storage class and then use it for all the
Sysdig platform stateful sets.
storageclassName
: Leave empty.
Use Case 2: Dynamic, predefined
It is also possible that you are using dynamic storage but have already created storage classes there. This dynamic storage could be AWS, GKE, or any other functioning dynamic storage you use. In this case, you would enter:
storageclassProvisioner
: Leave empty; anything put here would be
ignored.
storageclassName
: Provide the name of the pre-configured storage
class you want to use. The installer will use this storage class for
all the Sysdig platform stateful sets.
Use Case 3: Static Storage
In cases where dynamic storage is not available, you can use static storage for the Sysdig stateful sets. In this case, you would use:
storageclassProvisioner
: Enter hostpath
, then define the nodes
for the four main Sysdig components: ElasticSearch, Cassandra,
MySQL, and Postgres.storageclassProvisioner
See Frequently Used Installer Configurations for details.
This install assumes the Kubernetes cluster has network access to pull images from quay.io.
Have your Sysdig Technical Account Manager download the installer binary that matches your OS from the the sysdigcloud-kubernetes releases page.
For Upgrades Only: Copy the current version of values.yaml to your working directory.]
./installer-image import -n sysdig --certs-directory certs -o values.yaml
If you will be editing for an OpenShift installation and want to review a sample, see openshift-with-hostpath values.yaml. .
Edit the following values:
size: Specifies the size of the cluster. Size defines CPU, Memory, Disk, and Replicas. Valid options are: small, medium and large
quaypullsecret: quay.io provided with your Sysdig purchase confirmation mail
storageClassProvisioner: Review Storage Requirements, above.
If you have the default use case, enter aws
or gke
in the
storageClassProvisioner
field. Otherwise, refer to Use Case 2
or 3.
sysdig.license: Sysdig license key provided with your Sysdig purchase confirmation mail
sysdig.dnsname: The domain name the Sysdig APIs will be served on. Note that the master node may not be used as the DNS name when using hostNetwork mode.
sysdig.collector.dnsName: (OpenShift installs only) Domain name the Sysdig collector will be served on. When not configured it defaults to whatever is configured for sysdig.dnsName. Note that the master node may not be used as the DNS name when using hostNetwork mode.
deployment: (OpenShift installs only) Add
deployment: openshift
to the root of the values.yaml
file.
sysdig.ingressNetworking: The networking construct used to expose the Sysdig API and collector.Options are:
hostnetwork: sets the hostnetworking in the ingress daemonset and opens host ports for api and collector. This does not create a Kubernetes service.
loadbalancer: creates a service of type loadbalancer and expects that your Kubernetes cluster can provision a load balancer with your cloud provider.
nodeport: creates a service of type nodeport.The node ports can be customized with:
sysdig.ingressNetworkingInsecureApiNodePort
sysdig.ingressNetworkingApiNodePort
sysdig.ingressNetworkingCollectorNodePort
When not configured, sysdig.ingressNetworking
defaults to
hostnetwork
.
If doing an airgapped install , you would also edit the following values:
airgapped_registry_name: The URL of the airgapped (internal) docker registry. This URL is used for installations where the Kubernetes cluster can not pull images directly from Quay
airgapped_repository_prefix: This defines custom
repository prefix for airgapped_registry. Tags and pushes
images as
airgapped_registry_name/airgapped_repository_prefix/image_name:tag
airgapped_registry_password: The password for the configured airgapped_registry_username. Ignore this parameter if the registry does not require authentication.
airgapped_registry_username: The username for the configured airgapped_registry_name. Ignore this parameter if the registry does not require authentication.
[For Upgrades Only:]
[Generate and review the diff of changes the installer is about to introduce:
./installer diff
This will generate the differences between the installed environment and the upgrade version. The changes will be displayed in your terminal.
If you want to override a change, based on your environment’s custom settings, then contact Sysdig Support for assistance.]
Run the installer:
./installer deploy
See Output (below) to finish.
Save the values.yaml
file in a secure location; it will be used for
future upgrades.
There will also be a generated directory containing various Kubernetes
configuration yaml
files that were applied by the Installer against
your cluster. It is not necessary to keep the generated directory, as
the Installer can regenerate it consistently with the same
values.yaml
file.
The installer can be used in airgapped environments, either with a multi-homed installation machine that has internet access, or in an environment with no internet access.
This assumes a private docker registry is used and the installation machine has network access to pull from quay.io and push images to the private registry.
The Prerequisites and workflow are the same as in the Quickstart Install (above) with the following exceptions:
In step 2, add the airgap registry information
After step 3, make the installer push Sysdig images to the airgapped registry by running:
./installer airgap
That will pull all the images into the images_archive
directory as
tar
files and push them to the airgapped registry.
If you are upgrading, run the diff as directed in Step 4.
Run the installer:
./installer deploy
This assumes a private docker registry is used and the installation machine does not have network access to pull from quay.io, but can push images to the private registry.
In this situation, a machine with network access (called the “jump machine”) will pull an image containing a self-extracting tarball which can be copied to the installation machine.
Access Requirements
Sysdig license key (Monitor and/or Secure)
Quay pull secret
Anchore license file (if Sysdig Secure is licensed)
Requirements for jump machine
Network access to quay.io
Docker
Requirements for installation machine
Network access to Kubernetes cluster
Docker
Network and authenticated access to the private registry
Edited values.yaml with airgap registry details updated
Host Disk Space Requirements:/tmp
> 4 GB; directory from
which the installer is run >8GB; and /var/lib/docker
> 4GB.
NOTE: The environment variable TMPDIR
can be used to override
the /tmp
directory.
Retrieve Quay username and password from Quay pull secret. For example:
AUTH=$(echo <REPLACE_WITH_quaypullsecret> | base64 --decode | jq -r '.auths."quay.io".auth'| base64 --decode)
QUAY_USERNAME=${AUTH%:*}
QUAY_PASSWORD=${AUTH#*:}
Log in to quay.ioUse the username and password retrieved above.
docker login -u "$QUAY_USERNAME" -p "$QUAY_PASSWORD" quay.io
On the Jump Machine
Follow the Docker Log In to quay.io steps, above.
Pull the image containing the self-extracting tar:
docker pull quay.io/sysdig/installer:5.1.2-1-uber
Extract the tarball:
docker create --name uber_image quay.io/sysdig/installer:5.1.2-1-uber
docker cp uber_image:/sysdig_installer.tar.gz .
docker rm uber_image
Copy the tarball to the installation machine.
On the Installation Machine:
Copy the current version values.yaml to your working directory.
wget https://raw.githubusercontent.com/draios/sysdigcloud-kubernetes/installer/installer/values.yaml
Edit the following values:
size: Specifies the size of the cluster. Size defines CPU, Memory, Disk, and Replicas. Valid options are: small, medium and large
quaypullsecret: quay.io provided with your Sysdig purchase confirmation mail
storageClassProvisioner: Review Storage Requirements, above.
If you have the default use case, enter aws
or gke
in the
storageClassProvisioner
field. Otherwise, refer to Use Case 2
or 3.
sysdig.license: Sysdig license key provided with your Sysdig purchase confirmation mail
sysdig.dnsname: The domain name the Sysdig APIs will be served on. Note that the master node may not be used as the DNS name when using hostNetwork mode.
sysdig.collector.dnsName: (OpenShift installs only) Domain name the Sysdig collector will be served on. When not configured it defaults to whatever is configured for sysdig.dnsName. Note that the master node may not be used as the DNS name when using hostNetwork mode.
deployment: (OpenShift installs only) Add
deployment: openshift
to the root of the values.yaml
file.
sysdig.ingressNetworking: The networking construct used to expose the Sysdig API and collector.Options are:
hostnetwork: sets the hostnetworking in the ingress daemonset and opens host ports for api and collector. This does not create a Kubernetes service.
loadbalancer: creates a service of type loadbalancer and expects that your Kubernetes cluster can provision a load balancer with your cloud provider.
nodeport: creates a service of type nodeport.The node ports can be customized with:
sysdig.ingressNetworkingInsecureApiNodePort
sysdig.ingressNetworkingApiNodePort
sysdig.ingressNetworkingCollectorNodePort
airgapped_registry_name: The URL of the airgapped (internal) docker registry. This URL is used for installations where the Kubernetes cluster can not pull images directly from Quay
airgapped_repository_prefix: This defines custom
repository prefix for airgapped_registry. Tags and pushes
images as
airgapped_registry_name/airgapped_repository_prefix/image_name:tag
airgapped_registry_password: The password for the configured airgapped_registry_username. Ignore this parameter if the registry does not require authentication.
airgapped_registry_username: The username for the configured airgapped_registry_name. Ignore this parameter if the registry does not require authentication.
Copy the tarball file to the directory where you have your
values.yaml
file.
Run:
installer airgap --tar-file sysdig_installer.tar.gz
NOTE: This step will extract the images into the
images_archive
directory relative to where the installer was run
and push the images to the airgapped_registry
.
[For Upgrades Only:]
[Generate and review the diff of changes the installer is about to introduce:
./installer diff
This will generate the differences between the installed environment and the upgrade version. The changes will be displayed in your terminal.
If you want to override a change, based on your environment’s custom settings, then contact Sysdig Support for assistance.]
Run the installer:
./installer deploy
See Output (below) to finish.
Save the values.yaml
file in a secure location; it will be used for
future upgrades.
There will also be a generated directory containing various Kubernetes
configuration yaml
files that were applied by the Installer against
your cluster. It is not necessary to keep the generated directory, as
the Installer can regenerate it consistently with the same
values.yaml
file.
NOTE: Sysdig Secure users who install in an airgapped environment do not have internet access to the continuous checks of vulnerability databases that are used in image scanning. (See also: How Sysdig Image Scanning Works.)
As of installer version 3.2.0-9, airgapped environments can also receive periodic vulnerability database updates.
When you install with the “airgapped_
” parameters enabled (see Full
Airgap
Install
instructions), the installer will automatically push the latest
vulnerability database to your environment. Follow the steps below to
reinstall/refresh the vuln db, or use the script and chron job to
schedule automated updates (daily, weekly, etc.).
To automatically update the vulnerability database, you can:
Download the image file quay.io/sysdig/vuln-feed-database-ubi:latest from the Sysdig registry to the jump box server and save it locally.
Move the file from the jump box server to the airgapped environment (if needed)
Load the image file and push it to the airgapped image registry.
Restart the pod sysdigcloud-feeds-db
Restart the pod feeds-api
The following script (feeds_database_update.sh
) performs the five
steps:
#!/bin/bash
QUAY_USERNAME="<change_me>"
QUAY_PASSWORD="<change_me>"
# Download image
docker login quay.io/sysdig -u ${QUAY_USERNAME} -p ${QUAY_PASSWORD}
docker image pull quay.io/sysdig/vuln-feed-database-ubi:latest
# Save image
docker image save quay.io/sysdig/vuln-feed-database-ubi:latest -o vuln-feed-database-ubi.tar
# Optionally move image
mv vuln-feed-database-ubi.tar /var/shared-folder
# Load image remotely
ssh -t user@airgapped-host "docker image load -i /var/shared-folder/vuln-feed-database-ubi.tar"
# Push image remotely
ssh -t user@airgapped-host "docker tag vuln-feed-database-ubi:latest airgapped-registry/vuln-feed-database-ubi:latest"
ssh -t user@airgapped-host "docker image push airgapped-registry/vuln-feed-database-ubi:latest"
# Restart database pod
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-db --replicas=0"
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-db --replicas=1"
# Restart feeds-api pod
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-api --replicas=0"
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-api --replicas=1"
Schedule a chron job to run the script on a chosen schedule (e.g. every day):
0 8 * * * feeds-database-update.sh >/dev/null 2>&1
A successful installation should display output in the terminal such as:
All Pods Ready.....Continuing
Congratulations, your Sysdig installation was successful!
You can now login to the UI at "https://awesome-domain.com:443" with:
username: "configured-username@awesome-domain.com"
password: "awesome-password"
There will also be a generated directory containing various Kubernetes
configuration yaml
files which were applied by installer against your
cluster. It is not necessary to keep the generated directory, as the
installer can regenerate consistently with the same values.yaml
file.
To see all the configuration parameters available, as well as their definitions, values, and examples, see configuration_parameters.md on GitHub.
For advanced options, including static storage and patching, see Frequently Used Installer Configurations.
Example values.yaml files:
The available fields for SMTP configuration are documented in the
configuration_parameters.md
. Each includes SMTP
in its name. For
example:
sysdig:
...
smtpServer: smtp.sendgrid.net
smtpServerPort: 587
#User,Password can be empty if the server does not require authentication
smtpUser: apikey
smtpPassword: XY.abcdefghijk...
smtpProtocolTLS: true
smtpProtocolSSL: false
#Optional Email Header
smtpFromAddress: sysdig@mycompany.com
To configure email settings to be used for a notification
channel, copy the
parameters and appropriate values into your values.yaml.
The available fields for AWS credentials are documented in the configuration_parameters.md. They are:
sysdig:
accessKey: my_awesome_aws_access_key
secretKey: my_super_secret_secret_key
The Installer assumes the usage of a dynamic storage provider (AWS or
GKE). In case these are not used in your environment, add the entries
below to thevalues.yaml
to configure static storage.
Based on the size
entered in the values.yaml
file
(small/medium/large
), the Installer assumes a minimum number of
replicas and nodes to be provided. You will enter the names of the nodes
on which you will run the Cassandra, ElasticSearch, mySQL and Postgres
components of Sysdig in the values.yaml,
as in the parameters and
example below.
Parameters
storageClassProvisioner:
hostPath.
sysdig.cassandra.hostPathNodes:
The number of nodes configured
here needs to be at minimum 1 when configured size is small
, 3
when configured size is medium
and 6 when configured size is
large
.
elasticsearch.hostPathNodes:
The number of nodes configured here
needs to be at minimum 1 when configured size is small
, 3 when
configured size is medium
and 6 when configured size is large
.
sysdig.mysql.hostPathNodes:
When sysdig.mysqlHA is configured to
true
, this must be at least 3 nodes. When sysdig.mysqlHA is not
configured, it should be at least 1 node.
sysdig.postgresql.hostPathNodes:
This can be ignored if Sysdig
Secure is not licensed or used in this environment. If Secure is
used, then the parameter should be set to 1, regardless of the
size
settingstorageClassProvisioner: hostPath
elasticsearch:
hostPathNodes:
- my-cool-host1.com
- my-cool-host2.com
- my-cool-host3.com
- my-cool-host4.com
- my-cool-host5.com
- my-cool-host6.com
sysdig:
cassandra:
hostPathNodes:
- my-cool-host1.com
- my-cool-host2.com
- my-cool-host3.com
- my-cool-host4.com
- my-cool-host5.com
- my-cool-host6.com
mysql:
hostPathNodes:
- my-cool-host1.com
postgresql:
hostPathNodes:
- my-cool-host1.com
If you have a large shared Kubernetes cluster and want to dedicate a few nodes for just the Sysdig backend component installation, you can use the Kubernetes concept of taints and tolerations.
The basic process is:
Assign labels and taints to the relevant nodes.
Review the sample node-labels-and-taints values.yaml in the Sysdig github repo.
Copy that section to your own values.yaml
file and edit with
labels and taints you assigned.
Example from the sample file:
# To make the 'tolerations' code sample below functional, assign nodes the taint
# dedicated=sysdig:NoSchedule. E.g:
# kubectl taint my-awesome-node01 dedicated=sysdig:NoSchedule
tolerations:
- key: "dedicated"
operator: "Equal"
value: sysdig
effect: "NoSchedule"
# To make the Label code sample below functional, assign nodes the label
# role=sysdig.
# e.g: kubectl label nodes my-awesome-node01 role=sysdig
nodeaffinityLabel:
key: role
value: sysdig
Patching can be used to customize or “tweak” the default behavior of the
Installer to accommodate the unique requirements of a specific
environment. Use patching to modify the parameters that are not exposed
by thevalues.yaml
. Refer to the configuration_parameters.md
for more
detail about various parameters.
The most common use case for patching is during upgrades. When generating the differences between an existing installation and the upgrade, you may see previously customized configurations that the upgrade would overwrite, but that you want to preserve.
If you have run generate diff
and found a configuration that you
need to tweak (e.g. the installer will delete something you want to
keep, or you need to add something that isn’t there), then follow these
general steps:
Create an overlays
directory in the same location as the
values.yaml
.
This directory, and the PATCH.yaml you create for it, must be kept. The installer will use it during future upgrades of Sysdig.
Create a .yaml
file to be used for patching. You can name it
whatever you want; we will call it PATCH.yaml for this example.
Patch files must include, at a minimum:
apiVersion
kind
metadata.name
of the object to be patched.
Then you add the specific configuration required for your needs. See one example below.
You will need this patch definition for every Kubernetes object you want to patch.
Run generate diff
again and check that the outcome will be what
you want.
When satisfied, complete the update by changing the scripts value to deploy and running the installer (see Installer Upgrade (2.5.0+).
If you want to add another patch, you can either add a separate
.yaml
file or a new YAML document separated by ---
.
The recommended practice is to use a single patch per Kubernetes object.
Presume you have the following generated configuration:
apiVersion: v1
kind: Service
metadata:
annotations: {}
labels:
app: sysdigcloud
role: api
name: sysdigcloud-api
namespace: sysdigcloud
spec:
clusterIP: None
ports:
- name: api
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: sysdigcloud
role: api
sessionAffinity: None
type: ClusterIP
Suppose you want to add an extra label
my-awesome-label: my-awesome-value
to the Service object. Then in the
PATCH.yaml, you would put the following:
apiVersion: v1
kind: Service
metadata:
name: sysdigcloud-api
labels:
my-awesome-label: my-awesome-value
Run the installer again, and the configuration would be as follows:
apiVersion: v1
kind: Service
metadata:
annotations: {}
labels:
app: sysdigcloud
role: api
my-awesome-label: my-awesome-value
name: sysdigcloud-api
namespace: sysdigcloud
spec:
clusterIP: None
ports:
- name: api
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: sysdigcloud
role: api
sessionAffinity: None
type: ClusterIP
Supposed you wanted to remove all the labels. Then in the PATCH.yaml, you would put the following:
apiVersion: v1
kind: Service
metadata:
name: sysdigcloud-api
labels:
Run the installer again, and the configuration would be as follows:
apiVersion: v1
kind: Service
metadata:
annotations: {}
name: sysdigcloud-api
namespace: sysdigcloud
spec:
clusterIP: None
ports:
- name: api
port: 8080
protocol: TCP
targetPort: 8080
selector:
app: sysdigcloud
role: api
sessionAffinity: None
type: ClusterIP
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
The Sysdig platform includes both Sysdig Monitor and Sysdig Secure, which are licensed separately. All installations include Sysdig Monitor, while some of the Secure components are installed and configured as additional steps, as noted.
When installing the Sysdig platform with Kubernetes as the
orchestrator, you install each backend component with separate kubectl
commands.
Installation with the Installer tool is recommended from version 2.5.0 onwards.
To perform a manual install on OpenShift, see Manual Install (OpenShift).
The manual install on Kubernetes 1.9+ is described below.
Access to a running Kubernetes cluster 1.9+
(Note: if your environment is installed elsewhere, such as your own data center, contact Sysdig Professional Services to customize the installation instructions appropriately.)
Two items from your Sysdig purchase-confirmation email:
Your Sysdig license key
Your Sysdig quay.io pull secret
kubectl
installed on your machine and communicating with the
Kubernetes cluster
(Note that your kubectl
and Kubernetes versions should match to
avoid errors.)
An External Load Balancer (required for production – see below)
If installing in a cloud-provider environment (such as AWS, GCloud, or Azure), you will deploy an HAProxy load balancer and point a DNS record to that load balancer.
If installing in your own data center, then you will need two DNS records, one for the collector and one for the UI.
A DNS server and control over a DNS name that you can point to Sysdig
By default, the Elasticsearch container will be installed in
privileged
(root-access) mode. This mode is only needed so the
container can reconfigure the hosts’ Linux file descriptors if
necessary. See Elasticsearch’s description
here.
If you prefer not to allow Elasticsearch to run with root access to the host, you will need to:
Set your own file descriptors on all Linux hosts in the Kubernetes cluster.
If one host were to go down, Kubernetes could choose a different node for Elasticsearch, so each Linux host must have the file descriptors set.
Set privileged:false
in the elasticsearch-statefulset.yaml
file.
See the step under Coonfigure Backend Components, below, for details.
If you are using EKS or GKE, default storage classes are provided; check for them (step 1).
In other environments, you may need to create a storage class (step 2).
Finally, enter the storageClassName
in the appropriate .yaml
files
(step 3).
Verify whether a storage class has been created, by running the command:
kubectl get storageclass
If no storage class has been defined, create a manifest for one, and then deploy it.
For example, a manifest could be named
sysdigcloud-storageclass.yaml
and contain the following contents
(for a storage class using GP2 volumes in AWS):
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2
annotations:
storageclass.beta.kubernetes.io/is-default-class: "true"
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
Now run the command:
kubectl apply -f sysdigcloud-storageclass.yaml
Sysdig provides the necessary scripts, images, and .yaml
files in a
GitHub repository. The first step is to clone those files and check out
the latest version. (These examples use 1234.)
Find the current release tag from https://github.com/draios/sysdigcloud-kubernetes/releases/latest.
Run the command:
git clone https://github.com/draios/sysdigcloud-kubernetes.git
cd sysdigcloud-kubernetes
git checkout tags/<1234>
Create a namespace called sysdigcloud
:
kubectl create namespace sysdigcloud
Create a TCP load balancer (i.e., AWS NLB) that forwards ports 80, 443,
6443 to the Kubernetes worker nodes, with a healthcheck to /healthz
on
port 10253.
This can be done in three ways:
Use an existing external load balancer. Sysdig relies heavily on DNS; you need a DNS record pointing to the load balancer.
Create a load balancer in your cloud provider. (For example in AWS,
see
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-network-load-balancer.html.)
You need a DNS record that points to the load balancer. This is the
fully qualified domain name required later in the
config.yaml, api-ingress.yaml
and/or
api-ingress-with-secure.yaml
.
Create a yaml with the following content and apply it to the sysdigcloud namespace. This automatically creates a load balancer in the cloud provider environment, with an external DNS name.
This is the fully qualified domain name required later in the
config.yaml, api-ingress.yaml
and/or
api-ingress-with-secure.yaml
.
---
apiVersion: v1
kind: Service
metadata:
name: haproxy-ingress-lb-service
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 80
- name: https
port: 443
targetPort: 443
- name: https2
port: 6443
targetPort: 6443
selector:
run: haproxy-ingress
Apply the changes to the sysdigcloud namespace.
kubectl -n sysdigcloud apply -f <yourlbfile.yamlservice.yaml>
To get the DNS name, run the command:
$ kubectl get svc -o wide -n sysdigcloud
The output shows the External-IP (DNS name):
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
haproxy-ingress-lb-service LoadBalancer 100.66.118.183 sample123.us-east-1.elb.amazonaws.com 80:31688/TCP,443:32324/TCP,6443:30668/TCP 1d run=haproxy-ingress
Not for production environments.
Create a DNS entry for your Sysdig install using the fully qualified domain name that contains all the external IPs as A records. This will use DNS round-robin to load balance your clients to the Kubernetes cluster.
The install images, scripts, and other files are located in a GitHub repository:https://github.com/draios/sysdigcloud-kubernetes
The ConfigMap
(config.yaml
)
is populated with information about usernames, passwords, SSL certs, and
various application-specific settings.
The steps below give the minimum edits that should be performed in a test environment.
It is necessary to review and customize the entries in config.yaml
before launching in a production environment.
See To Make Configuration
Changes
for the kubectl
format to use for post-install edits, such as adding
third-party authenticators like LDAP.
If you are not installing Sysdig Secure, set the following attributes to
false in the config.yaml
:
nats.enabled: "false"
nats.forward.enabled: "false"
Add your license key:
In config.yaml,
enter the key that was emailed to you in the
following parameter:
# Required: Sysdig Cloud license
sysdigcloud.license: "
Change the super admin name and password, which are the super admin credentials for the entire system. See here for details.
Find the settings in config.yaml
here:
sysdigcloud.default.user: test@sysdig.com
# Required: Sysdig Cloud super admin user password
# NOTE: Change upon first login
sysdigcloud.default.user.password: test
Change the mysql.password from change_me
to desired
credentials.
mysql.password: change_me
# Required: Cassandra endpoint DNS/IP. If Cassandra is deployed as a
Kubernetes service, this will be the service name.
# If using an external database, put the proper address (the address of a
single node will be sufficient)
**Edit the collector endpoint and api-url:**Change the defaults
(sysdigcloud-collector
and sysdigcloud-api:443)
to point to the
DNS name you have established for Sysdig.
Note: The collector port should remain 6443.
collector.endpoint: <DNS_NAME>
collector.port: "6443"
api.url: https://<DNS_NAME>:443
Recommended: edit the file to set the JVM options for Cassandra, Elasticsearch, and API, worker, and collector as well.
(To use the AWS implicit key, edit the JVM options as described in AWS: Integrate AWS Account and CloudWatch Metrics (Optional).)
For installations over 100 agents, it is recommended to allocate 8 GB per JVM.
cassandra.jvm.options: "-Xms8G -Xmx8G"
elasticsearch.jvm.options: "-Xms8G -Xmx8G"
sysdigcloud.jvm.api.options: "-Xms8G -Xmx8G"
sysdigcloud.jvm.worker.options: "-Xms8G -Xmx8G"
sysdigcloud.jvm.collector.options: "-Xms8G -Xmx8G"
Note: If you do not wish to use SSL between the agent and the collector, use the following settings instead:
cassandra.jvm.options: "-Xms8G -Xmx8G"
elasticsearch.jvm.options: "-Xms8G -Xmx8G"
sysdigcloud.jvm.api.options: "-Xms8G -Xmx8G -Ddraios.agents.installParams.sslEnabled=false"
sysdigcloud.jvm.worker.options: "-Xms8G -Xmx8G -Ddraios.agents.installParams.sslEnabled=false"
sysdigcloud.jvm.collector.options: "-Xms8G -Xmx8G -Ddraios.agents.installParams.sslEnabled=false"
Optional: Change ElasticSearch container setting to non-privileged.
See Consider Elasticsearch Default Privileges, above.
To change the default setting, edit the file
elasticsearch-statefulset.yaml
and set privileged: false
.
containers:
- name: elasticsearch
image: quay.io/sysdig/elasticsearch:5.6.16.15
securityContext:
privileged: false
Deploy the configuration map and secrets for all services by running the commands:
For Sysdig Monitor:
kubectl -n sysdigcloud apply -f sysdigcloud/config.yaml
To add Sysdig Secure:
kubectl -n sysdigcloud apply -f sysdigcloud/scanning-secrets.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/anchore-secrets.yaml
Apply the secret for the policy advisor:
kubectl -n sysdigcloud apply -f sysdigcloud/policy-advisor-secret.yaml
Configure DNS name in api-ingress.yaml
(or
api-ingress-with-secure.yaml
if using Secure). (Files located in
sysdigcloud/)
Edit: host: <EXTERNAL-DNS-NAME>
to suit your DNS name
Define namespace in ingress-clusterrolebinding.yaml
. (File
located in
sysdigcloud/ingress_controller/) Edit namespace: sysdigcloud
Using either the existing storage class name from step 1, or the storage
class name defined in the previous step, edit the storageClassName
in
the following .yaml
files:
For Monitor:
datastores/as_kubernetes_pods/manifests/cassandra/cassandra-statefulset.yaml
datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-statefulset.yaml
datastores/as_kubernetes_pods/manifests/mysql/mysql-deployment.yaml
With Secure:
datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
If using Sysdig Secure:
Edit the MySQL deployment to uncomment the MYSQL_EXTRADB_* environment variables.
This forces MySQL to create the necessary scanning database on startup.
File location:
datastores/as_kubernetes_pods/manifests/mysql/mysql-deployment.yaml
- name: MYSQL_EXTRADB_SCANNING_DBNAME
valueFrom:
configMapKeyRef:
name: sysdigcloud-config
key: scanning.mysql.dbname
- name: MYSQL_EXTRADB_SCANNING_USER
valueFrom:
configMapKeyRef:
name: sysdigcloud-config
key: scanning.mysql.user
- name: MYSQL_EXTRADB_SCANNING_PASSWORD
valueFrom:
secretKeyRef:
name: sysdigcloud-scanning
key: scanning.mysql.password
The scanning service will not start unless MySQL creates the scanning database.
A specific Quay pull secret is sent via email with your license key.
Edit the file sysdigcloud/pull-secret.yaml
and change the place
holder <PULL_SECRET>
with the provided pull secret.
Deploy the pull secret object:
kubectl -n sysdigcloud apply -f sysdigcloud/pull-secret.yaml
SSL-secured communication is used between user browsers and the Sysdig API server(s), and between the Sysdig agent and the collectors.
To set this up, you must:
Use existing standard certs for API and collector, or
Create self-signed certificates and keys for API and collector
To disable SSL between agents and collectors, set JVM options when configuring backend components.
Run these commands (edit to add your API_DNS_NAME
and
COLLECTOR_DNS_NAME
):
openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -subj "/C=US/ST=CA/L=SanFrancisco/O=ICT/CN=<API_DNS_NAME>" -keyout server.key -out server.crt
openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -subj "/C=US/ST=CA/L=SanFrancisco/O=ICT/CN=<COLLECTOR_DNS_NAME>" -keyout collector.key -out collector.crt
This uses two different certificates, one for the API/UI, and one for the collector.
kubectl -n sysdigcloud create secret tls sysdigcloud-ssl-secret --cert=server.crt --key=server.key
kubectl -n sysdigcloud create secret tls sysdigcloud-ssl-secret-collector --cert=collector.crt --key=collector.key
The Sysdig platform may sometimes open connections over SSL to certain external services, including:
LDAP over SSL
SAML over SSL
OpenID Connect over SSL
HTTPS Proxies
If the signing authorities for the certificates presented by these services are not well-known to the Sysdig Platform (e.g., if you maintain your own Certificate Authority), they are not trusted by default.
To allow the Sysdig platform to trust these certificates, use the command below to upload one or more PEM-format CA certificates. You must ensure you’ve uploaded all certificates in the CA approval chain to the root CA.
kubectl -n sysdigcloud create secret generic sysdigcloud-java-certs --from-file=certs1.crt --from-file=certs2.crt
For Sysdig Monitor:
Create the datastore statefulsets for Elasticsearch and Cassandra.
Elasticsearch and Cassandra are automatically set up with
--replica=3
generating full clusters.
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/cassandra/cassandra-service.yaml
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/cassandra/cassandra-statefulset.yaml
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-service.yaml
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-statefulset.yaml
Wait for those processes to be running, then create the database and caching systems: MySQL, and Redis.
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/mysql/mysql-deployment.yaml
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/redis/redis-deployment.yaml
To add Sysdig Secure: Create the PostgreSQL database:
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/postgres/postgres-service.yaml
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
Wait until datastore pods are in ready
state:
Run the command:
kubectl -n sysdigcloud get pods
Then look in the READY column to ensure all pods are ready. For example, displaying a 1/1 means 1 of 1 pods is ready
Apply the NATS service and deployment to deliver events to Sysdig backend components:
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/nats-streaming/nats-streaming-deployment.yaml
kubectl -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/nats-streaming/nats-streaming-service.yaml
Apply the API deployment. Pause until all containers in the API pod are running, then apply the collector and worker deployments.
kubectl -n sysdigcloud apply -f sysdigcloud/api-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/collector-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/worker-deployment.yaml
Create the service for the API and collector:
kubectl -n sysdigcloud apply -f sysdigcloud/api-headless-service.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/collector-headless-service.yaml
Sysdig Secure only Create anchore-engine
deployments and
service (used in scanning):
kubectl -n sysdigcloud apply -f sysdigcloud/anchore-service.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/anchore-core-config.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/anchore-core-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/anchore-worker-config.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/anchore-worker-deployment.yaml
Wait 60 seconds to ensure the Anchore components are up and running. Then deploy custom Sysdig Secure scanning components:
kubectl -n sysdigcloud apply -f sysdigcloud/scanning-service.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/scanning-api-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/scanning-alertmgr-service.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/scanning-alertmgr-deployment.yaml
Sysdig Secure only Create services, deployments, and a janitor job for the activity audit and policy advisor features:
kubectl -n sysdigcloud apply -f sysdigcloud/policy-advisor-service.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/activity-audit-api-service.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/activity-audit-api-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/policy-advisor-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/activity-audit-worker-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/activity-audit-janitor-cronjob.yaml
kubectl create clusterrolebinding cluster-admin-binding --clusterrole cluster-admin --user $(gcloud config get-value account)
For Sysdig Monitor:
To permit incoming connections to the Sysdig API and collector, deploy
the following ingress
yamls.
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-clusterrole.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-clusterrolebinding.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-role.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-rolebinding.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-serviceaccount.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/default-backend-service.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/default-backend-deployment.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-configmap.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-tcp-services-configmap.yaml
kubectl -n sysdigcloud apply -f sysdigcloud/ingress_controller/ingress-daemonset.yaml
If NOT using Sysdig Secure, then apply the following ingress.yaml:
kubectl -n sysdigcloud apply -f sysdigcloud/api-ingress.yaml
For Sysdig Secure:
If you ARE using Secure, replace the api-ingress.yaml
with the
following line:
kubectl -n sysdigcloud apply -f sysdigcloud/api-ingress-with-secure.yaml
When the terminal messages indicate that installation was successfully completed:
Point your browser to https://API_DNS_NAME
.
You will be prompted to log in with the Admin credentials you set in Step 2 Configure Backend Components.
Log in as Super Admin.
The Welcome Wizard is launched and prompts you to install your first Sysdig agent.
Install the agent(s).
The Welcome Wizard should be populated with install parameters from your environment (access key, collector name, and collector port). For example:
{docker run -d --name sysdig-agent --restart always --privileged --net host --pid host -e ACCESS_KEY=xxxxxxxxxx -e COLLECTOR=abc.us-west.elb.amazonaws.com -e COLLECTOR_PORT=6443 -e CHECK_CERTIFICATE=false -e TAGS=example_tag:example_value -v /var/run/docker.sock:/host/var/run/docker.sock -v /dev:/host/dev -v /proc:/host/proc:ro -v /boot:/host/boot:ro -v /lib/modules:/host/lib/modules:ro -v /usr:/host/usr:ro --shm-size=350m quay.io/sysdig/agent
Replace kubectl
with oc
for OpenShift.
There are two ways to change the original installation parameters in the config map: edit or overwrite.
To edit the config map, run the following command:
kubectl edit configmap/sysdigcloud-config --namespace sysdigcloud
A text editor is presented with the config map to be edited. Enter parameters as needed, then save and quit.
Then restart the config map (below).
To overwrite the config map that is edited on the client-side, (e.g. to keep it synced in a git repository), use the following command:
kubectl replace -f sysdigcloud/config.yaml --namespace sysdigcloud
Then restart the config map (below).
After updating the configmap, the Sysdig components must be restarted for the changed parameters to take effect. This can be done by forcing a rolling update of the deployments.
A possible way to do so is to change something innocuous, which forces a rolling update. E.g.:
kubectl -n sysdigcloud patch deployment [deploymnet] -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"$(date +'%s')\"}}}}}"
Replace kubectl
with oc
for OpenShift.
For Sysdig installations on Kubernetes or OpenShift, version 2.5.0 and above.
The Sysdig Installer tool is a Docker image containing a collection of scripts that help automate the on-premises deployment of the Sysdig platform (Sysdig Monitor and/or Sysdig Secure), for environments using Kubernetes or OpenShift. Use the Installer to install or upgrade your Sysdig platform. It is recommended as a replacement for the earlier Kubernetes manual installation and upgrade procedures.
To install, you will log in to quay.io, download a values.yaml file, provide a few basic parameters in it, and launch the Installer. In a normal installation, the rest is automatically configured and deployed.
You can perform a quick install if your environment has access to the internet, or a partial or full airgapped installation, as needed. Each is described below.
See Frequently Used Installer Configurations to:
Customize or override settings
Use hostPath for static storage of Sysdig components
Use Kubernetes node labels and taints to run only Sysdig pods on selected nodes in a cluster
The installer must be run from a machine with kubectl/oc
configured
with access to the target cluster where the Sysdig platform will be
installed. Note that this cluster may be different than where the Sysdig
agent will be deployed.
Requirements for Installation Machine with Internet Access
Network access to Kubernetes cluster
Docker
Bash
Network access to quay.io (See Docker Login to quay.io, below.)
A domain name you are in control of.
Additional Requirements for Airgapped Environments
Edited values.yaml with airgap registry details updated
Network and authenticated access to the private registry
Access Requirements
Sysdig license key (Monitor and/or Secure)
Quay pull secret
You may use dynamic or static storage on a variety of platforms to store the Sysdig platform components (stateful sets). Different configuration parameters and values are used during the install, depending on which scenario you have.
Use Case 1: Default, undefined (AWS/GKE)
If you will use dynamic storage on AWS or GKE and haven’t configured any storage class there yet, then the Quick Install streamlines the process for you.
storageclassProvisioner:
Enter aws
or gke
. The installer will
create the appropriate storage class and then use it for all the
Sysdig platform stateful sets.
storageclassName
: Leave empty.
Use Case 2: Dynamic, predefined
It is also possible that you are using dynamic storage but have already created storage classes there. This dynamic storage could be AWS, GKE, or any other functioning dynamic storage you use. In this case, you would enter:
storageclassProvisioner
: Leave empty; anything put here would be
ignored.
storageclassName
: Provide the name of the pre-configured storage
class you want to use. The installer will use this storage class for
all the Sysdig platform stateful sets.
Use Case 3: Static Storage
In cases where dynamic storage is not available, you can use static storage for the Sysdig stateful sets. In this case, you would use:
storageclassProvisioner
: Enter hostpath
, then define the nodes
for the four main Sysdig components: ElasticSearch, Cassandra,
MySQL, and Postgres.storageclassProvisioner
See Frequently Used Installer Configurations for details.
Retrieve the Quay username and password from Quay pull secret.
For example
AUTH=$(echo <REPLACE_WITH_quaypullsecret> | base64 --decode | jq -r '.auths."quay.io".auth'| base64 --decode)
QUAY_USERNAME=${AUTH%:*}
QUAY_PASSWORD=${AUTH#*:}
**Log in to quay.io.**Use the username and password retrieved above.
docker login -u "$QUAY_USERNAME" -p "$QUAY_PASSWORD" quay.io
This install assumes the Kubernetes cluster has network access to pull images from quay.io.
Copy the current version values.yaml to your working directory.
wget https://raw.githubusercontent.com/draios/sysdigcloud-kubernetes/installer/installer/values.yaml
If you will be editing for an OpenShift installation and want to review a sample, see openshift-with-hostpath values.yaml.
Edit the following values:
size: Specifies the size of the cluster. Size defines CPU, Memory, Disk, and Replicas. Valid options are: small, medium and large
quaypullsecret: quay.io provided with your Sysdig purchase confirmation mail
storageClassProvisioner: Review Storage Requirements, above.
If you have the default use case, enter aws
or gke
in the
storageClassProvisioner
field. Otherwise, refer to Use Case 2
or 3.
sysdig.license: Sysdig license key provided with your Sysdig purchase confirmation mail
sysdig.dnsname: The domain name the Sysdig APIs will be served on. Note that the master node may not be used as the DNS name when using hostNetwork mode.
sysdig.collector.dnsName: (OpenShift installs only) Domain name the Sysdig collector will be served on. When not configured it defaults to whatever is configured for sysdig.dnsName. Note that the master node may not be used as the DNS name when using hostNetwork mode.
deployment: (OpenShift installs only) Add
deployment: openshift
to the root of the values.yaml
file.
sysdig.ingressNetworking: The networking construct used to expose the Sysdig API and collector.Options are:
hostnetwork: sets the hostnetworking in the ingress daemonset and opens host ports for api and collector. This does not create a Kubernetes service.
loadbalancer: creates a service of type loadbalancer and expects that your Kubernetes cluster can provision a load balancer with your cloud provider.
nodeport: creates a service of type nodeport.The node ports can be customized with:
sysdig.ingressNetworkingInsecureApiNodePort
sysdig.ingressNetworkingApiNodePort
sysdig.ingressNetworkingCollectorNodePort
When not configured, sysdig.ingressNetworking
defaults to
hostnetwork
.
If doing an airgapped install , you would also edit the following values:
airgapped_registry_name: The URL of the airgapped (internal) docker registry. This URL is used for installations where the Kubernetes cluster can not pull images directly from Quay
airgapped_repository_prefix: This defines custom
repository prefix for airgapped_registry. Tags and pushes
images as
airgapped_registry_name/airgapped_repository_prefix/image_name:tag
airgapped_registry_password: The password for the configured airgapped_registry_username. Ignore this parameter if the registry does not require authentication.
airgapped_registry_username: The username for the configured airgapped_registry_name. Ignore this parameter if the registry does not require authentication.
Run the installer. (This step differs in Airgapped Installation, below.)
docker run \
-e HOST_USER=$(id -u) \
-e KUBECONFIG=/.kube/config \
-v ~/.kube:/.kube:Z \
-v $(pwd):/manifests:Z \
quay.io/sysdig/installer:
See Output (below) to finish.
Save the values.yaml
file in a secure location; it will be used for
future upgrades.
There will also be a generated directory containing various Kubernetes
configuration yaml
files that were applied by the Installer against
your cluster. It is not necessary to keep the generated directory, as
the Installer can regenerate it consistently with the same
values.yaml
file.
The installer can be used to install in airgapped environments, either with a multi-homed installation machine that has internet access, or in an environment with no internet access.
NOTE: Sysdig Secure users who install in an airgapped environment do not have internet access to the continuous checks of vulnerability databases that are used in image scanning. (See also: How Sysdig Image Scanning Works.)
As of installer version 3.2.0-9, airgapped environments can also receive periodic vulnerability database updates.
When you install with the “airgapped_
” parameters enabled (see Full
Airgap
Install
instructions), the installer will automatically push the latest
vulnerability database to your environment. Follow the steps below to
reinstall/refresh the vuln db, or use the script and chron job to
schedule automated updates (daily, weekly, etc.).
To automatically update the vulnerability database, you can:
Download the image file quay.io/sysdig/vuln-feed-database-ubi:latest from the Sysdig registry to the jump box server and save it locally.
Move the file from the jump box server to the airgapped environment (if needed)
Load the image file and push it to the airgapped image registry.
Restart the pod sysdigcloud-feeds-db
Restart the pod feeds-api
The following script (feeds_database_update.sh
) performs the five
steps:
#!/bin/bash
QUAY_USERNAME="<change_me>"
QUAY_PASSWORD="<change_me>"
# Download image
docker login quay.io/sysdig -u ${QUAY_USERNAME} -p ${QUAY_PASSWORD}
docker image pull quay.io/sysdig/vuln-feed-database-ubi:latest
# Save image
docker image save quay.io/sysdig/vuln-feed-database-ubi:latest -o vuln-feed-database-ubi.tar
# Optionally move image
mv vuln-feed-database-ubi.tar /var/shared-folder
# Load image remotely
ssh -t user@airgapped-host "docker image load -i /var/shared-folder/vuln-feed-database-ubi.tar"
# Push image remotely
ssh -t user@airgapped-host "docker tag vuln-feed-database-ubi:latest airgapped-registry/vuln-feed-database-ubi:latest"
ssh -t user@airgapped-host "docker image push airgapped-registry/vuln-feed-database-ubi:latest"
# Restart database pod
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-db --replicas=0"
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-db --replicas=1"
# Restart feeds-api pod
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-api --replicas=0"
ssh -t user@airgapped-host "kubectl -n sysdigcloud scale deploy sysdigcloud-feeds-api --replicas=1"
Schedule a chron job to run the script on a chosen schedule (e.g. every day):
0 8 * * * feeds-database-update.sh >/dev/null 2>&1
This assumes a private docker registry is used and the installation machine has network access to pull from quay.io and push images to the private registry.
The Prerequisites and workflow are the same as in the Quickstart Install (above) with the following exceptions:
In step 2, add the airgap registry information
In step 3, run the installer as follows:
docker run \
-e HOST_USER=$(id -u) \
-e KUBECONFIG=/.kube/config \
-e IMAGE_EXTRACT_PUSH=true \
-v ~/.kube:/.kube:Z \
-v $(pwd):/manifests:Z \
-v /var/run/docker.sock:/var/run/docker.sock:Z \
-v ~/.docker:/root/docker:Z \
quay.io/sysdig/installer:
This assumes a private docker registry is used and the installation machine does not have network access to pull from quay.io, but can push images to the private registry.
In this situation, a machine with network access (called the “jump machine”) will pull an image containing a self-extracting tarball which can be copied to the installation machine.
Requirements for jump machine
Network access to quay.io
Docker
Requirements for installation machine
Network access to Kubernetes cluster
Docker
Bash
Network and authenticated access to the private registry
Edited values.yaml with airgap registry details updated
Host Disk Space Requirements:/tmp
> 4 GB; directory from
which the installer is run >8GB; and /var/lib/docker
> 4GB.
NOTE: The environment variable TMPDIR
can be used to override
the /tmp
directory.
On the Jump Machine
Follow the Docker Log In to quay.io steps.
Pull the image containing the self-extracting tar:
docker pull quay.io/sysdig/installer:5.1.2-1-uber
Extract the tarball:
docker create --name uber_image quay.io/sysdig/installer:5.1.2-1-uber
docker cp uber_image:/sysdig_installer.tar.gz .
docker rm uber_image
Copy the tarball to the installation machine.
On the Installation Machine:
Copy the current version values.yaml to your working directory.
wget https://raw.githubusercontent.com/draios/sysdigcloud-kubernetes/installer/installer/values.yaml
Edit the following values:
size: Specifies the size of the cluster. Size defines CPU, Memory, Disk, and Replicas. Valid options are: small, medium and large
quaypullsecret: quay.io provided with your Sysdig purchase confirmation mail
storageClassProvisioner: Review Storage Requirements, above.
If you have the default use case, enter aws
or gke
in the
storageClassProvisioner
field. Otherwise, refer to Use Case 2
or 3.
sysdig.license: Sysdig license key provided with your Sysdig purchase confirmation mail
sysdig.dnsname: The domain name the Sysdig APIs will be served on. Note that the master node may not be used as the DNS name when using hostNetwork mode.
sysdig.collector.dnsName: (OpenShift installs only) Domain name the Sysdig collector will be served on. When not configured it defaults to whatever is configured for sysdig.dnsName. Note that the master node may not be used as the DNS name when using hostNetwork mode.
deployment: (OpenShift installs only) Add
deployment: openshift
to the root of the values.yaml
file.
sysdig.ingressNetworking: The networking construct used to expose the Sysdig API and collector.Options are:
hostnetwork: sets the hostnetworking in the ingress daemonset and opens host ports for api and collector. This does not create a Kubernetes service.
loadbalancer: creates a service of type loadbalancer and expects that your Kubernetes cluster can provision a load balancer with your cloud provider.
nodeport: creates a service of type nodeport.The node ports can be customized with:
sysdig.ingressNetworkingInsecureApiNodePort
sysdig.ingressNetworkingApiNodePort
sysdig.ingressNetworkingCollectorNodePort
airgapped_registry_name: The URL of the airgapped (internal) docker registry. This URL is used for installations where the Kubernetes cluster can not pull images directly from Quay
airgapped_repository_prefix: This defines custom
repository prefix for airgapped_registry. Tags and pushes
images as
airgapped_registry_name/airgapped_repository_prefix/image_name:tag
airgapped_registry_password: The password for the configured airgapped_registry_username. Ignore this parameter if the registry does not require authentication.
airgapped_registry_username: The username for the configured airgapped_registry_name. Ignore this parameter if the registry does not require authentication.
Copy the tarball file to the directory where you have your
values.yaml
file.
Run the tar file:
bash sysdig_installer.tar.gz
NOTE: The above step extracts images, runs the installer, and pushes
images to the remote repository in a single step. The extract/push
images can be redundant for successive installer runs. Setting
IMAGE_EXTRACT_PUSH=false
runs only the installer:
IMAGE_EXTRACT_PUSH=false bash sysdig_installer.tar.gz
See Output (below) to finish.
Save the values.yaml
file in a secure location; it will be used for
future upgrades.
There will also be a generated directory containing various Kubernetes
configuration yaml
files that were applied by the Installer against
your cluster. It is not necessary to keep the generated directory, as
the Installer can regenerate it consistently with the same
values.yaml
file.
A successful installation should display output in the terminal such as:
All Pods Ready.....Continuing
Congratulations, your Sysdig installation was successful!
You can now login to the UI at "https://awesome-domain.com:443" with:
username: "configured-username@awesome-domain.com"
password: "awesome-password"
There will also be a generated directory containing various Kubernetes
configuration yaml
files which were applied by installer against your
cluster. It is not necessary to keep the generated directory, as the
installer can regenerate consistently with the same values.yaml
file.
To see all the configuration parameters available, as well as their definitions, values, and examples, see configuration_parameters.md on GitHub.
For advanced options, including static storage and patching, see Frequently Used Installer Configurations.
Example values.yaml files:
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
As of Sysdig Platform v 2.5.0, a semi-automated install option is available and is preferred.
This section describes how to install the backend components of the Sysdig platform using an existing OpenShift cluster. It applies to backend versions 1929 and higher.
The Sysdig platform includes both Sysdig Monitor and Sysdig Secure,
which are licensed separately. All installations include Sysdig Monitor,
while some of the Secure components are installed and configured as
additional steps within the overall installation process. When
installing the Sysdig platform on OpenShift manually, you will install
each backend component with separate oc
commands.
Access to a running OpenShift 3.11+ instance
Two items from your Sysdig purchase-confirmation email:
Your Sysdig license key
Your Sysdig quay.io pull secret
octools
installed on your machine and communicating with the
OpenShift cluster. (Note that your oc
and OpenShift versions
should match to avoid errors.)
If you want more information on OpenShift’s DNS requirements; see the OpenShift documentation.
Option 1: DNS without Wildcard
You need to request two different DNS records from your DNS team:
one for the Sysdig API/UI and another for the Sysdig collector.
These records should point to your infrastructure nodes and are the
two routes that will be exposed, i.e., sysdig.api.example.com
and
sysdig.collector.example.com
.
Option 2: DNS with Wildcard
With wildcard DNS, you do not have to make an official request from
the DNS team. Your implementation team can pick any two DNS names to
use for the API/UI and Collector. These will be exposed to the
infrastructure nodes once the configuration is completed. (i.e.
sysdig.api.example.com
and sysdig.collector.example.com
.)
Step 5: Set Up SSL Connectivity to the Backend discusses how to implement SSL; decide ahead of time whether you will use SSL with wildcard or without.
SSL with Wildcard
With wildcard SSL, you use the same certificate for both the API and the collector.
SSL without Wildcard
You need two SSL certs, one for each DNS record.
By default, the Elasticsearch container will be installed in
privileged
(root-access) mode. This mode is only needed so the
container can reconfigure the hosts’ Linux file descriptors if
necessary. See Elasticsearch’s description
here.
If you prefer not to allow Elasticsearch to run with root access to the host, you will need to:
Set your own file descriptors on all Linux hosts in the Kubernetes cluster.
If one host were to go down, Kubernetes could choose a different node for Elasticsearch, so each Linux host must have the file descriptors set.
Set privileged:false
in the elasticsearch-statefulset.yaml
file.
See the step under Coonfigure Backend Components, below, for details.
Download the latest release from https://github.com/draios/sysdigcloud-kubernetes/releases/latest
Unpack the .tar ball.
The source link has the format:
https://github.com/draios/sysdigcloud-kubernetes/archive/<v1234>.tar.gz.
To
unpack it, run the following commands (replacing version number as
appropriate):
wget https://github.com/draios/sysdigcloud-kubernetes/archive/<v1234>.tar.gz
tar zxf <1234>.tar.gz
cd sysdigcloud-kubernetes-<1234>
Create a new project called sysdigcloud
and copy the cloned
folders into it:
oc new-project sysdigcloud
Apply the correct security contexts to the namespace. (This allows you to run privileged containers in the sysdigcloud namespace)
oc adm policy add-scc-to-user anyuid -n sysdigcloud -z default
oc adm policy add-scc-to-user privileged -n sysdigcloud -z default
The ConfigMap
(config.yaml
)
is populated with information about usernames, passwords, SSL certs, and
various application-specific settings.
The steps below give the minimum edits that should be performed in a test environment.
It is necessary to review and customize the entries in config.yaml
before launching in a production environment.
See Making Configuration Changes, below, for the oc
format to use for
post-install edits, such as adding 3rd-party authenticators such as
LDAP.
Add your license key:
In config.yaml,
enter the key that was emailed to you in the
following parameter:
# Required: Sysdig Cloud license
sysdigcloud.license: ""
Change the super admin name and password, which are the super admin credentials for the entire system. See here for details.
Find the settings in config.yaml
here:
sysdigcloud.default.user: test@sysdig.com
# Required: Sysdig Cloud super admin user password
# NOTE: Change upon first login
sysdigcloud.default.user.password: test
**Edit the collector endpoint and API URL:**Change the placeholder to point to the DNS names you have established for Sysdig.
Remember that you must have defined one name for the collector and another for the API URL.
Note: Change the collector port to 443.
collector.endpoint: <COLLECTOR_DNS_NAME>
collector.port: "443"
api.url: https://<API_DNS_NAME>:443
Recommended: edit the file to set the JVM options for Cassandra, Elasticsearch, and API, worker, and collector as well.
(To use the AWS implicit key, edit the JVM options as described in AWS: Integrate AWS Account and CloudWatch Metrics (Optional).)
For installations over 100 agents, it is recommended to allocate 8 GB of heap per JVM.
cassandra.jvm.options: "-Xms8G -Xmx8G"
elasticsearch.jvm.options: "-Xms8G -Xmx8G"
sysdigcloud.jvm.api.options: "-Xms4G -Xmx8G"
sysdigcloud.jvm.worker.options: "-Xms4G -Xmx8G"
sysdigcloud.jvm.collector.options: "-Xms4G -Xmx8G"
Note: If you do not wish to use SSL between the agent and the collector, use the following settings instead:
cassandra.jvm.options: "-Xms8G -Xmx8G"
elasticsearch.jvm.options: "-Xms8G -Xmx8G"
sysdigcloud.jvm.api.options: "-Xms8G -Xmx8G -Ddraios.agents.installParams.sslEnabled=false"
sysdigcloud.jvm.worker.options: "-Xms8G -Xmx8G -Ddraios.agents.installParams.sslEnabled=false"
sysdigcloud.jvm.collector.options: "-Xms8G -Xmx8G -Ddraios.agents.installParams.sslEnabled=false"
Optional: Change ElasticSearch container setting to non-privileged.
See Consider Elasticsearch Default Privileges, above.
To change the default setting, edit the file
elasticsearch-statefulset.yaml
and set privileged: false
.
containers:
- name: elasticsearch
image: quay.io/sysdig/elasticsearch:5.6.16.15
securityContext:
privileged: false
Deploy the configuration maps and secrets for all services by running the commands:
For Sysdig Monitor:
oc -n sysdigcloud apply -f sysdigcloud/config.yaml
**(Sysdig Secure only) Edit and apply secrets for Anchore and the
scanning component:**Edit theyaml
files:
scanning-secrets.yaml
stringData:
scanning.mysql.password: change_me
anchore-secrets yaml
stringData:
anchore.admin.password: change_me
anchore.db.password: change_me
policy-advisor-secret.yaml
stringData:
padvisor.mysql.password: change_me
Then apply the files:
oc -n sysdigcloud apply -f sysdigcloud/scanning-secrets.yaml
oc -n sysdigcloud apply -f sysdigcloud/anchore-secrets.yaml
oc -n sysdigcloud apply -f sysdigcloud/policy-advisor-secret.yaml
Edit the API DNS name in either api-ingress.yaml
or
api-ingress-with-secure.yaml
(if using Secure).
The files are located in sysdigcloud/
spec:
rules:
- host: <API_DNS_NAME>
...
tls:
- hosts:
- <API_DNS_NAME>
secretName: sysdigcloud-ssl-secret
Edit the collector DNS name in the file
openshift-collector-router.yaml
. Use the collector DNS name you
created in the Prerequisites.
The file is located in sysdigcloud/openshift/
.
spec:
host: <COLLECTOR_DNS_NAME>
If using Sysdig Secure :
Edit the MySQL deployment to uncomment the MYSQL_EXTRADB_* environment variables. This forces MySQL to create the necessary scanning database on startup.
File location:
datastores/as_kubernetes_pods/manifests/mysql/mysql-deployment.yaml
- name: MYSQL_EXTRADB_SCANNING_DBNAME
valueFrom:
configMapKeyRef:
name: sysdigcloud-config
key: scanning.mysql.dbname
- name: MYSQL_EXTRADB_SCANNING_USER
valueFrom:
configMapKeyRef:
name: sysdigcloud-config
key: scanning.mysql.user
- name: MYSQL_EXTRADB_SCANNING_PASSWORD
valueFrom:
secretKeyRef:
name: sysdigcloud-scanning
key: scanning.mysql.password
The scanning service will not start unless MySQL creates the scanning database.
A specific Quay pull secret is sent via email with your license key.
Edit the file sysdigcloud/pull-secret.yaml
and change the
place holder <PULL_SECRET>
with the provided pull secret.
vi sysdigcloud/pull-secret.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: sysdigcloud-pull-secret
data:
.dockerconfigjson: <PULL_SECRET>
type: kubernetes.io/dockerconfigjson
Deploy the pull secret object:
oc -n sysdigcloud apply -f sysdigcloud/pull-secret.yaml
SSL-secured communication is used between user browsers and the Sysdig API server(s), and between the Sysdig agent and the collectors.
To set this up, you must:
Use an existing wildcard SSL certificate and key, or
Use existing standard certs for API and collector, or
Create self-signed certificates and keys for API and collector
If you are not using wildcard SSL, you have to use two separate certificates, one for API URL and one for the collector.
To disable SSL between agent and collector:
To disable SSL between agent and collectors, you set a JVM option when configuring backend components (below).
To create self-signed certs:
Run these commands (edit to add your API_DNS_NAME
and
COLLECTOR_DNS_NAME
):
openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -subj "/C=US/ST=CA/L=SanFrancisco/O=ICT/CN=<API_DNS_NAME>" -keyout server.key -out server.crt
openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -subj "/C=US/ST=CA/L=SanFrancisco/O=ICT/CN=<COLLECTOR_DNS_NAME>" -keyout collector.key -out collector.crt
To use an existing wildcard cert:
Obtain the respective server.crt
and server.key
files.
With Wildcard
Uses the same certificate for both the API/UI and the collector.
Run these commands:
oc -n sysdigcloud create secret tls sysdigcloud-ssl-secret --cert=server.crt --key=server.key
oc -n sysdigcloud create secret tls sysdigcloud-ssl-secret-collector --cert=server.crt --key=server.key
Without Wildcard
Uses two different certificates, one for the API/UI, and one for the collector.
Run these commands:
oc -n sysdigcloud create secret tls sysdigcloud-ssl-secret --cert=server.crt --key=server.key
oc -n sysdigcloud create secret tls sysdigcloud-ssl-secret-collector --cert=collector.crt --key=collector.key
The Sysdig platform may sometimes open connections over SSL to certain external services, including:
LDAP over SSL
SAML over SSL
OpenID Connect over SSL
HTTPS Proxies
If the signing authorities for the certificates presented by these services are not well-known to the Sysdig Platform (e.g., if you maintain your own Certificate Authority), they are not trusted by default.
To allow the Sysdig platform to trust these certificates, use the command below to upload one or more PEM-format CA certificates. You must ensure you’ve uploaded all certificates in the CA approval chain to the root CA.
oc -n sysdigcloud create secret generic sysdigcloud-java-certs --from-file=certs1.crt --from-file=certs2.crt
You need a storage class; step 2 shows how to create one if needed.
Enter the storageClassName in the appropriate .yaml
files (see step
3).
Verify whether a storage class has been created, by running the command:
oc get storageclass
If no storage class has been defined, create a manifest for one, and then deploy it.
For example, a manifest could be named
sysdigcloud-storageclass.yaml
and contain the following contents
(for a storage class using GP2 volumes in AWS):
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: gp2
labels:
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: EnsureExists
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
Now run the command:
oc apply -f sysdigcloud-storageclass.yaml
Using either the existing storage class name from step 1, or the
storage class name defined in step 2, edit the storageClassName
in
the following .yaml
files:
For Monitor:
datastores/as_kubernetes_pods/manifests/cassandra/cassandra-statefulset.yaml
datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-statefulset.yaml
datastores/as_kubernetes_pods/manifests/mysql/mysql-deployment.yaml
With Secure:
datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
In each file, the code snippet looks the same:
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 50Gi
storageClassName: <STORAGECLASS_NAME>
For Sysdig Monitor
Create the datastore statefulsets for Elasticsearch and Cassandra.
Elasticsearch and Cassandra are automatically set up with
--replica=3
generating full clusters.
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/cassandra/cassandra-service.yaml
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/cassandra/cassandra-statefulset.yaml
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-service.yaml
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-statefulset.yaml
Wait for those processes to be running, then create the MySQL and Redis databases:
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/mysql/mysql-deployment.yaml
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/redis/redis-deployment.yaml
To add Sysdig Secure: Create the PostgreSQL database:
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/postgres/postgres-service.yaml
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
Wait until datastore pods are in ready
state, then deploy the
backend deployment sets (worker, collector, and API).
Run the command:
kubectl -n sysdigcloud get pods
Then look in the READY column to ensure all pods are ready. For example, displaying a 1/1 means 1 of 1 pods is ready.
Apply the NATS service and deployment to deliver events to Sysdig backend components:
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/nats-streaming/nats-streaming-deployment.yaml
oc -n sysdigcloud apply -f datastores/as_kubernetes_pods/manifests/nats-streaming/nats-streaming-service.yaml
Then deploy the backend deployment sets (worker, collector, and API). Pause for 60 seconds after creating the API deployment.
oc -n sysdigcloud apply -f sysdigcloud/api-deployment.yaml
oc -n sysdigcloud apply -f sysdigcloud/openshift/openshift-collector-deployment.yaml
oc -n sysdigcloud apply -f sysdigcloud/worker-deployment.yaml
Create the service for the API and collector:
oc -n sysdigcloud apply -f sysdigcloud/api-headless-service.yaml
oc -n sysdigcloud apply -f sysdigcloud/openshift/openshift-collector-service.yaml
For Sysdig Secure Wait for the API, worker, and collector to come up before proceeding.
Then create anchore-engine
deployments and service (used in
scanning):
oc -n sysdigcloud apply -f sysdigcloud/anchore-service.yaml
oc -n sysdigcloud apply -f sysdigcloud/anchore-core-config.yaml
oc -n sysdigcloud apply -f sysdigcloud/anchore-core-deployment.yaml
oc -n sysdigcloud apply -f sysdigcloud/anchore-worker-config.yaml
oc -n sysdigcloud apply -f sysdigcloud/anchore-worker-deployment.yaml
Wait 60 seconds to ensure the core-deployment is in Running
status, then deploy the rest of the Secure-related yamls:
oc -n sysdigcloud apply -f sysdigcloud/scanning-service.yaml
oc -n sysdigcloud apply -f sysdigcloud/scanning-api-deployment.yaml
oc -n sysdigcloud apply -f sysdigcloud/scanning-alertmgr-service.yaml
oc -n sysdigcloud apply -f sysdigcloud/scanning-alertmgr-deployment.yaml
Sysdig Secure only Create services, deployments, and a janitor job for the activity audit and policy advisor features:
oc -n sysdigcloud apply -f sysdigcloud/policy-advisor-service.yaml
oc -n sysdigcloud apply -f sysdigcloud/activity-audit-api-service.yaml
oc -n sysdigcloud apply -f sysdigcloud/activity-audit-api-deployment.yaml
oc -n sysdigcloud apply -f sysdigcloud/policy-advisor-deployment.yaml
oc -n sysdigcloud apply -f sysdigcloud/activity-audit-worker-deployment.yaml
oc -n sysdigcloud apply -f sysdigcloud/activity-audit-janitor-cronjob.yaml
Apply the appropriate ingress yaml. (The API_DNS
name was entered in
step 7, in Step 2: Configure Backend
Components
This configures the route to the Sysdig UI.
For Sysdig Monitor
oc -n sysdigcloud apply -f sysdigcloud/api-ingress.yaml
With Sysdig Secure:
oc -n sysdigcloud apply -f sysdigcloud/api-ingress-with-secure.yaml
Configure connectivity to the collector for the agent:
oc -n sysdigcloud apply -f sysdigcloud/openshift/openshift-collector-router.yaml
Replace kubectl
with oc
for OpenShift.
There are two ways to change the original installation parameters in the config map: edit or overwrite.
To edit the config map, run the following command:
kubectl edit configmap/sysdigcloud-config --namespace sysdigcloud
A text editor is presented with the config map to be edited. Enter parameters as needed, then save and quit.
Then restart the config map (below).
To overwrite the config map that is edited on the client-side, (e.g. to keep it synced in a git repository), use the following command:
kubectl replace -f sysdigcloud/config.yaml --namespace sysdigcloud
Then restart the config map (below).
After updating the configmap, the Sysdig components must be restarted for the changed parameters to take effect. This can be done by forcing a rolling update of the deployments.
A possible way to do so is to change something innocuous, which forces a rolling update. E.g.:
kubectl -n sysdigcloud patch deployment [deploymnet] -p \
"{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"date\":\"$(date +'%s')\"}}}}}"
Replace kubectl
with oc
for OpenShift.
Sysdig will deprecate support for Replicated installs in the coming months. If you are a new customer considering installing with Replicated, please contact Sysdig support.
When planning an on-premises installation, the following choice points must be decided upon.
Infrastructure Managers: To install Sysdig on-premises, administrators choose one of two infrastructure managers:
Kubernetes (see Installer (Kubernetes | OpenShift), or
Replicated: an easy-to-use orchestrator that includes a GUI management tool.
This guide describes how to install the Replicated client and use it to install and manage the Sysdig platform.
Single-Host or Multi-Host Install: For test or proof-of-concept installations, a single-host install will include all components; for production, a distributed environment is needed.
Airgapped or non-airgapped environment:
If your environment is accessible to the Internet during the install process, then the installation options include both script-based or GUI-based.
In airgapped environments (no Internet access), you must download components into your airgapped repository, and can only use the GUI-based installation.
Where to put the Replicated Management Console: When installing on-premises using Replicated as the orchestrator, the following Replicated components will be installed on your system:
Replicated UI (on a host you designate to host the Replicated Management Console)
Replicated retraced containers that handle logging (on the Management Console host only)
Replicated operator component (will go on all hosts)
In a multi-host installation, one server will be the Replicated Management Console host. The system load for these components is minor.
No matter which installation options you choose, you will use the Replicated GUI post-installation to:
Start/stop Sysdig components
Upgrade Sysdig backend components
Collect and view logs (support bundles).
Review and complete the Pre-Install requirements.
If installing on multiple nodes, decide which node will host the Replicated Management Console.
If using an airgapped environment, set up for an Airgapped Installation.
Install the Replicated Clienton a host.
Log In to the Replicated Management Console and set the Replicated Management Console Password.
Configure Sysdig Admin Password and Basic Settings.
Configure Sysdig Application Advanced Settings (if necessary).
Complete Distributed Install Steps (if necessary).
Restart the host(s).
Sysdig will deprecate support for Replicated installs in the coming months. If you are a new customer considering installing with Replicated, please contact Sysdig support.
To install the Sysdig platform on-premises, in an environment that has no inbound or outbound paths available to internet traffic, you must use the Replicated GUI-based installation option. No script-based option is currently available.
Perform the following steps to download the required Sysdig installation files, the Replicated components, and the Sysdig license file, and save them to a repository on your airgapped server. Then perform the setup steps in the Replicated Management Console, as described below.
A server instance with Docker version 1.7.1 or later installed is required prior to installation.
The Replicated .airgap
installation script does not install
docker-engine
. Sysdig recommends using the latest version of Docker
available for the server operating system.
For more information on installing Docker in an airgapped environment, refer to the Installing Docker in an Airgapped Environment documentation.
Download the latest Sysdig installation files using the links provided by the Sysdig Sales Engineer:
The Sysdig platform application .airgap
package
The Sysdig application license file (.rli
)
(Optional) The Sysdig Agent Docker image
Download the latest Replicated installation file from:
https://s3.amazonaws.com/replicated-airgap-work/replicated.tar.gz
Copy all downloaded files to a designated location on your airgapped server. For example:
/var/tmp/sysdig
(Note this path to be used when you complete the Install Components (Replicated).)
Open a command shell on the airgapped server and extract
the replicated.tar.gz
file:
sudo tar xzvf replicated.tar.gz
Run the following command to install the Replicated infrastructure manager:
sudo cat ./install.sh | sudo bash -s airgap
In a browser, navigate to the Replicated Management Console:
https://server_address:8800
(Replace server_address
with the
server name/IP address.)
Accept the default self-signed certificate, or provide a custom one,
and click Continue
.
On the next screen, once the “preflight” checks have been resolved,
select the Airgapped
option, and click Continue
.
Upload the .rli
license file.
Provide a path to the Sysdig application .airgap
file.
Should you need to upgrade an airgapped license at a future time, see Upgrade an On-Premises License. For general license information, see Subscription.
Continue with “Setting the Replication Management Password” and the rest of the installation steps in Install Components (Replicated).
Sysdig will deprecate support for Replicated installs in the coming months. If you are a new customer considering installing with Replicated, please contact Sysdig support.
You can use the Replicated UI to install the Sysdig platform on either a single host or on multiple hosts. If multi-host, decide which machine will also run the Replicated Admin Console and begin there.
If your environment is “airgapped” (no access to inbound or outbound internet traffic), there are some setup steps you must perform before doing the GUI-based Replicated installation.
See Airgapped Installation for details.
Log in to the chosen machine with a shell and run a command to install the Replicated components. You can also install Docker if it’s not already on the environment.
Log into the designated server instance with SSH.
Run the following commands:
a. To install the Replicated Infrastructure and Docker:
sudo curl -sSL https://install.sysdigcloud.com/docker | sudo bash
b. If Docker is already installed on the server instance,
add-s --no-docker
to the command:
sudo curl -sSL https://install.sysdigcloud.com/docker | sudo bash -s -- no-docker
c. If installing the Replicated Infrastructure behind a proxy, modify the installation command as shown below:
sudo curl -sSL -x http://<proxy>:<port> -o /tmp/sdc-onpremises-installer.sh https://install.sysdigcloud.com/docker && bash /tmp/sdc-onpremises-installer.sh http-proxy=http://<proxy>:<port>
As prompted, open the Replicated Client at
https://<yourserver>:8800.
Supply the DNS hostname for the Replicated Admin Console.
Accept the self-signed certificate, or upload a custom SSL certificate and private key.
Note: If a self-signed certificate is uploaded, it must include the end user, all intermediate, and the root certificates, as the certificate will be used by the Sysdig platform, as well as for the Replicated Admin Console.
To later replace a self-sign cert with a custom cert, see Replace a Self-Signed Cert with Custom Cert.
Click the Choose License
button, and upload the Sysdig license
file supplied from Sysdig Sales.
Choose Online installation option if prompted.
Once the Sysdig license validation is complete, secure the Replicated Admin Console using a local password, LDAP user account, or anonymous access (insecure).
Sysdig recommends securing the console with either a local password or LDAP user account.
Click Continue
.
After clicking Continue, the Settings page is displayed. Here you enter the configuration information that will be used by Replicated to orchestrate the Sysdig installation.
|| ||
These settings are typically defined with consultation from a Sysdig Sales Engineer.
Any JVM options to be passed to the application, such as memory constraint settings for the Java Virtual Machine components, proxy settings, etc.
At a minimum, it is recommended to define the memory constraints, in the format:
-Xms###g Xmx###g
.
Note that if multiple components are on a single machine, adjust the percentages as needed so JVMs all fit in a node.
Cassandra JVM options: recommended allocating 50% of the host’s memory to this JVM
(in a multi-node environment)
Elasticsearch JVM options: recommended allocating 50% of the host’s memory to this JVM
(in a multi-node environment)
Sysdig Cloud application JVM options: recommended to allocate up to 80% of the host’s memory to this JVM.
This is also used to set proxy settings; see HTTP/HTTPS and Proxy Support.
It is also used to set an implicit key in AWS; see AWS: Integrate AWS Account and CloudWatch Metrics (Optional).
NOTE: If you do not want to use SSL between the agent and the collectors, you append the following settings to the Sysdig Cloud application JVM options entry:
-Ddraios.agents.installParams.sslEnabled=false
For example:
-Xms8G Xmx8G -Ddraios.agents.installParams.sslEnabled=false
Ports and Security
Sysdig UI port: default 80. Port used for the Sysdig Monitor/ Sysdig Secure GUI.
Sysdig UI secure port: default 433. SSL port used for Sysdig Monitor/ Sysdig Secure GUI.
Force HTTPS: This turns off the unsecured port (80) access.
Forward Sysdig application logs to stdout
: switches logging
from the application log files to Linux standard output (stdout)
.
Sysdig collector port: default 6443. Port used for agent metrics collection. See also Agent Installation.
In earlier versions, the Sysdig Agent connected to port 6666. This behavior has been deprecated, as the Sysdig agent now connects to port 6443.
Sysdig secure collector port: default 6443. Port used for agent metrics collection. See also Agent Installation.
Exposed port for HTTP traffic inbound to Sysdig Platform backend container: 27878 – do not change without the recommendation of Sysdig Support.
Exposed port for Collector traffic inbound: 27877 – do not change without the recommendation of Sysdig Support.
Database Entries
Store Sysdig Captures in Cassandra (recommended): Default checked. Used for Sysdig trace file storage when capture function is used. If you do not store files in the Cassandra DB, you can alternately configure an AWS S3 bucket storage location.
See also: Storage: Configure AWS Capture File Storage (Optional) and Captures.
Sysdig data directory: default /opt
. Where Cassandra, MySQL,
and Elasticsearch databases will be created on a host.
Cassandra CQL native client’s port: The default port is 9042. Change the default port if you are running your own Cassandra cluster with non-standard ports.
Cassandra replication factor: The value should be either 1 or 3, never 2.
Sysdig MySQL user: default admin, recommend changing
Sysdig MySQL password: Enter a unique password and store securely.
This password is needed for future updates and will not be visible in the Replicated Admin Console. Retain this password for future use.
Sysdig MySQL max connections: The default is 1024.
Cassandra CQL native client’s port: The default is 9042.
External MySQL service: The secure end of your MySQL service. This is external to the Sysdig platform.
External Cassandra service: The secure end of your Cassandra service. This is external to the Sysdig platform.
External Redis service: The secure end of your Redis Service. This is external to the Sysdig platform.
Sysdig Redis password: The password associated with the Redis account.
External Elasticsearch service URL: An external service URL with user name and password embedded
OAuth allowed domains: List of secure Elasticsearch domains.
Google OAuth client ID: Used when integrating Google-based user login.
Google OAuth client secret: Used when integrating Google-based user login. See Google OAuth (On-Prem)
SSO CA certificate: CA certificate for single sign-on.
Datastore Authorization and SSL: See Authenticating Backend Components on Cassandra and Authenticating Backend Components on Elasticsearch.
When fields are complete, click Save
.
After Saving, click Start Now
to apply settings to the environment
immediately. Click Cancel
to apply settings at a later time.
As of version 2.4.1, authenticating Sysdig backend components on Sysdig’s Cassandra nodes or for your own Cassandra nodes is supported. In order to authenticate the backend components to Cassandra, enable the option, specify credentials of the identity you want to establish with Cassandra, and enable secure communication. This is the additional layer of defense against unauthorized access to the datastore.
Enable Cassandra authentication: Select this option if you want to authenticate Sysdig backend components to use Cassandra datastore. The option by default is disabled.
Cassandra password for authentication: The password associated with the username. If running Sysdig’s Cassandra database, create a password here. If you are using your own Cassandra database, enter the appropriate user password for Sysdig access.
Enable Cassandra TLS: (Mandatory) Establish TLS communication between the Sysdig backend components and the Cassandra node. The option by default is unchecked.
Cassandra username for authentication: The username of the identity that you want to establish with Cassandra. If running Sysdig’s Cassandra database, create a user here. If you are using your own Cassandra database, enter the appropriate user account for Sysdig access.
As of version 2.4.1, authenticating Sysdig backend components on both Sysdig’s Elasticsearch cluster or for your own Elasticsearch cluster is supported. In order to authenticate the backend components to Elasticsearch datastore, configure TLS-based authentication. You generate certificates and keys for Elasticsearch server, client, and admin user, and specify them along with Elasticsearch user credentials while setting up Sysdig platform. This is the additional layer of security to safeguard the datastore.
Before you configure Elasticsearch authentication, ensure that you set up Sysdig Agent for data collection and TLS generate certificates.
Log into Quay:
Locate your Quay pull_secret. Contact Support if you are unable to locate it.
Get your credentials by running:
# Note: For MacOS users, change "base64 -d" to "base64 -D"echo <quay_pull_secret> | base64 -d | awk NR==4 | cut -d'"' -f4 | xargs | base64 -d
The Output should look as follows:
sysdig+<your_username>:<your_password>
Log into Quay by running the following:
docker login quay.io -u sysdig+<your_username> -p <your_password>
Run the following docker command to generate the root/admin certificates for Elasticsearch to a directory within the current working directory:
docker run -d -v "$(pwd)"/generated_elasticsearch_certs:/tools/out quay.io/sysdig/elasticsearch:1.0.1-es-certs
The following files are generated in the generated_elasticsearch_certs directory. Retain the certificates and key files to upload as part of the TLS configuration as described in Configure TLS Authentication.
Elasticsearch root CA
root-ca.pem
root-ca.key
Elasticsearch Admin (Kirk)
kirk.pem
kirk.key
EElasticsearch Client (Spock)
spoke.pem
spoke.key
Sysdig Replicated install supports Search Guard to establish secure authentication with Elasticsearch datastore. You set up two users in order to access Elasticsearch datastore on behalf of the Sysdig backend components: Admin user and read-only user.
Amin user: The admin user will have the read and write permissions on Elasticsearch clusters and indices. Sysdig backend components use this identity to write data to Elasticsearch clusters. This is the same as the Search Guard admin user.
Read-only user: As the name implies, the read-only user will only have the read permission on Elasticsearch indices. Sysdig Agent uses this identity to read data from Elasticsearch datastore. This is the same as the Search Guard sg_readonly user that is created as part of the installation.
Enable Elasticsearch Authentication and TLS: Select this option to enable authentication and secure communication between Sysdig backend components and the Elasticsearch datastore. To gain access to Elasticsearch datastore, you must prove your identity, by using credentials and certificates. The Elastic Stack authenticates users by identifying the users behind the requests that hit the datastore and verifying that they are who they claim to be.
Elasticsearch admin username: The admin user is created by default. You can edit the user name if desired. The default user is admin.
Elasticsearch admin password: The password associated with the Elasticsearch admin user.
Elasticsearch read-only username: Specify the username for the read-only access to the Elasticsearch indices. If running your own secure Elasticsearch cluster, enter the username for the read-only Search Guard user.
Elasticsearch read-only password: The password associated with Elasticsearch read-only username.
When fields are complete, click Save.
After saving, click Restart Now to apply settings to the environment immediately.
Click Cancel to apply settings at a later time.
If you are monitoring Elasticsearch with sysdig-agent, ensure the sysdig-agent configuration file, dragent.yaml, has the following Elasticsearch configuration in the data.dragent.yaml.app_checks section below:
app_checks:
- name: elasticsearch
check_module: elastic
pattern:
port: 9200
comm: java
conf:
url: https://<DNS_or_ip_address_to_elasticsearch>:9200
username: <your_read_only_username>
password: <your_read_only_password>
ssl_verify: false
Follow these steps if you are running the Agent in a Docker container:
READONLY_USERNAME=<your_readonly_username>
READONLY_PASSWORD=<your_readonly_username_password>
ELASTICSEARCH_PORT=9200
URL_TO_SECURE_ELASTICSEARCH=https://<your_url_to_secure_elasticsearch>
ADDITIONAL_CONF="$(echo "app_checks:
- name: elasticsearch
check_module: elastic
pattern:
port: $ELASTICSEARCH_PORT
comm: java
conf:
url: $URL_TO_SECURE_ELASTICSEARCH:$ELASTICSEARCH_PORT
username: $READONLY_USERNAME
password: $READONLY_PASSWORD
ssl_verify: false
" | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g')"
Remove the existing Agent container:
Make sure that you remove the existing Agent container instead of just stopping it. By default, the Agent container is named sysdig-agent. If you stop the Agent container and attempt to create a new one, you will get a name-conflict error:
docker: Error response from daemon: Conflict. The container name “/sysdig-agent” is already in use by container <ontainer-id>. You have to remove (or rename) that container to be able to reuse that name.
Run the Agent container with the new additional config. For example:
docker run \
--name sysdig-agent \
--restart always \
--privileged \
--net host \
--pid host \
-e ACCESS_KEY=1234-your-key-here-1234 \
-e COLLECTOR=collector_ip \
-e COLLECTOR_PORT=6443 \
-e SECURE=true \
-e TAGS=dept:sales,local:NYC \
-e ADDITIONAL_CONF="$ADDITIONAL_CONF" \
-v /var/run/docker.sock:/host/var/run/docker.sock \
-v /dev:/host/dev \
-v /proc:/host/proc:ro \
-v /boot:/host/boot:ro \
-v /lib/modules:/host/lib/modules:ro \
-v /usr:/host/usr:ro \
quay.io/sysdig/agent
You may encounter an error in the sysdig-agent logs stating that an unverified HTTPS request has been made. You can safely ignore the error for now.
Do the following if you are running the Agent directly on the machine (non-containerized environment):
Add the app_check configuration to your /opt/draios/etc/dragent.yaml configuration:
app_checks:
- name: elasticsearch
check_module: elastic
pattern:
port: 9200
comm: java
conf:
url: https://<DNS_or_ip_address_to_elasticsearch>:9200
username: <your_read_only_username>
password: <your_read_only_password>
ssl_verify: false
Restart the agent:
service dragent restart
After completing the Settings and restarting, no further installation steps are required for a single-host install.
The dashboard will remain in Starting mode for approximately 4-5 minutes, depending on the internet connection bandwidth, while Sysdig application software is downloaded and installed. Once the installation is complete, the dashboard will move to Started mode.
Click the Open
link to navigate to the Sysdig Monitor login panel.
Input the Super Admin user login credentials defined in the basic settings, above.
Next Steps
To start, stop, and update the application, or to retrieve support
information, use the Replicated Admin Console:
https://<yourserver>:8800
.
To login as a user and see metrics for hosts with the Sysdig Agent
installed, use the Sysdig Monitor Web Interface:
https://<yourserver>:80
After configuring the settings and clicking Start Now
, an error will
indicate the need to assign and install the remaining components. You
will need to define the hosts/nodes to be used and will assign the
Sysdig components to be installed on them. The steps below describe the
actions on one host; they must be repeated on all applicable hosts and
all the Sysdig components must be assigned.
Choose the Cluster
tab in the Replicated Admin Console.
From here, you can tag components to be run on the local host, and/or add new nodes.
To add and configure new nodes:
Click Add Node
.
The Add Node worksheet is displayed. Here you enter the IP address and then tag the Sysdig component(s) to be installed on that node.
Replicated will compile either an installation script or a Docker run command out of your entries, which you will copy and use on the given node.
On the Add Node worksheet page, do the following:
Choose Installation script
or Docker run command
option.
Enter private and/or public IP address, depending on the type of access you want to permit.
Select the Sysdig components to be installed by checking the appropriate “Tags” buttons.
Descriptions in the table below:
Name | Tag | Role Description |
api | api | Application Programming Interface server |
cassandradb | cassandra | Cassandra database server |
elasticsearch | elasticsearch | Elasticsearch server for events storage/search |
collector | collector | Agent metrics collector |
lb_collector | lb_collector | Load balancer for collector service; handles connections from the agents |
lb_api | lb_api | Load balancer for API service; handles user connection requests to the Sysdig application. Use the address for this node as the DNS entry for the cluster. |
mysql, redis | mysql & redis | MySQL & Redis databases |
worker | worker | Metrics history processor |
emailrenderer | emailrenderer | Email renderer |
nginxfrontend | nginxfrontend | Frontend static server |
When setting up a DNS entry for the cluster, use the address for the
‘lb_api'
node.
At the bottom of the page, a curl
script or Docker run command is
compiled for you.
Copy the command and issue it on the targeted host.
Repeat this procedure on all desired hosts.
Restart the Sysdig application from the Replicated console.
The dashboard will be in “Starting” mode for several minutes while software is downloaded and installed onto each server component (depending on your internet connection bandwidth).
You should see green check marks for each host next to the Provisioned and Connected columns, as the software is installed and the node connects successfully to the Replicated Admin server.
Once the installation is fully completed, the infrastructure admin dashboard will be in “Started” mode and will also show the “Open” link that will bring you to Sysdig Monitor web interface login screen.
At the login screen, use the credentials configured earlier (Default User) to log in and start using the Sysdig application on-premises solution.
To start, stop, and update the application or retrieve support information use the Replicated Admin dashboard: https://server_address:8800
To log in as a user and see metrics about hosts where Sysdig agents are installed, use the Sysdig Monitor UI: https://server_address:80
Sysdig will deprecate support for Replicated installs in the coming months. If you are a new customer considering installing with Replicated, please contact Sysdig support.
These configurations are optional.
This process differs depending on how you installed the Sysdig Platform.
If you installed the Sysdig Platform on Kubernetes or OpenShift using the Installer, the Installer automatically generates a self-signed cert on the fly. To use a different certificate you would:
Add your cert and key to the /certs directory ex: (server.crt, server.key)
Update values.yaml:
sysdig:
certificate:
crt: certs/server.crt
key: certs/server.key
Rerun the Installer.
The configuration_parameter.md
Readme
gives full details on sysdig.certificate.crt
and
sysdig.certificate.key
.
If you installed the Sysdig Platform manually on Kubernetes or OpenShift, the steps for managing the certs are described in Step 5 of the installation procedures:
If you installed the Sysdig Platform using Replicated and you accepted the self-signed certificate for SSL/TLS communication when installing the Sysdig components (see Define Basic Settings & License Info ), you can exchange for a custom certificate as follows:
Log in to the Replicated Management Console and select the
Gear icon > Console Settings.
Click Upload certificate
and it will automatically replace the
original self-signed certificate.
Sysdig Monitor/Cloud/etc uses a self-signed SSL/TLS security certificate, unless a custom certificate is provided.
The example command below creates a custom, unsigned certificate called
MyCert.pem
; the certificate has a private key called MyCert.key
, and
is valid for five years:
sudo openssl req -new -x509 -sha256 -days 1825 -nodes -out ./MyCert.pem -keyout ./MyCert.key
When experiencing issues, you can collect troubleshooting data that can
help the support team. The data can be collected by hand, or Sysdig
provides a very simple get_support_bundle.sh
script that takes as an
argument the namespace where Sysdig is deployed and will generate a
tarball containing some information (mostly log files). The script is
located in the GitHub repository.
$ ./scripts/get_support_bundle.sh sysdigcloud
Getting support logs for sysdigcloud-api-1477528018-4od59
Getting support logs for sysdigcloud-api-1477528018-ach89
Getting support logs for sysdigcloud-cassandra-2987866586-fgcm8
Getting support logs for sysdigcloud-collector-2526360198-e58uy
Getting support logs for sysdigcloud-collector-2526360198-v1egg
Getting support logs for sysdigcloud-mysql-2388886613-a8a12
Getting support logs for sysdigcloud-redis-1701952711-ezg8q
Getting support logs for sysdigcloud-worker-1086626503-4cio9
Getting support logs for sysdigcloud-worker-1086626503-sdtrc
Support bundle generated: 1473897425_sysdig_cloud_support_bundle.tgz
Some issues with IPv4 and IPv6 interconnectivity between on-premises containers and the outside world have been detected.
IP packet forwarding is governed by the ip_forward
system parameter.
Packets can only pass between containers if this parameter is 1
.
Usually, you will simply leave the Docker server at its default setting
--ip-forward=true
and Docker will go set ip_forward
to 1
for you
when the server starts up. If you set --ip-forward=false
and your
system’s kernel has it enabled, the --ip-forward=false
option has no
effect.
To check the setting on your kernel use:
sysctl net.ipv4.conf.all.forwarding
To turn it on use:
sysctl net.ipv4.conf.all.forwarding=1
Please see this article from docker for more details on Docker Connectivity.
Prior to installing ensure your proxy settings are valid for the
session. You can use curl
, lynx
, or wget
to test internet
connectivity:
export http_proxy="http://user:password@proxy_server:port"
export https_proxy="https://user:password@proxy_server:port"
echo $http_proxy
You can then attempt a curl or docker hub call to ensure outside connectivity.
Prior to installation, you may want to disable local firewall (iptables) to rule out local connectivity issues.
However here are some details around Sysdig connectivity and backend connectivity requirements.
Sysdig Connectivity:
6443 Agent communication
443 Sysdig Monitor UI access
8800 Management console access
Here are specifics around what is used for connectivity for the Sysdig backend for on-premises solution:
https://www.replicated.com/docs/kb/supporting-your-customers/firewalls/
During the install, you may see errors writing to volumes such as
(/var
or /opt
) from either the onprem install scripts or Docker. You
should disable SELINUX
(CENTOS/RHEL) or Apparmor
(UBUNTU/DEBIAN)
during the course of the install so the valid directories can be
created. This can be accomplished by:
Centos (SELINUX)
From the command line, edit the /etc/sysconfig/selinux
file. This file
is a symlink to /etc/selinux/config
. The configuration file is
self-explanatory. Changing the value of SELINUX
or
*SELINUXTYPE
*changes the state of SELinux and the name of the policy
to be used the next time the system boots.
[root@host2a ~]# cat /etc/sysconfig/selinux
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - SELinux is fully disabled.
SELINUX=permissive
# SELINUXTYPE= type of policy in use. Possible values are:
# targeted - Only targeted network daemons are protected.
# strict - Full SELinux protection.
SELINUXTYPE=targeted
# SETLOCALDEFS= Check local definition changes
SETLOCALDEFS=0
See SELinux Modes for more information.
UBUNTU/Debian (AppArmor)
AppArmor can be disabled, and the kernel module unloaded by entering the following:
sudo systemctl stop apparmor.service
sudo update-rc.d -f apparmor remove
To re-enable AppArmor enter:
sudo systemctl start apparmor.service
sudo update-rc.d apparmor defaults
In the preflight check step with Replicated, if you come across the error:
getsockopt: no route to host
Please do the following:
For CentOS 7/RedHat:
Log in as root or run these commands via sudo:
service firewalld stop
systemctl disable firewalld
sysctl -w net.ipv4.ip_forward=1
iptables -F
setenforce 0
service docker restart
For Ubuntu:
Log in as root or run these commands via sudo:
sysctl -w net.ipv4.ip_forward=1
systemctl stop apparmor.service
update-rc.d -f apparmor remove
ufw disable
iptables -F
service docker restart
This section describes how to upgrade an on-premise installation, depending on whether it was installed using a Kubernetes or a Replicated orchestrator.
As needed, version-specific upgrade or migration instructions will be added to this section.
As part of our continued focus on our customers, we are now offering oversight services for all on-premise installs and upgrades. Your Technical Account Manager (TAM), in conjunction with our support organization and Professional Services [where applicable], will work with you to:
Assess your environment to ensure it is configured correctly
Review your infrastructure to validate the appropriate storage capacities are available
Review and provide recommendations for backing up your Sysdig data
Work with you to ensure our teams are ready to assist you during the install and upgrade process
Provide the software for the install
Be available during the process to ensure a successful deployment
You can always review the process in the documentation on GitHub (v. 3.6.0+) or the standard docs site (for older versions).
If you are a new customer looking to explore Sysdig, please head over here to sign up for a trial on our SaaS Platform. Alternatively, you can contact us here.
This section provides roadmaps for upgrading Sysdig Platform components. Review the topics appropriate to your environment.
Topics | Description |
---|---|
Supported Upgrade Paths | Understand the upgrade and migration paths for on-prem installations. |
Installer Upgrade (3.5.0-3.5.1) | Upgrading Sysdig Platform to v 3.5.0 from v. 3.0, 3.2.x using the installer tool. There is no manual install option as of version 3.5.0. |
Installer Upgrade (2.5.0+) | Upgrading Sysdig Platform v2.5.0 - 3.2.2 by using the Installer tool. As of version 2.5.0, the Sysdig platform on Kubernetes or OpenShift should be upgraded using the Installer tool. |
Manual Upgrade (3.0.0+) | Upgrading Sysdig Platform v3.0.0 and above manually on Kubernetes. |
Upgrading v2.4.1- v2.5.0 on Kubernetes | Upgrading the Sysdig Platform versions between 2.4.1and 2.5.0 manually on Kubernetes. |
Upgrading v2.3.0 on Kubernetes | Upgrading Sysdig Platform v2.3.0 manually on Kubernetes. |
Upgrading v2435 on Kubernetes | Upgrading Sysdig Platform v2435 manually on Kubernetes. |
Basic Upgrade on Replicated | Upgrading the mandatory components of the Sysdig Platform on a Replicated environment. |
In general, Sysdig tests and supports direct upgrade from five releases back to the current version. Release-specific requirements are listed in the table below.
Install Version | Incl. Hotfixes | Supported Upgrade From | Notes | Baseline Documentation |
---|---|---|---|---|
3.6.0 (by request) | 3.2.2, 3.5.1 | Platform changes: new inline scanner version, interactive session expiration. Sysdig Secure modules added/changed, including Compliance, Event Forwarding, Capture improvements, Image Scan results. Sysdig Monitor improvements in UI. | ||
3.5.1 (by request) | 3.0, 3.2.x, (3.5.0 if it was installed) | New/changed modules in both Sysdig Secure and Sysdig Monitor, including: New Getting Started and Overview, new Dashboards, overhauled Secure Events Feed, new navigation bar icons and layout, changed Image scanning navigation and usage, new Secure vulnerability feed and benchmark test, | Installer Upgrade (3.5.0-3.5.1) with oversite assistance from Sysdig Support | |
2.5.0, 3.2.0 | Hot fix | |||
3.2.1, 3.2.2 | 2.5.0, 3.0.0 | In Sysdig Secure: Data retention limits for scan results, vulnerabiity comparison in scan results, redesigned Captures page, RBAC capability, activity audit improvement. In Sysdig Monitor and Secure: S3-compatible storage for Capture files. | ||
2.4.1, 2.5.0 | Beta Activity Audit feature, Beta Policy Advisor feature. | |||
2.3.0, 2.4.1 | New Installer tool for upgrading; new documentation site; inline scanning for Secure, other enhancements | |||
2.3.0 | Report service added; upgrade to Anchore license required | |||
1929, 2435 | Ability to secure Elasticsearch and the Cassandra DB with password authentication or SSL/TLS protection. | |||
(2304) (2266) (2172) | 1929, 1765 | Architecture changes to Dashboards & API pods. | Note that this replaces 2172, 2266, and 2304. | |
legacy | ||||
Migration Tool (MySQL Connector) Architecture & Port 443 change | legacy | |||
((1586) | legacy | |||
(1472) (1402) | legacy | |||
legacy | ||||
Migration Tool (Unified Events) | legacy | |||
legacy |
Most Replicated environments can be upgraded directly to the current version. See also: Basic Upgrade (Replicated).
It is recommended to follow upgrade best practices:
Keep upgrades current.
Test upgrades in a non-mission-critical or staging environment before rolling into production.
As of version 3.5.0/3.5.1, Sysdig has changed its upgrade procedure.
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
With version 3.5.0, both installing and upgrading with the installer has
been simplified from previous versions. Upgrade differs from Install in
that you run an installer diff
to discover the differences between the
old and new versions and then installer deploy for the new version.
Some guidance from Sysdig Support may be warranted in highly customized installations.
Upgrading is performed just like a fresh install, with the addition of
the generate diff
step. Refer to the appropriate workflow, depending
on your environment:
Version 3.5.0 upgrade includes an automatic Postgres version upgrade. Depending on the size of your database, that can take some time.
The data migration takes approximately 1 min per 1 GiB of data. The speed of data migration ultimately depends on the underlying storage media.
To complete the Postgres upgrade, you must also ensure there is sufficient disk space in the volume when using a local-disk storage provisioner in Kubernetes. For example, if your current Postgres size is 100 GiB, ensure there is at least another 100 GiB space free in the volume. This is required temporarily for copying the data during the upgrade.
Upgrading from version 3.5.0 to 3.5.1 also upgrades Elasticsearch. Due
to the Elasticsearch update strategy of ondelete
, the pods will only
be upgraded when they are restarted:
image: quay.io/sysdig/elasticsearch:6.8.3.7
image: quay.io/sysdig/elasticsearch:6.8.3.9
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
The Installer tool can be used to upgrade a Sysdig implementation. Just
as in an installation, you must meet the prerequisites, download the
values.yaml
, edit the values as indicated, and run the installer. The
main difference is that you run it twice: once to discover the
differences between the old and new versions and the second time to
deploy the new version.
As this is a new feature, some guidance from Sysdig Professional Services may be warranted in highly customized installations.
On-Prem Version | Installer Version |
---|---|
3.0.0 | 3.0.0-7 |
3.2.0 | 3.2.0-9 |
3.2.2 | 3.2.2-1 |
To upgrade:
Download the latest installer binary that matches your OS from the sysdigcloud-kubernetes releases page.
Copy the current version of values.yaml to your working directory.
wget https://raw.githubusercontent.com/draios/sysdigcloud-kubernetes/installer/installer/values.yaml
Edit the following values:
scripts: set to generate diff.
This setting will generate the differences between the installed environment and the upgrade version. The changes will be displayed in your terminal.
size: Specifies the size of the cluster. Size defines CPU, Memory, Disk, and Replicas. Valid options are: small, medium and large
quaypullsecret: quay.io provided with your Sysdig purchase confirmation mail
storageClassProvisioner: The name of the storage class
provisioner to use when creating the configured
storageClassName
parameter. When installing, if you use AWS or
GKE as your storage provisioner for Kubernetes, enter aws
or
gke
in the storageClassProvisioner
field. If you do not use
one of those two dynamic storage provisioners, enter: hostPath
and then refer to the Advanced examples for how to configure
static storage provisioning using this option.
sysdig.license: Sysdig license key provided with your Sysdig purchase confirmation mail
sysdig.anchoreLicensePath: The path relative to the
values.yaml
where the Anchore enterprise license yaml is
located. (For Sysdig Secure users only.)
sysdig.dnsname: The domain name the Sysdig APIs will be served on. Note that the master node may not be used as the DNS name when using hostNetwork mode.
sysdig.collector.dnsName: (OpenShift installs only) Domain name the Sysdig collector will be served on. When not configured it defaults to whatever is configured for sysdig.dnsName. Note that the master node may not be used as the DNS name when using hostNetwork mode.
deployment: (OpenShift installs only) Add
deployment: openshift
to the root of the values.yaml
file.
sysdig.ingressNetworking: The networking construct used to expose the Sysdig API and collector.Options are:
hostnetwork: sets the hostnetworking in the ingress daemonset and opens host ports for api and collector. This does not create a Kubernetes service.
loadbalancer: creates a service of type loadbalancer and expects that your Kubernetes cluster can provision a load balancer with your cloud provider.
nodeport: creates a service of type nodeport.The node ports can be customized with:
sysdig.ingressNetworkingInsecureApiNodePort
sysdig.ingressNetworkingApiNodePort
sysdig.ingressNetworkingCollectorNodePort
If doing an airgapped install , you would also edit the following values:
airgapped_registry_name: The URL of the airgapped (internal) docker registry. This URL is used for installations where the Kubernetes cluster can not pull images directly from Quay
airgapped_registry_password: The password for the configured airgapped_registry_username. Ignore this parameter if the registry does not require authentication.
airgapped_registry_username: The username for the configured airgapped_registry_name. Ignore this parameter if the registry does not require authentication.
Run the installer.
For environments with access to the internet:
docker run -e HOST_USER=$(id -u) -e KUBECONFIG=/.kube/config
-v ~/.kube:/.kube:Z -v $(pwd):/manifests:Z
quay.io/sysdig/installer:<version>
For other supported installer versions, see Supported Installer Versions.
For partial-airgap (installation machine has access to the internet):
docker run -e HOST_USER=$(id -u) -e KUBECONFIG=/.kube/config
-v ~/.kube:/.kube:Z
-v $(pwd):/manifests:Z
-v /var/run/docker.sock:/var/run/docker.sock:Z
-v ~/.docker:/root/docker:Z
quay.io/sysdig/installer:<version>
For other supported installer versions, see Supported Installer Versions.
For full airgapped environment:
Run steps 1-4 in the Full Airgap Install, then run:
bash sysdig_installer.tar.gz
If you are fine with the differences displayed, then set scripts
to deploy
and rerun the installer as in Step 3.
If you want to override a change, based on your environment’s custom settings, then contact Sysdig Support for assistance.
The datastores Cassandra and ElasticSearch have an onDelete
update
strategy and must be manually restarted to complete the upgrade.
As of August 2020, Sysdig has changed its upgrade procedure.
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
Sysdig platform on-premise releases are listed here. Each release has a version number and specific release notes.
This release has the following significant changes:
Added NATS service to deliver events to the Sysdig backend
Added services for the beta Policy Advisor, which permits a user to auto-generate Pod Security Policies and perform dry tests or “simulations” of them before committing them to an environment.
Added services for activity audit, which allows users to view different data sources in-depth for monitoring, troubleshooting, diagnostics, or to meet regulatory controls
Some Anchore reporting components are not needed anymore and have been removed.
Download the new version from Sysdig’s GitHub and unzip it.
wget https://github.com/draios/sysdigcloud-kubernetes/archive/<version_number>.tar.gz && tar xvf <version_number>.tar.gz
It is important to use the latest YAML files for a successful upgrade.
Edit the following files within the sysdigcloud
directory to match any
customizations you may have made in your existing production system.
Please do not edit the image:
property.
Ensure that any passwords or user names are transferred from your existing config.yaml to the new one. Suggested areas to review are listed below.
config.yaml:
The following variables are always customized in Sysdig installations:
api.url
collector.endpoint
sysdigcloud.license
mysql.password
Modifying following variables is optional but commonly done:
cassandra.jvm.options
elasticsearch.jvm.options
sysdigcloud.jvm.api.options
sysdigcloud.jvm.collector.options
sysdigcloud.jvm.worker.options
Check deployment YAML files for CPU/memory settings.
Update the spec.replicas
definition in the following files:
sysdigcloud/api-deployment.yaml
sysdigcloud/collector-deployment.yaml
sysdigcloud/worker-deployment.yaml
If running Sysdig Secure:
sysdigcloud/anchore-core-config.yaml
sysdigcloud/anchore-worker-config.yaml
sysdigcloud/anchore-core-deployment.yaml
sysdigcloud/anchore-worker-deployment.yaml
sysdigcloud/scanning-api-deployment.yaml
sysdigcloud/scanning-alertmgr-deployment.yaml
postgres-statefulset.yaml
: Edit the storage class name in this
file.
The file is located in
datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulsets.yaml
Storage class name appears as
spec.volumeClaimTemplates[].spec.storageClassName
elasticsearch-statefulset.yaml
: For example, your environment may
have customized the values for the number of replicas, resource
constraints, amount of storage, and the storage class name:
spec.replicas and spec.template.spec.containers[elasticsearch].env[ELASTICSEARCH_GOSSIP_NODES_NUM].value
spec.template.spec.containers[].resources
spec.volumeClaimTemplates[].spec.resources.requests.storage
spec.volumeClaimTemplates[].spec.storageClassName
cassandra-statefulset.yaml
: As with Elasticsearch, your
environment may have customized the values for the number of
replicas, resource constraints, amount of storage, and the storage
class name:
spec.replicas
spec.template.spec.containers[].resources
spec.volumeClaimTemplates[].spec.resources.requests.storage
spec.volumeClaimTemplates[].spec.storageClassName
The --force
flag deletes the object and re-creates it whereas the
--replace
flag automatically creates an object if it doesn’t exist.
For the upgrade, assume NAMESPACE=sysdigcloud
.
In version 3.0, a NATS datastore was introduced for handling events inside the Sysdig platform:
kubectl -n $NAMESPACE apply -f datastores/as_kubernetes_pods/manifests/nats-streaming/nats-streaming-deployment.yaml
kubectl -n $NAMESPACE apply -f datastores/as_kubernetes_pods/manifests/nats-streaming/nats-streaming-service.yaml
Run the kubectl
commands to apply the relevant files to your cluster.
kubectl -n $NAMESPACE apply -f sysdigcloud/config.yaml
kubectl -n $NAMESPACE replace --force -f datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-statefulset.yaml
kubectl -n $NAMESPACE replace --force -f datastores/as_kubernetes_pods/manifests/cassandra/cassandra-statefulset.yaml
Pause to allow Elasticsearch and Cassandra to come up. then continue:
kubectl -n $NAMESPACE apply -f sysdigcloud/api-deployment.yaml
Pause to allow api to come up, then continue:
kubectl -n $NAMESPACE apply -f sysdigcloud/collector-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/worker-deployment.yaml
Run the kubectl
commands to apply the relevant files to your cluster.
kubectl -n $NAMESPACE replace --force -f datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-core-config.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-worker-config.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-core-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-worker-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/scanning-api-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/scanning-alertmgr-deployment.yaml
Create secrets for the new policy advisor and activity audit components by deploying the policy-advisor-secret.yaml.
kubectl -n $NAMESPACE apply -f sysdigcloud/policy-advisor-secret.yaml
Deploy the components:
kubectl -n $NAMESPACE apply -f sysdigcloud/policy-advisor-service.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/activity-audit-api-service.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/activity-audit-api-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/policy-advisor-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/activity-audit-worker-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/activity-audit-janitor-cronjob.yaml
You can delete the Anchore reporting components to free up system resources:
kubectl -n $NAMESPACE delete -f sysdigcloud/anchore-enterprise-license.yaml
kubectl -n $NAMESPACE delete -f sysdigcloud/anchore-reports-config.yaml
kubectl -n $NAMESPACE delete -f sysdigcloud/anchore-reports-deployment.yaml
kubectl -n $NAMESPACE delete -f sysdigcloud/anchore-reports-service.yaml
As of August 2020, Sysdig has changed its upgrade procedure.
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
Sysdig platform on-premise releases are listed here. Each release has a version number and specific release notes.
This release has the following significant component change:
The Report service is now available for Sysdig Secure. Installing it requires first applying an Anchore license and then applying the appropriate report yamls, as listed below.
Download the new version from Sysdig’s GitHub and unzip it.
Note that as of this release, versioning standards have changed from a single build number (e.g. v1929) to semantic versioning (e.g. 2.3.0)
wget https://github.com/draios/sysdigcloud-kubernetes/archive/<version_number>.tar.gz && tar xvf <version_number>.tar.gz
It is important to use the latest YAML files for a successful upgrade.
Edit the following files within the sysdigcloud
directory to match any
customizations you may have made in your existing production system.
Customization involves copying the existing settings from your environment and modifying the files listed in this section.
Update the following files with customizations from your existing environment:
sysdigcloud/config.yaml:
Pull configurations from your
sysdigcloud-config configmap
to the downloaded
sysdigcloud/config.yaml
.
The following variables are mandatory for Sysdig installations:
api.url
collector.endpoint
sysdigcloud.license
The following variables are optional but commonly modified for Sysdig installations:
cassandra.jvm.options
elasticsearch.jvm.options
sysdigcloud.jvm.api.options
sysdigcloud.jvm.collector.options
sysdigcloud.jvm.worker.options
If you have modified the previous config.yaml
, copy the modified
options such as the external endpoints. You must also check
deployment YAML files for CPU/memory settings.
Copy configurations from your existing deployment and update the
spec.replicas
definition in the following files:
sysdigcloud/api-deployment.yaml
sysdigcloud/collector-deployment.yaml
sysdigcloud/worker-deployment.yaml
If running Sysdig Secure:
Please do not edit the image:
property.
sysdigcloud/anchore-core-config.yaml
sysdigcloud/anchore-worker-config.yaml
sysdigcloud/anchore-core-deployment.yaml
sysdigcloud/anchore-worker-deployment.yaml
sysdigcloud/scanning-api-deployment.yaml
sysdigcloud/scanning-alertmgr-deployment.yaml
Update the following file with customizations from your existing environment:
Please do not edit the image:
property.
spec.volumeClaimTemplates[].spec.storageClassName
in the
datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
file.In version 2.3.0, Elasticsearch and Cassandra yaml configurations have been updated. Update the new files with customizations from your existing environment.
Please do not edit the image:
property.
elasticsearch-statefulset.yaml
- For example, your environment may
have customized the values for the number of replicas, resource
constraints, amount of storage, and the storage class name:
spec.replicas and spec.template.spec.containers[elasticsearch].env[ELASTICSEARCH_GOSSIP_NODES_NUM].value
spec.template.spec.containers[].resources
spec.volumeClaimTemplates[].spec.resources.requests.storage
spec.volumeClaimTemplates[].spec.storageClassName
cassandra-statefulset.yaml
- As with Elasticsearch, your
environment may have customized the values for the number of
replicas, resource constraints, amount of storage, and the storage
class name:
spec.replicas
spec.template.spec.containers[].resources
spec.volumeClaimTemplates[].spec.resources.requests.storage
spec.volumeClaimTemplates[].spec.storageClassName
Run the kubectl
commands to apply the relevant files to your cluster.
The --force
flag deletes the object and re-creates it whereas the
--replace
flag automatically creates an object if it doesn’t exist.
NAMESPACE=sysdigcloud
kubectl -n $NAMESPACE apply -f sysdigcloud/config.yaml
kubectl -n $NAMESPACE replace --force -f datastores/as_kubernetes_pods/manifests/elasticsearch/elasticsearch-statefulset.yaml
kubectl -n $NAMESPACE replace --force -f datastores/as_kubernetes_pods/manifests/cassandra/cassandra-statefulset.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/api-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/collector-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/worker-deployment.yaml
For versions 2.4.1 and higher: To use the Reports functionality in
Sysdig Secure, it is necessary to enter a license key in the
anchore-license.yaml
. If you are upgrading or installing and do not
have an anchore license please contact support. This license is used for
additional 3rd party vulnerability feed entitlements.
Edit the license YAML
file: sysdigcloud/anchore-enterprise-license.yaml
. Replace <LICENSE>
with
the key received from Sysdig.
---
apiVersion: v1
kind: Secret
metadata:
name: anchore-enterprise-license
data:
# <LICENSE> is derived from `cat anchore-license.yaml | base64`
anchore-license.yaml: <LICENSE>
type: Opaque
Run the command:
kubectl -n sysdigcloud apply -f sysdigcloud/anchore-enterprise-license.yaml
Run the following commands, preserving the order:
kubectl -n $NAMESPACE replace --force -f datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-core-config.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-worker-config.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-core-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-worker-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/scanning-alertmgr-service.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/scanning-api-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/scanning-alertmgr-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-enterprise-license.yaml #version 2.4.1 or higher
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-reports-config.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-reports-deployment.yaml
kubectl -n $NAMESPACE apply -f sysdigcloud/anchore-reports-service.yaml
As of August 2020, Sysdig has changed its upgrade procedure.
All on-premises installations and upgrades are now scheduled with and guided by Sysdig technical account managers and professional services division. See Oversight Services Now Offered for All Installs and Upgrades .
For customers, the instructions in this section are for review purposes only.
Sysdig platform on-premise releases are listed here. Each release has a version number and specific Release Notes.
This release has the following significant component changes:
Includes the option of securing Elasticsearch and/or Cassandra with username/password authentication and TLS-encrypted data in transit. This prevents both unauthorized access to the clusters and network eavesdropping. The upgrade instructions below incorporate this new capability when using the Sysdig-provided Cassandra and Elasticsearch components.
If you are running your own Cassandra or Elasticsearch clusters, you can skip the section Elasticsearch and Cassandra Files.
Updates of the Postgres database and Anchore engine (if you are running Sysdig Secure).
The following parameter has been renamed in config.yaml:
elasticsearch.url
to elasticsearch.hostname
The value of elasticsearch.hostname
does not include the protocol
(e.g.http://); just use the hostname itself. For example, if you had
elasticsearch.url: ``http://sysdigcloud-elasticsearch
, it would
now be elasticsearch.hostname: sysdigcloud-elasticsearch
.
Download the new version from Sysdig’s GitHub and unzip it.
Note that as of this release, versioning standards have changed from a single build number (e.g. v1929) to semantic versioning (e.g. 2.3.0)
wget https://github.com/draios/sysdigcloud-kubernetes/archive/<version_number>.tar.gz && tar xvf <version_number>.tar.gz
It is important to use the latest YAML files for a successful upgrade.
Edit the following files within the sysdigcloud
directory to match any
customizations you may have made in your existing production system.
Customization involves copying the existing settings from your environment and modifying the files listed in this section.
Update the following files with customizations from your existing environment:
sysdigcloud/config.yaml:
Pull configurations from your
sysdigcloud-config configmap
to the downloaded
sysdigcloud/config.yaml
.
The following variables are mandatory for Sysdig installations:
api.url
collector.endpoint
sysdigcloud.license
The following variables are optional but commonly modified for Sysdig installations:
cassandra.jvm.options
elasticsearch.jvm.options
sysdigcloud.jvm.api.options
sysdigcloud.jvm.collector.options
sysdigcloud.jvm.worker.options
If you have modified the previous config.yaml
, copy the modified
options such as the external endpoints. You must also check
deployment YAML files for CPU/memory settings.
Copy configurations from your existing deployment and update the
spec.replicas
definition in the following files:
sysdigcloud/api-deployment.yaml
sysdigcloud/collector-deployment.yaml
sysdigcloud/worker-deployment.yaml
If running Sysdig Secure:
Please do not edit the image:
property.
sysdigcloud/anchore-core-config.yaml
sysdigcloud/anchore-worker-config.yaml
sysdigcloud/anchore-core-deployment.yaml
sysdigcloud/anchore-worker-deployment.yaml
sysdigcloud/scanning-api-deployment.yaml
sysdigcloud/scanning-alertmgr-deployment.yaml
Update the following file with customizations from your existing environment:
Please do not edit the image:
property.
spec.volumeClaimTemplates[].spec.storageClassName
in the
datastores/as_kubernetes_pods/manifests/postgres/postgres-statefulset.yaml
file.In version 2.3.0, Elasticsearch and Cassandra yaml configurations have been updated. Update the new files with customizations from yo