Vulnerability Management
About Vulnerability Management
Sysdig Secure Vulnerabilities module addresses the top requirements for effective vulnerability management:
- Provides highly accurate views of vulnerability risk at scale
- Rich details provide precision about vulnerability risk (ex. CVSS vector, score, fix age) and insights from multiple expert feeds (ex. VulnDB)
- Access to public exploits allows you to verify security controls and patch efficiently
- Prioritized risk data focused on the vulnerabilities that are tied to the packages loaded at runtime
- Accepting risks when the vulnerability matching does not apply for your particular deployment, or you want to postpone remediation
At this time, the Vulnerability Management engine supports: CI/CD pipeline and runtime image scanning, policies, notifications, and reporting for runtime. Registry scanning does not yet support policies.
Understand Vulnerability Management Stages
To create an effective vulnerability management plan, it’s important to recognize the various stages of the lifecycle that need to be addressed.
Key Concepts
During the build phase, software vulnerabilities may be introduced when container images are defined and assembled. Container images are, by definition, immutable. Altering the contents of an image will update the ImageID and, thus, will be considered a different image by Sysdig Secure.
Even though unique container images (ImageIDs) cannot be modified, Sysdig can identify new vulnerabilities in running containers (for example, in Kubernetes workloads) as security feeds are continuously updated. For instance, a container image with no known vulnerabilities during its build phase might be affected by a critical vulnerability discovered ten days after being deployed into the runtime. The image remains unchanged, but new security information related to the software it contains has been found.
Sysdig uses the same fundamental concepts to analyze the contents of an image (SBOM) and match vulnerabilities but treats images differently based on their location. Sysdig can analyze the vulnerabilities of images in a development pipeline, stored in a container image registry, or used as the template for a running container, known as runtime workloads.
Stages: Pipeline - Registry - Runtime
Pipeline
Any analysis conducted before the registry phase is considered a pipeline. A clear example is CI/CD builds (Jenkins, Github, etc.), but also the execution of the sysdig-cli-scanner binary performed on a developer laptop or using a custom scanning script.
- Pipeline images do not have runtime context.
- The scan happens outside of the execution nodes where the agent is installed:
- CI/CD
- External instrumentation
- Custom scripts or image scanning plugins
- Pipeline scans are one-off vulnerability reports; the information is a static snapshot with its corresponding execution date.
- If you want to evaluate a newer version of the image or reevaluate the same image with newer feed information, the analysis needs to be triggered again.
- Images analyzed using the sysdig-cli-scanner will show up in the Pipeline section of the vulnerability management interface.
Registry
Container Registry scanning allows you to integrate Sysdig with image registries from a range of vendors. Registry scanning provides an extra layer of defense between pipeline and runtime, where:
- Software that sits in the registry before being deployed is checked for newly discovered vulnerabilities
- Third-party software that may have been installed without going through pipeline scanning will be checked
Registry scans occur as scheduled through a cron job in the installation Helm chart once per week (Saturdays at 6:00 AM) by default.
- Batch scanning is done asynchronously and separately from the development pipeline; regardless of the time it takes to scan the batch, the pipeline is unaffected.
See Install Registry scanner(s).
Runtime
Runtime workloads are executed using a container image. Accessing the Runtime section of the Vulnerabilities menu, you will be able to see those images and their vulnerability and policy evaluation.
- Runtime workloads are located in an execution node and are being monitored by a Sysdig agent/node analyzer, for example a Kubernetes node that is instrumented using the Sysdig agent bundle.
- Runtime workloads will offer a live, auto-refreshing state. This means:
- Workloads that are no longer running will be removed from the runtime view
- Vulnerabilities and policies evaluations will automatically refresh without any user interaction, offering always the most up-to-date information known.
- At least once per day
- Runtime workload have a runtime context associated with them, i.e. Kubernetes cluster and namespace.
- Workloads analyzed during runtime will show up in the Runtime section of the vulnerability management interface.
See Runtime.
Asset Types: Workload, Host, and Container
The way things are scanned and how results are filtered relies on how Sysdig defines and uses the concepts of “image,” “workload,” “host,” and “container.”
Image
Image is not officially an asset type, but is the primary unit underlying all types of scanning.
In Pipeline and Registry scanning, Sysdig scans images used in container-based environments.
In Runtime, images have the context of either Kubernetes or the container host environment.
These images are strictly evaluated based on the image definition without reference to the deployed context (neither runtime nor orchestrator).
All images are extracted into a Software Bill of Materials (SBOM) format that is compatible with the CycloneDX standard. See CycloneDX for details.
Workload
When scanning images that are deployed in Kubernetes environments, the scanner evaluates the image in the context of the container running in a Kubernetes workload. This context makes it contextually different from images and containers.
Host
A host is a full-stack Operating System (OS) running on virtual infrastructure such as Cloud IaaS or Virtual Machines. The host will also have an SBOM, without the layers that are seen in images. All the context data for a host is relative to a full-stack OS with network connectivity (OS, Host Name, IPs, etc.).
Container
When Sysdig scans a container (either agentlessly or with the agent) it extracts a container from a Host image via the filesystem. In container scans, only the parent host context is available, not the runtime and orchestration context.
To be scanned by Sysdig, containers must all be compliant with the Open Container Initiative (OCI). See OCI for details.
Usage
When defining a Vulnerability policy or filtering Runtime vulnerability results, you can define the asset type to use.
Key Vulnerability Management Terminology
A cybersecurity vulnerability is a flaw in a computer system that enables an adversary to gain direct access to a network or system to compromise security and inflict harm.
CVE
Common Vulnerabilities and Exposures (CVE) is the database of such known cybersecurity vulnerabilities and exposures.
You can view the following information for all the asset types in all the vulnerability management stages:
- The severity level reported by National Vulnerability Database
- The package where vulnerability was found
- The directory path to the package
- The version in which the vulnerability was fixed
CVSS
The Common Vulnerability Scoring System (CVSS) is used to assess the severity of information security vulnerabilities. Each entry in the CVE database is assigned a corresponding CVSS score, which quantifies the severity of the vulnerability.
You can view the following information for all the asset types in all the vulnerability management stages:
The sources selected for score calculation. Fo rexample, National Vulnerability Database
The CVSS Score and version
The date when the vulnerability was discovered.
CISA KEV
The Known Exploited Vulnerabilities (KEV) Catalog maintained by CISA (Cybersecurity and infrastructure Security Agency). It is the list of software flaws that have already been exploited in the real-world.
You can view the following information for all the asset types in all the vulnerability management stages:
- The date when the vulnerability was added to the KEV catalog
- The deadline for when the vulnerability should have a fix
- If the vulnerability has been used in any Ransomware campaigns
Exploit
An exploit is a software program created to identify and exploit security flaws or vulnerabilities present in a system.
You can view the following information for all the asset types in all the vulnerability management stages:
- Link to the existing exploit PoC or source code
- The date when the exploit was disclosed.
Vulnerability Databases
Sysdig Secure continuously checks against a wide range of vulnerability databases, updating the Runtime scan results with any newly detected CVEs.
The current database list includes:
- NIST NVD
- VulnDB
- NPM
- Python
- Ruby
- Alpine Linux
- Centos
- Debian
- Red Hat
- Ubuntu
- Amazon Linux
- Alibaba Linux
- Oracle Linux
- Chainguard
- Wolfi
Getting Started with Vulnerability Management
For full vulnerability coverage, ensure you have completed the prerequisites:
A current Sysdig agent that includes the vulnerability management engine and runtime scanner.
The installed host scanner, whether agent-based (for Kubernetes, as a container, or as a package), or agentless (for AWS cloud accounts)
The downloaded cli-scanner for pipeline.
The installed container registry scanners for registries.
Advanced User+ permissions
Log in to Sysdig Secure and select Vulnerabilities.
The out-of-the-box policies for Pipeline and Runtime vulnerabilities will work without further setup.
Choose Pipeline, Registry, or Runtime to see the scanning results.
Choose Reporting to configure schedules for creating downloadable reports on runtime vulnerability results.
To create or edit Pipeline or Runtime vulnerability Policies and Rule Bundles, select the relevant links from the Policies tab in the navigation bar.
To accept the risk of detected vulnerabilities, configure an acceptance based on scope, justification, and length of time.
Optionally, download the vulnerability report in PDF format. To get a report in PDF format, select the entity and click Download PDF.
Understand Image Scanning
Image scanning enables you to examine container images for vulnerabilities, misconfigurations, and license violations. You can integrate image scanning into your development build process to verify images added to your container registry, as well as scan images used by active containers in your infrastructure.
Image scanning provides registry insights into where your images are stored, initiates scans, and allows you to review the scan results.
Behind the scenes:
SBOM is analyzed
SBOM is compared against multiple vulnerability databases
SBOM is then evaluated against default or user-defined policies.
Results are reported, both in Sysdig Secure and (if applicable) in a developer’s external CI tool.
Platform-Based Scanning
Sysdig Vulnerability Management module offers platform-based scanning by default. The scanning tools analyze images and host filesystems to extract the Software Bill of Materials (SBOM) and send them to the Sysdig backend for vulnerability matching and policy evaluation. You can retrieve the scanning result by using an API.
Prerequisites
VM Tools installed
The following version of VM tools support SBOM scanning:
Host Scanner v0.7.0 and above for non-Kubernetes containers
Host scanner for host filesystem is not currently supported
Cluster Scanner v1.23 and above
CLI Scanner v1.8.0 and above
Registry Scanner v1.1.26 and above
Agentless scanning
For on-prem environment, use v6.8.0
View SBOM
To retrieve the SBOM of your asset, issue a curl GET request against the Sysdig Secure endpoint:
curl -XGET -H 'Authorization: Bearer <API_TOKEN>' 'https://<HOSTNAME>/secure/vulnerability/v1beta1/sboms?assetId=<sha256:xxxxxx>&assetType=container-image'
Query Parameters
assetId
: The ID of the asset for which you want to retrieve the SBOM. It’s theimageId
for container-image and thehostId
for hosts. For example,assetId=sha256:xxxxxx
. You can providebomIdentifier
or bothassetId
andassetType
assetType
: The type of the asset for which you want to retrieve the SBOM. For example,container-image
. Provide this with theassetId
if you are not providingbomIdentifier
.bomIdentifier
: The ID of a single SBOM. Either you can providebomIdentifier
or bothassetId
andassetType
to retrieve the SBOM result. For example,bomIdentifier=urn:uuid:45667rrrr-b8f2-42345-b996-dkffllflp
.
Instant Scanning with Scan Now
You can skip deploying CLI-based scanning tools such as Registry Scanner and directly scanning the images in your running environment using the Scan Now button. This initiates platform-based scanning, making it easier to get started with Vulnerability Management. You can effortlessly scan and view image results without needing to deploy any extra components. For more information, see Manual Image Scanning.
Layered Analysis
Container images are optimized for distribution by composing by layers. Each change or instruction executed when building an image generates a new layer. Every set of filesystem changes you make, be it additions, deletions, or modifications causes the previous image to change, thus creating a new layer.
Layered Analysis helps detect and display vulnerabilities and packages associated with each image layer and identify different remediation actions and ownership depending on the layer that introduces them. For example, vulnerabilities in the base OS layer are remediated by updating the base image version, typically performed by the security team. In contrast, the ones belonging to the application or non-os layers are remediated by bumping the versions of libraries and dependencies in the corresponding application or service owned by the development teams.
Additionally, layered analysis can detect sibling images that are built on top of the same base image and enables routing remediation actions to the right layer and the right owner.
Prerequisites and Guidelines
CLI Scanner v1.12.0 or higher
Use the
--separate-by-layer
and--separate-by-image
options to change the output of the CLI scanner to show image hierarchy or layer information. For more information, see Display Image Layers.Cluster Shield
Platform scanning is enabled by default
Registry Scanner v1.1.26 or higher
Platform scanning is enabled by default
Legacy runtime-scanner is not supported
Use Scan Results API JSON to view the new fields for layer information
Caveats
The FROM shown in the Image Hierarchy should not be confused with the
FROM
in the Dockerfile, as they might differ. Sysdig detects base images by matching a common set of layers and displays these base images using the known name and digest as FROM in the Image Hierarchy. Since tags and image names might mutate or be aliases for the same image, the displayed name might differ. For example, an image built using the commandFROM alpine
in the Dockerfile might be shown asFROM alpine:3.18.1, alpine@sha256:1ab61723...
in the image hierarchy.The SBOM of the base images must be known to Sysdig to identify them as part of the image hierarchy. For example, you can scan the
alpine:3.18.1
base image using pipeline or registry scanning, and runtime scanning helps recognize it when running any workload based onalpine:3.18.1
.Under the Image Hierarchy:
All layers shows the total number of vulnerabilities in the final image composition, including vulnerabilities from both the application layers and OS layers. Commonly the total number displayed will be the sum of the base images and the application layers, but this is not always the case. If a vulnerability is fixed by upgrading or removing a package or library in one of the intermediate layers, the count in the All layers won’t include that vulnerability.
The base images (prefixed with FROM) display the vulnerabilities present in that base image, including those inherited from parent images. However, the Application layers counter includes vulnerabilities introduced in the application layers only. It doesn’t include the ones from base images.
View Image Layers
See the Recommendations tab associated with an image or pipeline scan to view list of CVEs, fixable packages, and the commands to fix the issues.
The Vulnerabilities tab provides you with the hierarchy of the image, all the layers of the image, and the total number of CVEs. Select a layer to view the CVEs associated with that particular layer.
See the Packages tab to view the list of layers associated with the selected image and packages associated with each layer.
Report Scanning Results
The analysis generates a detailed report of the image contents, including:
Official OS packages
Unofficial OS packages
Configuration files
Credentials files
Image layers
Localization modules and software-specific installers:
Javascript with NPM
Python PiP
Ruby with GEM
Java/JVM with .jar archives
Image metadata and configuration attributes
Understanding Risk Acceptance for Vulnerabilities
As of November, 2022, users can choose to accept the risk of a detected vulnerability or asset. Accept Risk is available for both Runtime and Pipeline, and for specific CVEs or specified hosts or images.
Prerequisites
Accept Risk requires Sysdig Secure SaaS to be installed with:
sysdig-deploy
Helm chart version 1.5.0+cluster-shield
latest versionssysdig-cli-scanner
version 1.13.0+
Because Accept Risk is applied to both pipeline and runtime vulnerability results impartially, the required versions of both components are required.
If the minimum enablement requirements are not met, the Accept Risk
button and panel will show in your interface, but will not activate. The created Acceptance will appear in Pending
status for 20 minutes, then disappear as if you had never created it.
Check Your Versions
Check sysdig-deploy
Helm Chart: Must be 1.5.0+
helm list -n <namespace>
(default namespace is sysdig-agent)
Example:
$ helm list -n sysdig-agent
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
sysdig-agent sysdig-agent 5 2022-11-11 17:57:54.109917081 +0100 CET deployed sysdig-deploy-1.5.0
Upgrade Helm Chart Instructions here.
Check Cli Scanner: must be 1.3.0+.
./sysdig-cli-scanner --version
Upgrade Cli Scanner: Instructions here.
When to Use
When faced with a large number of reported vulnerabilities, organizations need to know which are the most relevant for their security posture. Sysdig already highlights critical vulnerability with a fix available, and vulnerabilities that occur in images actually in use.
An additional feature is the targeted ability to accept the risk of a vulnerability and not count it towards a policy violation, for example, when:
- An internal security team has analyzed the vulnerability and declared it a false positive
- The preconditions of the vulnerability don’t apply
- Deployment in production is required and it is reasonable to postpone the fix
What Types of Risk
You can accept risk for different entities:
- Individual CVE IDs
- Assets
- Container images
- Hosts
Accepting Risk in the context of vulnerability management applies an exception to the Vulnerability policy. Adding an accept to a CVE doesn’t make the CVE disappear. It still shows in the list, but voids the policy violation associated with that CVE.
When accepting risks it is important to:
- Be careful with the accept scope or context; overly broad exceptions can create false negatives.
- Sysdig offers several scoping options for the accepts created.
- Remain aware of what is accepted so it doesn’t become a visibility gap.
- The Sysdig UI presents clear indications of what is accepted and why.
See:
Remediate with Jira
Sysdig has an API-enabled remediation workflow with Jira .
To create a Jira ticket from the Sysdig Secure Vulnerabilities module:
Navigate to any results page.
Open the Vulnerabilities tab.
Click on a vulnerability
The details panel will open on right.
Click Create a Jira ticket.
Fill in the details, such as Project, Ticket type, assignee, and customer labels.
Submit the ticket.
Once submitted the CVE drawer will note that there is an existing Jira ticket for that remediation.
Supported Operating Systems
The Vulnerability Management scanners can scan any OS, but for the following they will also report vendor-specific information.
Alibaba Linux
Alpine
Amazon Linux 2022/2023
CentOS 7
CentOS 8-stream
CentOS 9-stream
Debian
RHEL
The Sysdig Secure UI displays Red Hat Security Updates in the UI and in CLI version > 1.6.1 using the option
--output-json
. Note that this information is not included in Reports.Ubuntu
Ubuntu kinetic v22.10
Ubuntu Lunar v23.04
Runtime
- Kubernetes Runtime and Hosts
- Supported container runtimes:
- Docker daemon
- ContainerD
- CRI-O
Agentless Host Scanning
Cloud Hosts (AWS EC2): Discovery of running hosts
Supported Operating Systems
- Debian 10 and Debian 11
- Ubuntu 20.04 LTS and Ubuntu 22.04 LTS
- Amazon Linux 2 and Amazon Linux 2023
- RedHat 8 and RedHat 9
- Operating Systems that the agent supports
CI/CD Pipeline
Supported Container Image Formats
- Docker Registry V2 - compatible
- Docker Daemon
- Podman
- Docker Archive (tar)
- OCI Archive
Supported Package Types
Debian
Alpine
RHEL: The Sysdig Secure UI displays Red Hat Security Updates in the UI and in CLI version > 1.6.1 using the option
--output-json
. Note that this information is not included in Reports.Ubuntu
Java JAR, WAR, EAR
Golang (built with go 1.13+)
Pypi (Python)
NPM (JS)
Ruby Gems
NuGet (.Net)
Cargo (Rust)
Composer (PHP)
Supported Container Image CPU Architectures
- linux/amd64
- linux/arm64
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.