Respond

• Response Actions lets you take action directly from the Events feed.
• Rapid Response lets you connect to a remote shell within your environment and execute commands there.

These features supplement the response capabilities of Threat Detection Policies. Several policy types, such as Container Drift Policy, let you configure actions to occur when an event is detected in the workload.

Response Actions

Response Actions allow you to execute response actions from the Events feed. This can be preferable to Threat Detection Policy Actions, because you can assess an event in the Events feed to ensure it is not a false positive before you take action.

Prerequisites

• A role with Response Actions permissions. Contained by default in Admin, Advanced User, Team Manager. See Role management
• Host Shield version 13.9 or higher with response actions enabled in values.yaml:

features:
  respond:
    response_actions:
      enabled: true

• For more installation instructions, see:
  • Install Shield on Linux Nodes
  • Install Shield as a Container
  • Install Shield as Package

Configuration

See the Configuration Library to customize the Response Actions.

Execute a Response Action

To execute a response action from an event:

1. Log in to Sysdig Secure.

2. Select Threats > Activity | Events.

3. Open an event happening on a Kubernetes, Host or Container workload.

   The Respond button will be available.

4. Select the Respond button.

   

   • Optional: Select Ask for Response Advice to open Sysdig Sage. Sage will suggest remediation actions based on your current context.

5. If a response action is available for you on that host, select Run Response Action to see the actions available for that context, as well as the last executed action, if any.

   

6. After you select an action, you can review the parameters.

   

7. To execute the action, select Run Action.

   After the action executes, a message appears reporting the status and, in case of an error, the reason.

   

Container Kill

The Container Kill action instantly terminates the container, along with all of its process, without waiting for them to shut down. This can stop a malicious activity quickly, but can result in data loss (in stateful applications) and service disruption. It is the best choice for containing severe and urgent threats.

If the container is orchestrated with Kubernetes, the control plane will typically attempt to restart the container based on its configuration.

Drawbacks: Any possible forensic data will be lost. Possible service disruption, in cases where there’s no container orchestration or it’s not working as expected.

Container Stop

Container Stop gracefully terminates the container with its processes, deleting the memory state and the file system upper layer. It’s best for stateful applications.

If the container is orchestrated with Kubernetes and it has a liveness probe defined, the control plane will restart the pod as soon as the liveness probe fails. If no liveness prove is defined, Kubernetes will restart the container based on the restartPolicy. By default, the behavior is to restart terminated containers. In case of Kubernetes environments, this action cannot be reverted, i.e. containers cannot be restarted directly.

On non-orchestrated environments, or when the control plane doesn’t restart the container, you can start the container again from Sysdig. Follow the steps in the Revert a Response Action section to understand how to do it.

Drawbacks: Possible service disruption, in cases where there’s no container orchestration or it’s not working as expected.

Container Pause

Container Pause suspends the execution of a container, freezing its processes and preserving its runtime state. The container memory state disk state are maintained. This is ideal for forensic analysis and limiting disruption, but may result in higher resource consumption and prolonged risk due to lack of resolution.

If the container is orchestrated with Kubernetes and the pod’s liveness probe will start failing, it will be restarted, making this have the same effect of a Container Stop action.

In other cases, the execution will be temporarily stopped until further actions are taken. You can start the container again by reverting the action

Drawbacks: Possible service disruption, in cases where there’s no container orchestration or it’s not working as expected.

Process Kill

The Process Kill action sends a SIGKILL signal to the process, causing it to terminate instantly.

When you execute this action from an event, you can decide to kill the process that originated the event itself or any of the processes in its ancestor line.

Drawbacks: Possible service disruption, when the process is in charge of executing business critical applications without orchestrations/watchdog mechanisms.

File Quarantine

File Quarantine is a Response Action you can use to isolate a suspicious or vulnerable file, so that it cannot be executed and can be accessed later for further investigations.

This action compresses the file, to make it non-executable, and moves it inside a quarantine folder. Optional: To change the default quarantine folder, see Configure File Quarantine.

As an example, this action can be useful if you see an Event warning you about a process receiving an unexpected connection. You can use the File Quarantine action to move the executable elsewhere and make it non-executable. Later, you can inspect the file in its safe new location.

This action can be reverted, moving the file back to its original location. To do this, see Revert a Response Action.

Drawbacks: If the action is performed on a business critical application executable, this can cause possible service disruptions.

Configure File Quarantine

You can customize File Quarantine behavior to:

• Change the quarantine path.
• Change the maximum file size

Check out how to Configure Response Actions and the available Additional Agent Configurations in the Configuration Library.

Troubleshoot File Quarantine

If you receive an error message, such as failed to remove quarantined file or file system mounted as read-only, the file may be in a folder the agent does not have permission to perform write actions in. The File Quarantine action requires write permissions on the target file and on the quarantine folder path.

Verify the instructions for customizing the quarantine folder path and the write permissions in the Agent Configuration Library

File Acquire

The File Acquire action allows you to fetch a file from a workload. This can be useful to:

• Download suspicious executable files for further examining them
• Access log files to analyze the system activities and trace user actions
• Check for modifications performed by actors to understand if they are malicious

Drawbacks: whoever is entitled to execute this action will be able to pull data from workloads, if the Sysdig agent has such a permissions.

Configure File Acquire

In order to acquire a file, you need to setup a Custom Storage, where the file is uploaded from the workload. See how to do it in the dedicated page.

File Acquire also supports limiting the maximum file size to acquire, to avoid excessive resource usage. See the Response Actions Additional Agent configurations to learn more about it.

Download an acquired file

Once you terminate a File Acquire action with success, you will see the execution in the Response History. In the case of successful File Acquire actions, you will also be able to download the file from the context menu or from the execution details.

See Previous Executions

To view the history of response actions:

1. Log in to Sysdig Secure.

2. Select Threats > Activity | Events.

3. Open an event happening on a Kubernetes, Host or Container workload.

   The Event Details panel appears.

   

4. In the Response History section, you can see response actions previously taken in that context. For example, container actions targeting the same host or cluster. Each row in the section represents an execution request sent to the agent.

   The actions displayed are limited by the time frame of currently applied to the Events page. See Change Time Span to expand or reduce it.

5. By using the “View Detail” contextual action, you can see the action parameters:

   • Metadata, containing the execution status and the execution timestamp
   • Inputs, showing the action target information
   • Configuration, having details about the setup when that action was run

   

Revert a Response Action

When you’re looking at the execution of a Response Action that can be reverted in the Response History, you will see the possibility to execute the reciprocal revert action from the context menu in the Response History and from the execution details.

From the Response History:

From the execution details:

Once you select that, you will be prompted to review the parameters, like with normal executions.

Upon submission, the revert action will trigger, trying to revert the previous action execution.

You will be able to see the outcome of that action and the details as for any other action.

Rapid Response

Rapid Response is a tool that lets security teams connect to a remote shell within your environment to start troubleshooting and investigating an event using the commands they are already accustomed to, with the flexibility they need to run the security tools at their disposal, directly from the event alert.

Prerequisites

To Download Session Information, the following prerequisites are required:

• OpenSSL v1.1.1 or above

• A properly configured Custom S3 storage bucket.

Process Overview

• Install and configure the Rapid Response container.

• Create or configure teams of Sysdig Secure Advanced Users who should have Rapid Response privileges.

• Team members log in and manage workloads using Rapid Response shell.

• Review session logs to keep track of what has been done using Rapid Response.

Configure Rapid Response Teams

You can configure a team to grant Rapid Response permissions to particular users. For example, if you have an existing team called CustomerResponse with 40 members and you’d like to grant 5 of those users Rapid Response capabilities, you could create a team called CustomerResponse_RR and add the 5 designated Advanced Users to it.

To create a team with Rapid Response permission:

1. Log in to Sysdig Secure as Admin.

2. Select Settings > Teams.

3. Select Add team.

4. Enter the team name and configure the team details.

5. Check the Rapid Response additional permission checkbox.

6. Select Save.

Enable Rapid Response for an Existing Team

To enable Rapid Response for an existing team:

1. Go to Settings > Teams.

2. Choose the applicable team to display the Edit Teams page.

3. Select the Rapid Response checkbox in the additional permissions section and click Save.

Start Rapid Response

1. Log in to the Sysdig Secure UI as a Rapid Response team member.

2. Select Threats > Start Rapid Response.

3. Select the host as prompted and click Start Session.

4. Enter the passphrase for that host.

5. Enter the two-factor authentication code that was emailed to you and click Confirm.

6. Begin your session.

   You can dock the terminal window at the bottom or right panel of your page, or as a separate screen.

Launch a Rapid Response Session from Events Feed

1. Log in to the Sysdig Secure UI as a Rapid Response team member.

2. Select Events and choose an event from the list to open the detail pane.

3. Click Respond: Open Shell.

4. Enter the two-factor authentication code that was emailed to you and click Confirm.

5. Begin your session.

   You can dock the terminal window at the bottom or right panel of your page, or as a separate screen.

Manage Rapid Response Logs

When reviewing the logs, you can download completed log sessions or close sessions that are live.

If you’re using Sysdig SaaS, configure a storage for Rapid Response logs in order to store them.

The logs visible to you depend on the team and role under which you are logged in. Administrators will see the entire log list.

The Session Log list includes the session initiator, the timestamp, and the host name accessed.

Download Session Information

If the session has been closed, you can download the content of the session from the UI (input and output) as an Open SLL-compatible gzip encrypted file.

To open the file, use the following command (session-file is the name of the downloaded file and <passphrase> is the passphrase you set up for that host during the installation):

gzip -dc <session-file> | openssl enc -d -aes-256-ctr -pbkdf2 -k <passphrase>

Close an Active Session

Rapid Response team members can review the Session Log list and close any active session by clicking Close from the Actions column.