Rapid Response: Installation

With Rapid Response, Sysdig has introduced a way to grant designated Advanced Users in Sysdig Secure the ability to remote connect into a host directly from the Event stream and execute desired commands there.

Rapid Response team members have access to a full shell from within the Sysdig Secure UI. Responsibility for the security of this powerful feature rests with you: your enterprise and your designated employees.

See also: Rapid Response.

Install and Configure Rapid Response

The Rapid Response agent can be installedvia Helm, as a Kubernetes DaemonSet, or as a Docker container.

Using Helm

Rapid Response can be installed using the sysdig-deploy Helm chart (preferred method).

Alternatively, you can use the Rapid Response Helm chart directly.

Other Methods

Prerequisites

  • Sysdig Secure On-Premises 4.0+ or Sysdig Secure SaaS

  • Have on hand:

    • Your Sysdig agent access key

    • Your Sysdig API endpoint

      • On-prem: custom, depending on your on-prem installation

      • SaaS: Region-dependent (use the “endpoint” entries, e.g. https://us2.app.sysdig.com for US West)

    • A password used to encrypt all traffic between the user and host.

      NOTE: Sysdig cannot recover this password. If lost, a user will not be able to start a session, nor will any session logs be recoverable.

    Optionally, these can be added to the environment variables:

    export API_ENDPOINT=https://sysdig.mycompany.com
    export ACCESS_KEY=$YOUR_SYSDIG_AGENT_INSTALLATION_KEY
    export PASSWORD=$ENCRYPTION_PASSWORD
    export API_TLS_SKIP_CHECK=false
    

As Kubernetes DaemonSet

  1. Create a namespace and secrets for the Rapid Response agent:

    kubectl create ns rapid-response
    kubectl create secret generic sysdig-rapid-response-host-component-access-key --from-literal=access-key=$ACCESS_KEY -n rapid-response
    kubectl create secret generic sysdig-rapid-response-host-component-password --from-literal=password=$PASSWORD -n rapid-response
    
  2. Create the configmap and change the API-ENDPOINT parameter:

    echo "apiVersion: v1
    kind: ConfigMap
    metadata:
      name: sysdig-rapid-response-host-component
    data:
      api-endpoint: ${API_ENDPOINT}
      api-tls-skip-check: 'false'" | kubectl apply -n rapid-response -f -
    
  3. Deploy the DaemonSet.

    Note: The agent does not automatically have access to the host filesystem; there are several mounts commented-out in the manifest that must be uncommented to investigate the host.

    echo "# apiVersion: extensions/v1beta1  # If you are in Kubernetes version 1.8 or less please use this line instead of the following one
    apiVersion: apps/v1
    kind: DaemonSet
    metadata:
      name: sysdig-rapid-response-host-component
      labels:
        app: sysdig-rapid-response-host-component
    spec:
      selector:
        matchLabels:
          app: sysdig-rapid-response-host-component
      updateStrategy:
        type: RollingUpdate
      template:
        metadata:
          labels:
            app: sysdig-rapid-response-host-component
        spec:
          hostNetwork: true
          volumes:
            # Add custom volume here
            # Uncomment these lines if you'd like to map /root/ from the
            # host into the container.
            #- hostPath:
            #    path: /
            #  name: host-root-vol
            - name: sysdig-rapid-response-host-component-config
              configMap:
                name: sysdig-rapid-response-host-component
                optional: true
          tolerations:
            - effect: NoSchedule
              key: node-role.kubernetes.io/master
          containers:
            - name: sysdig-rapid-response-host-component
              image: quay.io/sysdig/rapid-response-host-component
              #securityContext:
                # The privileged flag is necessary for OCP 4.x and other Kubernetes setups that deny host filesystem access to
                # running containers by default regardless of volume mounts. In those cases, access to the CRI socket would fail.
              #  privileged: true
              imagePullPolicy: Always
              resources:
                limits:
                  cpu: 500m
                  memory: 500Mi
                requests:
                  cpu: 250m
                  memory: 250Mi
              # Add custom volume mount here
              # Uncomment these lines if you'd like to map /root/ from the
              # host into the container.
              #volumeMounts:
              #- mountPath: /host
              #  name: host-root-vol
              env:
                - name: API_ENDPOINT
                  valueFrom:
                    configMapKeyRef:
                      name: sysdig-rapid-response-host-component
                      key: api-endpoint
                - name: API_TLS_SKIP_CHECK
                  valueFrom:
                    configMapKeyRef:
                      name: sysdig-rapid-response-host-component
                      key: api-tls-skip-check
                - name: ACCESS_KEY
                  valueFrom:
                    secretKeyRef:
                      name: sysdig-rapid-response-host-component-access-key
                      key: access-key
                - name: PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: sysdig-rapid-response-host-component-password
                      key: password" | kubectl apply -n rapid-response -f -
    

As Docker Container

  1. Mount the host directories and binaries to gain access to the host.

    docker run --hostname $HOST_NAME -d quay.io/sysdig/rapid-response-host-component:latest --endpoint $API_ENDPOINT --access-key $ACCESS_KEY --password $PASSWORD
    
  2. Customize the Docker image.

    The container is simply bash shell. To add custom scripts without needing to mount the underlying host filesystem, you can bake this into the Docker container, e.g. by installing kubectl, gcloud, netstat, or another command-line utility.

    Since Alpine doesn’t support bash by default, the code must be installed, as below.

    FROM quay.io/sysdig/rapid-response-host-component:latest AS base-image
    
    FROM alpine:3.13
    COPY --from=base-image /usr/bin/host /usr/bin/host
    
    RUN apk update && \
        apk add bash && \
        rm -rf /var/cache/apk/*
    
    # add custom scripts and other directives
    
    ENTRYPOINT ["host"]
    

Configure Log Storage

In order to save session logs, Rapid Response requires custom storage to be configured.

If you are using the default storage for Capture files, you will need to configure an AWS or custom S3 bucket to store Rapid Response log files after a session. If you have already configured an S3 bucket for Captures, then Rapid Response logs will be routed there automatically, into their own folder.

For SaaS Users with Sysdig Secure Only

Contact Sysdig Support for assistance creating a custom S3 bucket for rapid response logs.

Post-Installation

After Rapid Response is installed, team(s) must be configured to use it.

Troubleshooting

In order to check that Rapid Response if fully operational and everything works as expected:

  • Ensure the host component can reach the Sysdig collector. Its address and port are defined here if you are a SaaS customer or provided during the installation in case of on-prem.
  • Ensure there are no intermediate proxies that could enforce maximum time to live (since sessions could potentially have long durations)
  • Ensure that the host component can reach the object storage (S3 bucket) when configured.