AWS Fargate Serverless Agents

Introduction

Check the Overview for an explanation of when and why to use serverless agents in “container-as-a-service” cloud environments.

Architecture

The Sysdig serverless agent provides runtime detection through policy enforcement with Falco. At this time, the serverless agent is available for AWS Fargate on ECS. It is comprised of an orchestrator agent and (potentially multiple) workload agents.

  • The Sysdig serverless orchestrator agent is a collection point installed on each VPC to collect data from the serverless workload agent(s) and to forward them to the Sysdig backend. It also syncs the Falco runtime policies and rules to the workload agent(s) from the Sysdig backend.

  • The Sysdig serverless workload agent is installed in each task and requires network access to communicate with the orchestrator agent.

Understanding Install Options

  • Install Option 1: As of Serverless Agent 3.0, a simplified installer lets you deploy the orchestrator agent and instrument the Fargate task definitions for each workload agent using a dowloaded YAML file from Sysdig and then editing your own CloudFormation Template and deploying.
  • Install Option 2 (legacy): Here you deploy the orchestrator agent and the workload agents in two separate processes. A range of tools are supported for the workload agent deployment: you can use Kilt (automated), Terraform (automated), a manual Task Definition, or a manually instrumented container image.

The prerequisites, upgrade instructions, and downloaded YAML files differ between the two versions.

Option 1 Installation for Fargate ECS

This option presumes your services use an existing CFT and you will install the Workload Agent using an automated process which will instrument all the Fargate Task Definitions in your CFT.

  • For the Orchestrator Agent, Sysdig provides a YAML to use as a CloudFormation Template which you can deploy through the AWS Console. You need one Orchestrator deployment per VPC in your environment which your organization wants to secure.
  • For the Workload Agents, you need one Workload Agent per Fargate Task Definition. For example, if you have ten services and ten Task Definitions, each needs to be instrumented.

Prequisites

Before starting, ensure you have the following.

On AWS Side
  • A custom CFT containing Fargate Task Definitions you want to instrument through the Sysdig Serverless Agent
  • Two VPC subnets that can connect with the Internet via a NAT Gateway or an Internet Gateway
On Sysdig Side
  • Sysdig Secure
  • Sysdig Agent Key to install Sysdig Agents
  • The endpoint of the Sysdig Collector for your region
  • The Sysdig CFT serverless-instrumentation.yaml to instrument the Fargate Task Definitions in your CFT

Deployment Steps

Deploy the Sysdig Instrumentation and Orchestration Stack

Deploy the serverless-instrumentation.yaml for each desired VPC using CloudFormation:

  1. Log in to the AWS Console, select CloudFormation, Create Stack with new resources, and specify the serverless-instrumentation.yaml as the Template source.

  2. Specify the stack details to deploy the Orchestrator Agent on the same VPC where your service is running. For standard deployments, provide the parameters highlighted in the figure below:

  3. Click Next, complete the stack creation, and wait for the deployment to complete (usually a few minutes).

Edit your CFT

Once the stack is deployed, the Output tab provides the Transformation Instruction as shown in the figure below. Copy and paste the Transformation Instruction at the root level of your CFT.

Deploy your CFT

All new deployments of your CFT will be instrumented.

When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.

Upgrade from a Prior Version

Up through version 2.3.0, the installation process deployed two stacks (using Installation Option 2):

  • Orchestration stack, deployed via YAML
  • Instrumentation stack, deployed via command-line installer.

From version 3.0.0, you will deploy the “Instrumentation & Orchestration” stack only, using option 1.

To upgrade to version 3.0.0:

  1. Deploy the new “Instrumentation and Orchestration” stack and the Workload Agents, as described in Option1 When deploying the “Instrumentation and Orchestration” stack, assign a unique name to your macro (Transform: MyV2SysdigMacro). You now have two versions of the serverless agent components. When you are ready to switch from the earlier version, proceed with the next step.
  2. Stop all running tasks and use CloudFormation to delete the earlier stacks.
  3. Clean up the earlier Macro using the installer: ./installer-linux-amd64 cfn-macro delete MyEarlierSysdigMacro
  4. Redeploy the workload stack with the updated CFT (Transform: MyV2SysdigMacro).

Installation Option 2 for Fargate ECS

In this scenario, the two components of the serverless agent are installed separately.

  • For the orchestrator agent, Sysdig provides a yaml to use as a CloudFormation Template which you can deploy through the AWS Console. You need one orchestrator deployment per VPC in your environment which your organization wants to secure.

  • For the workload agents, you need one workload agent per Fargate task definition. (If you have ten services and ten task definitions, each needs to be instrumented.)

    We assume your services use an existing CFT and you will install the workload agent using an automated process which will instrument all the task definitions in your CFT.

Prerequisites

On the AWS side:

  • AWS CLI configured and permissions to create and use an S3 bucket

  • Permissions to upload images to repos, deploy CloudFormation Templates (CFTs), and create task definitions for Fargate

  • The Fargate tasks you want to instrument with the Sysdig serverless agent

  • Two subnets that can connect with the internet. (Your service on Fargate must reach the orchestrator agent, and the orchestrator agent must reach the internet to communicate with Sysdig’s back end.)

  • A NAT gateway or an AWS Internet Gateway so the orchestrator agent can reach to Sysdig.

On the Sysdig side:

Install the Orchestrator Agent

  1. Obtain the Sysdig Orchestrator Agent yaml to be used as the CloudFormation Template source.

    For more information on CloudFormation (CFN), see AWS documentation.

  2. Deploy the orchestrator agent for each desired VPC, using CloudFormation.

    The steps below are an outline of the important Sysdig-related parts.

    1. Log in to the AWS Console. Select CloudFormation and Create Stack with new resources and specify the orchestrator-agent.yaml as the Template source.

    2. Specify the stack details to deploy the orchestrator agent on the same VPC where your service is running.

      Stack name: self-defined

      Sysdig Settings

      • Sysdig Access Key: Use the agent key for your Sysdig platform.

      • Sysdig Collector Host: collector.sysdigcloud.com (default); region-dependent in Sysdig SaaS; custom in Sysdig on-prem.

      • Sysdig Collector Port: 6443 (default), or could be custom for on-prem installations.

      Network Settings

      • VPC Id: Choose your VPC.

      • VPC Gateway: Choose the type of Gateway in your VPC to balance the load of the orchestrator service.

      • Subnet A & B: These depend on the VPC you choose; select from the drop-down menu.

      Advanced Settings

      • Sysdig Agent Tags: Enter a comma-separated list of tags (eg. role:webserver,location:europe) Note: tags will also be created automatically from your infrastructure’s metadata, including AWS, Docker, etc.

      • Sysdig Orchestrator Agent Image:

        quay.io/orchestrator-agent:latest (default)

      • Check Collector SSL Certificate: Default: true. False means no validation will be done on the SSL certificate received from the collector, used for dev purposes only.

      • Sysdig Orchestrator Agent Port: 6667 (default). Port where the orchestrator service will be listening for instrumented task connections.

    3. Click Next, complete the stack creation, and wait for the deployment to complete (usually less than 10 minutes.)

    4. In Output, take note of the OrchestratorHost and OrchestratorPort values.

      As of Serverless Agent Version 2.2.0, the orchestrator port can be configured via either SYSDIG_ORCHESTRATOR_PORT (default 6667, which is hardcoded into other parts of the CFT) or the SysdigOrchestratorAgentPort (alternate) parameter in the CloudFormation template.

If AWS Internet Gateway is used (as opposed to a NAT Gateway), uncomment the line AssignPublicIp: ENABLED in the orchestrator.yaml.

Install the Workload Agents

Choose one of the four options below (A, B, C1, C2).

A: Automated Process (Kilt)

  1. Prerequisite: Have the orchestrator agent deployed in the appropriate VPC and have the Orchestrator Host and Port information handy.

  2. Download the appropriate installer for your OS.

    These set up Kilt, an open-source library mechanism for injection into Fargate containers.

  3. Create a macro for the serverless worker agents, using the installer. Any service tagged with this macro will have the serverless worker agent(s) added and Fargate data will be collected.

    1. Log in to AWS CLI.

    2. Create a CFN macro that applies instrumentation. You will need the outputs from previous task. Example: 

      ./installer-linux-amd64 cfn-macro install -r us-east-1 MySysdigMacro $OrchestratorHost $OrchestratorPort

  4. Add the macro you created to the CFT that you use for your own service at the root.

    Use: Transform: MySysdigMacro.

    All new deployments of that template will be instrumented.

    Defining Entrypoint and Command in CFT:

    In this step, the macro will go over the original CloudFormation Template looking for ContainerDefinitions to instrument. This includes replacing the original entry point for the Sysdig entry point, so ideally the CFT should explicitly describe Entrypoint and Command. Otherwise, the macro will try to pull the image to retrieve a default Entrypoint and Command, which only works if the image is publicly available. Otherwise the pull will fail and the instrumentation will not be completed.

  5. Complete!

    When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.

B: Automated Process Terraform

Alternatively, you can use the Sysdig Provider on the Terraform registry to deploy the workload agents, as described below.

  1. Prerequisite: Have the orchestrator agent deployed in the appropriate VPC and have the Orchestrator Host and Port information handy.

  2. Set up the Sysdig Terraform provider:

    terraform {
  required_providers {
    sysdig = {
      source = "sysdiglabs/sysdig"
      version = ">= 0.4.0"
    }
  }
}
provider "sysdig" {
  sysdig_secure_api_token = var.sysdig_access_key
}

  3. Pass the container-definition, orchestrator host, and orchestrator port to the sysdig_fargate_workload_agent data source.

    Notes:

    • The input container definitions must be in JSON format, following CloudFormation naming conventions.

    • When deploying the orchestrator agent with CFT, the port is always 6440.

    • When deploying the orchestrator agent with CFT, the host is the collector endpoint for your Sysdig region; e.g. "collector-static.sysdigcloud.com" for US East. For other regions, see SaaS Regions and IP Ranges.

      data "sysdig_fargate_workload_agent" "instrumented" {
  container_definitions = file("container-definitions/hello.json")

  sysdig_access_key = var.sysdig_access_key

  workload_agent_image = "quay.io/sysdig/workload-agent:latest"

  orchestrator_host = module.sysdig_orchestrator_agent.orchestrator_host
  orchestrator_port = module.sysdig_orchestrator_agent.orchestrator_port
}

  4. Include the instrumented JSON in your Fargate task definition:

    resource "aws_ecs_task_definition" "fargate_task" {
  ...

  network_mode = "awsvpc"
  requires_compatibilities = ["FARGATE"]

  container_definitions = "${data.sysdig_fargate_workload_agent.instrumented.output_container_definitions}"
}

  5. Complete!

    When instrumentation is complete, Fargate events should be visible in the Sysdig Secure Events feed.

/en/docs/installation/serverless-agents/aws-fargate-serverless-agents/manage-serverless-agent-logs)

Environment Variables for Workload Agents

The most commonly used variables include:

  • SYSDIG_ORCHESTRATOR: (required) use the output OrchestratorHost value
  • SYSDIG_ORCHESTRATOR_PORT: (required) use the output from OrchestratorPort
  • SYSDIG_EXTRA_ARGS: use to pass any extra arguments needed to the instrumentation
  • SYSDIG_EXTRA_CONF: use to add any extra configuration needed to the instrumentation
  • SYSDIG_LOGGING: add this variable and set to debug if needed. Any other value will print minimal logs at startup. See also: Manage Serverless Agent Logs

C. Manual Instrumentation for Workload Agents

In some cases, you may prefer not to use the Kilt or Terraform installers and instead to either:

  • Instrument a Task Definition, or
  • Instrument a Container Image (rare)

C1: Instrument a Task Definition

  1. Review the installation overview and prerequisites.

  2. Install the Orchestrator Agent, as described above. Take note of the OrchestratorHost and OrchestratorPort values.

  3. Instrument the CloudFormation task definition to deploy the workload agent manually.

  4. Add a new container to your existing CloudFormation task definition.

    • For the example, we will use SysdigInstrumentation for the image name.
    • The entrypoint/command fields can be left empty
    • Obtain the image from quay.io/sysdig/workload-agent:latest

  5. Edit the containers you want to instrument.

    • Mount volumes from SysdigInstrumentation

    • Add SYS_PTRACE capability to your container. See AWS Documentation for detail if needed.

    • Set /opt/draios/bin/instrument as the entry point of your container.

      Note: if you are using the entry point in your image, then prepend it: /opt/draios/bin/instrument,your,original,entry,point

  6. Configure the Sysdig instrumentation through environment variables.

Example Task Definitions
CloudFormation Task Before Instrumentation `
TestApp:
  Type: AWS::ECS::TaskDefinition
  Properties:
    NetworkMode: awsvpc
    RequiresCompatibilities:
      - FARGATE
    Cpu: 2048
    Memory: 8GB
    ExecutionRoleArn: !Ref TestAppExecutionRole
    TaskRoleArn: !Ref TestAppTaskRole
    ContainerDefinitions:
      - Name: App
        Image: !Ref TestAppImage
        EntryPoint:
        - /bin/ping
        - google.com
    Tags:
      - Key: application
        Value: TestApp
Task After Manual Instrumentation
TestApp:
  Type: AWS::ECS::TaskDefinition
  Properties:
    NetworkMode: awsvpc
    RequiresCompatibilities:
      - FARGATE
    Cpu: 2048
    Memory: 8GB
    ExecutionRoleArn: !Ref TestAppExecutionRole
    TaskRoleArn: !Ref TestAppTaskRole
    ContainerDefinitions:
      - Name: App
        Image: !Ref TestAppImage
        EntryPoint:
### Added for manual instrumentation
        - /opt/draios/bin/instrument
### End added propertis
        - /bin/ping
        - google.com
### Added for manual instrumentation
        LinuxParameters:
          Capabilities:
            Add:
            - SYS_PTRACE
        VolumesFrom:
        - SourceContainer: SysdigInstrumentation
          ReadOnly: true
        Environment:
        - Name: SYSDIG_ORCHESTRATOR # Required
          Value: <host orchestrator output, region dependent>
        - Name: SYSDIG_ORCHESTRATOR_PORT # Required
          Value: "6667"
        - Name: SYSDIG_ACCESS_KEY
          Value: ""
        - Name: SYSDIG_COLLECTOR
          Value: ""
        - Name: SYSDIG_COLLECTOR_PORT
          Value: ""
        - Name: SYSDIG_LOGGING
          Value: ""
      - Name: SysdigInstrumentation
        Image: !Ref WorkloadAgentImage
### End of added properties
    Tags:
      - Key: application
        Value: TestApp

C2: Instrument in a Container Image

Alternatively, you can include the workload agent in your container at build time. To do so, update your Dockerfile to copy the required files:

ARG sysdig_agent_version=latest
FROM quay.io/sysdig/workload-agent:$sysdig_agent_version AS workload-agent

FROM my-original-base

COPY --from=workload-agent /opt/draios /opt/draios

Prepend the /opt/draios/bin/instrument command to the entrypoint of your container:

ENTRYPOINT ["/opt/draios/bin/instrument", "my", "original", "entry", "point"]

Note about Logs

To separate instrumentation logs from your workload logs, the instrument entrypoint will attempt to send logs to localhost:32000, where they will be mixed with the workload logs.

  • If you want to separate instrumentation logs in a manual install, we advise using Option A.
  • To avoid log spam, you can set SYSDIG_LOGGING to silent.

See also: Manage Serverless Agent Logs

Upgrade (Up to v2.3.0)

Note: In most cases, it is advised to upgrade directly to v3.0.0+, using the Option 1 upgrade path. These instructions are kept for special cases.

To upgrade the serverless agents, you install a second version of both components then kill all the running tasks and restart with the new version. You then delete and clean up the old CloudFormation and Kilt residuals.

  1. Obtain the Sysdig Orchestrator Agent yaml to be used as the CloudFormation Template source.

    At this time, the yaml metadata does not specify the agent version, but this link always downloads the latest file.

  2. Perform the steps to Install the Orchestrator Agent.

    Note that in step 2.4, the OrchestratorHost and OrchestratorPort values will be unique.

  3. Perform the steps to Install the Workload Agents (if using Kilt. Workload agents installed with Terraform are automatically updated to _latest.)Manual instrumentation requires redoing the manual process.)

    In step 4, assign a unique name to your macro (Transform: MyV2SysdigMacro).

    You now have two versions of the serverless agent components. When you are ready to switch from the earlier version, proceed with the next step.

  4. Stop all running tasks and use CloudFormataion to delete the earlier stack. Redeploy the new stack with the updated CFT. (Transform: MyV2SysdigMacro)

  5. Clean up macros: Delete the previous kilt macro using the installer:

    ./installer-linux-amd64 cfn-macro delete MySysdigMacro


Last modified June 23, 2022