Storage: Configure Options for Capture Files

The Sysdig Capture feature allows you to record detailed system trace data via remote connection from any of your agent-installed hosts. In SaaS installations, by default, this data will be stored on Sysdig's secure Amazon S3 storage location, under a separate partition for your account. In on-premises installations, by default, the data will be stored in the Cassandra database.

This page describes two custom alternatives: using an AWS S3 bucket (available for SaaS and on-prem) and using custom S3 storage (on-prem only).

Configure AWS S3 Storage

To configure this option, use the fields provided Sysdig Settings UI and then append some code to the IAM Policy you created in AWS for Sysdig integration.

Prerequisites

  • Your AWS account must be integrated with Sysdig, but the CloudWatch feature is not required to be enabled.

    See AWS: Integrate AWS Account and CloudWatch Metrics (Optional)

  • Have your S3 bucket name available.

  • To use your own AWS S3 bucket to store Sysdig capture files, append the following code snippets to the AWS Identity and Access Management (IAM) page.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Put*",
                "s3:List*",
                "s3:Delete*",
                "s3:Get*"
            ],
            "Effect": "Allow",
            "Resource": [
            "arn:aws:s3:::BUCKET_NAME",
            "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}

In the Sysdig UI

  1. Log in as Administrator to Sysdig Monitor or Sysdig Secure.

  2. From the Selector button in the lower-left navigation, select Settings >Sysdig Storage.

    aws_s3_policy.png

  3. Enable the Use a custom S3 buckettoggle and enter your AWS S3 bucket name.

To Test: Capture a Trace File in Sysdig Monitor UI

When enabled, you will have the option to select between "Sysdig Monitor Storage" or your own storage bucket when configuring a file capture. See Create a Sysdig Capture File.

Configure Custom S3 Endpoint

You can set up a custom Amazon-S3-compatible storage, such as Minio or IBM Cloud Object Storage, for storing Captures in a Sysdig on-premises deployment. The capture storage location can be used for both Sysdig Monitor and Sysdig Secure. This is an API-only functionality and currently, no UI support is available.

You must configure values.yaml corresponding to your Sysdig installation in order for this configuration to work.

Prerequisites

  • Your on-premise installation is Installer-based. If you have installed Sysdig Platform manually and you want to configure custom S3 buckets to store your capture files, contact your Sysdig representative.

  • Ensure that AWS-client compatible credentials used for authentication are present in the environment.

  • Ensure that the list, get, and put operations are functional on the S3 bucket that you wish to use. Confirm this by using the S3 native tools, for example, as described in AWS CLI for IBM Cloud.

Configure Installer

Configure the following parameters in the values.yaml file so that collectors, workers, and the API server are aware of the custom endpoint configuration.

  • sysdig.s3.enabled

    Required: true
Description: Specifies if storing Sysdig Captures in S3 or S3-compatible storage is enabled or not.
Options:true|false
Default:false

    For example:

    sysdig:
  s3:
    enabled: true

  • sysdig.s3.endpoint

    Required: true
Description: The S3 or S3-compatible endpoint for the bucket. This option is ignored if sysdig.s3.enabled is not configured.

    For example:

    sysdig:
  s3:
    endpoint: <your S3-Compatible custom bucket>

  • sysdig.s3.bucketName

    Required: true
Description: The name of the S3 or S3-compatible bucket to be used for captures. This option is ignored if sysdig.s3.enabled is not configured

    For example:

    sysdig:
  s3:
    bucketName: <Name of the S3-compatible bucket to be used for captures>

  • sysdig.accessKey

    Required: true
Description: The AWS or AWS-compatible access key to be used by Sysdig components to write captures in the S3 bucket.

    For example:

    sysdig:
  accessKey: <AWS-compatible access key>

  • sysdig.secretKey

    Required: true
Description: The AWS or AWS-compatible secret key to be used by Sysdig components to write captures in the s3 bucket.

    For example:

    sysdig:
  secretKey: <AWS-compatible secret key>

For example, the following AWS CLI command uploads a Sysdig Capture file to a Minio bucket:

aws --profile minio --endpoint http://10.101.140.1:9000 s3 cp <Sysdig Capture filename> s3://test/

In this example, the endpoint is http://10.101.140.1:9000/ and the name of the bucket is test.

When you finish the S3 configuration, continue with the instructions on on-premise installation by using the Installer.