Storage: Configure Options

The Sysdig Capture feature allows you to record detailed system trace data via remote connection from any of your agent-installed hosts. In SaaS installations, by default, this data will be stored on Sysdig’s secure Amazon Simple Storage Service (S3) location. This location will have a separate partition for your account. In on-premises installations, by default, the data will be stored in the Cassandra database.

The Rapid Response feature requires a custom AWS or S3-compatible bucket to be configured, for both on-prem and SaaS users.

This page describes two custom alternatives: using an AWS S3 bucket (available for SaaS and on-prem) and using custom S3 storage.

Configure AWS S3 Storage

To configure this option, use the fields provided by Sysdig Settings UI and then append some code to the Identity Access Management (IAM) Policy you created in AWS for Sysdig integration.

Prerequisites

• Your AWS account must be integrated with Sysdig, but the CloudWatch feature is not required to be enabled.

  See AWS: Integrate AWS Account and CloudWatch Metrics (Legacy)

• Ensure that your S3 bucket name is available.

• To use your own AWS S3 bucket to store Sysdig capture files, append the following code snippets to the AWS Identity and Access Management (IAM) page.

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                  "s3:Put*",
                  "s3:List*",
                  "s3:Delete*",
                  "s3:Get*"
              ],
              "Effect": "Allow",
              "Resource": [
              "arn:aws:s3:::BUCKET_NAME",
              "arn:aws:s3:::BUCKET_NAME/*"
              ]
          }
      ]
  }
  

• If you are using AWS Key Management Service (KMS) for AWS S3 encryption, ensure that necessary privileges are given to the Sysdig Account or Role to use the custom key.

  

  Use the Key users option to do so:

  

In the Sysdig UI

1. Log in as an Administrator to Sysdig Monitor.

2. From the user menu in the lower-left navigation, do one of the following:

   • Select Integrations > Capture Storage.
   • Select Settings > Capture Storage.

   

3. Enable the Use a custom S3 buckettoggle and enter your AWS S3 bucket name.

To Test: Capture a Trace File in Sysdig Monitor UI

When enabled, you will have the option to select between Sysdig Monitor Storage or your own storage bucket when configuring a file capture. See Create a Sysdig Capture File.

(SaaS) Configure Custom S3 Storage Endpoint

You can set up a custom Amazon-S3-compatible storage, such as Minio or IBM Cloud Object Storage, for storing Captures in a Sysdig SaaS deployment. The capture storage location can be used for both Sysdig Monitor and Sysdig Secure. This is an API-only functionality and currently, no UI support is available.

The following APIs are supported for this functionality:

• List existing AWS integrations

• Create a new AWS integration

• Update an existing AWS integration

• Configure storage configuration

Prerequisites

• Ensure the feature is enabled for your account.

• Use the access key and secret as the credentials.

• Configure a new AWS integration. Set the skipFetch field to true. This will cause the AWS integration to ignore fetching data from this account. Therefore, both the AWS metadata and AWS CloudWatch will not be fetched and you can use this storage exclusively for Sysdig Capture.

• Configure the storage interface with the new account, by specifying the AWS integration ID to use to authenticate the endpoint, bucket name, and the path specified in the bucket.

List Existing AWS Integration

The API returns the list of configured AWS integrations.

REST Resource: Providers

GET {{host}}/api/providers
Authorization: Bearer {{API_Token}}

Response Parameters

Sample Response

{
  "providers": [
    {
      "id": 2398,
      "name": "aws",
      "credentials": {
        "id": "AKIA4JRXW5ZVZU6MHNPE",
        "role": null
      },
      "skipFetch" : false,
      "status": {
        "status": "done",
        "lastUpdate": 1617274193293,
        "percentage": 100,
        "lastProviderMessages": []
      },
      "alias": null,
      "accountId": "845151661675"
    }
  ]
}

Create a New AWS Integration

REST Resource: Providers

POST {{host}}/api/providers
content-type: application/json
Authorization: Bearer {{API_Token}}

{
    "name":"aws",
    "skipFetch": false,
    "credentials": {
        "id":"<AWS_Access_Key_ID>",
        "role":null,
        "key":"<AWS_SecretKey>"
    }
}

Request Parameters

Update Custom Storage Settings

To update existing storage settings, perform a PUT HTTP call to the endpoint as follows:

REST Resource: Settings

PUT {{host}}/api/sysdig/settings
content-type: application/json
Authorization: Bearer {{API_Token}}

{
    "enabled":true,
    "buckets":[
        {
            "folder":"/folder1/folder2",
            "name":"bucketName",
            "providerKeyId": 3,
            "endpoint": "http://127.0.0.1:9009"
        }
    ]
}

Request Parameters

(On-Prem) Configure Custom S3 Endpoint

You can set up a custom Amazon-S3-compatible storage, such as Minio or IBM Cloud Object Storage, for storing Captures in a Sysdig on-premises deployment. The capture storage location can be used for both Sysdig Monitor and Sysdig Secure. This is an API-only functionality and currently, no UI support is available.

You must configure values.yaml corresponding to your Sysdig installation in order for this configuration to work.

Prerequisites

• Your on-premise installation is Installer-based. If you have installed Sysdig Platform manually and you want to configure custom S3 buckets to store your capture files, contact your Sysdig representative.

• Ensure that AWS-client compatible credentials used for authentication are present in the environment.

• Ensure that the list, get, and put operations are functional on the S3 bucket that you wish to use. Confirm this by using the S3 native tools, for example, as described in AWS Command Line Interface (CLI) for IBM Cloud.

Configure Installer

Configure the following parameters in the values.yaml file so that collectors, workers, and the API server are aware of the custom endpoint configuration.

• sysdig.s3.enabled

  Required: true
  Description: Specifies if storing Sysdig Captures in S3 or S3-compatible storage is enabled or not.
  Options:true|false
  Default:false
  

  For example:

  sysdig:
    s3:
      enabled: true
  

• sysdig.s3.endpoint

  Required: true
  Description: The S3 or S3-compatible endpoint for the bucket. This option is ignored if sysdig.s3.enabled is not configured.
  

  For example:

  sysdig:
    s3:
      endpoint: <your S3-Compatible custom bucket>
  

• sysdig.s3.capturesFolder

   Required: false
   Description: Name of the folder in S3 bucket to be used for storing captures, this option is ignored if sysdig.s3.enabled is not configured.
  

  For example:

  sysdig:
    s3:
      capturesFolder: my_captures_folder
  

  The path to the capture folder in the S3 bucket will be ​{​​customerId​​}/{​my_captures_folder​​}​​. For on-prem deployments, the ​customerID​ is ​1​​. If ​finance​ is your capture folder, the path to the folder in the S3 bucket will be 1/finance​​.

• sysdig.s3.bucketName

  Required: true
  Description: The name of the S3 or S3-compatible bucket to be used for captures. This option is ignored if sysdig.s3.enabled is not configured
  

  For example:

  sysdig:
    s3:
      bucketName: <Name of the S3-compatible bucket to be used for captures>
  

• sysdig.accessKey

  Required: true
  Description: The AWS or AWS-compatible access key to be used by Sysdig components to write captures in the S3 bucket.
  

  For example:

  sysdig:
    accessKey: <AWS-compatible access key>
  

• sysdig.secretKey

  Required: true
  Description: The AWS or AWS-compatible secret key to be used by Sysdig components to write captures in the s3 bucket.
  

  For example:

  sysdig:
    secretKey: <AWS-compatible secret key>
  

For example, the following AWS CLI command uploads a Sysdig Capture file to a Minio bucket:

aws --profile minio --endpoint http://10.101.140.1:9000 s3 cp <Sysdig Capture filename> s3://test/

In this example, the endpoint is http://10.101.140.1:9000/ and the name of the bucket is test.

When you finish the S3 configuration, continue with the instructions on on-premise installation by using the Installer.