Storage: Configure Options for Capture Files

The Sysdig Capture feature allows you to record detailed system trace data via remote connection from any of your agent-installed hosts. In SaaS installations, by default, this data will be stored on Sysdig's secure Amazon S3 storage location. This location will have a separate partition for your account. In on-premises installations, by default, the data will be stored in the Cassandra database.

This page describes two custom alternatives: using an AWS S3 bucket (available for SaaS and on-prem) and using custom S3 storage.

Storage Options	SaaS	On-Prem
Sysdig Provided Storage	Sysdig provided	Installation provided Cassandra AWS S3 S3 Compatible
AWS S3	Configure using the API or the UI	Installation provided Configure using the API or the UI
S3 Compatible	Using the API	Installation provided

Note

if Google Cloud Storage is used as the S3 compatible storage, you will not be able to bulk delete captures due to compatibility issues with Google's S3 API implementation. You can delete captures one by one or delete them directly from the Google console.

Configure AWS S3 Storage

To configure this option, use the fields provided by Sysdig Settings UI and then append some code to the IAM Policy you created in AWS for Sysdig integration.

Prerequisites

Your AWS account must be integrated with Sysdig, but the CloudWatch feature is not required to be enabled.
See AWS: Integrate AWS Account and CloudWatch Metrics (Optional)
Ensure that your S3 bucket name is available.

To use your own AWS S3 bucket to store Sysdig capture files, append the following code snippets to the AWS Identity and Access Management (IAM) page.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Put*",
                "s3:List*",
                "s3:Delete*",
                "s3:Get*"
            ],
            "Effect": "Allow",
            "Resource": [
            "arn:aws:s3:::BUCKET_NAME",
            "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}

If you are using AWS KMS for AWS S3 encryption, ensure that necessary privileges are given to the Sysdig Account or Role to use the custom key.
Use the Key users option to do so:

In the Sysdig UI

Log in as Administrator to Sysdig Monitor or Sysdig Secure.
From the Selector button in the lower-left navigation, select Settings >Sysdig Storage.
Enable the Use a custom S3 buckettoggle and enter your AWS S3 bucket name.

To Test: Capture a Trace File in Sysdig Monitor UI

When enabled, you will have the option to select between "Sysdig Monitor Storage" or your own storage bucket when configuring a file capture. See Create a Sysdig Capture File.

(SaaS) Configure Custom S3 Storage Endpoint

You can set up a custom Amazon-S3-compatible storage, such as Minio or IBM Cloud Object Storage, for storing Captures in a Sysdig SaaS deployment. The capture storage location can be used for both Sysdig Monitor and Sysdig Secure. This is an API-only functionality and currently, no UI support is available.

The following APIs are supported for this functionality:

List existing AWS integrations
Create a new AWS integration
Update an existing AWS integration
Configure storage configuration

Prerequisites

Ensure that the feature is enabled for your account.
Use the access key and secret as the credentials.
Configure a new AWS integration. Set the skipFetch field to true. This will cause the AWS integration to ignore fetching data from this account. Therefore, both the AWS metadata and AWS CloudWatch will not be fetched and you can use this storage exclusively for Sysdig Capture.
Configure the storage interface with the new account, by specifying the AWS integration ID to use to authenticate the endpoint, bucket name, and the path specified in the bucket.

Limitation

The AWS account ID is currently shown as null on the UI.

List Existing AWS Integration

The API returns the list of configured AWS integrations.

REST Resource: Providers

GET {{host}}/api/providers
Authorization: Bearer {{API_Token}}

Response Parameters

Field	Response
`id`	String The unique identifier of the integration.
`name`	String The name of the integration and by default is set to `aws`.
`credentials`	String The dictionary containing the information about how Sysdig authenticates to AWS: `id`: The AWS access key ID. `role` : The AWS Role if role-based authentication is used.
`skipFetch`	Boolean `true` : The account will be used only for configuring storage. In this case, it will not fetch AWS metadata and CloudWatch metrics. `false`: The account will be used only for fetching AWS metadata and CloudWatch metrics.
`status`	String Status denotes the status of the integration.
`accountId`	String The unique identifier of the AWS account. The value will be `null` if you are using AWS-compatible storage.
`tags`	Ignore this deprecated field.

Sample Response

{
  "providers": [
    {
      "id": 2398,
      "name": "aws",
      "credentials": {
        "id": "AKIA4JRXW5ZVZU6MHNPE",
        "role": null
      },
      "skipFetch" : false,
      "status": {
        "status": "done",
        "lastUpdate": 1617274193293,
        "percentage": 100,
        "lastProviderMessages": []
      },
      "alias": null,
      "accountId": "845151661675"
    }
  ]
}

Create a New AWS Integration

REST Resource: Providers

POST {{host}}/api/providers
content-type: application/json
Authorization: Bearer {{API_Token}}

{
    "name":"aws",
    "skipFetch": false,
    "credentials": {
        "id":"<AWS_Access_Key_ID>",
        "role":null,
        "key":"<AWS_SecretKey>"
    }
}

Request Parameters

Field	Description
`name`	String The name of the integration and by default is set to `aws`.
`skipFetch`	Boolean `false`: The account will be used only for fetching AWS metadata and CloudWatch metrics. `true` : The account will be used only for configuring storage. In this case, it will not fetch AWS metadata and CloudWatch metrics.
`credentials`	The dictionary containing the information about how Sysdig authenticates to AWS: `id`: The AWS access key ID. `role`: The AWS Role if role-based authentication is used. `key`: The secret associated with the account.

Field

Description

name

String

The name of the integration and by default is set to aws.

skipFetch

Boolean

false: The account will be used only for fetching AWS metadata and CloudWatch metrics.

true : The account will be used only for configuring storage. In this case, it will not fetch AWS metadata and CloudWatch metrics.

credentials

The dictionary containing the information about how Sysdig authenticates to AWS:

id: The AWS access key ID.
role: The AWS Role if role-based authentication is used.
key: The secret associated with the account.

Update Custom Storage Settings

To update existing storage settings, perform a PUT HTTP call to the endpoint as follows:

REST Resource: Settings

PUT {{host}}/api/sysdig/settings
content-type: application/json
Authorization: Bearer {{API_Token}}

{
    "enabled":true,
    "buckets":[
        {
            "folder":"/folder1/folder2",
            "name":"bucketName",
            "providerKeyId": 3,
            "endpoint": "http://127.0.0.1:9009"
        }
    ]
}

Request Parameters

Field	Description
`enabled`	Boolean Indicates whether the custom storage is configured. If the value is `false`, Sysdig-provided storage is used.
`buckets`	Returns the list of buckets that you can set. Currently, only one bucket is supported. `folder`: The bucket path where the captures will be stored. `name`: The unique bucket name. `providerKeyId`: The unique ID of the AWS or compatible provider to be used to configure storage. Currently, only one bucket can be configured at a time. `endpoint`: The endpoint of the provider. It is configured only if you wish to set AWS compatible and is automatically set for AWS. For example, https://storage.googleapis.com/ for Google Cloud https://s3.us-south.cloud-object-storage.appdomain.cloud for IBM. For a full list of endpoints, please refer to the relevant provider documentation.

Field

Description

enabled

Boolean

Indicates whether the custom storage is configured.

If the value is false, Sysdig-provided storage is used.

buckets

Returns the list of buckets that you can set. Currently, only one bucket is supported.

folder: The bucket path where the captures will be stored.
name: The unique bucket name.
providerKeyId: The unique ID of the AWS or compatible provider to be used to configure storage. Currently, only one bucket can be configured at a time.
endpoint: The endpoint of the provider. It is configured only if you wish to set AWS compatible and is automatically set for AWS.
For example, https://storage.googleapis.com/ for Google Cloud
https://s3.us-south.cloud-object-storage.appdomain.cloud for IBM.
For a full list of endpoints, please refer to the relevant provider documentation.

(On-Prem) Configure Custom S3 Endpoint

You can set up a custom Amazon-S3-compatible storage, such as Minio or IBM Cloud Object Storage, for storing Captures in a Sysdig on-premises deployment. The capture storage location can be used for both Sysdig Monitor and Sysdig Secure. This is an API-only functionality and currently, no UI support is available.

You must configure values.yaml corresponding to your Sysdig installation in order for this configuration to work.

Prerequisites

Your on-premise installation is Installer-based. If you have installed Sysdig Platform manually and you want to configure custom S3 buckets to store your capture files, contact your Sysdig representative.
Ensure that AWS-client compatible credentials used for authentication are present in the environment.
Ensure that the list, get, and put operations are functional on the S3 bucket that you wish to use. Confirm this by using the S3 native tools, for example, as described in AWS CLI for IBM Cloud.

Configure Installer

Configure the following parameters in the values.yaml file so that collectors, workers, and the API server are aware of the custom endpoint configuration.

sysdig.s3.enabled

Required: true
Description: Specifies if storing Sysdig Captures in S3 or S3-compatible storage is enabled or not.
Options:true|false
Default:false

For example:

sysdig:
  s3:
    enabled: true

sysdig.s3.endpoint

Required: true
Description: The S3 or S3-compatible endpoint for the bucket. This option is ignored if sysdig.s3.enabled is not configured.

For example:

sysdig:
  s3:
    endpoint: <your S3-Compatible custom bucket>

sysdig.s3.bucketName

Required: true
Description: The name of the S3 or S3-compatible bucket to be used for captures. This option is ignored if sysdig.s3.enabled is not configured

For example:

sysdig:
  s3:
    bucketName: <Name of the S3-compatible bucket to be used for captures>

sysdig.accessKey

Required: true
Description: The AWS or AWS-compatible access key to be used by Sysdig components to write captures in the S3 bucket.

For example:

sysdig:
  accessKey: <AWS-compatible access key>

sysdig.secretKey

Required: true
Description: The AWS or AWS-compatible secret key to be used by Sysdig components to write captures in the s3 bucket.

For example:

sysdig:
  secretKey: <AWS-compatible secret key>

For example, the following AWS CLI command uploads a Sysdig Capture file to a Minio bucket:

aws --profile minio --endpoint http://10.101.140.1:9000 s3 cp <Sysdig Capture filename> s3://test/

In this example, the endpoint is http://10.101.140.1:9000/ and the name of the bucket is test.

When you finish the S3 configuration, continue with the instructions on on-premise installation by using the Installer.

Storage: Configure Options for Capture Files

Note

Configure AWS S3 Storage

Prerequisites

In the Sysdig UI

To Test: Capture a Trace File in Sysdig Monitor UI

(SaaS) Configure Custom S3 Storage Endpoint

Prerequisites

Limitation

List Existing AWS Integration

REST Resource: Providers

Response Parameters

Sample Response

Create a New AWS Integration

REST Resource: Providers

Request Parameters

Update Custom Storage Settings

REST Resource: Settings

Request Parameters

(On-Prem) Configure Custom S3 Endpoint

Prerequisites

Configure Installer

Search results