Forwarding to Elasticsearch

Prerequisites

• Event forwards originate from region-specific IPs. For the full list of outbound IPs by region, see SaaS Regions and IP Ranges. Update your firewall and allow inbound requests from these IP addresses to enable Sysdig to handle event forwarding.

• You must have an instance of Elasticsearch running and permissions to access it.

Configure Standard Integration

1. Log in to Sysdig Secure as Admin and navigate to Event Forwarding via Integrations or Settings.

2. Click +Add Integration and choose Elasticsearch from the drop-down menu.

3. Configure the required options:

   Integration Name: Define an integration name.

   Endpoint: Enter the specific Elasticsearch instance where the data will be saved. For ES Cloud and ES Cloud Enterprise, the endpoint can be found under the Deployments page:

   

   Index Name: Enter the name of the index under which the data will be stored. See What is an Elasticsearch Index? for details.

   Authentication: Choose a method to authenticate API calls. Basic authentication uses the most common format of username:password. The given user must have write privileges in Elasticsearch; you can query the available users.

   Secret: If you selected Basic authentication: specify the username:password of a user with write privileges.

   Labels and fields format: Choose Original format or Key value pairs. See Labels and Fields Format

   Data to Send: Select from the drop-down the types of Sysdig data that should be forwarded. The available list depends on the Sysdig features and products you have enabled.

   Allow insecure connections: Check the box if you wish to skip certificate validations when using HTTPS. You may wish to do this if you use self-signed ceriticates.

   Toggle the Enabled switch as necessary. You will need to Test Integration with the button below before enabling the integration.

4. Click Save.

Labels and Fields Format

There are two formatting options for Elasticsearch event forwarding.

Original format sends events in the format shown in Event Fowarding. This format is not compatible with dynamic mapping.

Key value pairs sends events labels and content.field objects, such as {"foo": "bar", ...}, as arrays of objects: [{"key": "foo", "value": "value"}, ...]. This format is compatible with dynamic mapping.

Key value pairs is recommended for Runtime Policy Events or Activity Audit. If you prefer using Original format for these types of events, we recommend you disable dynamic mapping in Elasticsearch.

Timestamp Mapping

To handle timestamps directly in Elasticsearch, you might want to map them to the appropriate field type. Timestamps have nanosecond resolution in Sysdig and they are available both in epoch timestamp and in RFC 3339 format.

The best approach is to use the date_nanos field type and define an explicit mapping in your Elasticsearch instance.

You will need to perform a PUT /<index>/_mapping API call with the index into which you are storing data, using the following payload:

{
 "properties": {
    "timestampRFC3339Nano": {
      "type": "date_nanos",
      "format": "strict_date_optional_time_nanos"
    }
  }
}

If you use the Kibana interface, you can do it there instead.

Configure Agent Local Forwarding

Review the configuration steps and use the following parameters for this integration.