This the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

  • 1:

    Activity Audit

    From Sysdig Secure 3.0 (with Sysdig Agent 0.93.0+), the Commands Audit feature has been replaced by the more powerful and intuitive Activity Audit module.

    Contact Sysdig Support to have Activity Audit enabled for your SaaS Sysdig Secure environment.

    To track kubectl exec and other Kubernetes commands, you must enable Kubernetes audit logging.

    If you are running on a GKE cluster, review the GKE Limitations.

    From June, 2021, Activity Audit has been moved under the Investigate menu in the nav bar.

    Activity Audit takes the high-value data from captures and makes it always-on, searchable, and indexed against your cloud-native assets. This stream includes executed commands, network activity, file activity, and kube exec requests to the Kubernetes API.

    Understanding How Activity Audit is Used

    Activity Audit allows users to view different data sources in-depth for monitoring, troubleshooting, diagnostics, or to meet regulatory controls.

    Troubleshooting

    A system investigation may be triggered by an event generated by Sysdig, or by an alert from another tool or person.

    • Find contextualized, relevant data Activity Audit allows easy access to the underlying data to help trace the event, evaluate its impact, and resolve the issue.

      From Policy Events in Sysdig Secure, jump directly to the relevant Activity Audit to investigate details.

    • Trace commands and connections back to users Activity Audit can correlate the interactive requests from a Kubernetes user with the commands and network connections performed inside the container, allowing an operator to trace this activity back to a user identity.

    Regulatory

    The Activity Audit can also provide data about the infrastructure to help prove to auditors that proper data visibility and security measures are in place

    Activity Audit is a critical requisite for many compliance standards:

    Activity Audit displays a continuously updated list of activities. Use the UI features to find and filter the information you need.

    Select Investigate > Activity Audit to access.

    Groupings

    Filter activities by pre-defined groupings from the drop-down menu.

    Note that each element in a group shows a summary of the activity entries for that entity. You can use this number to discover areas of high activity; filtering changes the number.

    Data Sources

    Use the top-right drop-down to filter information from one particular data set. Current data sources are:

    • User commands

    • Network connections

    • File activities

    • Kube exec commands

    Activity audit feature captures only interactive commands and the network connections and file modifications related to those commands. Kube exec commands, on the other hand, are extracted from the Kubernetes/OpenShift audit log.

    Frequency Graph

    The graph shows the activity frequency for each data source, allowing users easily to zero-in on anomalies.

    The image above shows a spike in network activity (orange line) between 7:00 - 8:00 pm.

    Drag the mouse over the peak to auto-zoom on the time frame and see more detail.

    Time Navigation Buttons

    Use the time window navigation bar to show only activities run within that window. (For more information, see also Time Windows.)

    Sysdig Secure does not currently provide the functionality to configure a custom time window.

    Detail View

    Select an activity row to see its details, including the attributes by which you can filter (whitelist/blacklist).

    See Review Activity Details for the attributes of each data source.

    You can also use the whitelist (=)/blacklist (!=) attribute filters from the detail view.

    Whitelist (=) and Blacklist (!=) Attribute Options

    Next to attributes in the Detail View are =/!= signs used for filtering.

    Click = to include an attribute; click != to exclude an attribute from a filter.

    Trace Button (kube exec activity)

    Beside each kube exec item is a Trace button .

    Using this feature you can correlate container activity to the original Kubernetes user and IP. See Follow a kubectl exec Trace.

    This button does not appear if you are running on a GKE cluster.

    Filter

    Filtering is the heart of Activity Audit’s power. Filters allow you to search, sort, parse, and surface meaningful data and connections as they are needed.

    Filters are used instead of free text search in Activity Audit.

    Ways to filter Activity Data:

    • Data Source: Choose a data source from the drop-down: network activity, commands, kubectl exec, or file activity.

    • Attribute (=/!=): Choose = or -!= next to an attribute to include/exclude that attribute from the filter

    • Attribute (manual): If you know the attribute, you can type it into the filter box manually, with the following syntax:

      Include an attribute

      attribute_name="attribute_value" e.g. comm="grep"

      Exclude an attribute

      attribute_name!="attribute_value" e.g. comm!="grep"

    • Trace Trace kube exec entries to see all relevant activity in that session from that user

    • Time graph: Select a section of the graph to zoom in on a time frame and see detailed activity

    • Combine: These methods can be combined as needed.

      For example, the filter below surfaces activity on a particular pod, while excluding activity from one IP address that is known to be normal.

      resource="pods" name="woocommerce-6877958" sourceaddresses!="172.20.41.2

    Review Activity Details

    Command Details

    Name

    Description

    When

    The date and time the command was executed.

    Command

    The command executed.

    Full Command Line

    The complete command, including all variables/options.

    Working Directory

    The directory the command was executed in.

    Scope

    The entities within the infrastructure impacted by the command.

    Host

    The hostname and MAC address of the host the command was executed on.

    Container

    The container ID, container name, and image that the command was executed on.

    Additional Details

    Detailed user/host information:

    • The Process ID (PID) of the command.

    • The Parent Process ID (PPID) of the command.

    • The user ID of the user that executed the command.

    • The Shell ID.

    Network Connection Details

    Only TCP or UDP connections are currently captured in activity audit.

    Name

    Description

    Time

    The date and time of the network connection

    Connection Direction

    Incoming or outgoing connection

    Connection Details

    Including:

    • Transport-level protocol (lp4)

    • Client address, server address (lp4)

    • Client port, server port

    Scope

    The entities impacted by the network connection

    Host

    The host name and MAC address of the host where the connection was made

    Additional Details

    The process name and ID (Parent Process ID/PID) that launched or received the network connection

    Kubectl Exec Details

    Name

    Description

    Time

    The date and time of the kubectl command

    Kubernetes resource

    Including:

    • resource: The kind of Kubernetes resource affected (currently only pods)

    • name: name of the resource (pod name)

    • subresource: currently exec

    • command: the command executed

    • container: the high-level name in the Kubernetes defintion

    Kubernetes user and group

    Including:

    • user: user name performing the kubectl command. Can be either a service account or a human user.

    • groups: groups the user belongs to

    • userAgent: client userAgent

    Sources addresses

    External IP address that initiated the connection

    Scope

    Including

    • Kubernetes namespace name

    • Kubernetes deployment name

    • Kubernetes pod name

    • Container ID

    Host

    Host name and MAC address of the host where the kubectl exec was made

    File Activity Details

    Name

    Description

    When

    Date and time the file was modified

    File access details

    - File name

    - File directory

    - Command used to access the file

    - Access mode

    Scope

    Entities impacted by the file activity

    Host

    The host name and MAC address of the host where the file activity occurred

    Sample Use Cases

    Blacklist: Commands

    I have a noisy command happening every 0.2s, which I know is totally normal in my environment. There are other commands that look more suspicious to me:

    Foo
    Foo
    Foo
    Foo
    Foo
    Suspicious command - curl
    Foo
    Foo
    Foo
    … 200 Foo
    Suspicious command  - dpkg
    …. 500 Foo
    Suspicious command  - shred
    

    This is a clear case for blacklisting “Foo” and concentrating on the rest.

    Filtering for Incident Response

    A Policy Event reports a dangerous peak in network connections coming from a specific pod. This example describes one way to search for the root cause.

    What user and what activity triggered this issue?

    1. Use the Activity Audit button next to the policy event to jump directly to the relevant audit trail.

    2. Here one can determine at a glance:

      • The pod/namespace on which the heightened activity is occurring (where the 3396 network entries are)

      • The process related to the activity (in this case, ab, or the Apache Benchmark tool)

      • Related activities in the graph (cmd and kube exec lines)

      • Repetitive entries that can be screened out

    3. Refine the view through filtering:

      1. Switch from network data source to cmd and kube exe.

      2. Filter out noisy, repetitive entries (e.g. comm!="bash")

      3. Investigate details of a kube exec item for user information.

    4. After filtering, you have a focused incident report detailing:

      • The Kubernetes user “johndoe”

      • The external IP he used to connect

      • The set of commands he used to install and launch the Apache Benchmark stress-testing tool.

    Follow a kubectl exec Trace

    In a production environment, kubectl exec commands are typically suspicious. Also, because such commands are interactive sessions, it can be difficult to pinpoint which individual has issued the command(s) and what other activities the individual performed. This is where Sysdig’s Trace functionality comes in, correlating kubectl exec commands with a specific user and the network and command activities performed in that user’s session.

    In this example, suspicious activity has been detected and you want to determine whether someone has downloaded and executed a Trojan horse.

    1. Use the Groups to display your Kubernetes hierarchy by namespace and deployment. Focus on the pod displaying unexpected high levels of activity (based on the number in parentheses).

    2. Checking the corresponding activity graph, you zero in on a time frame and see kube exec activity among the hundreds of commands and network events.

    3. Select the kube exec item and click the Trace button on left.

      This session trace will display a formatted report of any container activity (network, commands) that the user performed inside the container.

      This button does not appear if you are running on a GKE cluster.

    1 -

    [Legacy] Commands Audit

    The Commands Audit module provides Sysdig Secure users with a searchable and sortable audit trail of user commands executed within the infrastructure.

    While policy events are an inherently suspicious activity that warrants investigation, commands are not themselves considered suspicious.

    The Sysdig Agent examines all execve events. Information about commands that meet the following criteria is saved by the Sysdig backend, and made available for review as a command entry in the Commands Audit module table:

    • A program was launched by a shell associated with a terminal (i.e. is related to a user-entered command).

    • The parent process was launched in a running container (i.e. the result of a docker exec <container> command).

    If an excessive volume of commands occurs in a given second, some commands may be excluded from the information sent from the agent to the Sysdig backend.

    The table below outlines the information displayed in the Command Audits module:

    DataDescription
    TimeThe date and time the command was executed.
    ShellThe terminal shell the command was executed in.
    Command LineThe full command executed, including flags/variables.
    ScopeThe affected scope within the infrastructure.

    Review a Command

    Individual commands can be reviewed by selecting the line item in the Commands Audit module table. This opens the Command Details window:

    The table below outlines the information displayed in the Command Details window:

    Name

    Description

    When

    The date and time the command was executed.

    Command

    The command executed.

    Full Command Line

    The complete command, including all variables/options.

    Working Directory

    The directory the command was executed in.

    Scope

    The entities within the infrastructure impacted by the command.

    Host

    The hostname and MAC address of the host the command was executed on.

    Container

    The container ID, container name, and image that the command was executed on.

    Additional Details

    Detailed user/host information:

    • The Process ID (PID) of the command.

    • The Parent Process ID (PPID) of the command.

    • The user ID of the user that executed the command.

    • The Shell ID.

    • The distance from the root of the process hierarchy.

    Filtering the Commands Table

    The Commands Audit module’s table can be filtered to display only the most relevant commands for a particular issue, or to provide greater visibility of a more targeted scope within the infrastructure. There are three ways to filter the table, which can be used in tandem to refine the information presented.

    Groupings

    Groupings are hierarchical organizations of labels, allowing users to organize their infrastructure views in a logical hierarchy. Users can switch between pre-configured groupings via the Browse By menu, or configure custom groupings, and then dive deeper into the infrastructure.

    Time Navigation

    Use the time window navigation bar to show only activities run within that window. (For more information, see also Time Windows.)

    Sysdig Secure does not currently provide the functionality to configure a custom time window.

    Search Filters

    Search filters can be applied by either using the search bar directly or by adding pre-configured search strings via the Command Details panel. The search bar example below displays only table items that include apt-get:

    To use a pre-configured search string:

    1. From the Commands Audit module, select a command from the table to open the Command Details window.

    2. Add a filter by click the Add link beside one of the available options:

    The example below shows the table filtered by the working directory:

    Pre-configured filters exist for the following information:

    • Command

    • Working Directory

    • Process ID

    • Parent Process ID

    • User ID

    • Shell ID

    • Shell Distance

    Search filters can be deleted by either deleting the text in the search bar or clicking the Remove link beside the filter in the Command Details window.