kubectl exec and other Kubernetes commands, you must enable
Kubernetes audit logging.
If you are running on a GKE cluster, review the GKE Limitations.
Activity Audit takes the high-value data from captures and makes it
always-on, searchable, and indexed against your cloud-native assets.
This stream includes executed commands, network activity, file activity,
kube exec requests to the Kubernetes API.
Understanding How Activity Audit is Used
Activity Audit allows users to view different data sources in-depth for monitoring, troubleshooting, diagnostics, or to meet regulatory controls.
Using to Investigate Events
A system investigation may be triggered by an event generated by Sysdig, or by an alert from another tool or person.
Find contextualized, relevant data Activity Audit allows easy access to the underlying data to help trace the event, evaluate its impact, and resolve the issue.
From Policy Events in Sysdig Secure, jump directly to the relevant Activity Audit to investigate details.
Trace commands and connections back to users Activity Audit can correlate the interactive requests from a Kubernetes user with the commands and network connections performed inside the container, allowing an operator to trace this activity back to a user identity.
Using for Regulatory Audits
The Activity Audit can also provide data about the infrastructure to help prove to auditors that proper data visibility and security measures are in place
Activity Audit is a critical requisite for many compliance standards:
Navigate the Audit Interface
Activity Audit displays a continuously updated list of activities. Use the UI features to find and filter the information you need.
Investigate > Activity Audit to access.
Filtering is the heart of Activity Audit’s power. Filters allow you to search, sort, parse, and surface meaningful data and connections as they are needed.
Ways to filter Activity Data:
Scope: Reduce the scope of your investigation by focusing on a more specific area of your infrastructure. By default, your scope will be set to
Everywhere, unless a team scope is defined for your currenly selected team.
Data Source: Choose a data source from the right side of the graph. Currently available data sources are:
kubectl exec, or
file. You can also select more than one source.
Attribute (=/!=): Choose = or -!= next to an attribute, either from the list or from the detail view, to include/exclude that attribute from the filter
Attribute (manual): If you know the attribute, you can type it into the filter box manually, with the following syntax:
Include an attribute
Exclude an attribute
Trace Trace entries to see all relevant activity in that session from that user. See the Trace Button paragraph for more detailed information.
Frequency graph: Select a section of the graph to zoom in on a time frame and see detailed activity. More info in the Frequency Graph paragraph
Combine: These methods can be combined as needed.
For example, the filter below surfaces activity on a particular pod, while excluding activity from one IP address that is known to be normal.
resource="pods" name="woocommerce-6877958" sourceaddresses!="172.20.41.2
The graph shows the activity frequency for each data source, allowing users easily to zero-in on anomalies.
The image above shows a spike in network activity (purple line) between 12:00 - 3:00 pm.
Drag the mouse over the peak to auto-zoom on the time frame and see more detail.
Use the legend at the right side of the graph to filter information from one particular data set. The currently available data sources are:
Kube exec commands
Activity audit feature captures only interactive commands and the
network connections and file modifications related to those commands.
Kube exec commands, on the other hand, are extracted from the
Kubernetes/OpenShift audit log.
Time Navigation Buttons
Use the time window navigation bar to show only activities run within that window.
Activity Row and Details
Select an activity row to see its details on the right panel, including all the collected attributes.
See Review Activity
for the attributes of each data source.
Some attributes allow you to quickly add filters including
= or excluding
!= such a value.
You can also perform quick filtering by selecting attribute and filter type directly from the activity row.
Trace Button for kube exec and Shell Activities
Beside each activity originating a trace there is a Trace button
- kube exec
- kube attach
- shells (*sh)
This feature allows you to correlate activities from the originator to each single performed operation. See Follow a kubectl exec Trace use case to see it in action.
This button does not appear if you are running on a GKE cluster.
The Kubernetes event in the activity audit list labeled kube run is received from the AC in the following cases:
- A kubectl run is performed
- A kubectl attach is performed
Review Activity Details
The date and time the command was executed.
The command executed.
Full Command Line
The complete command, including all variables/options.
The directory the command was executed in.
The entities within the infrastructure impacted by the command, including
The hostname and MAC address of the host the command was executed on.
Detailed user/host information:
Network Connection Details
UDP connections are currently captured in activity
The date and time of the network connection
Incoming or outgoing connection
The entities impacted by the network connection, including
The host name and MAC address of the host where the connection was made
The process name and ID (Parent Process ID/PID) that launched or received the network connection
Kubectl Exec Details
The date and time of the kubectl command
Kubernetes user and group
External IP address that initiated the connection
Host name and MAC address of the host where the kubectl exec was made
File Activity Details
Date and time the file was modified
File access details
- File name
- File directory
- Command used to access the file
- Access mode
Entities impacted by the file activity, including
The host name and MAC address of the host where the file activity occurred
Sample Use Cases
Look for Suspicious Commands
During an investigation you usually need to find suspicious activities among the most normal and recurring ones.
In this example, we have a
ps command being executed very frequently,
being noisy and making investigations harder.
In order facilitate focusing on other, more suspicious commands, filters can be used to reduce noise and make the investigation easier.
In this case, quick filters can be used to exclude the
so that other more interesting commands emerge instantly.
Filtering for Incident Response
A Policy Event reports a dangerous peak in network connections coming from a specific pod. This example describes one way to search for the root cause.
What user and what activity triggered this issue?
Use the Respond button next to the policy event to jump directly to the relevant data.
Here one can determine at a glance:
The pod/namespace on which the heightened activity is occurring
The process related to the activity (in this case,
ab, or the Apache Benchmark tool)
Related activities in the graph (
Repetitive entries that can be screened out
Refine the view through filtering:
networkdata source to
Filter out noisy, repetitive entries (e.g.
Investigate details of a kube exec item for user information.
After filtering, you have a focused incident report detailing:
The Kubernetes user “johndoe”
The external IP he used to connect
The set of commands he used to install and launch the Apache Benchmark stress-testing tool.
Follow a kubectl exec Trace
In a production environment,
kubectl exec commands are typically
suspicious. Also, because such commands are interactive sessions, it can
be difficult to pinpoint which individual has issued the command(s) and
what other activities the individual performed. This is where Sysdig’s
Trace functionality comes in, correlating kubectl exec commands with a
specific user and the network and command activities performed in that
In this example, suspicious activity has been detected and you want to determine whether someone has downloaded and executed a Trojan horse.
Use the Groups to display your Kubernetes hierarchy by namespace and deployment. Focus on the pod displaying unexpected high levels of activity (based on the number in parentheses).
Checking the corresponding activity graph, you zero in on a time frame and see
kube execactivity among the hundreds of commands and network events.
kube execitem and click the Trace button on left.
This session trace will display a formatted report of any container activity (network, commands) that the user performed inside the container.
This button does not appear if you are running on a GKE cluster.
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.