Kubernetes Audit Integration

This integration allows you to observe and monitor:

• Creation and destruction of pods, services, deployments, daemon sets
• Creating/updating/removing config maps or secrets
• Attempts to subscribe to changes to any endpoint

and other operations performed on Kubernetes resources.

Setup methods

There are two alternative ways to set up this integration:

• Emulated audit logs (default): uses a Dynamic Admission Control to get notified whenever a resource change is submitted:
  • It’s standard across all Kubernetes distributions, and easy to set up
  • It doesn’t provide logs for Get/List/Watch operations
  • It doesn’t provide IPs/User Agents
• Audit backend: relies on logs generated by the API server for any operation, as per the Audit Policy.
  • It has the most complete visibility on all APIs, along with IPs and User Agents
  • It officially needs to be set up to access the control plane, so set it up on managed when that access is not available, like in Managed distributions, is different on each use case, and logs can also have different formats.
  • It is only available on Linux

Prerequisites

• Helm v3.8 or above
• Install Sysdig Shield on Kubernetes using Helm

Emulated audit logs method setup

1. Recommended: Replace helm install with helm upgrade --install.

2. To enable Kubernetes audit logging in your existing Sysdig Secure Helm install command, add the following flag:

    --set features.detections.kubernetes_audit.enabled=true
   

3. For On-Prem < 7.4, set the Sysdig Secure API token. The API Token needs to belong to a user with a Role containing the permission Policies | Runtime Policies = READ.

   

Set that in Helm as well:

 --set sysdig_endpoint.secure_api_token="<YOUR_SYSDIG_API_TOKEN_HERE>"

For additional configuration options, refer to the installation guide and the configuration reference.

Audit backend method setup

This setup exposes a webhook through a Kubernetes Service, using the Shield chart. It then needs to be manually connected to the Kubernetes cluster audit log.

This setup heavily depends on Kubernetes distributions and their versions. We guide you through setting up the Service and only provide examples to connect this in some distributions.

For further support, please contact your Sysdig representative or Sysdig support.

Audit Log Webhook deployment

1. Recommended: Replace helm install with helm upgrade --install.

2. To enable the Audit Log Webhook in your existing Shield chart Helm install command, add the following flags:

    --set features.detections.kubernetes_audit.method=audit_backend
   

This will allow the Cluster Shield to receive Kubernetes Audit logs at http://sysdig-shield-cluster:6443/k8s_audit. Kubernetes Audit Logs are defined in the audit.k8s.io/v1 API Group. Read more in the official Kubernetes documentation.

With Cluster Shield < 1.22

1. Recommended: Replace helm install with helm upgrade --install.

2. To enable the Audit Log Webhook in your existing Sysdig Secure Helm install command, add the following flags:

    --set host.additional_settings.security.k8s_audit_server_enabled=true
    --set host.additional_settings.security.k8s_audit_server_url=0.0.0.0
    --set host.additional_settings.security.k8s_audit_server_port=7765
   

3. Create a Service to expose it:

apiVersion: v1
kind: Service
metadata:
  name: sysdig-shield-audit
  namespace: sysdig
spec:
  type: ClusterIP
  ports:
    - port: 7765
      protocol: TCP
      name: audit
  selector:
    app.kubernetes.io/instance: shield
    app.kubernetes.io/name: shield
    sysdig/component: host

This will create a Webhook listening on 0.0.0.0:7765, which can be reached via http://sysdig-shield-audit.sysdig.svc.cluster.local:7765/k8s_audit. Use this URL as the destination for the Audit logs instead of the one suggested, as it applies to setups using the new version. Kubernetes Audit Logs are defined in the audit.k8s.io/v1 API Group. Read more in the official Kubernetes documentation.

Sending Audit Logs to the Webhook

This section provides examples of how to configure this in different Kubernetes distributions.

OpenShift

Starting from Logging 5.7, it’s possible to set up a ClusterLogForwarder to send Audit Logs to an HTTP backend.

It relies on a Service Account to operate, so:

1. Start the creation with kubectl create serviceaccount logging-sa -n sysdig
2. Bind it to the ClusterRole required kubectl create clusterrolebinding logging-sa-binding \ --clusterrole=cluster-admin --serviceaccount=sysdig:logging-sa

Then, install the operator:

1. Open the OpenShift Web Console.
2. Navigate to Ecosystem -> Software Catalog
3. Select sysdig in the project list.
4. Search for “openshift logging” in the search bar.
5. Install the Red Hat OpenShift Logging operator.
6. Choose sysdig as the Installed namespace.
7. Create a ClusterLogForwarder using the configuration below.

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: logging
  namespace: sysdig
spec:
  outputs:
    - name: cluster-shield-audit-webhook
      http:
        method: POST
        url: "http://sysdig-shield-cluster:6443/k8s_audit"
      type: http
  pipelines:
    - inputRefs:
        - audit
      name: audit-log-pipeline
      outputRefs:
        - cluster-shield-audit-webhook
  serviceAccount:
    name: logging-sa

Kops

To implement this in kops you need to edit the cluster configuration.

1. Start by obtaining the current cluster configuration and saving it to a file:

   kops get cluster <your cluster name> -o yaml > cluster-current.yaml
   

2. Edit cluster-current.yaml, merging it with the YAML below, replacing what already exists and adding what’s missing. You can find the content of the webhook-config and audit-policy file assets in the dedicated sections:

   • Audit Policy, to replace AUDIT_POLICY_FILE_CONTENT
   • Webhook configuration, to replace WEBHOOK_CONFIG_FILE_CONTENT.

   spec:
     fileAssets:
       - name: webhook-config
         path: /var/lib/k8s_audit/webhook-config.yaml
         roles: [Master]
         content: |
           WEBHOOK_CONFIG_FILE_CONTENT
       - name: audit-policy
         path: /var/lib/k8s_audit/audit-policy.yaml
         roles: [Master]
         content: |
           AUDIT_POLICY_FILE_CONTENT
     kubeAPIServer:
       auditLogPath: /var/lib/k8s_audit/audit.log
       auditLogMaxBackups: 1
       auditLogMaxAge: 10
       auditLogMaxSize: 100
       auditWebhookBatchMaxWait: 5s
       auditPolicyFile: /var/lib/k8s_audit/audit-policy.yaml
       auditWebhookConfigFile: /var/lib/k8s_audit/webhook-config.yaml
   

3. Configure Kops with the new cluster configuration:

   kops replace -f cluster.yaml
   

4. Update the cluster configuration to prepare changes to the cluster:

   kops update cluster <your cluster name> --yes
   

5. Perform a rolling update to redeploy the master nodes with the new files and API server configuration:

   kops rolling-update cluster --yes
   

GKE

GKE provides Kubernetes audit logs, but unlike standard Kubernetes, the logs are stored in Cloud Logging and use a different format than the native one used by Kubernetes. For more information, see Google’s official documentation.

To set up the audit log delivery infrastructure, you can use the command-line interface (CLI) or the Google Cloud Console. You can also provision the log delivery pipeline with Terraform.

As a preliminary step, enable audit log generation in your cluster.

• CLI
• Console

Additional information is available in the Google Cloud documentation.

Once these steps are completed, logs are available to be queried in Logs Explorer:

protoPayload.@type="type.googleapis.com/google.cloud.audit.AuditLog"
resource.labels.cluster_name="<CLUSTER_NAME>"

Next, enable Workload Identity in your cluster so Sysdig Shield can use a Google Cloud service account to access those logs.

• CLI
• Console

gcloud container clusters update <CLUSTER_NAME> \
  --region=<CLUSTER_REGION> \
  --workload-pool=<PROJECT_ID>.svc.id.goog \
  --project=<PROJECT_ID>

gcloud container node-pools list \
  --cluster=<CLUSTER_NAME> --region=<CLUSTER_REGION> --project=<PROJECT_ID>

gcloud container node-pools update <POOL_NAME> \
  --cluster=<CLUSTER_NAME> --region=<CLUSTER_REGION> \
  --workload-metadata=GKE_METADATA \
  --project=<PROJECT_ID>

Next, route those logs to Sysdig Shield through a delivery pipeline. Sysdig Shield consumes the logs from Pub/Sub using the StreamingPull API.

• Terraform
• CLI
• Console

After this is done, update the Helm configuration for Sysdig Shield:

 --set features.detections.kubernetes_audit.method=audit_backend
 --set features.detections.kubernetes_audit.gke_pub_sub_subscription=<SUBSCRIPTION>
 --set 'cluster.rbac.annotations.iam\.gke\.io/gcp-service-account=<SYSDIG_SERVICE_ACCOUNT>@<PROJECT_ID>.iam.gserviceaccount.com'

After applying the changes, restart deployment/shield-cluster to finalize the setup.

Note that Pub/Sub and Logging are paid GCP offerings.

GKE uses a Kubernetes audit policy that emits a more limited set of information than the one recommended by Sysdig. As a result, there are several limitations when retrieving Kubernetes audit information for the Events feed and Activity Audit features in Sysdig Secure.

Request Object

In particular, audit events for config maps in GKE generally do not contain a requestObject field that contains the object being created/modified.

Pod exec does not Include command/container

For many Kubernetes distributions, an audit event representing a pod exec includes the command and specific container as arguments to the requestURI. For example:

"requestURI": "/api/v1/namespaces/default/pods/nginx-deployment-7998647bdf-phvq7/exec?command=bash&container=nginx1&container=nginx1&stdin=true&stdout=true&tty=true

In GKE, the audit event is missing those request parameters.

Implications for the Event Feed

If the rule condition includes a field that is not available in the Kubernetes audit log provided by GKE, the rule will not trigger.

For example, rules targeting changes to ConfigMap content containing specific information or specific commands being executed will not trigger, leading to missed detections/false negatives.

This will also limit the information that can be displayed in the outputs of rules. For example the command=%ka.uri.param[command] output variable in the Attach/Exec Pod rule will always return N/A.

Implications for Activity Audit

• kubectl exec elements will not be scoped to the cluster name; they will only be visible scoping by entire infrastructure"

• A kubectl exec item in Activity Audit will not display command or container information

• Drilling down into a kubectl exec will not provide the container activity as there is no information that allows Sysdig to correlate the kubectl exec action with an individual container.

EKS

Amazon EKS does not provide webhooks for audit logs, but it allows audit logs to be forwarded to CloudWatch. You can check the status and enable them via the Console or the AWS CLI.

• Console
• AWS CLI

Next, you need to register the IAM OIDC provider for the cluster. First, obtain the cluster OIDC ID:

• Console
• AWS CLI

aws eks describe-cluster --name <CLUSTER_NAME> --region <REGION> \
  --query "cluster.identity.oidc.issuer" --output text

Then, verify whether the provider is already registered and configured correctly.

• Console
• AWS CLI

aws iam list-open-id-connect-providers \
  --query "OpenIDConnectProviderList[?contains(Arn, '<OIDC_ID>')].Arn" \
  --output text

eksctl utils associate-iam-oidc-provider \
  --cluster <CLUSTER_NAME> --region <REGION> --approve

THUMBPRINT=$(aws eks describe-cluster \
  --name <CLUSTER_NAME> --region <REGION> \
  --query "cluster.certificateAuthority.data" --output text \
  | base64 --decode | openssl x509 -fingerprint -sha1 -noout \
  | sed 's/://g; s/.*=//' | tr '[:upper:]' '[:lower:]')

aws iam create-open-id-connect-provider \
  --url https://oidc.eks.<REGION>.amazonaws.com/id/<OIDC_ID> \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list "$THUMBPRINT"

Afterward, you need to provision an IAM Role that allows Sysdig Shield to access the audit logs. You can use either the Terraform snippet or the CloudFormation stack template below. To run either one, you will need:

• Cluster name
• Region (set as a value in Terraform; selected in the Console for CloudFormation)
• Cluster OIDC ID
• Sysdig Shield namespace
• Sysdig Cluster Shield service account name, if customized
• Cluster CloudWatch Log Group, if customized

Each produces an IAM Role whose ARN you will use later.

• Terraform
• CloudFormation

locals {
  # ── Required: fill these in ──────────────────────────────────────────────

  # AWS region where the EKS cluster lives.
  eks_audit_reader_region = "us-east-1"

  # Name of the EKS cluster.
  eks_audit_reader_cluster_name = "cluster-name"

  # OIDC issuer URL for the EKS cluster (no trailing slash).
  # Retrieve with:
  #   aws eks describe-cluster --name <cluster> --query 'cluster.identity.oidc.issuer' --output text
  eks_audit_reader_oidc_issuer_url = "https://oidc.eks.<region>.amazonaws.com/id/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

  # Kubernetes namespace and service account name for the Cluster Shield.
  eks_audit_reader_k8s_namespace       = "sysdig"
  eks_audit_reader_k8s_service_account = "shield-cluster"

  # ── Optional: override defaults ──────────────────────────────────────────

  # Override CloudWatch log group. Leave empty to use /aws/eks/<cluster>/cluster.
  eks_audit_reader_cw_log_group = ""

  eks_audit_reader_tags = {
    product = "sysdig-secure"
  }

  # ── Derived (do not edit) ─────────────────────────────────────────────────

  _eks_audit_reader_log_group_name = (
    local.eks_audit_reader_cw_log_group != ""
    ? local.eks_audit_reader_cw_log_group
    : "/aws/eks/${local.eks_audit_reader_cluster_name}/cluster"
  )

  # Strip the "https://" prefix to get the bare OIDC provider hostname used in
  # the IAM trust policy condition keys and OIDC provider ARN.
  _eks_audit_reader_oidc_provider = trimprefix(local.eks_audit_reader_oidc_issuer_url, "https://")

  _eks_audit_reader_oidc_provider_arn = "arn:${data.aws_partition.eks_audit_reader.partition}:iam::${data.aws_caller_identity.eks_audit_reader.account_id}:oidc-provider/${local._eks_audit_reader_oidc_provider}"

  _eks_audit_reader_log_group_arn = "arn:${data.aws_partition.eks_audit_reader.partition}:logs:${local.eks_audit_reader_region}:${data.aws_caller_identity.eks_audit_reader.account_id}:log-group:${local._eks_audit_reader_log_group_name}"
}

# ── Data sources ──────────────────────────────────────────────────────────────

data "aws_caller_identity" "eks_audit_reader" {}
data "aws_partition"       "eks_audit_reader" {}

# ── IAM role (IRSA) ───────────────────────────────────────────────────────────

resource "aws_iam_role" "eks_audit_reader" {
  name = "sysdig-eks-audit-reader-${local.eks_audit_reader_cluster_name}"
  tags = local.eks_audit_reader_tags

  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Effect = "Allow"
      Principal = {
        Federated = local._eks_audit_reader_oidc_provider_arn
      }
      Action = "sts:AssumeRoleWithWebIdentity"
      Condition = {
        StringEquals = {
          "${local._eks_audit_reader_oidc_provider}:sub" = "system:serviceaccount:${local.eks_audit_reader_k8s_namespace}:${local.eks_audit_reader_k8s_service_account}"
          "${local._eks_audit_reader_oidc_provider}:aud" = "sts.amazonaws.com"
        }
      }
    }]
  })
}

resource "aws_iam_role_policy" "eks_audit_reader" {
  name = "cloudwatch-read"
  role = aws_iam_role.eks_audit_reader.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        # DescribeLogGroups does not support resource-level permissions.
        Sid      = "DescribeLogGroups"
        Effect   = "Allow"
        Action   = ["logs:DescribeLogGroups"]
        Resource = "*"
      },
      {
        Sid    = "ReadEksAuditLogs"
        Effect = "Allow"
        Action = [
          "logs:DescribeLogStreams",
          "logs:GetLogEvents",
          "logs:FilterLogEvents",
        ]
        Resource = [
          local._eks_audit_reader_log_group_arn,
          "${local._eks_audit_reader_log_group_arn}:log-stream:*",
        ]
      },
      {
        # Lets the Cluster Shield resolve the real cluster name from a node's
        # aws:eks:cluster-name EC2 tag, so the correct CloudWatch log group is
        # used even when it differs from the configured cluster name.
        # ec2:DescribeTags does not support resource-level permissions.
        Sid      = "ResolveClusterName"
        Effect   = "Allow"
        Action   = ["ec2:DescribeTags"]
        Resource = "*"
      },
    ]
  })
}

# ── Output ────────────────────────────────────────────────────────────────────

output "eks_audit_reader_role_arn" {
  description = "IAM role ARN. Set as the value of the eks.amazonaws.com/role-arn annotation on the Kubernetes service account."
  value       = aws_iam_role.eks_audit_reader.arn
}

AWSTemplateFormatVersion: "2010-09-09"
Description: >
  Provisions the identity to connect Sysdig Cluster Shield with EKS audit logs in CloudWatch Logs.

# ── Parameters ────────────────────────────────────────────────────────────────

Parameters:
  EksClusterName:
    Type: String
    Description: >
      Name of the EKS cluster. The CloudWatch log group
      /aws/eks/<EksClusterName>/cluster is used unless
      CloudWatchLogGroupName is set explicitly.

  OidcIssuerUrl:
    Type: String
    Description: >
      OIDC issuer URL for the EKS cluster (no trailing slash).
      Retrieve with:
      aws eks describe-cluster --name <cluster>
        --query 'cluster.identity.oidc.issuer' --output text

  K8sNamespace:
    Type: String
    Default: "sysdig"
    Description: >
      Kubernetes namespace for the Cluster Shield.

  K8sServiceAccount:
    Type: String
    Default: "shield-cluster"
    Description: >
      Kubernetes Service Account name for the Cluster Shield

  CloudWatchLogGroupName:
    Type: String
    Default: ""
    Description: >
      Override the CloudWatch log group name.
      Leave empty to use /aws/eks/<EksClusterName>/cluster.

# ── Conditions ────────────────────────────────────────────────────────────────

Conditions:
  UseDefaultLogGroup: !Equals [!Ref CloudWatchLogGroupName, ""]

# ── Resources ─────────────────────────────────────────────────────────────────

Resources:

  ReaderRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "sysdig-eks-audit-reader-${EksClusterName}"
      Tags:
        - Key: product
          Value: sysdig-secure
      # Use Fn::Sub on a JSON string so the OIDC provider hostname can appear
      # as a map key inside the Condition block of the trust policy.
      AssumeRolePolicyDocument: !Sub
        - |
          {
            "Version": "2012-10-17",
            "Statement": [{
              "Effect": "Allow",
              "Principal": {
                "Federated": "arn:${AWS::Partition}:iam::${AWS::AccountId}:oidc-provider/${OidcProvider}"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                "StringEquals": {
                  "${OidcProvider}:sub": "system:serviceaccount:${K8sNamespace}:${K8sServiceAccount}",
                  "${OidcProvider}:aud": "sts.amazonaws.com"
                }
              }
            }]
          }
        # Strip the "https://" prefix to get the bare hostname used in both
        # the OIDC provider ARN and the trust-policy condition keys.
        - OidcProvider: !Select [1, !Split ["https://", !Ref OidcIssuerUrl]]
      Policies:
        - PolicyName: cloudwatch-read
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - # DescribeLogGroups does not support resource-level permissions.
                Sid: DescribeLogGroups
                Effect: Allow
                Action:
                  - logs:DescribeLogGroups
                Resource: "*"
              - Sid: ReadEksAuditLogs
                Effect: Allow
                Action:
                  - logs:DescribeLogStreams
                  - logs:GetLogEvents
                  - logs:FilterLogEvents
                Resource:
                  - !Sub
                    - "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${LogGroup}"
                    - LogGroup: !If
                        - UseDefaultLogGroup
                        - !Sub "/aws/eks/${EksClusterName}/cluster"
                        - !Ref CloudWatchLogGroupName
                  - !Sub
                    - "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${LogGroup}:log-stream:*"
                    - LogGroup: !If
                        - UseDefaultLogGroup
                        - !Sub "/aws/eks/${EksClusterName}/cluster"
                        - !Ref CloudWatchLogGroupName
              - # Lets the Cluster Shield resolve the real cluster name from a node's
                # aws:eks:cluster-name EC2 tag, so the correct CloudWatch log group is
                # used even when it differs from the configured cluster name.
                # ec2:DescribeTags does not support resource-level permissions.
                Sid: ResolveClusterName
                Effect: Allow
                Action:
                  - ec2:DescribeTags
                Resource: "*"

# ── Outputs ───────────────────────────────────────────────────────────────────

Outputs:
  RoleArn:
    Description: >
      IAM role ARN. Set as the value of the eks.amazonaws.com/role-arn
      annotation on the Kubernetes service account.
    Value: !GetAtt ReaderRole.Arn

  RoleName:
    Description: Name of the IAM role.
    Value: !Ref ReaderRole

Once you have the IAM Role ARN, update the Helm setup for the Sysdig Shield:

 --set features.detections.kubernetes_audit.method=audit_backend
 --set 'cluster.rbac.annotations.eks\.amazonaws\.com/role-arn=<IAM_ROLE_ARN>'

Once applied, restart deployment/shield-cluster to finalize the setup.

Please note that CloudWatch is an additional AWS paid offering. In addition, with this solution, all the pods running on the worker nodes will be allowed to read CloudWatch logs through AWS APIs.

AKS

In AKS, you can enable Audit Logs generation using Diagnostic Settings for the cluster. You can refer to the official documentation to set this up.

Those Diagnostic Settings will be used to route those logs to a destination which, unfortunately, can’t be the Sysdig webhook, directly. There are a few alternative architectures, but our suggestion is to employ Event Hub as a target.

If you choose this route, you will need something to subscribe to the Event Hub stream and send those events to the webhook. This can be achieved in many ways, among those:

• Use a Kubernetes deployment, like in the example available in this repository
• Use Logic App, triggering on events available on Event Hub and sending requests to the service, correctly exposed.

RKE/K3s

Audit support is already enabled by default, but the audit policy must be updated to provide additional granularity. These instructions enable a webhook backend pointing to the agent’s service.

The following steps need to be applied on each node in the control plane:

• Follow the official documentation to edit the Audit policy. This is the link to v2.13; verify the exact procedure for your version. You can find a suggested Policy here.

• Next, create the Webhook Configuration. Based on the distribution/version: /var/lib/rancher/{rke2,k3s}/server/audit-webhook.yaml. You can find the content here, where you can use the pod network option.

• This configuration then needs to be used by the Cluster. This is applied via the control plane configuration (/etc/rancher/rke2/config.yaml) or as arguments (SystemD approach)

  • audit-policy-file=/var/lib/rancher/rke2/server/audit.yaml
  • audit-webhook-config-file=/var/lib/rancher/rke2/server/audit-webhook.yaml
  • audit-webhook-batch-max-size=100
  • audit-webhook-batch-max-wait=5s

• Finally, restart the nodes

IKS

IKS supports generating Audit Logs and forwarding them to an HTTP endpoint. The sole caveat is that the API server runs outside the cluster network and, therefore, needs to reach the webhook through its IP. The other limitation is on the generated logs, where the configured policy cannot be changed.

You can refer to IBM documentation to set this up. The Webhook URL can be obtained by looking at the Service previously set up"

echo "http://$(kubectl get service sysdig-shield-cluster -n sysdig -o jsonpath='{.spec.clusterIP}'):6443/k8s_audit"

Minikube

Minikube lets you manage the Control plane directly, so you can specify the Audit Policy and the Audit backend directly. This is accomplished using the --extra-config parameter of the minikube start command, changing the apiserver configurations, as shown in the tutorial of Minikube. See the reference for the minikube start command in the Minikube documentation and the apiserver configurations in the official Kubernetes documentation.

In each node of the Control plane, as suggested by the tutorial, create the audit-policy.yaml as well as the webhook-config.yaml files in the ~/.minikube/files/etc/ssl/certs directory. Afterward, you can restart Minikube by specifying the following options:

• --extra-config=apiserver.audit-policy-file=/etc/ssl/certs/audit-policy.yaml
• --extra-config=apiserver.audit-log-path=-
• --extra-config=apiserver.audit-webhook-config-file=/etc/ssl/certs/webhook-config.yaml
• --extra-config=apiserver.audit-webhook-batch-max-size=10
• --extra-config=apiserver.audit-webhook-batch-max-wait=5s

Kubeadm

With Kubeadm you manage the Control Plane directly so you can create the audit-policy.yaml and the webhook-config.yaml files in the /etc/kubernetes/ folder.

Then, make the Control plane use them. Create /etc/kubernetes/kubeadm-audit-config.yaml with:

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
apiServer:
  extraArgs:
    - name: "audit-policy-file"
      value: "/etc/kubernetes/audit-policy.yaml"
    - name: "audit-log-path"
      value: "-"
    - name: "audit-webhook-config-file"
      value: "/etc/kubernetes/webhook-config.yaml"
    - name: "audit-webhook-batch-max-size"
      value: "10"
    - name: "audit-webhook-batch-max-wait"
      value: "5s"

Finally, run kubeadm init --config /etc/kubernetes/kubeadm-audit-config.yaml

See the official documentation for additional information.

Reference: Audit Policy

This is a general-purpose Audit Policy you can use to audit all the relevant events in the Control Plane. Adjust it based on your use case, if needed.

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - "RequestReceived"

rules:
  # Ignore low-impact read-only URLs
  - level: None
    userGroups: ["system:authenticated"]
    nonResourceURLs:
      - "/api*"
      - "/version"
      - "/healthz"
      - "/readyz"
      - "/livez"

  # Ignore system noise
  - level: None
    users:
      - "system:kube-proxy"
      - "system:kube-scheduler"
      - "system:kube-controller-manager"
    verbs: ["get", "list", "watch"]

  # Log exec, attach, port-forward at RequestResponse (capture commands)
  - level: RequestResponse
    verbs: ["create"]
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # Log security-sensitive resources at RequestResponse level
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["secrets", "serviceaccounts"]
      - group: "rbac.authorization.k8s.io"
        resources:
          ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
      - group: "networking.k8s.io"
        resources: ["networkpolicies"]
      - group: "policy"
        resources: ["podsecuritypolicies"]

  # Log pod and deployment changes at RequestResponse
  - level: RequestResponse
    resources:
      - group: ""
        resources:
          ["pods", "services", "persistentvolumes", "persistentvolumeclaims"]
      - group: "apps"
        resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]

  # Log configmap changes
  - level: Request
    resources:
      - group: ""
        resources: ["configmaps"]

  # Catch-all at Metadata level
  - level: Metadata
    omitStages:
      - "RequestReceived"

Reference: Webhook configuration

This is a standard configuration for the audit webhook. Based on the networking your Control Plane uses, you should set AUDIT_WEBHOOK_ENDPOINT differently:

• sysdig-shield-cluster if it uses the pod networking and can access the Cluster DNS
• The ClusterIP corresponding to the Service, if it runs outside the pod networking. This can be obtained via:

  kubectl get service sysdig-shield-cluster -n sysdig -o jsonpath='{.spec.clusterIP}')
  

apiVersion: v1
kind: Config
clusters:
  - name: sysdig
    cluster:
      server: http://AUDIT_WEBHOOK_ENDPOINT:6443/k8s_audit
    users:
      - name: sysdig
    contexts:
      - context:
          cluster: sysdig
          user: sysdig
        name: sysdig
    current-context: sysdig

Next Steps

Once this is set up, you will see:

• Events in the Events Feed, if you enabled Kubernetes Audit Threat detection policies and any Audit log matches the rules they contain.
• Activity Audit entries for the Kubernetes Data source (attach/exec).