Kubernetes Audit Logging

Kubernetes log integration enables Sysdig Secure to use Kubernetes audit log data for Falco rules, activity audit, and to test the impact of Pod Security Policies. We now provide examples for the distributions and platforms listed below.

The integration allows auditing of:

  • Creation and destruction of pods, services, deployments, daemon sets, etc.

  • Creating/updating/removing config maps or secrets

  • Attempts to subscribe to changes to any endpoint

Prerequisites

Install Sysdig Agent and Apply the Agent Service

These instructions assume that the Sysdig agent has already been deployed to the Kubernetes cluster. See Agent Installation for details. When the agent(s) are installed, have the Sysdig agent service account, secret, configmap, and daemonset information on hand.

If the sysdig-agent-service.yaml was not explicitly deployed during agent installation, you need to apply it now:

kubectl apply -f https://raw.githubusercontent.com/draios/sysdig-cloud-scripts/master/agent_deploy/kubernetes/sysdig-agent-service.yaml -n sysdig-agent

Choose Enablement Steps

Sysdig has tested Kubernetes audit log integration on a variety of platforms and distributions. Each requires different steps, as detailed in the sections below.

The routing of Kubernetes audit events has changed rapidly between Kubernetes versions. For more information, review the Kubernetes documentation.

Routing is accomplished via either:

  • Webhook backend: Kubernetes version >= 1.11, or

  • Dynamic backend with Audit Sink: Kubernetes version >= 1.13

When a distribution supports both 1.11/1.12 and 1.13+, the dynamic backend version is preferred.

The table below summarizes the tested options:

Distro/Platform

Version

Uses Webhook

Uses Dynamic

Uses Other

OpenShift

3.11

X

OpenShift

4.2, 4.3

X

MiniShift

3.11

X

Kops

1.15

X

GKE (Google)

1.13

X (bridge)

EKS (Amazon)

eks.5/ Kubernetes 1.14

X CloudWatch

RKE (Rancher)

RKE v1.0.0/Kubernetes 1.13+

X

IKS (IBM)

1.15

X

Minikube

1.11/1.12

X

Minikube

1.13+

X

Enable Kubernetes Audit Logging

These instructions assume that the Kubernetes cluster has NO audit configuration or logging in place. The steps add configuration only to route audit log messages to the Sysdig agent.

There is a beta script automating many of these steps, which is suitable for proof-of-concept/non-production environments. In any case, we recommend reading the step-by-step instructions carefully before continuing.

OpenShift 3.11

Note

Openshift 3.11 only supports webhook backends (described as "Advanced Audit" in the Openshift Documentation).

Follow the steps below on the Kubernetes API master node:

  1. Copy the provided audit-policy.yaml file to the Kubernetes API master node in the /etc/origin/master directory.

    (The file will be picked up by OpenShift services running in containers because this directory is mounted into the Kube API server container at /etc/origin/master.)

  2. Create a Webhook Configuration File and copy it to the Kubernetes API master node, in the /etc/origin/master directory.

  3. Modify the master configuration by adding the following to your /etc/origin/master/master-config.yaml file, replacing any existing auditConfig: entry.

    auditConfig:
      enabled: true
      maximumFileSizeMegabytes: 10
      maximumRetainedFiles: 1
      auditFilePath: "/etc/origin/master/k8s_audit_events.log"
      logFormat: json
      webHookMode: "batch"
      webHookKubeConfig: /etc/origin/master/webhook-config.yaml
      policyFile: /etc/origin/master/audit-policy.yaml 

    One way to do this is to use oc ex config patch.

    Assuming the above content were in a file audit-patch.yaml and you had copied /etc/origin/master/master-config.yaml to /tmp/master-config.yaml.original, you could run:

    oc ex config patch /tmp/master-config.yaml.original -p "$(cat audit-patch.yaml)" > /etc/origin/master/master-config.yaml
  4. Restart the API server by running the following:

    # sudo /usr/local/bin/master-restart api
    # sudo /usr/local/bin/master-restart controllers

    Once restarted, the server will route Kubernetes audit events to the Sysdig agent service.

MiniShift 3.11

Like OpenShift 3.11, Minishift 3.11 supports webhook backends, but the way Minishift launches the Kubernetes API server is different. Therefore, the command line arguments are somewhat different than in the instructions above.

  1. Copy the provided audit-policy.yaml file to the Minishift VM into the directory /var/lib/minishift/base/kube-apiserver/.

    (The file will be picked up by Minishift services running in containers because this directory is mounted into the kube API server container at /etc/origin/master.)

  2. Create a Webhook Configuration File and copy it to the Minishift VM into the directory /var/lib/minishift/base/kube-apiserver/.

  3. Modify the master configuration by adding the following to /var/lib/minishift/base/kube-apiserver/master-config.yaml on the Minishift VM, merging/updating as required. 

    Note:master-config.yaml also exists in other directories such as /var/lib/minishift/base/openshift-apiserver and /var/lib/minishift/base/openshift-controller-manager/.

    You should modify the one in kube-apiserver:

    kubernetesMasterConfig:
      apiServerArguments:
      audit-log-maxbackup:
      - "1"
      audit-log-maxsize:
      - "10"
      audit-log-path:
      - /etc/origin/master/k8s_audit_events.log
      audit-policy-file:
      - /etc/origin/master/audit-policy.yaml
      audit-webhook-batch-max-wait:
      - 5s
      audit-webhook-config-file:
      - /etc/origin/master/webhook-config.yaml
      audit-webhook-mode:
      - batch
    
  4. Restart the API server by running the following:

    (For minishift)
    # minishift openshift restart 

    Once restarted, the server will route Kubernetes audit events to the Sysdig agent service.

OpenShift 4.2, 4.3

By default, Openshift 4.2/4.3 enables Kubernetes API server logs and makes them available on each master node, at the path /var/log/kube-apiserver/audit.log. However, the API server is not configured by default with the ability to create dynamic backends.

You must first enable the creation of dynamic backends by changing the API server configuration. You then create audit sinks to route audit events to the Sysdig agent.

  1. Run the following to update the API server configuration:

    oc patch kubeapiserver cluster --type=merge -p '{"spec":{"unsupportedConfigOverrides":{"apiServerArguments":{"audit-dynamic-configuration":["true"],"feature-gates":["DynamicAuditing=true"],"runtime-config":["auditregistration.k8s.io/v1alpha1=true"]}}}}' 
  2. Wait for the API server to restart with the updated configuration.

  3. Create a Dynamic Audit Sink.

    Once the dynamic audit sink is created, it will route Kubernetes audit events to the Sysdig agent service.

Kops

As the latest released version Kops (1.15) does not yet support configuring dynamic backends, these instructions use a webhook. You will modify the cluster configuration using kops set, update the configuration using kops update, and then perform a rolling update using kops rolling-update.

Note: A fix for Kops is merged in https://github.com/kubernetes/kops/pull/7424 but is not yet part of any Kops release.

  1. Create a Webhook Configuration File and save it locally.

  2. Get the current cluster configuration and save it to a file:

    kops get cluster <your cluster name> -o yaml > cluster-current.yaml
  3. To ensure that webhook-config.yaml is available on each master node at /var/lib/k8s_audit, and that the kube-apiserver process is run with the required arguments to enable the webhook backend, you will edit cluster.yaml to add/modify the fileAssets and kubeAPIServer sections as follows:

    apiVersion: kops.k8s.io/v1alpha2
    kind: Cluster
    spec:
      ...
      fileAssets:
        - name: webhook-config
          path: /var/lib/k8s_audit/webhook-config.yaml
          roles: [Master]
          content: |
            <contents of webhook-config.yaml go here>
        - name: audit-policy
          path: /var/lib/k8s_audit/audit-policy.yaml
          roles: [Master]
          content: |
            <contents of audit-policy.yaml go here>
      ...
      kubeAPIServer:
        auditLogPath: /var/lib/k8s_audit/audit.log
        auditLogMaxBackups: 1
        auditLogMaxSize: 10
        auditWebhookBatchMaxWait: 5s
        auditPolicyFile: /var/lib/k8s_audit/audit-policy.yaml
        auditWebhookConfigFile: /var/lib/k8s_audit/webhook-config.yaml
      ...

    A simple way to do this using yq would be with the following script:

    cat <<EOF > merge.yaml
    spec:
      fileAssets:
        - name: webhook-config
          path: /var/lib/k8s_audit/webhook-config.yaml
          roles: [Master]
          content: |
    $(cat webhook-config.yaml | sed -e 's/^/        /')
        - name: audit-policy
          path: /var/lib/k8s_audit/audit-policy.yaml
          roles: [Master]
          content: |
    $(cat audit-policy.yaml | sed -e 's/^/        /')
      kubeAPIServer:
        auditLogPath: /var/lib/k8s_audit/audit.log
        auditLogMaxBackups: 1
        auditLogMaxSize: 10
        auditWebhookBatchMaxWait: 5s
        auditPolicyFile: /var/lib/k8s_audit/audit-policy.yaml
        auditWebhookConfigFile: /var/lib/k8s_audit/webhook-config.yaml
    EOF
    
    yq m -a cluster-current.yaml merge.yaml > cluster.yaml
  4. Configure Kops with the new cluster configuration:

    kops replace -f cluster.yaml
  5. Update the cluster configuration to prepare changes to the cluster:

    kops update cluster <your cluster name> --yes
  6. Perform a rolling update to redeploy the master nodes with the new files and API server configuration:

    kops rolling-update cluster --yes 

GKE (Google)

Note

These instructions assume you have already created a cluster and configured the gcloud and kubectl command-line programs to interact with the cluster.

Note that the kubectl exectrace feature is not currently supported on GKE clusters, but all the other activity audit features should function normally.

GKE already provides Kubernetes audit logs, but the logs are exposed using Stackdriver and are in a different format than the native format used by Kubernetes.

To simplify things, we have written a bridge program that reads audit logs from Stackdriver, reformats them to match the Kubernetes-native format, and sends the logs to a configurable webhook and to the Sysdig agent service.

  1. Create a Google Cloud (not Kubernetes) service account and key that has the ability to read logs:

    $ gcloud iam service-accounts create swb-logs-reader --description "Service account used by stackdriver-webhook-bridge" --display-name "stackdriver-webhook-bridge logs reader"
    $ gcloud projects add-iam-policy-binding <your gce project id> --member serviceAccount:swb-logs-reader@<your gce project id>.iam.gserviceaccount.com --role 'roles/logging.viewer'
    $ gcloud iam service-accounts keys create $PWD/swb-logs-reader-key.json --iam-account swb-logs-reader@<your gce project id>.iam.gserviceaccount.com 
  2. Create a Kubernetes secret containing the service account keys:

    kubectl create secret generic stackdriver-webhook-bridge --from-file=key.json=$PWD/swb-logs-reader-key.json -n sysdig-agent
  3. Deploy the bridge program to your cluster using the provided stackdriver-webhook-bridge.yaml file:

    kubectl apply -f stackdriver-webhook-bridge.yaml -n sysdig-agent 

The bridge program routes audit events to the domain name sysdig-agent.sysdig-agent.svc.cluster.local, which corresponds to the sysdig-agent service you created either when deploying the agent or as a prerequisite step.

EKS (Amazon)

These instructions were verified with eks.5 on Kubernetes v1.14.

Amazon EKS does not provide webhooks for audit logs, but it allows audit logs to be forwarded to CloudWatch. To access CloudWatch logs from the Sysdig agent, proceed as follows:

  1. Enable CloudWatch logs for your EKS cluster.

  2. Allow access to CloudWatch from the worker nodes.

  3. Add a new deployment that polls CloudWatch and forwards events to the Sysdig agent.

You can find an example configuration that can be implemented with the AWS UI, along with the code and the image for an example audit log forwarder. (In a production system this would be implemented as IaC scripts.)

Please note that CloudWatch is an additional AWS paid offering. In addition, with this solution, all the pods running on the worker nodes will be allowed to read CloudWatch logs through AWS APIs.

RKE (Rancher) with Kubernetes 1.13+

These instructions were verified with RKE v1.0.0 and Kubernetes v1.16.3. It should work with versions as old as Kubernetes v1.13.

Audit support is already enabled by default, but the audit policy must be updated to provide additional granularity. These instructions enable a webhook backend pointing to the agent's service. Dynamic audit backends are not supported as there isn't a way to enable the audit feature flag.

  1. On each Kubernetes API Master Node, create the directory /var/lib/k8s_audit.

  2. On each Kubernetes API master node, copy the provided audit-policy.yaml file to to the master node into the directory /var/lib/k8s_audit. (This directory will be mounted into the API server, giving it access to the audit/webhook files.)

  3. Create a Webhook Configuration File and copy it to each Kubernetes API master node, into the directory /var/lib/k8s_audit.

  4. Modify your RKE cluster configuration cluster.yml to add extra_args and extra_binds sections to the kube-api section. Here's an example:

    kube-api:
    ...
        extra_args:
          audit-policy-file: /var/lib/k8s_audit/audit-policy.yaml
          audit-webhook-config-file: /var/lib/k8s_audit/webhook-config.yaml
          audit-webhook-batch-max-wait: 5s
        extra_binds:
        - /var/lib/k8s_audit:/var/lib/k8s_audit
    ... 

    This changes the command-line arguments for the API server to use an alternate audit policy and to use the webhook backend you created.

  5. Restart the RKE cluster via rke up.

IKS (IBM)

IKS supports routing Kubernetes audit events to a single configurable webhook backend URL. It does not support dynamic audit sinks and does not support the ability to change the audit policy that controls which Kubernetes audit events are sent.

The instructions below were adapted from the IBM-provided documentation on how to integrate with Fluentd. It is expected that you are familiar with (or will review) the IKS tools for forwarding cluster and app logs described there.

Limitation: The Kubernetes default audit policy generally does not include events at the Request or RequestResponse levels, meaning that any rules that look in detail at the objects being created/modified (e.g. rules using the ka.req.* and ka.resp.* fields) will not trigger. This includes the following rules:

  • Create Disallowed Pod

  • Create Privileged Pod

  • Create Sensitive Mount Pod

  • Create HostNetwork Pod

  • Pod Created in Kube Namespace

  • Create NodePort Service

  • Create/Modify Configmap With Private Credentials

  • Attach to cluster-admin Role

  • ClusterRole With Wildcard Created

  • ClusterRole With Write Privileges Created

  • ClusterRole With Pod Exec Created

These instructions describe how to redirect from Fluentd to the Sysdig agent service.

  1. Set the webhook backend URL to the IP address of the sysdig-agent service:

    http://$(kubectl get service sysdig-agent -o=jsonpath={.spec.clusterIP} -n sysdig-agent):7765/k8s_audit 
  2. Verify that the webhook backend URL has been set:

    ibmcloud ks cluster master audit-webhook get --cluster <cluster_name_or_ID>
  3. Apply the webhook to your Kubernetes API server by refreshing the cluster master. It may take several minutes for the master to refresh.

    ibmcloud ks cluster master refresh --cluster <cluster_name_or_ID>

Native Kubernetes or Minikube 1.11/1.12

These instructions were verified using Minikube 1.2.0. Other Minikube versions should also work as long as they run Kubernetes versions 1.11/1.12 (which only supports webhook backends). In all cases below, "the Minikube VM" refers to the VM created by Minikube. In cases where you're using --vm-driver=none, this means the local machine.

  1. Create the directory /var/lib/k8s_audit on the master node. (On Minikube, it must be on the Minikube VM.)

  2. Copy the provided audit-policy.yaml file into the directory /var/lib/k8s_audit. (This directory will be mounted into the API server, giving it access to the audit/webhook files. On Minikube, it must be on the Minikube VM.)

  3. Create a Webhook Configuration File and copy it to each Kubernetes API master node, into the directory /var/lib/k8s_audit.

  4. Modify the Kubernetes API server manifest at /etc/kubernetes/manifests/kube-apiserver.yaml, adding the following command-line arguments:

    --audit-log-path=/var/lib/k8s_audit/k8s_audit_events.log
    --audit-policy-file=/var/lib/k8s_audit/audit-policy.yaml
    --audit-log-maxbackup=1
    --audit-log-maxsize=10
    --audit-webhook-config-file=/var/lib/k8s_audit/webhook-config.yaml
    --audit-webhook-batch-max-wait=5s 

    Command-line arguments are provided in the container spec as arguments to the program /usr/local/bin/kube-apiserver. The relevant section of the manifest will look like this:

    spec:
      containers:
      - command:
        - kube-apiserver --allow-privileged=true --anonymous-auth=false
          --audit-log-path=/var/lib/k8s_audit/audit.log
          --audit-policy-file=/var/lib/k8s_audit/audit-policy.yaml
          --audit-log-maxbackup=1
          --audit-log-maxsize=10
          --audit-webhook-config-file=/var/lib/k8s_audit/webhook-config.yaml
          --audit-webhook-batch-max-wait=5s
          ... 
  5. Modify the Kubernetes API server manifest at /etc/kubernetes/manifests/kube-apiserver.yaml to add a mount of /var/lib/k8s_audit into the kube-apiserver container. The relevant sections look like this:

    volumeMounts:
    - mountPath: /var/lib/k8s_audit/
      name: k8s-audit
      readOnly: true
         ...
    volumes:
    - hostPath:
      path: /var/lib/k8s_audit
      type: DirectoryOrCreate
      name: k8s-audit
        ...
  6. Modifying the manifest will cause the Kubernetes API server automatically to restart. Once restarted, it will route Kubernetes audit events to the Sysdig agent's service.

Native Kubernetes or Minikube 1.13+

These instructions were verified using Minikube 1.2.0. Other Minikube versions should also work as long as they run Kubernetes versions 1.13+. When using 1.13, these instructions enable dynamic sinks, but still require changes to the kube-apiserver command-line arguments.

In all cases below, "the Minikube VM" refers to the VM created by Minikube. In cases where you're using --vm-driver=none, this means the local machine.

  1. Create the directory /var/lib/k8s_audit on the master node. (On Minikube, it must be on the Minikube VM.)

  2. Copy the provided audit-policy.yaml file into the directory /var/lib/k8s_audit. (This directory will be mounted into the API server, giving it access to the audit/webhook files. On Minikube, it must be on the Minikube VM.)

  3. Modify the Kubernetes API server manifest at /etc/kubernetes/manifests/kube-apiserver.yaml, adding the following command-line arguments:

    --audit-log-path=/var/lib/k8s_audit/k8s_audit_events.log
    --audit-policy-file=/var/lib/k8s_audit/audit-policy.yaml
    --audit-log-maxbackup=1
    --audit-log-maxsize=10
    --audit-dynamic-configuration
    --feature-gates=DynamicAuditing=true
    --runtime-config=auditregistration.k8s.io/v1alpha1=true

    Command-line arguments are provided in the container spec as arguments to the program /usr/local/bin/kube-apiserver. The relevant section of the manifest will look like this:

    spec:
      containers:
      - command:
        - kube-apiserver --allow-privileged=true --anonymous-auth=false
          --audit-log-path=/var/lib/k8s_audit/audit.log
          --audit-policy-file=/var/lib/k8s_audit/audit-policy.yaml
          --audit-log-maxbackup=1
          --audit-log-maxsize=10
          --audit-dynamic-configuration
          --feature-gates=DynamicAuditing=true
          --runtime-config=auditregistration.k8s.io/v1alpha1=true
          ...
  4. Modify the Kubernetes API server manifest at /etc/kubernetes/manifests/kube-apiserver.yaml to add a mount of /var/lib/k8s_audit into the kube-apiserver container. The relevant sections look like this:

    volumeMounts:
    - mountPath: /var/lib/k8s_audit/
      name: k8s-audit
      readOnly: true
         ...
    volumes:
    - hostPath:
      path: /var/lib/k8s_audit
      type: DirectoryOrCreate
      name: k8s-audit
        ...

    Modifying the manifest will cause the Kubernetes API server automatically to restart.

  5. Create a Dynamic Audit Sink.

    Once the dynamic audit sink is created, it will route Kubernetes audit events to the Sysdig agent's service.

Prepare Webhook or Dynamic Backend

Most of the platform-specific instructions will use one of these methods.

Create a Webhook Configuration File

Sysdig provides a templated resource file that sends audit events to an IP associated with the Sysdig agent service, via port 7765.

It is "templated" in that the actual IP is defined in an environment variable AGENT_SERVICE_CLUSTERIP, which can be plugged in using a program like envsubst.

  1. Download webhook-config.yaml.in.

  2. Run the following to fill in the template file with the ClusterIP IP address associated with the sysdig-agent service you created, either when installing the agent or in the prereq step:

    AGENT_SERVICE_CLUSTERIP=$(kubectl get service sysdig-agent -o=jsonpath={.spec.clusterIP} -n sysdig-agent) envsubst < webhook-config.yaml.in > webhook-config.yaml

    Note: l Athough service domain names like sysdig-agent.sysdig-agent.svc.cluster.local can not be resolved from the Kubernetes API server (they're typically run as pods but not really a part of the cluster), the ClusterIPs associated with those services are routable.

Tip

Using a webhook backend to route audit events is a feature available from Kubernetes v1.11+. See Kubernetes' documentation for background info.

Create a Dynamic Audit Sink

When using dynamic audit sinks, you must create an AuditSink object that directs audit events to the sysdig agent service.

Sysdig provides a template file that can be used to create the sink.

  1. Download audit-sink.yaml.in .

  2. Run the following to fill in the template file with the ClusterIP IP address associated with the sysdig-agent service you created, either when installing the agent or in the prereq step:

    AGENT_SERVICE_CLUSTERIP=$(kubectl get service sysdig-agent -o=jsonpath={.spec.clusterIP} -n sysdig-agent) envsubst < audit-sink.yaml.in > audit-sink.yaml

Tip

Dynamic audit webhooks using AuditSink API objects are available from Kubernetes v1.13+. See Kubernetes' documentation for background info.

Test the Integration

To test that Kubernetes audit events are being properly passed to the agent, you can do any of the following:

  • Enable the All K8s Object Modifications policy and create a deployment, service, configmap, or namespace to see if the events are recorded and forwarded.

  • Enable other policies, such as Suspicious K8s Activity,if and test them.

  • You can use the falco-event-generator Docker image to generate activity that maps to many of the default rules/policies provided in Sysdig Secure. You can run the image via a command line like the following:

    docker run -v $HOME/.kube:/root/.kube -it falcosecurity/falco-event-generator k8s_audit 

    This will create resources in a namespace falco-event-generator.

    See also: Using Falco within Sysdig Secure and the native Falco documentation for more information about this tool.

(BETA) Script to Automate Configuration Changes

As a convenience, Sysdig has created a script: enable-k8s-audit.sh, which performs the necessary steps for enabling audit log support for all Kubernetes distributions described above, except EKS.

You can run it via: bash enable-k8s-audit.sh <distribution> where <distribution> is one of the following:

  • minishift-3.11

  • openshift-3.11

  • openshift-4.2, openshift-4.3

  • gke

  • iks

  • rke-1.13 (implies Kubernetes 1.13)

  • kops

  • minikube-1.13 (implies Kubernetes 1.13)

  • minikube-1.12 (implies Kubernetes 1.11/1.12)

It should be run from the sysdig-cloud-scripts/k8s_audit_config directory.

In some cases, it may prompt for the GCE project ID, IKS cluster name, etc..

For Minikube/Openshift-3.11/Minishift 3.11, it will use ssh/scp to copy files to and run scripts on the API master node. Otherwise, it should be fully automated.

View in the UI

Policies will need to be created to use the new Falco rules for Kubernetes audit logging. For information on creating policies, refer to the Policies documentation.

View Audit Logging Rules

The Kubernetes audit logging rules can be viewed in the Sysdig Policies Rules Editor, found in the Policies module. To view the audit rules:

  1. From the Policies module, navigate to the Rules Editor tab.

  2. Open the drop-down menu for the default rules, and select k8s_audit_rules.yaml:

    374669742.png

View Audit Events

Kubernetes audit events will now be routed to the Sysdig agent daemon set within the cluster.

Once the policies are created, the audit events will be able to be observed via the Sysdig Secure Policy Events module.