Troubleshoot Cluster Shield

This page provides troubleshooting guidance for Sysdig Cluster Shield.

For an overview of Cluster Shield and installation instructions, see Install Shield on Kubernetes.

API Slowness in EKS

Overview

If you experience slowness with the Kubernetes API server and you have Cluster Shield’s Audit feature enabled, the issue could be connectivity problems between the Kubernetes control plane and the audit webhook endpoint.

Every API call to the Kubernetes API server must be validated by admission controllers before processing. These controllers enforce security and compliance checks on incoming requests. If the admission controllers are unreachable (often due to networking issues preventing the Kubernetes control plane from reaching the webhook endpoints) the API server waits while attempting to contact them. This leads to retries and timeouts, significantly slowing down API response times.

To prevent this, ensure that the admission controllers are accessible and properly connected.

Solution

To ensure proper connectivity and prevent slowness, do the following:

Allow API Server Connectivity to Pods:
For Cluster Shield’s Audit feature to function efficiently, the Kubernetes API server must be able to connect to the webhook endpoint. In Kubernetes clusters network setup, direct communication from the control plane to certain ports of the data plane might be blocked. Ensure the necessary ports are open and accessible:
- Audit uses the port 6443 by default.
- Admission Control uses the port 8443 by default.
To customize these ports, see Customize Ports.
Update Security Group Rules:
- Update the inbound rules for the security group associated with your EKS worker nodes so the API server can reach the audit webhook endpoint without unnecessary delays.
- Protocol: TCP
- Port: The port you specified for Cluster Shield Audit. 6443 is the default port used by Cluster Shield Audit.
- Source: The EKS cluster’s control plane security group

If you have also enabled Cluster Shield’s admission control feature, allow TCP traffic on the admission controller port. Default: 8443.

Customize Ports

At times, you may need to change the default ports for Cluster Shield’s Audit and Admission Controller. For example, when the API server cannot reach the webhook endpoint.

To resolve this issue, when installing Cluster Shield via Helm, apply the following configurations:

features:
  admission_control:
    enabled: true
    http_port: 6000 # Or any other open and unused port > 1024
  detections:
    kubernetes_audit:
      enabled: true
      http_port: 5000 # Or any other open and unused port > 1024

Ensure that you update the inbound firewall rules to allow communication between the control plane and the data plane.

For Amazon EKS clusters, update the inbound rules in the security group associated with the EKS worker nodes. This allows TCP communication on the required ports from the EKS cluster security group. In the above example, ports 5000 and 6000 are opened for TCP traffic.

Race condition between Calico CNI and Sysdig Webhooks

When Calico is configured as the Container Network Interface (CNI), the Kubernetes API server may occasionally fail to connect to the Sysdig webhook (Audit or Admission Controller) during Calico initialization.

Sequence of Events

Calico pods are redeployed.
Before the overlay network is fully initialized, the API server attempts to reach the Sysdig webhook endpoint.
The request times out because the network is not ready.
If the value of timeoutSeconds in ValidatingWebhookConfiguration is greater than Calico’s retry window, the cluster network enters an unrecoverable state.

This race occurs only when Calico is running in overlay mode (encapsulation: VXLAN, CALICO_IPV4POOL_VXLAN=Always, or similar). Because the some of Kubernetes flavours control‑plane does not participate in that overlay, the API server cannot route to pod IPs until Calico is fully ready. Either run the webhook with hostNetwork: true, or ensure the webhook’s timeoutSeconds is shorter than Calico’s retry interval.

Mitigation Options

Enable host networking

cluster:
  host_network: true #Ensures webhook components are available even if the CNI is not yet initialized.

Shorten the webhook timeout

features:
  admission_control:
    enabled: true
    timeout: 1 #Default is 10 seconds
  detections:
    kubernetes_audit:
      enabled: true
      timeout: 1 #Default is 10 seconds

Failure Creating/Updating Control Plane Nodes

Overview

Some Kubernetes distributions started using Cluster API and allow kubeadm to talk directly with the local node API, making it unnecessary to wait for the CNI to come up before provisioning the node. Consequently, this means that the resources start getting created and reach out to the ValidatingWebhook for approval. Our ValidatingWebhook is implemented as a Deployment, and it doesn’t need to be on every node, or consume all those resources. This means the CNI needs to be available for a new node to contact it. The ValidatingWebhook should be configured to ignore failures, but due to concurrent timeouts, in some cases that might make the node provisioning fail.

See the related KB article for vSphere Kubernetes here.

Solution 1 - Temporarily Disable the ValidatingWebhook

Before provisioning new Control plane nodes (which includes upgrading them), you can temporarily disable the ValidatingWebhook and re-enable it afterward.

Set the features.detections.kubernetes_audit.enabled to false

features:
  detections:
    kubernetes_audit:
      enabled: false

Provision the new node and wait for it to be ready
Revert the change made at point 1: set features.detections.kubernetes_audit.enabled to true
```
features:
  detections:
    kubernetes_audit:
      enabled: true
```

This is a temporary solution, that must be executed every time you provision new nodes that are part of the Control plane. It creates a complete blind spot on detections based on Kubernetes actions, but only for a short time.

Solution 2 - Exclude Sensitive Namespaces from the ValidatingWebhook

You can tune the ValidatingWebhook to be bypassed for components required for node provisioning. This can be done either by ignoring namespaces, or by refining the ValidatingWebhook rule. The resources to bypass and how to bypass them depend on each single distribution. To understand it, you need to inspect the logs and see what resources fail to be provisioned.

To ignore namespaces, use features.detections.kubernetes_audit.excluded_namespaces attribute in the Shield chart and add those you want to exclude. For instance, with kube-system:

features:
  detections:
    kubernetes_audit:
      excluded_namespaces:
        - kube-system

You can also customize the ValidatingWebhook rule. For instance, to exclude Cluster-scoped resources:

features:
  detections:
    kubernetes_audit:
      webhook_rules:  # +doc-gen:break
        - apiGroups:
            - ""
            - apps
            - autoscaling
            - batch
            - networking.k8s.io
            - rbac.authorization.k8s.io
            - extensions
          apiVersions:
            - '*'
          operations:
            - '*'
          resources:
            - '*/*'
          scope: Namespaced

This is a longer-term solution, but it requires more effort and tuning to be set up. It’s also creating a persistent blind-spot, even if narrower.