View Agent Health

The Sysdig Agent uses ReadinessProbe to determine its readiness to accept incoming requests. Additionally, the agent can generate internal health metrics through a Prometheus exporter.

Readiness Probe for Sysdig Agent

Kubernetes uses readiness or liveness probes to determine the readiness or live status of a pod. Similar functionality is present in Sysdig Agent versions preceding v12.17.0, where the readiness state of the agent pod can be determined by checking for the existence of a file named running in the /opt/draios/logs directory. The presence of the running file indicates that the agent is connected both to the Sysdig backend and the Kubernetes API server.

In Sysdig Agent version 12.17.0, an HTTP port can be employed to query its health status, aligning with the convention followed by typical Kubernetes services. Beyond assessing backend and API server connectivity, this health status considers the stability and running status of all its sub-processes. In version 12.17.0, the health service defaults to listening on port 24483 across all host interface addresses.

Starting from agent version 12.19.0, the default behavior was modified to have the health service exclusively listen on the localhost (127.0.0.1) address.

To adjust the host address to listen to, specify the the following configuration in the dragent.yaml file.

status_host: 127.0.0.1

To change the port to listen to, add the following:

status_port: 24483

Starting from Helm chart v1.18.1, the new HTTP-based readiness endpoint is enabled by default.

If you are not using our Helm charts to install the agent, you can configure the readiness probe manually by adding the following configuration to the daemonset specification:

readinessProbe:
    failureThreshold: 3
    httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 24483
    initialDelaySeconds: 90
    periodSeconds: 5
    timeoutSeconds: 2

The initial delay can be customized depending on the typical time it takes for the agents to become ready in your cluster. This duration typically varies based on cluster size and the number of entities that the agents need to download initially to obtain the complete state of the cluster.

Collect Agent Health Metrics

You can export agent metrics using the Sysdig Agent Prometheus exporter. These metrics serve as a valuable tool for troubleshooting and retrieving status information from the agent, especially in scenarios where it encounters difficulties connecting to the backend.

Enable the Agent Prometheus Exporter

To enable the exporter using our Helm chart, set the following in your values.yaml:

agent:
  sysdig:
    settings:
      prometheus_exporter:
        enabled: true
        export_health_metrics: true

To enable the exporter, add the following to the dragent.yaml:

prometheus_exporter:
    enabled: true

By default the prometheus exporter will listen on port 9544 by using the standard path, /metrics.

To change the port or IP address to listen to, you can use the listen_url configuration parameter. For example:

prometheus_exporter:
		listen_url: 127.0.0.1:9544

Enable Authentication

To enable basic authentication in Prometheus Exporter, add the following to your values.yaml file:

agent:
  sysdig:
    settings:
      prometheus_exporter:
        enabled: true
        export_health_metrics: true
        basic_auth_users:
            <promuser>: <bcrypt-password-hash>

Replace and with the the credentials required to connect to Prometheus. Passwords must be hashed with bcrypt.

Enable TLS Authentication

To enable TLS authentication in Prometheus Exporter, add the following to your values.yaml file:


agent:
  sysdig:
    settings:
      prometheus_exporter:
        enabled: true
        export_health_metrics: true
        basic_auth_users:
            <promuser>: <bcrypt-password-hash>
        tls_server_config:
            cert_file: /opt/draios/etc/kubernetes/certs/example.com/example.com.crt
            key_file: /opt/draios/etc/kubernetes/certs/example.com/example.com.key

            client_ca_file: /opt/draios/etc/kubernetes/certs/client.com/ca.pem
            client_auth_type: RequireAndVerifyClientCert

Replace and with the the credentials required to connect to Prometheus. Passwords must be hashed with bcrypt.

For more information on basic_auth_users and tls_server_config, see web configuration.

Agent Health Metrics

Once enabled, the agent’s Prometheus exporter will automatically export the following metrics:

container_cpu_used_percent
container_file_time_in
container_file_time_other
container_file_time_out
container_memory_bytes_used
container_memory_swap_bytes_used
host_cpu_idle_percent
host_cpu_iowait_percent
host_cpu_nice_percent
host_cpu_stolen_percent
host_cpu_system_percent
host_cpu_used_percent
host_cpu_user_percent
host_file_time_in
host_file_time_other
host_file_time_out
host_memory_bytes_available
host_memory_bytes_total
host_memory_bytes_used
host_memory_bytes_virtual
host_memory_swap_bytes_available
host_memory_swap_bytes_total
host_memory_swap_bytes_used
promhttp_metric_handler_requests_in_flight
promhttp_metric_handler_requests_total
sysdig_sampling_ratio
sysdig_up

As of agent v12.19.0, you can export additional health metrics with:

prometheus_exporter:
    enabled: true
    export_health_metrics: true

You need to add the prometheus.io/scrape: "true" and prometheus.io/port: 9544 annotations to the agent daemonSet to allow the Agent’s Prometheus native service discovery to find the endpoint.

The additional metrics you can retrieve are:

sysdig_agent_host_info: 1.0
    host_hostname
    cluster_name
    agent_version
    agent_mode

sysdig_agent_healthy: 1/0


sysdig_agent_connected: 1/0

sysdig_agent_connection_error_code: [0-5]
    agent_error
    backend_error_code
    backend_error_msg

sysdig_agent_unlicensed: 1/0

sysdig_agent_analyzer_num_evts: <number>

sysdig_agent_analyzer_dropped_evts: <number>

host_uname: 1.0
    machine_type
    kernel_name
    kernel_release
    kernel_version

sysdig_agent_process_memory_kb: <memory used in kilo bytes >
    agent_process

sysdig_agent_feature_enabled: <Use the agent_feature label to retrieve the list of features enabled>

sysdig_agent_process_uptime_s: <Use the agent_process label to retrieve the  process uptime in seconds> 
sysdig_agent_analyzer_num_evts
sysdig_agent_analyzer_dropped_evts

where

sysdig_agent_host_info always return a value of 1.

sysdig_agent_healthy returns either 1 or 0. 1 indicates that it is healthy.

sysdig_agent_connected returns either 1 or 0. 1 indicates that it is connected.

sysdig_agent_unlicensed returns either 1 or 0. 1 indicates that it is unlicensed.

sysdig_agent_feature_enabled returns either 1 or 0. 1 indicates that the feature is enabled.

For example:

sysdig_agent_feature_enabled{agent_feature="cointerface"} 1
sysdig_agent_feature_enabled{agent_feature="jmx"} 0

[0-5] represents error codes returned by sysdig_agent_connection_error_code :

0 - No Error
1 - I/O Exception
2 - Connection timed out
3 - Invalid argument
4 - Handshake error
5 - Invalid message received
6 - Error received from collector or backend