HAProxy Ingress OpenShift

Metrics, Dashboards, Alerts and more for HAProxy Ingress OpenShift Integration in Sysdig Monitor.
HAProxy Ingress OpenShift

This integration is enabled by default.

Versions supported: > v3.11

This integration is out-of-the-box, so it doesn’t require any exporter.

This integration has 26 metrics.

Timeseries generated: The HAProxy ingress router generates ~400 time series per HAProxy router pod.

List of Alerts

AlertDescriptionFormat
[OpenShift-HAProxy-Router] Router DownRouter HAProxy down. No instances running.Prometheus
[OpenShift-HAProxy-Router] Percentage of routers lowLess than 75% Routers are upPrometheus
[OpenShift-HAProxy-Router] Route DownThis alert detects if all servers are down in a routePrometheus
[OpenShift-HAProxy-Router] High LatencyThis alert detects high latency in at least one server of the routePrometheus
[OpenShift-HAProxy-Router] Pod Health Check FailureThis alert triggers when there is a recurrent pod health check failure.Prometheus
[OpenShift-HAProxy-Router] Queue not empty in routeThis alert triggers when a queue is not empty in a routePrometheus
[OpenShift-HAProxy-Router] High error rate in routeThis alert triggers when the error rate in a route is higher than 15%.Prometheus
[OpenShift-HAProxy-Router] Connection errors in routeThis alert triggers when there are recurring connection errors in a routePrometheus

List of Dashboards

OpenShift HAProxy Ingress Overview

The dashboard provides information on the OpenShift HAProxy Ingress overview. OpenShift HAProxy Ingress Overview

OpenShift HAProxy Ingress Service Details

The dashboard provides information on the OpenShift HAProxy Ingress Service golden signals. OpenShift HAProxy Ingress Service Details

List of Metrics

Metric name
haproxy_backend_http_average_connect_latency_milliseconds
haproxy_backend_http_average_queue_latency_milliseconds
haproxy_backend_http_average_response_latency_milliseconds
haproxy_backend_up
haproxy_frontend_bytes_in_total
haproxy_frontend_bytes_out_total
haproxy_frontend_connections_total
haproxy_frontend_current_session_rate
haproxy_frontend_http_responses_total
haproxy_process_cpu_seconds_total
haproxy_process_max_fds
haproxy_process_resident_memory_bytes
haproxy_process_start_time_seconds
haproxy_process_virtual_memory_bytes
haproxy_server_bytes_in_total
haproxy_server_bytes_out_total
haproxy_server_check_failures_total
haproxy_server_connection_errors_total
haproxy_server_connections_total
haproxy_server_current_queue
haproxy_server_current_sessions
haproxy_server_downtime_seconds_total
haproxy_server_http_average_response_latency_milliseconds
haproxy_server_http_responses_total
haproxy_server_up
kube_workload_status_desired

Preparing the Integration

Openshift 3.11

Once the Sysdig agent is deployed, check if it is running on all nodes (compute, master, and infra):

oc get nodes
oc get pods -n sysdig-agent -o wide

Apply this patch in case the Agent is not running on infra/master.

oc patch namespace sysdig-agent --patch-file='sysdig-agent-namespace-patch.yaml'

sysdig-agent-namespace-patch.yaml file

apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/node-selector: ""

OpenShift integrates security by default. Therefore, if you want Sysdig agent to scrape HAProxy router metrics, provide it with the necessary permissions. To do so:

oc apply -f router-clusterrolebinding-sysdig-agent-oc3.yaml

router-clusterrolebinding-sysdig-agent-oc3.yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: haproxy-route-monitoring
rules:
- apiGroups:
  - route.openshift.io
  resources:
  - routers/metrics
  verbs:
  - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: sysdig-agent
  name: sysdig-router-monitoring
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: haproxy-route-monitoring
subjects:
- kind: ServiceAccount
  name: sysdig-agent
  namespace: sysdig-agent   # Remember to change to the namespace where you have the Sysdig agents deployed

Openshift 4.X

OpenShift integrates security by default. Therefore, if you want Sysdig agent to scrape HAProxy router metrics, provide it with the necessary permissions. To do so:

oc apply -f router-clusterrolebinding-sysdig-agent-oc4.yaml

router-clusterrolebinding-sysdig-agent-oc4.yaml file

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: router-monitoring-sysdig-agent
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: router-monitoring
subjects:
- kind: ServiceAccount
  name: sysdig-agent
  namespace: sysdig-agent   # Remember to change to the namespace where you have the Sysdig agents deployed

Installing

The installation of an exporter is not required for this integration.

Agent Configuration

This is the default agent job for this integration:

- job_name: 'haproxy-router'
  scheme: https
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_pod_host_ip]
    regex: __HOSTIPS__
  - action: replace
    source_labels: [__address__]
    regex: ([^:]+)(?::\d+)?
    replacement: $1:1936
    target_label: __address__
  - action: drop
    source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_omit]
    regex: true
  - action: replace
    source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    target_label: __scheme__
    regex: (https?)
  - action: replace
    source_labels:
    - __meta_kubernetes_pod_container_name
    - __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
    regex: (router);(.{0}$)
    replacement: openshift-haproxy
    target_label: __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
  - action: keep
    source_labels:
    - __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
    regex: "openshift-haproxy"
  - action: replace
    source_labels: [__meta_kubernetes_pod_uid]
    target_label: sysdig_k8s_pod_uid
  - action: replace
    source_labels: [__meta_kubernetes_pod_container_name]
    target_label: sysdig_k8s_pod_container_name
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: (haproxy_backend_http_average_connect_latency_milliseconds|haproxy_backend_http_average_queue_latency_milliseconds|haproxy_backend_http_average_response_latency_milliseconds|haproxy_backend_up|haproxy_frontend_bytes_in_total|haproxy_frontend_bytes_out_total|haproxy_frontend_connections_total|haproxy_frontend_current_session_rate|haproxy_frontend_http_responses_total|haproxy_process_cpu_seconds_total|haproxy_process_max_fds|haproxy_process_resident_memory_bytes|haproxy_process_start_time_seconds|haproxy_process_virtual_memory_bytes|haproxy_server_bytes_in_total|haproxy_server_bytes_out_total|haproxy_server_check_failures_total|haproxy_server_connection_errors_total|haproxy_server_connections_total|haproxy_server_current_queue|haproxy_server_current_sessions|haproxy_server_downtime_seconds_total|haproxy_server_http_average_response_latency_milliseconds|haproxy_server_http_responses_total|haproxy_server_up)
    action: keep