RabbitMQ

Metrics, Dashboards, Alerts and more for RabbitMQ Integration in Sysdig Monitor.
RabbitMQ

This integration is enabled by default.

Versions supported: > v3.8

This integration is out-of-the-box, so it doesn’t require any exporter.

This integration has 41 metrics.

Timeseries generated: 730 series per instance

List of Alerts

AlertDescriptionFormat
[RabbitMQ] Cluster Operator Unavailable ReplicasThere are kube_pod_names that are either running but not yet available or kube_pod_names that still have not been created.Prometheus
[RabbitMQ] Insufficient Established Erlang Distribution LinksInsuffient establised erland distribution linksPrometheus
[RabbitMQ] Low Disk Watermark PredictedThe predicted free disk space in 24 hours from now is lowPrometheus
[RabbitMQ] High Connection ChurnThere are a high connection churnPrometheus
[RabbitMQ] No MajorityOfNodesReadyThere are so many nodes not readyPrometheus
[RabbitMQ] Persistent Volume MissingThere is at least one pvc not boundPrometheus
[RabbitMQ] Unroutable MessagesThere were unroutable message within the last 5 minutes in RabbitMQ clusterPrometheus
[RabbitMQ] File Descriptors Near LimitThe file descriptors are near to the limitPrometheus
[RabbitMQ] Container RestartsOver the last 10 minutes a rabbitmq container was restartedPrometheus
[RabbitMQ] TCP Sockets Near LimitThe TCP sockets are near to the limitPrometheus

List of Dashboards

Rabbitmq Usage

The dashboard provides information on the usage of each queues, channels and connections. Rabbitmq Usage

Rabbitmq Overview

The dashboard provides information on the nodes, queues and erlang of a RabbitMQ cluster. Rabbitmq Overview

List of Metrics

Metric name
erlang_vm_dist_node_state
kube_deployment_status_replicas_unavailable
kube_kube_pod_name_container_status_restarts_total
kube_persistentvolumeclaim_status_phase
kube_statefulset_replicas
kube_statefulset_status_replicas_ready
rabbitmq_build_info
rabbitmq_channel_consumers
rabbitmq_channel_get_ack_total
rabbitmq_channel_get_empty_total
rabbitmq_channel_get_total
rabbitmq_channel_messages_acked_total
rabbitmq_channel_messages_confirmed_total
rabbitmq_channel_messages_delivered_ack_total
rabbitmq_channel_messages_delivered_total
rabbitmq_channel_messages_published_total
rabbitmq_channel_messages_redelivered_total
rabbitmq_channel_messages_unconfirmed
rabbitmq_channel_messages_unroutable_dropped_total
rabbitmq_channel_messages_unroutable_returned_total
rabbitmq_channels
rabbitmq_channels_closed_total
rabbitmq_channels_opened_total
rabbitmq_connections
rabbitmq_connections_closed_total
rabbitmq_connections_opened_total
rabbitmq_disk_space_available_bytes
rabbitmq_disk_space_available_limit_bytes
rabbitmq_process_max_fds
rabbitmq_process_max_tcp_sockets
rabbitmq_process_open_fds
rabbitmq_process_open_tcp_sockets
rabbitmq_process_resident_memory_bytes
rabbitmq_queue_messages_published_total
rabbitmq_queue_messages_ready
rabbitmq_queue_messages_unacked
rabbitmq_queues
rabbitmq_queues_created_total
rabbitmq_queues_declared_total
rabbitmq_queues_deleted_total
rabbitmq_resident_memory_limit_bytes

Preparing the Integration

Enable Prometheus Metrics

Rabbitmq instruments Prometheus metrics and annotates the metrics API pod with Prometheus annotations.

Make sure that Prometheus metrics are activated. If they are not, activate the plugin using this command inside of the rabbitmq container:

rabbitmq-plugins enable rabbitmq_prometheus

Installing

The installation of an exporter is not required for this integration.

Monitoring and Troubleshooting RabbitMQ

This document describes important metrics and queries that you can use to monitor and troubleshoot RabbitMQ.

Tracking metrics status

You can track RabbitMQ metrics status with following alerts: Exporter proccess is not serving metrics

# [RabbitMQ] Exporter Process Down
absent(rabbitmq_queue_messages_published_total{kube_cluster_name=~$cluster,kube_namespace_name=~$namespace,kube_workload_name=~$workload}) > 0

Exporter proccess is not serving metrics

# [RabbitMQ] Exporter Process Down
absent(rabbitmq_channels{kube_cluster_name=~$cluster,kube_namespace_name=~$namespace,kube_workload_name=~$workload}) > 0

Agent Configuration

This is the default agent job for this integration:

- job_name: rabbitmq-default
  tls_config:
    insecure_skip_verify: true
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_pod_host_ip]
    regex: __HOSTIPS__
  - action: replace
    source_labels:
    - __meta_kubernetes_pod_container_name
    - __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
    regex: (rabbitmq);(.{0}$)
    replacement: $1
    target_label: __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
  - action: keep
    source_labels:
    - __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
    regex: "rabbitmq"
  - action: replace
    source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
    target_label: __scheme__
    regex: (https?)
  - action: replace
    source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: replace
    source_labels: [__meta_kubernetes_pod_uid]
    target_label: sysdig_k8s_pod_uid
  - action: replace
    source_labels: [__meta_kubernetes_pod_container_name]
    target_label: sysdig_k8s_pod_container_name
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: (erlang_vm_dist_node_state|rabbitmq_build_info|rabbitmq_channel_consumers|rabbitmq_channel_get_ack_total|rabbitmq_channel_get_empty_total|rabbitmq_channel_get_total|rabbitmq_channel_messages_acked_total|rabbitmq_channel_messages_confirmed_total|rabbitmq_channel_messages_delivered_ack_total|rabbitmq_channel_messages_delivered_total|rabbitmq_channel_messages_published_total|rabbitmq_channel_messages_redelivered_total|rabbitmq_channel_messages_unconfirmed|rabbitmq_channel_messages_unroutable_dropped_total|rabbitmq_channel_messages_unroutable_returned_total|rabbitmq_channels|rabbitmq_channels_closed_total|rabbitmq_channels_opened_total|rabbitmq_connections|rabbitmq_connections_closed_total|rabbitmq_connections_opened_total|rabbitmq_disk_space_available_bytes|rabbitmq_disk_space_available_limit_bytes|rabbitmq_process_max_fds|rabbitmq_process_max_tcp_sockets|rabbitmq_process_open_fds|rabbitmq_process_open_tcp_sockets|rabbitmq_process_resident_memory_bytes|rabbitmq_queue_messages_published_total|rabbitmq_queue_messages_ready|rabbitmq_queue_messages_unacked|rabbitmq_queues|rabbitmq_queues_created_total|rabbitmq_queues_declared_total|rabbitmq_queues_deleted_total|rabbitmq_resident_memory_limit_bytes)
    action: keep