Consul
This integration is enabled by default.
Versions supported: > 1.11.1
This integration is out-of-the-box, so it doesn’t require any exporter.
This integration has 65 metrics.
Timeseries generated: 1800 timeseries
List of Alerts
Alert | Description | Format |
---|---|---|
[Consul] KV Store update time anomaly | KV Store update time anomaly | Prometheus |
[Consul] Transaction time anomaly | Transaction time anomaly | Prometheus |
[Consul] Raft transactions count anomaly | Raft transactions count anomaly | Prometheus |
[Consul] Raft commit time anomaly | Raft commit time anomaly | Prometheus |
[Consul] Leader time to contact followers too high | Leader time to contact followers too high | Prometheus |
[Consul] Flapping leadership | Flapping leadership | Prometheus |
[Consul] Too many elections | Too many elections | Prometheus |
[Consul] Server cluster unhealthy | Server cluster unhealthy | Prometheus |
[Consul] Zero failure tolerance | Zero failure tolerance | Prometheus |
[Consul] Client RPC requests anomaly | Consul client RPC requests anomaly | Prometheus |
[Consul] Client RPC requests rate limit exceeded | Consul client RPC requests rate limit exceeded | Prometheus |
[Consul] Client RPC requests failed | Consul client RPC requests failed | Prometheus |
[Consul] License Expiry | Consul License Expiry | Prometheus |
[Consul] Garbage Collection pause high | Consul Garbage Collection pause high | Prometheus |
[Consul] Garbage Collection pause too high | Consul Garbage Collection pause too high | Prometheus |
[Consul] Raft restore duration too high | Consul Raft restore duration too high | Prometheus |
[Consul] RPC requests error rate is high | Consul RPC requests error rate is high | Prometheus |
[Consul] Cache hit rate is low | Consul Cache hit rate is low | Prometheus |
[Consul] High 4xx RequestError Rate | High 4xx RequestError Rate | Prometheus |
[Consul] High Request Latency | Envoy High Request Latency | Prometheus |
[Consul] High Response Latency | Envoy High Response Latency | Prometheus |
[Consul] Certificate close to expire | Certificate close to expire | Prometheus |
List of Dashboards
Consul
The dashboard provides information on the status and latency of Consul.
Consul Envoy
The dashboard provides information on the Consul Envoy proxies.
List of Metrics
Metric name |
---|
consul_autopilot_failure_tolerance |
consul_autopilot_healthy |
consul_client_rpc |
consul_client_rpc_exceeded |
consul_client_rpc_failed |
consul_consul_cache_bypass |
consul_consul_cache_entries_count |
consul_consul_cache_evict_expired |
consul_consul_cache_fetch_error |
consul_consul_cache_fetch_success |
consul_kvs_apply_sum |
consul_raft_apply |
consul_raft_commitTime_sum |
consul_raft_fsm_lastRestoreDuration |
consul_raft_leader_lastContact |
consul_raft_leader_oldestLogAge |
consul_raft_rpc_installSnapshot |
consul_raft_state_candidate |
consul_raft_state_leader |
consul_rpc_cross_dc |
consul_rpc_queries_blocking |
consul_rpc_query |
consul_rpc_request |
consul_rpc_request_error |
consul_runtime_gc_pause_ns |
consul_runtime_gc_pause_ns_sum |
consul_system_licenseExpiration |
consul_txn_apply_sum |
envoy_cluster_membership_change |
envoy_cluster_membership_healthy |
envoy_cluster_membership_total |
envoy_cluster_upstream_cx_active |
envoy_cluster_upstream_cx_connect_ms_bucket |
envoy_cluster_upstream_rq_active |
envoy_cluster_upstream_rq_pending_active |
envoy_cluster_upstream_rq_time_bucket |
envoy_cluster_upstream_rq_xx |
envoy_server_days_until_first_cert_expiring |
go_build_info |
go_gc_duration_seconds |
go_gc_duration_seconds_count |
go_gc_duration_seconds_sum |
go_goroutines |
go_info |
go_memstats_buck_hash_sys_bytes |
go_memstats_gc_sys_bytes |
go_memstats_heap_alloc_bytes |
go_memstats_heap_idle_bytes |
go_memstats_heap_inuse_bytes |
go_memstats_heap_released_bytes |
go_memstats_heap_sys_bytes |
go_memstats_lookups_total |
go_memstats_mallocs_total |
go_memstats_mcache_inuse_bytes |
go_memstats_mcache_sys_bytes |
go_memstats_mspan_inuse_bytes |
go_memstats_mspan_sys_bytes |
go_memstats_next_gc_bytes |
go_memstats_stack_inuse_bytes |
go_memstats_stack_sys_bytes |
go_memstats_sys_bytes |
go_threads |
process_cpu_seconds_total |
process_max_fds |
process_open_fds |
Prerequisites
Enable Prometheus Metrics and Disable Hostname in Metrics
As seen in Consul documentation pages Helm Global Metrics and Prometheus Retention Time, to make Consul expose an endpoint for scraping metrics, you need to enable a few global.metrics configurations. You also need to enable the telemetry.disable_hostname “extra configurations” in the Consul Server and Client, so the metrics don’t contain the name of the instances.
If you install Consul with Helm, you need to use the following flags:
--set 'global.metrics.enabled=true'
--set 'global.metrics.enableAgentMetrics=true'
--set 'server.extraConfig="{"telemetry": {"disable_hostname": true}}"'
--set 'client.extraConfig="{"telemetry": {"disable_hostname": true}}"'
Installation
Installing an exporter is not required for this integration.
Related Blog Posts
Agent Configuration
The default agent jobs for this integration are as follows:
- job_name: 'consul-server-default'
metrics_path: '/v1/agent/metrics'
params:
format: ['prometheus']
tls_config:
insecure_skip_verify: true
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_pod_host_ip]
regex: __HOSTIPS__
- action: keep
source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: true
- action: drop
source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_omit]
regex: true
- source_labels: [__meta_kubernetes_pod_phase]
action: keep
regex: Running
- action: replace
source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
target_label: __scheme__
regex: (https?)
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
- __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
regex: (consul);(.{0}$)
replacement: consul
target_label: __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
- action: keep
source_labels:
- __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
regex: "consul"
- action: keep
source_labels: [__address__]
regex: (.*:8500)
- action: replace
source_labels: [__meta_kubernetes_pod_uid]
target_label: sysdig_k8s_pod_uid
- action: replace
source_labels: [__meta_kubernetes_pod_container_name]
target_label: sysdig_k8s_pod_container_name
- job_name: 'consul-envoy-default'
tls_config:
insecure_skip_verify: true
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
source_labels: [__meta_kubernetes_pod_host_ip]
regex: __HOSTIPS__
- action: keep
source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
regex: true
- action: drop
source_labels: [__meta_kubernetes_pod_annotation_promcat_sysdig_com_omit]
regex: true
- source_labels: [__meta_kubernetes_pod_phase]
action: keep
regex: Running
- action: replace
source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
target_label: __scheme__
regex: (https?)
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
- __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
regex: (envoy-sidecar);(.{0}$)
replacement: consul
target_label: __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
- action: keep
source_labels:
- __meta_kubernetes_pod_annotation_promcat_sysdig_com_integration_type
regex: "consul"
- action: replace
source_labels: [__address__]
regex: (.+?)(\\:\\d)?
replacement: $1:20200
target_label: __address__
- action: replace
source_labels: [__meta_kubernetes_pod_uid]
target_label: sysdig_k8s_pod_uid
- action: replace
source_labels: [__meta_kubernetes_pod_container_name]
target_label: sysdig_k8s_pod_container_name
metric_relabel_configs:
- source_labels: [__name__]
regex: (envoy_cluster_upstream_cx_active|envoy_cluster_upstream_rq_active|envoy_cluster_upstream_rq_pending_active|envoy_cluster_membership_total|envoy_cluster_membership_healthy|envoy_cluster_membership_change|envoy_cluster_upstream_rq_xx|envoy_cluster_upstream_cx_connect_ms_bucket|envoy_server_days_until_first_cert_expiring|envoy_cluster_upstream_rq_time_bucket)
action: keep
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.