Rancher RKE2 Etcd

Metrics, Dashboards, Alerts and more for Rancher RKE2 Etcd Integration in Sysdig Monitor.
Rancher RKE2 Etcd

This integration is disabled by default. See Enable and Disable Integrations to enable it in your account.

This integration is out-of-the-box, so it doesn’t require any exporter.

This integration has 52 metrics.

Timeseries generated: 1.5K TS

List of Alerts

AlertDescriptionFormat
[RKE2-Etcd] Etcd Members DownThere are members down.Prometheus
[RKE2-Etcd] Etcd Insufficient MembersEtcd cluster has insufficient membersPrometheus
[RKE2-Etcd] Etcd No LeaderMember has no leader.Prometheus
[RKE2-Etcd] Etcd High Number Of Leader ChangesLeader changes within the last 15 minutes.Prometheus
[RKE2-Etcd] Etcd High Number Of Failed GRPC RequestsHigh number of failed grpc requestsPrometheus
[RKE2-Etcd] Etcd GRPC Requests SlowgRPC requests are taking too much timePrometheus
[RKE2-Etcd] Etcd High Number Of Failed ProposalsHigh number of proposal failures within the last 30 minutes on etcd instancePrometheus
[RKE2-Etcd] Etcd High Fsync Durations99th percentile fync durations are too highPrometheus
[RKE2-Etcd] Etcd High Commit Durations99th percentile commit durations are too highPrometheus

List of Dashboards

Rancher RKE2 Etcd

If you are using Prometheus Remote Write you will need to add the following metric relabel config for this label.


    - action: replace
    source_labels: [ __address__ ]
    target_label: _sysdig_integration_rke2_etcd 
    replacement: true

The dashboard provides information on the RKE2 Etcd. Rancher RKE2 Etcd

List of Metrics

Metric name
etcd_debugging_mvcc_db_total_size_in_bytes
etcd_disk_backend_commit_duration_seconds_bucket
etcd_disk_wal_fsync_duration_seconds_bucket
etcd_grpc_proxy_cache_hits_total
etcd_grpc_proxy_cache_misses_total
etcd_mvcc_db_total_size_in_bytes
etcd_network_client_grpc_received_bytes_total
etcd_network_client_grpc_sent_bytes_total
etcd_network_peer_received_bytes_total
etcd_network_peer_received_failures_total
etcd_network_peer_round_trip_time_seconds_bucket
etcd_network_peer_sent_bytes_total
etcd_network_peer_sent_failures_total
etcd_server_has_leader
etcd_server_id
etcd_server_leader_changes_seen_total
etcd_server_proposals_applied_total
etcd_server_proposals_committed_total
etcd_server_proposals_failed_total
etcd_server_proposals_pending
go_build_info
go_gc_duration_seconds
go_gc_duration_seconds_count
go_gc_duration_seconds_sum
go_goroutines
go_info
go_memstats_buck_hash_sys_bytes
go_memstats_gc_sys_bytes
go_memstats_heap_alloc_bytes
go_memstats_heap_idle_bytes
go_memstats_heap_inuse_bytes
go_memstats_heap_released_bytes
go_memstats_heap_sys_bytes
go_memstats_lookups_total
go_memstats_mallocs_total
go_memstats_mcache_inuse_bytes
go_memstats_mcache_sys_bytes
go_memstats_mspan_inuse_bytes
go_memstats_mspan_sys_bytes
go_memstats_next_gc_bytes
go_memstats_stack_inuse_bytes
go_memstats_stack_sys_bytes
go_memstats_sys_bytes
go_threads
grpc_server_handled_total
grpc_server_handling_seconds_bucket
grpc_server_started_total
process_cpu_seconds_total
process_max_fds
process_open_fds
sysdig_container_cpu_cores_used
sysdig_container_memory_used_bytes

Prerequisites

None.

Installation

Installing an exporter is not required for this integration.

Agent Configuration

The default agent job for this integration is as follows:

- job_name: rancher-rke2-etcd-default
  scheme: https
  tls_config:
    insecure_skip_verify: true
    cert_file: /host/var/lib/rancher/rke2/server/tls/etcd/client.crt
    key_file: /host/var/lib/rancher/rke2/server/tls/etcd/client.key
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: keep
    source_labels: [__meta_kubernetes_pod_host_ip]
    regex: __HOSTIPS__
  - source_labels: [__meta_kubernetes_pod_phase]
    action: keep
    regex: Running
  - action: keep
    source_labels:
    - __meta_kubernetes_namespace
    - __meta_kubernetes_pod_name
    separator: '/'
    regex: 'kube-system/etcd-.+'
  - source_labels:
    - __address__
    action: replace
    target_label: __address__
    regex: (.+?)(\\:\\d)?
    replacement: $1:2379
    # Holding on to pod-id and container name so we can associate the metrics
    # with the container (and cluster hierarchy)
  - action: replace
    source_labels: [__meta_kubernetes_pod_uid]
    target_label: sysdig_k8s_pod_uid
  - action: replace
    source_labels: [__meta_kubernetes_pod_container_name]
    target_label: sysdig_k8s_pod_container_name
  - action: replace
    source_labels: [ __address__ ]
    target_label: _sysdig_integration_rke2_etcd 
    replacement: true