Change Alerts

Change Alerts trigger when a metric value substantially deviates compared to historical values.

Change Alerts let you receive alerts when your metrics considerably change over time. Instead of setting a fixed threshold for when a metric goes above or below a certain value, you can use Change Alerts to be notified when the metric changes by a certain percentage. This is particularly useful in environments with multiple regions where traffic and usage variations occur. By setting up a Change alert, you can detect changes relative to the baseline, rather than just using a static threshold.

When to Use Change Alerts

  • A historical baseline can be established. For example, steady network traffic
  • Metric is relatively steady. For example, database disk usage
  • An abrupt metric spike. For example, request latency

When Not to Use Change Alerts

  • Metric is expected to fluctuate significantly over time
  • Metric is noisy and unreliable
  • Metric has a low baseline

Define a Change Alert

Change alerts compare a shorter time range to a longer one and trigger an alert if the change between the two ranges exceeds a threshold that the user has defined.

For example, if you want to be alerted on an increase in database latency, you can configure a Change alert on the relevant metric, such as database_query_duration_seconds, comparing the last 5 minutes with the last 1 hour. If the change in the metric between the two time ranges exceeds the custom-defined threshold, the change alert will trigger and you will receive an alert notification.

Change Alert Resolution

Unlike alerts with static thresholds which resolve when the threshold is no longer met, Change Alerts resolve when difference between the shorter interval and the longer interval no longer violate the user-defined threshold.

For instance, the following graph shows database latency that increases significantly after 3:00 and remains high. Using a static threshold of 5s for a Metric Alert would result in an alert that remains triggered.

On the other hand, setting a threshold of 50% for a Change alert would result in an alert that triggers for the initial spike and then resolves. This is because the difference between the last 5 minutes and the last 1 hour is no longer significant.

Prevent Automatic Alert Resolution

In order to prevent an incident from being automatically closed when a Change alert no longer violates the threshold, you can configure an alert to not send alert resolutions to the notification channel when an alert resolves. This can help prevent confusion in the on-call process as an alert resolution does not necessarily mean that an incident has been resolved.

For more information, see Notify when Resolved.