Skip to content
Go to dashboard

9.2When an alert is sent (failure threshold)

The failure threshold is the number of failed checks in a row that opens an incident and sends an alert. By default it is 2. A passing check sets the count back to zero.

To see the value your team uses now, open alerting → thresholds.

Change it for one monitor#

You can set the threshold only when you edit a monitor, not when you create it. A new monitor starts with your team's defaults.

  1. Open the monitor and edit it (see 3.7).
  2. Find the paging block.
  3. In failures in a row before a page, enter a whole number from 1 to 20. The field shows the current default, for example default 2. Leave it blank to use the default.
  4. Select save changes →.

If you type something that is not a whole number, the form shows Whole numbers only, or blank for the default.

Latency anomalies#

URL, API, multi-step API, TCP, UDP, ping, gRPC, WebSocket, DNS and browser monitors also have a latency anomalies box in the paging block. It is off by default.

  • Tick latency anomalies to mark a check degraded when it is far slower than this monitor usually is.
  • It learns each region's usual time over its first 30 checks. Until then, it marks nothing.
  • A check it marks shows a message like latency anomaly: took 2400ms, usually about 300ms.
  • A degraded check counts as a failed check (see 9.1).
  • Through the API, send "latencyAnomaly": true with UpdateMonitor.

Through the API, send failureThreshold with UpdateMonitor. Send 0 to go back to the default.

Set the failure threshold
bash
curl -s "$API/monitor/obsrvly.v1.MonitorConfigService/UpdateMonitor" \
  -H "Authorization: Bearer $OBSRVLY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "<monitor-id>", "failureThreshold": 3}'

A value outside the range returns invalid_argument with failure_threshold must be 0 (default) or 1-20.

Which value applies#

  1. The monitor's own value, if it has one.
  2. Otherwise, your team's default (see 9.4).
  3. Otherwise, the obsrvly default.

To read the values that apply to one monitor, call GetAlertPolicy with its ID. Without an ID, it returns your team's and obsrvly's defaults.

Read the alert policy
bash
curl -s "$API/alert/obsrvly.v1.AlertService/GetAlertPolicy" \
  -H "Authorization: Bearer $OBSRVLY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"monitorId": "<monitor-id>"}'

The reply includes failureThreshold and cooldownSeconds.

Why didn't I get paged?#

The thresholds page has a section called why didn't I get paged? It shows the monitors with recent failed checks, failing ones first. For each one, it shows its last checks and whether an alert went out:

  • ▲ paged: an alert was sent.
  • ▲ paged · muted: an incident opened, but the monitor was muted.
  • no page: no alert was sent. A sentence explains why, for example too few failed checks in a row.

If nothing has failed, it says Nothing has failed recently. That is the good outcome.

Through the API, call ListEvaluations with a monitorId. It lists only failed checks. Each one has an outcome, such as below_threshold, cooldown, already_firing or fired. Use limit (default 200, at most 1000) and windowSeconds (default 24 hours, at most 90 days).

Last updated