Skip to content
Go to dashboard

9.1How you get alerted

An alert channel is a place obsrvly sends alerts to. It can be email, Slack, Microsoft Teams, Discord, Telegram, PagerDuty or a webhook. Chapter 10 shows how to add one.

Each monitor sends alerts only to the channels attached to it. You pick them when you create or edit the monitor.

  1. A monitor fails a check.
  2. The number of failed checks in a row reaches the failure threshold (see 9.2). obsrvly then opens an incident. An incident is the record of one problem, from start to end.
  3. obsrvly sends an alert to the monitor's channels. The headline says Down or Degraded.
  4. If you turned on reminders (see 9.7), obsrvly reminds you while the problem lasts. The headline says Still down or Still degraded.
  5. When the monitor passes again, the incident closes. obsrvly sends a resolved message.

Email subjects look like this:

json
[obsrvly] DOWN: <monitor>
[obsrvly] DEGRADED: <monitor>
[obsrvly] STILL DOWN: <monitor>
[obsrvly] STILL DEGRADED: <monitor>
[obsrvly] Resolved: <monitor>

<monitor> is the monitor's name. If it has no name, obsrvly uses its URL, then its ID.

What counts as a failure#

  • A degraded check counts as a failed check. Degraded means the check passed but was too slow, or a certificate or domain is close to expiry (see chapter 3 and 9.2).
  • TLS certificate monitors are the exception. Their expiry warnings work on their own (see 2.8).
  • If a monitor runs from several regions, each region counts its own failures. One region failing is enough to alert you. The incident closes when every failing region passes again.
  • Heartbeat monitors follow your alert policy, like every other monitor (see chapter 5).

Things to know#

  • If you add a channel to a monitor while its incident is open, the new channel gets the alert for that incident. This does not happen while the monitor is muted.
  • If a send fails because the channel did not answer, was busy or had an error, obsrvly tries again. It tries up to 3 more times, at 1, 11 and 41 minutes after the first try. Other refusals are not tried again.
  • An email channel whose address is not confirmed gets nothing. Confirm the address first (see 10.2).

Through the API, send alertChannelIds when you create a monitor. To change the list with UpdateMonitor, also send "replaceAlertChannels": true. Without it, the list is ignored.

Set a monitor's alert channels
bash
curl -s "$API/monitor/obsrvly.v1.MonitorConfigService/UpdateMonitor" \
  -H "Authorization: Bearer $OBSRVLY_KEY" \
  -H "Content-Type: application/json" \
  -d '{"id": "<monitor-id>",
       "alertChannelIds": ["<channel-id>"],
       "replaceAlertChannels": true}'

Last updated