Skip to content
Go to dashboard

11.6Delivery and failures

Reply fast with 2xx#

  • Reply with a 2xx status within 10 seconds. Reply first, then do slow work.
  • A 2xx reply counts as delivered. Any other reply, or no reply within 10 seconds, counts as failed.

Retries#

Your endpoint…What obsrvly does
Replies 2xxDelivered. Done.
Does not answer, or replies 429 or a 5xx statusTries again up to 3 more times: 1, 11 and 41 minutes after the first try. Nothing is retried after one hour.
Replies any other status, such as 400, 401, 404 or 410Failed. Not retried. Fix the endpoint or the channel settings.
  • If the incident closes before a retry, obsrvly drops the old alert.fire or reminder. You do not get a late alert.fire after the recovery.
  • An alert.resolved that gets no answer, 429 or a 5xx is still retried after the incident closes.
  • If you delete the channel, its pending retries are dropped.
  • Reply 401 or 403 to a request with a bad signature. It is not retried.

Duplicates#

After a timeout or a 5xx reply, obsrvly may send a request you already handled. To spot duplicates, use alert_id, event and reminder_number together as the key.

Check what was sent#

Open alerting → notification log (9.10). Each attempt has a row with the subject POST alert.fire or POST alert.resolved, the channel, the result and any error. Select a row to see the JSON that was sent. If a later retry gets through, the row changes to delivered.

Error in the logMeaning
webhook POST <status>: <reply>Your endpoint replied with this status. The start of your reply body follows.
webhook POST: <connection error>obsrvly could not reach your endpoint, or it did not answer in time.
webhook POST: destination is not a publicly routable addressYour host name now points to a private address. Point it to a public one.

With the API, call ListNotifications on NotificationService. It takes limit (default 50, up to 200), cursor and an optional monitorId.

Last updated