Webhook retries and dead letters

Nextmail posts every event once as it happens. When the receiver does not answer, or answers with a 5xx, a 408, a 425 or a 429, the event is tried again later, and how long it keeps trying is the retry policy of the endpoint. Each wait is varied by up to 10% so retries do not arrive in step, and a Retry-After of up to 6 hours is waited out. A 410 Gone switches the endpoint off, and any other answer counts as failed and is not tried again.

Choose it with retryPolicy when you create or change an endpoint: POST /webhooks and PATCH /webhooks/{id}, webhooks.create and webhooks.update in the SDKs, --retry-policy in the CLI, and the createWebhook and updateWebhook tools. Every read of an endpoint reports it. Extended suits a receiver that can be down for a weekend, such as a server behind a deploy freeze, at the cost of a slower answer when an event is really lost.

• —In the web app, Retry policy is a choice in the form that creates or edits an endpoint on the Webhooks page. In the phone app, the webhook screen has the same choice.
• —The policy is read at every attempt, so switching to extended also stretches the retries of events that are already waiting for one.
• —An event that has already had more attempts than a shorter policy allows gets the attempt it is waiting for, and is given up on if that fails.
• —An endpoint is switched off after 100 events in a row could not be delivered, whatever its policy, and its owner is told by email. Each of those events is kept as a dead letter.

When the last automatic attempt of an event fails, the event becomes a dead letter of its endpoint. That happens when its retries run out, when the receiver gives an answer that is not tried again (the 410 that switches the endpoint off included), and when the endpoint is switched off before the next retry is due. Every endpoint keeps dead letters, there is nothing to turn on, and keeping them changes nothing about how events are delivered.

GET /webhooks/{id}/dead-letters lists them newest first. status reads the open ones by default, and replayed, discarded or all the rest, and since and until narrow it to a window.

• —Events that happen while an endpoint is switched off are never attempted, so they are not kept for it.
• —An event the endpoint stopped listening for, or that left its allowlists before its next retry, is dropped without a dead letter.
• —Test events never become dead letters, and neither does a replay from the delivery log: a failed replay counts as the latest attempt of an open dead letter, and opens none of its own.

POST /webhooks/{id}/dead-letters/replay sends open dead letters again, each through the same replay as POST /webhooks/{id}/deliveries/{deliveryId}/replay: the stored event goes out once, now, with its original id and a fresh signature, so a receiver that drops ids it has seen handles it once. Name up to 100 in ids, or send no body to replay the 100 oldest open ones. The answer lists what happened to each.

• —Replays go out five at a time. When five in a row fail the call stops and leaves the rest open, so a receiver that is still down is not sent the whole backlog.
• —A call also stops starting new replays after about 20 seconds, or before it would use up the database work one request may do, so a long backlog can take a few calls. hasMore is true while open dead letters are left that it did not send.
• —While the endpoint is switched off the call is refused with 409 webhook_disabled and nothing is sent. Switch it back on first.
• —A replay from the delivery log that is delivered also closes the open dead letter of its event.

DELETE /webhooks/{id}/dead-letters/{deadLetterId} closes an open dead letter as discarded without sending anything, for an event the receiver should not get after all. The event stays in the delivery log, where it can still be replayed. Discarding one twice answers the same both times, and one that was replayed is a 409 dead_letter_resolved.

• —In the web app, the Dead letters tab on the page of an endpoint lists them. Replay and Discard sit in the menu of each row and act on the rows you select, Replay all open sends every open one, and Filters shows the replayed and discarded ones. Clicking a row opens its last attempt.
• —In the phone app, the webhook screen has a Dead letters tab that replays and discards them too.