Automatic Retry System for Failed Notifications

Why Automatic Retries Matter

In a perfect world, every notification would be delivered on the first attempt. In reality, things go wrong all the time. The Telegram API might be temporarily overloaded. Your email provider could hit a rate limit. A Slack webhook URL might temporarily return a 500 error. A Discord server might be experiencing downtime.

Without automatic retries, each of these failures means a lost notification. Your customer never gets their order confirmation. Your team misses the server alert. Your user does not receive the verification code. For critical notifications, a single point of failure is unacceptable.

One-Ping's retry system is built into the core of the platform. When a notification delivery fails, One-Ping automatically retries with exponential backoff, giving the downstream service time to recover. And if retries are exhausted on one channel, the fallback channel system kicks in to deliver your message through an alternative path. All of this happens transparently -- you send one API call, and One-Ping handles the rest.

How Retry Works: Exponential Backoff

When a notification delivery attempt fails, One-Ping does not immediately retry. Instead, it uses exponential backoff -- waiting progressively longer between each attempt. This approach gives the failing service time to recover and avoids overwhelming it with rapid-fire requests that could make the problem worse.

After the 3rd retry (4th total attempt), if the notification still has not been delivered, One-Ping marks it as failed. At this point, if you have configured fallback channels, the fallback system takes over. If not, a notification.failed event is sent to your webhook endpoint and the failure is logged in your dashboard.

What Triggers a Retry

Not every failure warrants a retry. One-Ping distinguishes between transient errors (which are likely to succeed on retry) and permanent errors (which will not). Here is what triggers and does not trigger automatic retries.

This distinction is important because retrying permanent errors wastes time and delays fallback channels from kicking in. One-Ping's retry logic is smart enough to skip retries for errors that will never succeed, moving directly to the fallback path if one is configured.

Fallback Channels: Your Safety Net

Retries handle transient failures on a single channel. But what if the entire channel is down for an extended period? What if the Telegram API is experiencing a major outage that lasts longer than your retry window? That is where fallback channels come in.

With One-Ping, you can specify backup channels that activate when the primary channel fails after all retry attempts. If your Telegram notification cannot be delivered, send it via Email instead. If Slack is unreachable, fall back to Discord. Your message gets through regardless of individual channel availability.

Configuring Fallback Channels via the API

curl -X POST https://api.one-ping.com/send \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "message": "Server CPU at 95% - immediate attention required",
    "channels": ["telegram"],
    "recipient": "+34612345678",
    "fallback_channels": ["email", "sms"],
    "retry": true
  }'

In this example, One-Ping will first try to deliver via Telegram. If all 3 retry attempts fail, it automatically sends the notification via Email. If Email also fails after retries, it falls back to SMS. The fallback chain is processed in the order you specify in the fallback_channels array.

Fallback Flow

Configuration Options

While the retry system works out of the box with sensible defaults, you can customize its behavior per notification or globally in your account settings.

Per-Request Configuration (via API)

Account-Level Settings (via Dashboard)

In your dashboard settings, you can configure default retry behavior that applies to all notifications unless overridden by per-request parameters.

• Default retry enabled: Turn retries on or off globally for your account
• Default fallback channels: Set fallback channels that apply to every notification automatically
• Retry notifications: Get notified in your dashboard when retries are triggered
• Failure alerts: Receive an alert when a notification fails after all retries and fallbacks

Common Retry Scenarios

Here are real-world scenarios where the retry system saves your notifications from being lost.

Monitoring Retries in the Dashboard

Every retry attempt is logged and visible in your One-Ping dashboard. When you view a notification that required retries, you can see the complete timeline of delivery attempts, including timestamps, the error that triggered each retry, and which attempt finally succeeded (or if all attempts failed).

The dashboard also provides aggregate retry metrics so you can understand the overall reliability of your notification channels.

Retry Metrics Available

• Retry rate by channel: See which channels trigger the most retries. If one channel consistently needs retries, it might indicate a configuration issue or a less reliable provider.
• Average retries to success: How many attempts it typically takes to deliver. If most notifications succeed on retry 2 or 3, the system is working as intended.
• Fallback activation rate: How often fallback channels are needed. A high rate suggests your primary channel needs attention.
• Final failure rate: The percentage of notifications that fail even after all retries and fallbacks. This is the metric you want to keep as close to zero as possible.

// Example notification timeline in the dashboard

Notification: msg_a1b2c3d4e5
Message: "Server CPU at 95%"
Primary channel: telegram

  10:30:00  ATTEMPT 1  telegram  FAILED  (timeout)
  10:30:15  ATTEMPT 2  telegram  FAILED  (timeout)
  10:31:15  ATTEMPT 3  telegram  FAILED  (503 Service Unavailable)
  10:36:15  ATTEMPT 4  telegram  FAILED  (503 Service Unavailable)
  10:36:16  FALLBACK  email     DELIVERED  (245ms)

Result: Delivered via fallback channel (email)
Total time: 6 minutes 16 seconds