How failover works when a provider goes down
Last week a major model vendor had a two-day partial outage. Here is what actually happened to traffic routed through bestokens, and how the routing layer decided what to do.
A single-provider architecture turns one vendor's bad day into your outage. The common fix is to add retry logic in your application — but retries only help if you already hold credentials for a second provider, know which model is equivalent, and have budgeted for it.
Three layers of protection
- Channel rotation: each model has multiple upstream channels. On 429, 500, 502, 503 or 504 we retry the next channel before the request is returned to you.
- Timeout budget: a channel that has not produced its first byte within the configured window is abandoned so the request can still complete elsewhere.
- Circuit breaking: a channel that fails repeatedly is removed from rotation for a cooldown period, so healthy traffic does not keep hitting a dead endpoint.
What it looks like from your side
Nothing. You send one request to one endpoint and receive one response. The retry happens inside the gateway, inside the same HTTP request, before your client times out. Streaming requests are only retried before the first byte is emitted — after that we cannot replay the response without duplicating content, so the stream is closed with an error event instead.
What it cannot do
Failover does not create capacity. If every channel for a model is degraded, the request fails. It also cannot silently swap to a different model: a request for a specific model id will only be served by that model, because a substitution would change your output quality without your consent.
If your workload can tolerate a model substitution, tell us and we will enable a documented fallback chain for your account. It is off by default.