Failover & circuit breaking

Providers fail independently. Prism's job is to make sure their failure is not yours.

Circuit breakers

Each provider carries its own breaker, in one of three states.

Closed - traffic flows normally and failures accumulate in a rolling window.

Open - the provider is removed from routing entirely. Requests go elsewhere with no penalty and no wasted attempt. This is the state that keeps one bad provider from becoming your outage.

Half-open - after a cooldown, a single probe request is allowed through. If it succeeds the breaker closes and normal traffic resumes. If it fails, the breaker reopens with a longer cooldown.

Defaults:

{
  "failureThreshold": 5,
  "windowMs": 30000,
  "cooldownMs": 15000,
  "halfOpenProbes": 1,
  "backoffMultiplier": 2,
  "maxCooldownMs": 300000
}

Five failures in thirty seconds trips the breaker. Fifteen seconds later one probe is tried. Repeated failures double the cooldown up to five minutes, so a provider having a long outage is checked occasionally rather than hammered.

What counts as a failure

The last row matters. A revert is a correct answer to an incorrect call. Counting it as provider failure would trip breakers across your whole pool because of a bug in your contract call.

Retries

When a request fails on one provider, Prism decides whether replaying it is safe, not merely whether it is likely to succeed.

Reads are retried. They are idempotent, so they are replayed on the next-best healthy provider with exponential backoff and full jitter, up to three attempts, within your request deadline.

Transaction submissions are not. eth_sendRawTransaction is sent to exactly one upstream. A timeout does not tell us whether it reached the mempool, and a duplicate submission is a worse outcome than an ambiguous one. The failure is returned to you unchanged.

{
  "maxAttempts": 3,
  "baseDelayMs": 120,
  "maxDelayMs": 2000,
  "jitter": "full",
  "retryOn": ["timeout", "5xx", "429", "connection-reset"],
  "neverRetry": ["eth_sendRawTransaction"]
}

Retries are not billed to you - see rate limits.

Handling a submission timeout

Never resubmit blindly. Compute the hash locally and check whether it landed:

import { keccak256 } from 'viem';

const hash = keccak256(signedTransaction);

try {
  await client.request({
    method: 'eth_sendRawTransaction',
    params: [signedTransaction],
  });
} catch (error) {
  // Poll before deciding it failed - it may already be in the mempool.
  for (let attempt = 0; attempt < 10; attempt++) {
    const receipt = await client.request({
      method: 'eth_getTransactionReceipt',
      params: [hash],
    });
    if (receipt) return receipt;
    await new Promise((resolve) => setTimeout(resolve, 2000));
  }
  throw error;
}

Health checking

Independent of live traffic, Prism probes every provider every few seconds with eth_blockNumber, and checks three things:

1. Reachability - does it answer at all
2. Latency - how long it took, feeding the rolling p50 used by routing
3. Drift - how far its head is behind the highest head in the pool

Drift is the one a naive health check misses. A provider that answers quickly and correctly for a block height twenty blocks old looks healthy by every conventional measure and will quietly serve your users stale state.

Do not add your own fallback

The most common mistake when adopting Prism is keeping the provider-fallback list it replaces.

Two balancers stacked on each other multiply requests during an incident - precisely when upstream capacity is scarcest - and your outer loop cannot see breaker state, so it retries providers Prism has already excluded.

Point at one Prism URL. Set your client's retry count to zero or one. Let the failure surface if the whole pool is down, because at that point retrying will not help.

When everything fails

If every provider's breaker is open, Prism returns -32603 with a body naming the state rather than hanging until your timeout:

{
  "jsonrpc": "2.0",
  "id": 1,
  "error": {
    "code": -32603,
    "message": "No healthy upstream available",
    "data": { "providersOpen": 5, "retryAfterMs": 4000 }
  }
}

This is a fast, explicit failure by design: it tells you the pool is exhausted instead of burning your request deadline. Surface it, back off for the interval given, and check status.