Back to Blog
Engineering PostgreSQL PolarDB ProxySQL Consistency High-Availability

Reader Failure Recovery Without Breaking Consistency

Semyon Binev

This article is about how Polar Proxy handles failure: a policy for what to do when a reader fails, a safe retry path, a stronger consistency mode for when the proxy cannot be sure, and the low-level discipline that keeps the failure path itself from introducing new bugs.

Part I — Background

Common ways a read fails

A read routed to a replica can fail in several distinct ways, and they deserve different handling:

  • Connection loss — the replica’s connection drops mid-query. The query may or may not have executed, and some output may already have reached the client. A dead reader is often one bad server, so the proxy gives the read one more chance on a sibling replica before falling back to the primary — but only if nothing has been forwarded to the client yet; once partial rows have reached the client, it stops retrying — a transaction-split read forwards the error and keeps the writer transaction, while a wait-wrapped or ordinary read closes the session.
  • Wait timeout — the replica is alive but did not catch up to the target LSN within the timeout. Re-reading the same or another replica would just repeat the same wait, so the retry target is the primary, which already has every write.
  • A reusable SQL error — the connection survives, but the query itself is wrong (an undefined table, a division by zero). Retrying it on a replica just fails again, so the proxy re-runs it on the primary and forwards the primary’s authoritative response — usually the same error, but real data when the primary succeeds.

There is one more failure the proxy deliberately does not retry: a query cancellation. A client cancel or a statement_timeout surfaces as SQLSTATE 57014 without the structured wait-timeout marker that distinguishes a real LSN wait timeout. Replaying it on another backend would either redo work the client asked to stop or mask the timeout, so the proxy keeps PostgreSQL’s ordinary cancellation semantics — it returns the error as-is and never replays it, not on another replica and not even on the primary.

The key insight is that each retryable failure class needs a different retry target: connection loss first tries another replica, while a wait timeout and a real SQL error go straight to the primary. Treating them all the same way either wastes retries on doomed queries or fails queries that would have succeeded.

Three failure classes map to three actions: connection loss retries on another replica and then the primary, wait timeout retries on the primary, and a reusable SQL error is re-run on the primary for an authoritative answer
Connection loss retries on another replica and then the primary; a wait timeout and a reusable SQL error both go to the primary, where the timeout disappears and the error's answer is authoritative.

A failover changes the writer identity set

When the primary fails, a replica is promoted to be the new primary. This is called a failover, and it has a subtle consequence for LSN tracking: the LSN a session recorded before the failover belongs to the old primary’s timeline. After the failover, the same numeric LSN may mean something different. A proxy that blindly trusted a pre-failover LSN would compare it against a post-failover replay position and could wrongly conclude a replica is caught up. So the proxy must track which writer the LSN came from, not just the number. Polar Proxy does this with a writer epoch — a counter, numbered per writer hostgroup, that increments every time that writer’s identity set changes. Any LSN recorded under an old epoch is discarded.

When the writer identity set changes, the writer epoch increments and any LSN recorded under the old epoch is discarded
A failover changes the writer identity set and increments the writer epoch; any LSN recorded under the old epoch belongs to the old timeline and is discarded.

Stale data is worse than an error

The principle of consistency reads is: silently returning stale data is worse than returning an error. An error is loud — the application knows something failed and can retry or report it. Stale data is silent — the application proceeds on data that is wrong, and the wrongness may not surface until much later. A proxy that optimizes for availability at the cost of silently serving stale reads is not being helpful; it is being dangerous. Every recovery rule below follows from this principle.

Part II — What upstream ProxySQL does

Upstream ProxySQL has no reader-failure policy for consistency reads. A replica that dies mid-read produces whatever error the connection layer happens to emit, with no notion of “this read was a consistency read that can be safely retried on the primary.” There is no primary-retry path, no retry-state preservation, and no way to express “under these conditions, prefer an error over a possibly-stale result.”

Beyond the missing policy, a pile of boundary cases in the correctness logic can break the guarantee:

  • “No LSN” vs. “LSN zero.” A ReadyForQuery that carries no LSN at all and one that carries a legitimate zero are different situations, but a naive parser treats both as “missing”. That confuses a valid session state with missing protocol data.
  • State migration during retry. Moving a request to another backend must move all of its state — retry limits, deadlines, cancellation flags — or the retried request inherits stale values. Cancellation and timeout markers are deliberately cleared on the target so they cannot leak into the retry.
  • Cleanup that mutates state in the middle of a read. The failure handler can read a backend field, then cleanup changes it, then the handler reads it again — a classic use-after-change bug.
  • Upgrade behavior. Old deployments encoded “route to the writer” as consistency_mode = 'primary'. An upgrade that converted that value to off would silently change where existing workloads run.

Part III — What the Polar Proxy team built

A policy: retry, return an error, or disconnect

The first piece is a real policy for what to do when a reader fails. Each failure class has its own configurable action, set through three options — pgsql-polardb_action_replica_loss, pgsql-polardb_action_lsn_timeout, pgsql-polardb_action_replica_error — and the defaults match the reasoning above: connection loss retries on another replica and then falls back to the primary, a wait timeout retries on the primary, and a reusable SQL error is re-run on the primary, whose result is forwarded to the client. Internally these options resolve to one of three client outcomes — retry, return an error, or disconnect the session — so an operator can, for example, choose to terminate the session on connection loss instead of retrying.

Retry safely: classified targets

When a consistency read fails on a replica and the policy says retry, the target depends on the failure class. Connection loss first retries on another replica of the same reader hostgroup — a dead reader is often one bad server, not a bad hostgroup — and only falls back to the primary when that sibling reader is unavailable or also fails. A wait timeout and a reusable SQL error go straight to the primary: the primary has every write, so there is no LSN to wait for, and its answer to a query that is itself wrong is authoritative.

Two rules keep any retry safe:

  • Only before results start. The moment the proxy starts forwarding a result to the client, it stops retrying. A retry after partial output would either duplicate rows or leave the client with a half-finished statement — both worse than surfacing the failure.
A failed replica read retries only if result forwarding to the client has not started; connection loss first tries another replica, then the primary, while a timeout or SQL error goes straight to the primary
A retry — to another replica or the primary, depending on the failure class — happens only while the proxy has not started forwarding a result to the client; after partial output, retrying would duplicate rows or leave a half-finished statement.
- **A distinct stream, not a new connection.** The retry never reuses the failed replica's stream: a sibling-replica retry draws an existing pooled reader connection with the failed server excluded (it never spins up a brand-new one), and a primary retry goes to a writer stream instead of the dead replica's. For a transaction-split read that writer must already exist and hold the active transaction; an ordinary or wait-wrapped read just acquires a writer backend and moves the packet there — no open transaction is required, so an autocommit read retries too. Reusing the failed stream would mix the failed dispatch's packet and connection state into the retry.

The failed replica connection is released or closed according to a decision captured before cleanup.

Preserve state across the retry

Retrying is only correct if the retried request carries the right state. A request owns several pieces of state — how many retries remain, when it should be cancelled, who owns the packet — and if any of it stays behind on the dead connection, the retry inherits stale values and can fail for the wrong reason.

The proxy therefore moves request-owned state explicitly from the failed stream to the retry stream, and clears anything that was only transient. Retry limits belong to the request, not to either stream: the source drops them and the target receives them, and the target starts without timeout or cancellation markers from an earlier request. Two rules keep the move safe: a fresh connection deadline is started only when the target still needs a connection, and a backend that is already ready is accepted without a deadline.

A stronger mode

The session-LSN mode guarantees read-your-writes: your own session’s reads see your own writes. It does not guarantee you see other sessions’ writes. For workloads that need cross-session causality, Polar Proxy adds a global-LSN mode: the read target is the maximum of the session’s LSN and the group LSN — the highest LSN any node in the replication group, writer or reader, has reported during the current writer epoch. The proxy must therefore consult that group position on every read, and critically, if the group position is unobservable, the proxy does not hand the read to an unchecked reader — it falls back to the primary or errors rather than risk a stale result. Global mode also forbids best-effort degradation: if the routing planner reaches a global-LSN policy that pairs a warning with a missing LSN or with an LSN wait timeout, it returns the query to the client as an error instead of executing it, because a possibly-stale read defeats the entire purpose of the mode.

Compatibility: don’t change behavior during an upgrade

Finally, upgrades must not silently move existing workloads. The old consistency_mode = 'primary' value — “route matching reads to the writer” — carried two meanings at once: a placement target and a consistency mode. On upgrade the proxy splits it in two: it inserts read_target = 'primary' to keep the placement behavior, then converts the legacy consistency_mode value from 'primary' to 'off'. A deployment upgrading across versions therefore keeps routing its reads to the writer exactly as it did before.

The result

The upshot is that failure becomes a handled case, not a correctness hole. A replica that dies mid-read triggers a classified, policy-driven response. It may retry on another replica or the primary, return the error, or disconnect the session. A failover invalidates stale LSN state through the writer epoch. Under the default fallback policy, a proxy that genuinely cannot enforce the wait falls back to the primary or to an explicit error — serving a possibly-stale read with only a warning is an opt-in degradation, not the default. And the failure handler itself is written to a single rule: capture state once, then act on the snapshot.

Across the five articles, the pattern is the same. Polar Proxy adds read-your-writes consistency to read/write splitting by tracking the LSN and making replicas wait — and it does so without ever gambling correctness for throughput, whether the query is simple or extended, whether the connection is fresh or pooled, and whether the replica is healthy or failing.