LSN Consistency for the Extended Protocol: The In-Band Wait Message
Every article so far has assumed the client sends a plain SQL string, one query at a time. Real applications usually do not. Most modern drivers — JDBC’s PreparedStatement, psycopg3, npgsql, and others — use the PostgreSQL extended protocol, in which a query is sent as a sequence of separate messages (Parse, Bind, Execute) rather than a single string. That changes everything for the LSN wait, because the “prepend three SET statements” trick simply has nowhere to go. This article is about how Polar Proxy solved that: a small binary message called W, negotiated at connection startup, that attaches a wait to an extended-protocol query without an extra round trip.
Part I — Background
Two ways to send a query
PostgreSQL has two query protocols.
The simple query protocol sends the whole SQL string in one message (Q). The server parses it, plans it, and runs it, all at once. It is what psql uses for a single statement. Our wait solution — prepend three SET statements to the string — works here because there is exactly one text payload to rewrite.
The extended protocol splits the same work into messages. The client first sends a Parse message (the SQL text and an optional statement name), optionally a Bind message (the parameter values), then an Execute message (run it). The server can reply with ParseComplete, BindComplete, and finally DataRow results. A Describe message asks for result-column metadata. Flush and Sync are pacing/barrier messages: Flush asks the server to send any pending output now; Sync closes an implicit transaction boundary and resets error recovery.
Drivers prefer the extended protocol for two reasons: prepared statements (Parse once, Execute many times with different parameters, skipping re-planning) and parameterized queries (values sent separately from the SQL, which sidesteps SQL-injection quoting entirely).
Implicit prepare
Prepared statements live on a single backend connection. Polar Proxy pools backend connections and reuses them across client sessions. When a client sends an Execute or Describe for a named statement that the borrowed backend has not prepared yet, the proxy sends a lazy Parse to that backend first, then the Execute or Describe. This hidden backend Parse is called an implicit prepare. It is invisible to the client, but it produces a ParseComplete result the proxy must consume and account for before forwarding the Execute result.
Part II — Why the wait wrapper cannot work here
The consistency mechanism from the first article depends on being able to prefix text to the query. The simple protocol gives you exactly one text blob to rewrite. The extended protocol does not:
- There is no single text payload. The SQL text is in the Parse message; the parameters are in Bind; neither is a place you can “insert a
SETstatement”. - A Parse message carries exactly one statement. You cannot prepend
SET polar_xact_split_wait_lsn = ...to the Parse text the way you prefix the simple-protocol string — the server rejects multiple commands in a prepared statement, so the hiddenSETwould have to become its own Parse/Bind/Execute, produce its own result set, and be drained before the real query runs. A named statement’s server-side plan is also fixed at Parse time, so changing the Parse text is a different statement, not a wrapped one. - A
SETis itself a command that must run and return a result, and the extended protocol has no spare slot to run a hidden command in-band.
The practical consequence is severe for any proxy that implements the wait by rewriting the query text: every extended-protocol read that needs an LSN wait falls back to the primary. Since the extended protocol is the default for most modern drivers, that means the read-offload feature silently does not apply to most real clients — the wait can only be applied to the minority that sends plain SQL strings.
Part III — What the Polar Proxy team built
The answer is a purpose-built wire message that carries the wait alongside the query, negotiated once at startup.
A binary wait message: W
We introduced a new client-to-server message, W. Its payload is 16 bytes:
| field | size | meaning |
|---|---|---|
| version | 1 byte | protocol version (1) |
| consistency mode | 1 byte | 0 = best-effort, 1 = strict |
| flags | 2 bytes | reserved (0) |
| timeout | 4 bytes | how long to wait, in milliseconds (0 = no deadline) |
| target LSN | 8 bytes | the WAL position the replica must reach |
W uses the standard PostgreSQL framing — a type byte, then a four-byte length, then the payload — but the message itself is a PolarDB extension, not part of the PostgreSQL wire protocol, so a native driver cannot emit it. The length field is 20 (the 16-byte payload plus the length field itself, excluding the type byte), making the whole frame 21 bytes on the wire. The consistency mode, timeout, and target LSN mirror the three SET values from the simple-protocol wrapper (polar_consistency_mode, polar_proxy_wait_timeout_ms, and polar_xact_split_wait_lsn).
Negotiate the capability
Sending W to a backend that does not understand it would be an error, so the capability is negotiated at connection startup. A connection opened with the v15_wait proxy protocol sends the startup parameter _pq_.polar_proxy_wait_v1 = 1; only a connection opened with that parameter will ever receive a W. The parameter is a one-way capability declaration — there is no backend acknowledgement. Connections opened with legacy or v15 instead of v15_wait do not negotiate the capability and keep the old behavior — the simple-protocol SET wrapper still works — while extended-protocol reads that need a wait fall back to the primary. A connection opened with off sends no PolarDB startup keys at all. session_lsn with off is rejected at configuration load, because no RFQ reply is available to learn the session LSN from; global_lsn with off is legal, because its wait target is the monitor-supplied group LSN rather than a backend RFQ reply. In that mode a simple-protocol read still gets the SET wrapper — which needs none of the startup keys that off withholds — but an extended-protocol read has no W to attach and falls back to the primary. Even on a v15_wait connection, the backend accepts W only on a replica and only for a proxy-identified connection (one that received _polar_proxy_client_host / _polar_proxy_client_port); the primary rejects it.
Attach the wait to the query — with no extra round trip
Three new libpq entry points place the W immediately before the semantic messages, all staged in the same output buffer and flushed together. The proxy’s runtime path uses two of them:
PQsendPreparePolarWait— sendsW, then Parse (plus Sync when not in pipeline mode);PQsendQueryPreparedPolarWait— sendsW, then Bind, Describe Portal, Execute, and Sync (only when not in pipeline mode) for an already-prepared statement (no Parse, because the statement is already prepared).
A third entry point, PQsendQueryParamsPolarWait (a parameterized W + Parse + Bind + Describe Portal + Execute + Sync), is exported by the patched libpq but is not on the proxy’s production path today — it exists in the API and is covered by unit tests.
The W and the command it qualifies are written into the same output buffer and sent in one flush. The replica reads W, knows it must first wait until it has replayed to the target LSN, and only then runs the Parse/Bind/Execute that follows. The wait adds no extra round trip — it is a few extra bytes on messages that were going out anyway. (An implicit prepare is the exception: the proxy sends W + Parse and waits for ParseComplete before sending the Bind/Execute/Sync, so that path is two backend round trips regardless of the W.)
Two correctness rules:
- Invalid waits are rejected before bytes are staged. If the target LSN is zero or the consistency mode is unknown, the send action fails before any of the message is written, so a later command cannot flush a half-formed
Winto the stream. - The output buffer is checkpointed and restored on failure. If constructing the compound message fails partway, the libpq output cursors are restored so no stale
Wbytes leak into the next command.
The consistency mode decides what a timeout means. In best-effort mode the replica returns the (possibly stale) rows with a warning. In strict mode a timeout is an error — the backend raises an ERROR and does not run the statement at all. The proxy does not recognize that error by its SQLSTATE; it matches two structured fields on it — the DETAIL must be polar_proxy_lsn_wait_timeout and the source function must be polar_proxy_wait_for_lsn. A 57014 (query-canceled) without those markers is an ordinary client cancellation or statement_timeout, not a wait timeout, and is never replayed on another backend.
Route a request as one unit, not message by message
In the extended protocol a single logical request is not one message but a batch of them — Parse, then Bind, then Execute, then Sync — that all belong together. The proxy cannot route them one message at a time: the whole batch must go to the same backend, and at the end the client must see one ReadyForQuery, not one per step.
The routing state was spread across several flags, and it was unclear who owned the final ReadyForQuery or when the proxy was allowed to switch backends. The Polar Proxy code makes a single piece of per-request state (called a “frame” internally) the primary owner of that routing state — the older phase flags remain as a fallback for when no frame is active. The frame:
- knows which kind of request is currently being processed — just a Parse? a Bind/Execute? the Flush/Sync that ends the batch?;
- picks the backend and pins it before the batch is dispatched, and refuses to switch backends halfway through — a failure retry moves the whole remaining batch to another backend as one unit, it never splits it across servers;
- owns the Flush/Sync cleanup, so the proxy sends out exactly one ReadyForQuery to the client per request.
The read-vs-write decision depends on which kind of extended-protocol request it is, and it is made at the batch’s Sync boundary, not message by message. A one-shot Parse/Bind/Execute frame is routed before its first backend Parse and emits W in front of the Parse. A named-statement batch whose shape is a single execution — one Execute, no Describe metadata, a Parse that precedes the Execute with matching statement names — is reader-eligible, so it emits W in front of the Bind/Execute (or, if the borrowed backend has not prepared the statement yet, in front of the implicit backend Parse that runs first). Metadata and multi-operation frames stay on the writer. Either way the whole batch is pinned to one backend, and the proxy never switches halfway through.
Handle the implicit-prepare edge cases
The hidden backend Parse that implicit prepare creates needs to be dealt with carefully.
- A failed Parse is a boundary. When a backend Parse fails during an implicit prepare, the connection state machine still completes with return code zero, but the error is present in the result. The rc0 completion path checks for that error before calling the register handler, so a failed implicit prepare becomes a client-visible error — never a registered statement followed by an Execute.
- A wait warning has one owner. In best-effort mode, when a replica does not catch up in time, the proxy does not fail the query — it returns the read anyway, and the backend raises a
WARNING(carried on aNoticeResponsewire message) that the data may not be fully up to date, which the proxy captures and forwards to the client. The problem is that an implicit prepare produces two results on one connection: a hidden backend Parse result the client never sees, and the Execute result it does. If that warning is raised during the hidden Parse, sending it with the Execute result would misattribute it, and sending it with the hidden Parse would silently drop it. The proxy records which result each warning belongs to: one bound for the backend Parse is held on the session until it can be surfaced, while one bound for the Execute goes out normally. Either way the client sees the warning exactly once — never lost, never duplicated.
The result
The extended protocol now carries the same consistency guarantee as the simple protocol, with one extra precondition the SQL wrapper does not have: the wait must arrive before the statement takes its snapshot. The backend rejects a W that would follow an open portal or an established transaction snapshot — W must precede Parse and Bind, and it does not refresh an existing cursor or a REPEATABLE READ snapshot. Where the wait is allowed to run, the guarantee is the same. A JDBC or psycopg3 application that writes and then reads can have its read served by a replica that waits for the LSN — because the wait rides the W message in front of the Parse (one-shot) or the Bind/Execute (reusable), with no extra round trip and no rewritten SQL. Extended-protocol reads that need a wait but cannot negotiate v15_wait still fall back to the primary (a simple-protocol read on a legacy or v15 backend still gets the SET wrapper). The fallback is per host group, not per backend: v15_wait is a host-group-level setting, and a v15_wait reader host group must contain only backends that understand W — an older backend would reject the message outright rather than degrade. Mixed old and new backends work safely only when they live in different host groups.
This completes the feature’s coverage: the LSN wait now works for the simple protocol (via the SET wrapper) and the extended protocol (via the W message) — which together cover essentially every PostgreSQL client in use. The remaining question is the one every distributed system eventually faces: what happens when a replica fails. That is the next article.