PolarFS: Disk-Paxos RW Leases That Keep Shared Storage Out of Split-Brain

PolarDB separates compute from storage. Many machines can reach the same block device. Exactly one of them may write to it. A second read-write mount corrupts data, and there is no recovery from it. This article is about how PolarFS, the filesystem under PolarDB, enforces that rule. It does so with a disk-paxos lease: a lock written to the shared disk itself, renewed every second, and backed by a watchdog that terminates the node when renewal stops, so it cannot write in split-brain.

Part I — Background

Compute is separate from storage

A PolarDB cluster does not put the database and its data files on the same machine. Compute nodes run the PostgreSQL engine. Storage nodes expose one shared block device that every compute node can reach. PolarFS is the user-space filesystem that turns that block device into a normal filesystem. To the engine it looks like a local disk.

Two terms matter here:

• PBD — the shared block device. Think of it as one virtual disk plugged into several machines at once.
• RW / RO mount — the PBD can be mounted read-write (RW) or read-only (RO). In a typical cluster the primary mounts RW and the read-only replicas mount RO. This mirrors the “one primary, many read replicas” shape of PostgreSQL.

Only one writer may exist

This rule is absolute. PostgreSQL assumes a page on disk has one authoritative writer. Two nodes writing the same pages at the same time corrupt them. Replication, crash recovery, and every replica all depend on there being a single source of truth for a given page. The shared disk itself offers no protection here — the filesystem must supply it.

The paxos file

Every PBD carries a small, dedicated file called the paxos file. It is split into sectors. The first sector holds the leader record: the log metadata, such as the tail and head transaction ids, the offsets, the log size, and the LSN. In effect it is the authoritative state of the RW node. The remaining sectors were always reserved as one slot per host.

Part II — The problem

A lock that stops at the machine boundary

The upstream code already refused a second RW mount — but only when the second mount ran on the same server as the first. A function named paxos_hostid_local_lock opens a file under /var/run/pfs/ and takes a write lock on it with fcntl. Each host id locks its own byte range in that file. If two processes on the same machine try to mount the same PBD read-write, the second lock fails and the mount is rejected.

Two nodes can each believe they own RW

But the situation changes when the two nodes are on different hosts. Both nodes run the acquire path. Both decide they are the writer. Both write to the same shared pages. This is split-brain: two primaries, one disk, no coordination. The first symptom is usually corruption, and by the time it appears the data is already corrupted.

The cluster’s higher layers — the control plane, the failover logic — can be wrong, race, or crash at the worst moment. A filesystem-level guarantee should not depend on them being correct.

The scaffolding was there, but unused

The code showed that cross-host exclusion was intended from the start, but never finished. Three signs stand out:

• mnt_host_generation, a per-mount epoch counter meant to tell a live instance from the leftover record of a crashed one, was initialized to zero and never incremented.
• The per-host sectors in the paxos file were allocated but never written.
• pfs_log_paxos_try_release was a stub that simply returned success, and the acquire path returned the node’s own in-memory leader state without reading or arbitrating the on-disk slots.

The lease timing machinery existed. The actual on-disk protocol did not.

Part III — What we built

The fix adds the missing cross-host protection. It writes a real, renewable lock record to the shared disk itself, and arbitrates contention with a ballot, using the Paxos protocol.

One 512-byte record per host

Each host owns one sector of the paxos file, indexed by host id. A new structure, pfs_host_record_t, fills exactly one 512-byte sector:

struct pfs_host_record {
    uint32_t hr_magic;      /* 0xcafebabe, or 0 when empty */
    uint32_t hr_flags;      /* FL_RW, FL_PREPARE */
    uint32_t hr_host_id;    /* 1 .. max_hosts */
    uint32_t hr_generation; /* prev+1 after crash, 1 after clean start */
    uint64_t hr_timestamp;  /* CLOCK_REALTIME seconds of last renewal */
    uint64_t hr_mbal;       /* promised ballot (prepare) */
    uint64_t hr_bal;        /* accepted ballot (acquire) */
    uint8_t  hr_unused[468]; /* pad to exactly 512 bytes */
    uint32_t hr_checksum;   /* crc32c of the bytes before it */
};

Three fields are central to the design:

• hr_flags marks which phase the record is in — FL_PREPARE for “I am trying”, FL_RW for “I hold the lease”.
• hr_generation tells a fresh mount from the leftover of a crashed one. A new mount that finds its own old record writes generation plus one; a clean start with no prior record begins at 1.
• hr_timestamp is the heartbeat. It stores CLOCK_REALTIME seconds of the last renewal. That choice means hosts must keep their clocks roughly in sync with NTP — a wall clock that drifts or steps backwards can make a live holder look expired, or a stale record look fresh. The judgement is made against a signed age and with a bounded skew allowance, paxos_clock_skew_max (2 seconds by default): a record stays live until it is lease duration + skew allowance + 1 seconds old by the reader’s clock, and a timestamp from the future is always live. Within that bound the lease tolerates ordinary NTP drift; past it, the lease’s liveness judgement depends on cluster timekeeping.

A two-phase, ballot-based acquire

Mounting read-write now runs a short protocol inside pfs_leader_load. The steps are:

1. Check — read every host sector. If some host holds a live RW record, note the freshest one and stop with -EBUSY. A sector that cannot be read, or that holds a corrupt record, is not treated as empty: the acquirer watches it until it becomes readable or its writer is presumed dead, and fails the mount with -EBUSY if it stays bad for a whole lease window.
2. Prepare — write FL_PREPARE to its own sector with its ballot. This announces intent.
3. Verify — read every sector again. A fresh RW record of another host preempts this host whatever its ballot is, because that host may already be past its own conflict check and would never yield; a higher promised ballot preempts it as well. Either way this host clears its sector and stops.
4. Acquire — write FL_RW to its own sector with its ballot. The host now claims the lease.
5. Conflict check — read every sector once more. This detects the case where two hosts both passed prepare before either acquired. The higher ballot wins, and the loser clears its sector.

Every step has to finish while the record the previous step wrote is still live to everybody else. If the prepare has already expired by the time the acquire lands — or the acquire record by the time the conflict check finishes — the host gives the lease back and reports -EBUSY, rather than mounting on top of a holder it has already lost.

The ballot is generation * num_hosts + host_id. Each host reads the previous record and increments its generation on a new mount, so no two hosts racing at the same time share a ballot, and a host re-mounting after a crash outranks the leftover record of its own earlier instance. A clean release zeroes the sector, so the counter starts over at 1 — it climbs only across crash-recovery remounts. When two hosts race to acquire at the same time, the host with the higher ballot wins the tie-break; the other host clears its sector and aborts. A record whose age has passed the lease duration and the skew allowance is ignored as stale, so a long-dead host’s old ballot cannot block a new mount. That is disk Paxos reduced to its single-writer form: each host writes its own slot, reads everyone else’s, and the freshest record decides the winner — a live RW record beats any ballot, and among concurrent acquirers the highest ballot wins.

Takeover that waits, then checks

What happens when a new node finds a live holder? It does not give up immediately. It polls the holder’s sector once a second, for at most one lease duration — 30 seconds by default (paxos_lease_duration), plus the clock-skew allowance — and checks for three outcomes:

• The timestamp advances — the holder is alive and renewing. The challenger returns -EBUSY.
• The sector is cleared — the holder unmounted cleanly. The challenger re-checks and proceeds.
• The timestamp stops moving for a full lease window — the holder is presumed dead. The challenger re-checks and takes over.

The check picks the holder with the freshest timestamp. Once that record expires, every older leftover has expired too. So one wait covers all of them — no sequential per-host delays.

Renewal and self-fencing

The lease is renewed continuously; it is not a one-time claim. The log thread rewrites the holder’s timestamp every log_paxos_lease seconds — one second by default. Renewal happens on the log thread, independent of metadata writes, so an idle database with only data traffic never lets the lease lapse.

The hard part is what happens when renewal fails — the storage is unreachable, the node is wedged, the kernel is stuck. In that moment the node can no longer prove it is alive. Other hosts will soon treat it as dead and mount RW. Continuing to write would mean two writers. So the node fences itself, in layers:

1. Abort. When renewal has failed for a full lease duration, the log thread calls abort(). It terminates itself rather than continue writing in split-brain.
2. Timer-kill. A POSIX timer armed at acquire sends SIGUSR2 if the log thread stops petting it. The handler writes one message and raises SIGKILL. This fires even if the process is otherwise stuck. The timer is not optional: if it cannot be armed, the mount refuses the RW lease and gives it back, because the log thread’s own self-fence cannot fire on a thread that is itself the thing that got stuck.
3. Kernel watchdog. With paxos_watchdog_enable set, the node opens /dev/watchdog and pets it on every renewal. If the whole kernel hangs in an uninterruptible state, the watchdog reboots the machine. This is the last-resort fence.

That layering is the real guarantee. A holder that cannot renew is killed by its own watchdog, and a killed holder cannot write.

Release and compatibility

On clean unmount the node disarms the watchdog, then zero-writes its own sector. A zero sector has hr_magic == 0, which the reader treats as “no holder” — identical to a released lease.

That choice made the change migration-free. The per-host sectors were always zero in existing deployments. An upgrading cluster sees empty sectors and simply runs the new protocol. A downgrade is safe too: older code never reads those sectors. The one place they are not necessarily zero is a copy of a PBD: fscp and the chunk backup/restore copy the paxos block verbatim, per-host lease records included, so the reset that runs after a restore clears them again.

A new pfs lease subcommand reads the paxos file and prints every host’s lease state — flags, generation, ballot, timestamp, age, the configured lease duration and clock-skew allowance, and whether it is active or expired. For a DBA, that is the difference between “the mount failed” and “host 7 held the lease and has not renewed for 42 seconds.”

The result

The outcome is a cross-host guarantee: at most one node holds a RW mount at any time. The guarantee holds across clean unmounts, crashes, and network partitions, because the lease lives on the disk all parties can reach. And it holds even in the worst case, because a node that loses the ability to renew kills itself rather than risk a second writer.

The guarantee is not assumption-free, and the first cut of the protocol did not earn it. A host that passed the live-holder check and then stalled could come back after another host had already acquired: the original verify step only compared ballots, so a lower-ballot holder was invisible to it, and the late host mounted RW next to the holder. A sector that could not be read, or that held a corrupt record, was skipped as if it were empty, so a single failed read of a live holder’s sector let a second host in. The age of a record was computed unsigned, so a holder whose clock ran even slightly ahead of the reader’s looked long expired. An acquisition that stalled for a whole lease window could complete and mount next to the host that had taken over in the meantime. And a lease could be granted with no kill timer armed, leaving nothing to stop a wedged process from writing.

The follow-up work closed all of that. The verify step now yields to any fresh RW record regardless of ballot; bad sectors fail closed instead of being skipped; the age is a signed comparison with a bounded skew allowance; every step of the acquisition has to finish while the previous record is still live; and the mount refuses the lease when the kill timer cannot be armed. With those in place the guarantee is real, subject to the two assumptions it always had — the hosts’ clocks stay inside the configured skew allowance, and a fenced holder really does stop writing.

The cost is small and well-bounded. A few extra sector reads and writes on mount, one sector write per second while mounted, and a wait of up to one lease duration (30 seconds by default, plus the skew allowance) on takeover. For a database, corruption is the costliest failure, so this cost is small by comparison.