Mark Kenneth Badilla

Room 3 of 3

Split-Brain Sandbox

Three nodes, one job, and only one may run it.

  1. Cut the primary off from the store and watch a second node take over.
  2. Let the first one come back still believing it is the leader.
What this shows
Making sure exactly one worker owns a job when the network cannot be trusted.
How
  • Lease, a timed claim on the job that expires unless the holder renews it, so a node that goes silent loses it automatically.
  • Fencing token, a counter that rises by one every time the lease changes hands.
  • Stale-token write, the database refuses anything carrying an old number, so a node that was asleep cannot act on a claim it no longer holds.
What this demo is and is not
What is realEvery lease claim and renewal is a real conditional write, and the nodes genuinely race each other.The token is checked at the storage layer, the only place that check counts.
What is stagedNothing is unplugged.Cutting a node off sets a flag on the node's own row, which its own code checks before it talks to the store.
What this does not proveNot that this survives a real network partition.The failover logic is what is tested here, and real network failures are messier in ways that matter.