Leader election

Leader election picks one node from a group to coordinate, while others follow. You need it for: write coordination, distributed locks, scheduled jobs that should run once, leader-based replication.

Why this is harder than it looks

Naive heartbeat election: each node pings others, lowest IP wins. Problem: under network partition, both sides think they are the only alive ones and both elect themselves. Split-brain. Now you have two leaders, both accepting writes, diverging.

Real leader election requires a consensus protocol: only the majority can elect a leader. Minority partitions cannot make progress, which is the correct behavior.

The three patterns

1. Coordinator service: use ZooKeeper, etcd, or Consul. They run Raft/ZAB internally. Your app does a CAS lease on a key. Cheapest and most reliable for most systems.

2. Embedded consensus: your app runs Raft directly. More work, but no external dependency. CockroachDB, Kafka KRaft, RethinkDB do this.

3. Cloud primitives: AWS DynamoDB conditional writes, Redis SETNX with TTL, GCS object versioning. Quick and dirty, decent at small scale.

Etcd election in 5 lines

session, _ := concurrency.NewSession(client)
election := concurrency.NewElection(session, "/leader-key")
election.Campaign(ctx, "node-id")
// you are now leader until session expires
defer election.Resign(ctx)

The session holds a TTL lease. If your node dies, lease expires, election fires for next campaigner. Etcd guarantees only one leader at a time across the cluster.

Fencing tokens

Even with consensus-based election, races happen. Old leader pauses (GC, network glitch), election promotes new leader, old leader wakes up thinking it's still leader. Now two leaders briefly.

Fix: every leader gets a monotonically increasing fencing token (term number). All downstream operations include the token. Followers and storage reject operations with stale tokens.

Spanner does this. ZooKeeper does this via zxid. Without fencing, you have a race.