Case study
Distributed Task Queue with Replicated State Store
Fault-tolerant task scheduler with 3-node quorum replication and exactly-once execution under node crashes. Workers auto-scale on queue depth via Kubernetes HPA.
Stack
A fault-tolerant distributed task scheduler built as an open-source learning project — exploring the engineering guarantees that production queuing systems (Kafka, Celery, BullMQ) implement under the hood.
Why
Most “background job” libraries are black boxes. When something fails — a worker dies mid-job, a node loses network, the leader crashes — you’re trusting the library to handle it. I wanted to build the guarantees from scratch: quorum replication, heartbeat-based failure detection, exactly-once execution, and idempotent workers.
Architecture
State store: 3-node cluster using Raft-lite consensus. Every task write requires a quorum acknowledgment (2 of 3 nodes) before confirming to the client. State is replicated synchronously on the write path; reads always go to leader.
Task lifecycle:
- Client submits task via gRPC → leader writes to WAL, replicates to followers, responds
- Worker pulls task → acquires distributed lease (5s TTL, heartbeat every 2s)
- Worker executes → writes result, releases lease
- On worker crash → lease expires, task re-enqueued, re-delivered to next available worker
Exactly-once semantics: Tasks carry a stable ID. Workers check idempotency token before executing — duplicate deliveries are detected and discarded. The result is persisted with the task ID, so re-delivered tasks return cached results.
Auto-scaling: Kubernetes HPA watches a custom metric (queue depth / active workers). Workers scale 1 → N on queue buildup and back to 1 on drain. Each pod is stateless; the state store is external.
What I verified
- Node crash mid-execution: task re-queued within lease TTL, re-executed once, exactly-once semantics hold
- Leader failure: follower election within ~3s, no task loss
- Network partition: minority partition rejects writes (quorum unavailable), rejoins on partition heal
- Duplicate submit: idempotent by task ID, second submit returns cached result
What’s in the repo
- Raft-lite leader election and log replication (no external dependency)
- gRPC task submission and worker polling API
- Lease-based worker locking with heartbeat renewal
- Idempotency token layer
- Kubernetes manifests + HPA config
- Test suite: chaos tests for node crashes, network partitions, duplicate submissions
← all work