A team of alarm clocks. The schedule book says what to do and when; several dispatchers watch their own pages of the book and ring a worker when a job is due. If a worker falls asleep mid-job, the job is handed to someone else.
Problem Statement
Design a distributed job scheduler: a service that runs jobs at a given time or on a recurring schedule ("send the daily report at 07:00", "retry this webhook in 5 minutes", "run the cleanup every hour"), across many worker machines, reliably, without running a job twice when it shouldn't or missing it when a machine dies. Think of a managed cron, or systems like Airflow's scheduler, Quartz clusters, or AWS EventBridge Scheduler.
Functional Requirements
Schedule one-off jobs (run at time T) and recurring jobs (cron expressions).
Run the job's handler (an HTTP call, a queue message or a container task) on a worker.
Retries with backoff on failure; timeouts; a dead-letter state after N failures.
Cancel/pause jobs; view run history and status.
Out of scope: DAG dependencies between jobs (mention), resource-aware bin packing.
Non-Functional Requirements
Reliability: every due job runs; at-least-once execution, with idempotent handlers for effectively-once.
Timeliness: jobs start within a few seconds of their due time.
Scale: 100M scheduled jobs, 10K jobs becoming due per second at peak (top of the hour).
No single point of failure in the scheduler.
Core Entities
Job: id, owner, schedule (cron or run_at), handler/target, payload, retry policy, timeout, status (active/paused).
JobRun (execution): job_id, scheduled_time, attempt, status (PENDING/RUNNING/SUCCEEDED/FAILED), worker_id, lease_expires_at.
Schedulers turn due entries into runs; workers lease runs, and an expired lease means the run is retried elsewhere.
clients -> Job API -> jobs DB (definitions)
|
Scheduler nodes (N, partitioned):
each owns a set of partitions (by job_id hash)
loop: find runs due in the next few seconds in MY partitions
-> create JobRun (PENDING) -> push to execution queue
-> for recurring jobs compute next_run_at
|
[ execution queue(s) ] (by priority / tenant)
|
Workers: pull run -> take lease -> execute handler (with timeout)
-> heartbeat / extend lease -> mark SUCCEEDED / FAILED
-> failed? schedule retry run at now + backoff
Reaper: runs whose lease expired (worker died) -> back to PENDING
Low-Level Design (LLD)
The unique (job_id, scheduled_time) key means an occurrence produces at most one run, even if a scheduler retries.
Finding due jobs efficiently
schedule_index(partition, next_run_at, job_id) -- indexed by (partition, next_run_at)
scheduler for partition p, every ~1 s:
SELECT job_id FROM schedule_index
WHERE partition = :p AND next_run_at <= now() + interval '5 s'
ORDER BY next_run_at LIMIT 1000
FOR UPDATE SKIP LOCKED; -- safe even if two schedulers overlap briefly
-> insert job_runs (unique on (job_id, scheduled_time)), enqueue,
update next_run_at = cron.next(scheduled_time)
Alternative for short delays: a Redis sorted set per partition, score = due time,
ZRANGEBYSCORE 0 now; or a timing wheel in memory backed by the DB.
The unique constraint on (job_id, scheduled_time) guarantees a scheduled occurrence creates at most one run even if a scheduler retries.
Partition ownership (no single scheduler)
Jobs are hashed into, say, 1,024 partitions; scheduler nodes hold leases on partitions through a coordination service (etcd/ZooKeeper) or a leases table. If a node dies, its leases expire and other nodes take over its partitions. Fencing tokens (the lease epoch stored with each write) stop a paused "zombie" scheduler from acting on partitions it no longer owns.
Execution: leases, retries, idempotency
A worker claims a run with a lease (e.g. 60 s) and extends it by heartbeat while running; if the worker dies, the lease expires and the run is retried elsewhere.
Because a run can execute more than once (a slow worker whose lease expired), handlers receive run_id as an idempotency key.
Retries use exponential backoff with jitter; after the maximum, the run is marked FAILED and alerts fire.
Thundering herd at 00:00 and top-of-hour: many jobs share a due time. Spread with small per-job jitter where acceptable, and size queues/workers for the peak.
Follow-up Questions
How do you guarantee a job doesn't run twice?
You can't fully, in a system that must also never miss a run: a worker can finish the work and die before recording success. Guarantee at-least-once and make handlers idempotent with the run id; the unique run constraint prevents duplicate scheduling.
The scheduler was down for 10 minutes. What about jobs due in that window?
When partitions are picked up again, the "due" query finds all past-due runs. Per-job policy decides: run all missed occurrences (catch-up), run once, or skip, which matters for recurring jobs.
How do you support dependencies (job B after job A)?
Model a DAG: B's run is created when A's run succeeds (event-driven), rather than by time. Workflow engines (Airflow, Temporal) specialise in this.