Proposed Design

This design presents a database-centric distributed scheduler that handles both one-time and recurring job execution at scale. The system uses a polling-based approach where a central scheduler continuously queries the database for due jobs and distributes them through a distributed queue. Key components include:

• Database as the single source of truth for job state
• Scheduler that polls and queues due jobs
• Executors that pull jobs with database locks for concurrency control
• External storage (S3) for job logs and results

The design emphasizes status-driven workflow with heartbeat mechanisms for fault detection and soft deletes for maintaining execution history.

Concurrency Control and Locking Mechanisms

In distributed schedulers, concurrency control prevents multiple executors from processing identical jobs through coordinated state management. This design leverages database-level pessimistic locking combined with job status transitions (NONE → QUEUED → RUNNING) to establish mutual exclusion. The distributed queue acts as a job broker while the database maintains authoritative job state, creating a two-step coordination process where queue delivery triggers database lock acquisition.

This approach ensures job execution integrity because lock acquisition atomically validates job availability and claims ownership. The status-based state machine prevents race conditions by making job transitions conditional on current state, while database ACID properties guarantee consistency across concurrent executor operations.

Data Lifecycle Management

Data Lifecycle Management in distributed job schedulers involves systematically managing job execution data from creation through deletion while maintaining accessibility and compliance. This encompasses active job tracking, historical data retention, and graceful data archival.

In this design, each job execution generates multiple data artifacts: execution metadata stored in the database, detailed logs written to cloud storage (S3), and results accessible through constructed file paths. The system maintains data accessibility through soft deletion strategies, where cancelled or deleted jobs remain queryable by marking their status rather than removing records entirely.

The approach enables users to access historical job information even after logical deletion, supporting compliance requirements while optimizing storage costs through tiered storage strategies.

Discussion Areas

The exploration of durability mechanisms reveals critical failure scenarios in distributed job processing. The question about queue persistence is essential because in-memory brokers lose jobs during failures, requiring external durability guarantees job reconstruction capabilities from persistent storage.

How does your distributed queue ensure the durability of scheduled jobs, and what happens if the scheduler or executor fails mid-processing?

This question exposes the fundamental tension between performance and reliability. Queue-based systems must balance message delivery guarantees with processing latency, particularly when jobs have varying execution times and failure probabilities.

The failure handling discussion illuminates executor coordination complexity where heartbeat mechanisms and timeout strategies must account for network partitions, GC pauses, and cascading failures across the distributed system.

Candidate's Approach

The design combines optimistic queue consumption with pessimistic database locking, creating a hybrid approach that balances throughput with consistency. The heartbeat mechanism enables proactive failure detection rather than relying solely on timeout-based recovery.

When the executor picks up the job from the queue, it'll update the status in the DB to say from queued to running and only if the status is queued will the executor pick up the job

The pessimistic concurrency control strategy using database locks prevents race conditions but introduces potential bottlenecks like Database contention, Connection pool exhaustion, Lock timeout cascades etc. Checkout Alternative approaches on how to solve those

Alternative approaches

• lease-based coordination using distributed consensus (Raft/Paxos): Instead of database row locks, executors acquire time-bound leases on jobs using a distributed coordination service like Consul, etcd, or Redis. When an executor picks a job, it acquires a lease (e.g., 5-minute TTL) that automatically expires if not renewed.

• queue-native deduplication with idempotency keys: Modern message queues like AWS SQS FIFO, Google Pub/Sub, or Apache Pulsar can be configured for exactly-once delivery guarantees. The scheduler can push jobs with deterministic message IDs, and the queue ensures no duplicates are delivered to executors.