Distributed Scheduler - API design
How do we make the API idenmpotent so that jobs are executed only once? Let's dive deeper to understand how to design an idempotent API
Summary
Distributed job scheduler APIs require careful design around three fundamental concepts: resource hierarchy, idempotency, and temporal scheduling. The job represents the scheduling definition while runs represent individual executions, creating a parent-child relationship that affects API endpoint design.
Idempotency mechanisms prevent duplicate job creation when clients retry requests due to network failures - critical in distributed environments where partial failures are common. This typically involves idempotency keys that allow servers to recognize and handle duplicate submissions gracefully.
Temporal scheduling requires the API to distinguish between one-time jobs (executed at specific timestamp) and recurring jobs (following cron patterns). The API design must accommodate both scheduling types while maintaining clear request structure and avoiding ambiguity in job definitions.
Designed APIs
api_name: /v1/jobs,
{
method_name POST:
{
description: Submitting new jobs.,
body:
{
// only one of schedule_recurring and schedule_oneoff can be present
schedule_recurring:"cron syntax"
schedule_oneoff: "timestamp"
job_url: "https://MYJOB_EXECUTION_URL"
job_payload: {
// forced to POST
additional_params
}
}
},
response: 201,400,403,401
authentication: OAuth or IAM
} api_name: /v1/jobs/{jobId}/run/{runId}/,
{
method_name DELETE:
{
description: Delete or cancel job
},
response_body: {
// to maintain history it is always a soft delete.
"status": "CANCELED|DELETE"
}
response: 201,400,403,401
authentication: OAuth or IAM
}