Chronos — Distributed Cron Scheduler

Add ?enabled= filter to GET /v1/jobs

Operators managing more than a handful of jobs need to quickly answer "what's currently disabled?" or "what's actually running?" without eyeballing a full unfiltered dump. Today GET /v1/jobs always returns everything, so anyone building tooling on top of the API (a dashboard, an alert that pages when too many jobs are disabled, a cleanup script) has to fetch and filter client-side -- fine at 10 jobs, painful and wasteful at 10,000.

HTTP 300 from a target is silently treated as success

A job's owner relies on chronos to tell the truth about whether their webhook actually did what it was supposed to. If a target returns an ambiguous or redirect-class status code and chronos silently marks it "succeeded," the job's owner has no signal that something needs attention -- no retry happens, nothing shows up as dead, nothing gets logged as unusual. This erodes trust in a scheduler specifically: the system is up and confidently wrong, which is worse for on-call because there's no alert to respond to.

Creating a job with both cron_expr and run_at returns a 500 instead of a 400

API consumers depend on clean 4xx errors to build correct client-side validation and debug their own integration quickly. When a client mistake (setting both scheduling fields) surfaces as an opaque 500 instead of a clear "these two fields are mutually exclusive" 400, the client's developer wastes time assuming it's a chronos outage or a database problem, files a support ticket, and on-call has to dig through logs to find a Postgres constraint violation a one-line validation check would have caught at the door. It also pollutes error-rate dashboards and alerting with 500s that aren't actually server failures.

Add ?status= filter to GET /v1/jobs/{id}/runs

The README explicitly calls out dead-letter inspection as a first-class workflow: record the full error so it's inspectable via the API, never just logged and dropped. In practice, a job with a long execution history buries the handful of dead runs someone actually cares about among hundreds of succeeded ones. Anyone triaging failures -- an on-call engineer, a support agent looking up why a customer's job stopped firing -- needs to filter straight to the failures, not page through everything and eyeball statuses.

Rescheduling a job via PATCH crashes if enabled is omitted

PATCH /v1/jobs/{id} is documented as supporting partial updates: enable/disable or reschedule, independently. A caller rescheduling a job -- the single most common PATCH use case, e.g. "change this job's cron expression" -- sends only {"cron_expr": "..."} and expects the job's enabled state to be left alone. Instead, every such request fails outright with a database constraint violation. This means the reschedule half of the documented PATCH functionality is completely broken for any caller that follows the API as documented, which is the normal, idiomatic way to call a PATCH endpoint.

Jobs silently get one extra retry beyond max_retries

max_retries is a contract: a job owner sets it expecting exactly that many attempts before chronos gives up and dead-letters the run for them to investigate. Silently granting one extra attempt beyond what was configured breaks that contract for every single job in the system, not just an edge case. For a job whose target_url triggers a paid API call, sends a customer notification, or charges a card, "one extra attempt beyond what I configured" is an unauthorized action the job owner explicitly tried to prevent by setting max_retries in the first place.

Add pagination to GET /v1/jobs

GET /v1/jobs today fetches and returns every single job in the system in one response, unbounded. That's fine for a demo with 3 seeded jobs; it's a real problem for any operator running chronos at scale with thousands of jobs -- every dashboard load, every polling script, every health check that touches this endpoint pays the cost of serializing and transferring the entire table, and that cost only grows. An unbounded list endpoint is a classic, easy-to-miss source of a production incident: it works fine for months and then one day the jobs table crosses a size threshold and the endpoint starts timing out or OOMing the process.

No cron job ever produces a run

This is the single most severe bug in this set: it silently disables the entire product. Chronos exists to run scheduled jobs; if the materializer never creates a job_run for a cron job's due occurrences, nothing is ever scheduled, ever, for any job, on any instance -- and there is no error anywhere. Tick() returns nil. Leader election is healthy. The HTTP API is healthy. /readyz reports a leader. Every dashboard looks green while the core function of the service has quietly stopped working entirely -- exactly the kind of bug that survives a superficial health check and is only discovered when a customer notices their job hasn't fired in a day.

Add cron.NextOccurrence to compute a job's next fire time

Anyone managing jobs through the API today has no way to answer "when will this job next run?" without doing cron-expression math in their head or in a separate tool. That's a small but constant friction for every operator-facing surface that could show it: a dashboard listing jobs alongside their next fire time, a "this job hasn't run recently, is that expected?" sanity check, or simply GET /v1/jobs being genuinely useful for a human glancing at it instead of just machine-consumable.

The reaper reclaims runs that are still actively executing

This bug inverts the entire purpose of the reaper. The reaper exists to recover work only after a worker has genuinely disappeared (crashed, network partition) -- it's the safety net for the rare case. As shipped, it does the opposite: it targets leases that are still comfortably within their validity window (runs a worker is right now actively and legitimately executing) and leaves truly abandoned/expired leases untouched forever. That means every currently-in-progress run is at constant risk of being handed to a second worker while the first is still processing it -- guaranteed duplicate execution on a much larger scale than the README's documented at-least-once tradeoff ever intended -- and a run whose worker genuinely crashed is never recovered at all.

Reclaimed runs never count toward max_retries

A job whose target never responds at all (hangs forever, or the network path is broken such that the worker's HTTP call never returns before the lease expires) is only ever recovered by the reaper's blind reclaim -- the worker itself never gets to call MarkRetry or MarkDead. If reclaiming doesn't increment attempt_count, that job retries forever: attempt_count stays frozen, it can never reach max_retries, and it never dead-letters. This is a silent resource leak with no ceiling -- a single misconfigured or unreachable target_url generates an unbounded stream of retries indefinitely, consuming worker capacity that should be going to healthy jobs, with nothing in the dead-letter queue to alert anyone that something is stuck.

Add Queue.HasActiveRun to detect overlapping executions

A job whose execution regularly takes longer than its own cron interval (a slow every-minute job that sometimes takes 90 seconds) will, today, happily pile up overlapping executions of itself: the materializer creates a new due occurrence on schedule regardless of whether the previous one has finished. For a job that isn't safely re-entrant on the target side, this means concurrent duplicate work against the same downstream system exactly when it's already under load -- the worst possible time to pile on more concurrent requests to it.

A shutdown/ticker scheduling race can leak the leadership lock

This is the subtlest bug in the set: it doesn't reproduce every time, which is exactly what makes it dangerous in production. A leader that steps down (graceful shutdown, deploy, rolling restart) will usually release the lock cleanly -- but on an unlucky timing coincidence between its own poll ticker and the shutdown signal, it can leak the lock instead, leaving the cluster with no cron-materialization leader until the leaked connection eventually times out or the process restarts again. Because it's intermittent, this is the kind of bug that passes staging and most production deploys, and then one day, during exactly the kind of high-traffic rolling restart where reliable failover matters most, no instance ever picks up leadership and every cron job silently stops materializing new runs.

Leader failover never happens -- the outgoing leader never unlocks

Leader election with automatic failover is the headline feature of this project -- the README's entire "kill the leader and watch failover" demo depends on it working. As shipped, it doesn't: every graceful shutdown (a deploy, a rolling restart, a scale-down) permanently leaks the leadership lock. Once that happens, no instance in the cluster can ever become leader again until someone notices cron jobs have stopped materializing and manually intervenes. This turns routine, expected operational events (every deploy) into the trigger for a total, silent outage of the scheduling function, with the API, health checks, and worker pool all reporting healthy the entire time.

WORKER_CONCURRENCY doesn't actually limit anything

WORKER_CONCURRENCY is the operator's only lever for controlling how much simultaneous load a single instance throws at downstream targets and how many goroutines/HTTP connections it opens at once. If it silently stops working, a burst of simultaneously-due work spawns one goroutine and one outbound HTTP request per due run, all at once, regardless of the configured limit. On a modest instance this can exhaust file descriptors, saturate the network, or hammer a downstream target hard enough to trip its rate limiting or take it down -- the exact failure mode WORKER_CONCURRENCY exists to prevent, happening silently because the config value is being read and stored correctly, just never enforced.

Add a linear backoff strategy alongside exponential

Exponential backoff is the right default for most jobs, but it's a poor fit for some: a job hitting a downstream system with a known, fixed recovery time (e.g. "this dependency's health check flaps for about 30 seconds after a deploy, then it's fine") is better served by evenly-spaced retries than by the rapidly-widening gaps exponential backoff produces. Without a configurable strategy, every job owner is forced into one retry shape regardless of what actually matches their target's behavior -- leading some to work around it by setting an artificially high max_retries just to get enough total retry attempts within their expected recovery window, which then also changes when the job gets dead-lettered.

Add API key authentication for the jobs API

Every write endpoint under /v1/jobs -- create, delete, enable/disable, reschedule, trigger -- is currently reachable by anyone who can reach the port, with zero authentication. For a local demo that's fine; for anything deployed beyond a fully trusted private network, it means anyone who can route to the service can create arbitrary jobs that POST to arbitrary URLs with arbitrary payloads (a server-side request forgery primitive), delete or disable existing jobs, or trigger jobs on demand. This is the kind of gap that's easy to overlook precisely because the demo/local experience works perfectly without it -- the risk only appears once someone actually deploys this somewhere reachable by more than themselves.

Add Postgres LISTEN/NOTIFY so pollers wake up immediately

Right now, a newly materialized job_run waits out the worker's IDLE_POLL_INTERVAL (default 3 seconds) before any instance notices it if all workers happened to just poll and find nothing. For most jobs this delay is invisible, but for POST /v1/jobs/{id}/trigger -- explicitly documented as the way to fire an immediate one-off run so a demo doesn't require waiting on a real cron interval -- a multi-second delay before it actually starts undercuts the whole point of the endpoint. At larger scale, shortening IDLE_POLL_INTERVAL to compensate means every idle instance hits the database far more often for no reason, trading latency for load in the wrong direction.