The reaper reclaims runs that are still actively executing

Nobody has started this yet — be first.

Business impact

This bug inverts the entire purpose of the reaper. The reaper exists to recover work only after a worker has genuinely disappeared (crashed, network partition) -- it's the safety net for the rare case. As shipped, it does the opposite: it targets leases that are still comfortably within their validity window (runs a worker is right now actively and legitimately executing) and leaves truly abandoned/expired leases untouched forever. That means every currently-in-progress run is at constant risk of being handed to a second worker while the first is still processing it -- guaranteed duplicate execution on a much larger scale than the README's documented at-least-once tradeoff ever intended -- and a run whose worker genuinely crashed is never recovered at all.

Problem

pkg/queue/queue.go's ReclaimExpiredLeases selects rows to requeue with WHERE status IN ('leased', 'running') AND leased_until > now(). leased_until > now() means "the lease has NOT YET expired" -- the opposite of what a reaper reclaiming expired leases should target. The comparison operator is inverted; it should be leased_until < now().

Current behavior

A run with leased_until in the past keeps status "leased" after ReclaimExpiredLeases instead of becoming "pending", while a run with leased_until in the future gets reclaimed incorrectly.

Expected behavior

ReclaimExpiredLeases should only requeue runs whose leased_until has already passed. Runs with an active, not-yet-expired lease must be left completely alone.

Steps to reproduce

Insert a job_runs row with status='running', leased_until one hour in the future (an active lease).
Insert another with status='leased', leased_until one minute in the past (a genuinely expired lease, simulating a crashed worker).
Call ReclaimExpiredLeases.
Observe: the active-lease row gets reclaimed (wrong), the expired-lease row is left alone (also wrong).

Why this matters

A flipped comparison operator on a time boundary is one of the most dangerous classes of bug precisely because both directions compile, both look superficially reasonable, and the bug only manifests under concurrent load in ways that are easy to attribute to something else ("why did this webhook fire twice?") rather than trace back to the reaper.

Suggested approach

Think about what leased_until represents (the moment a lease stops being valid) and which side of "now" a lease needs to be on for it to be eligible for reclaim.

Acceptance criteria

A run with leased_until in the past is reclaimed (its status becomes pending)
A run with leased_until in the future is NOT touched by ReclaimExpiredLeases
go build ./..., go vet ./..., gofmt -l . stay clean

Verification

TEST_DATABASE_URL="postgresql://chronos:chronos@localhost:5432/chronos?sslmode=disable" go test ./tickets/tests/... -run TestTicket12 -v

Hints (0/2)

Try it without hints first — the reading is the exercise.

Working on this ticket

Work on a branch named for the ticket — that's what you'll submit.

Branch off your fork

$git checkout -b fix/reclaim-inverted-comparison

Fix it and commit

Meet every acceptance criterion, and add a test that would have caught this.

Push the branch

$git push -u origin fix/reclaim-inverted-comparison

Submit it below

Paste your fork URL and the branch name, with a short write-up of the root cause.

Submit your fix

Questions

Stuck on something?

Ask about anything unclear in the ticket — the maintainer and anyone who has solved it can answer. Please don't post full solutions.

No questions yet. If something in this ticket reads ambiguously, you are probably not the only one — ask.

The project

Chronos — Distributed Cron Scheduler

A "cron-as-a-service" job scheduler backed entirely by Postgres -- SKIP LOCKED leasing and advisory-lock leader election, no etcd/ZooKeeper/Consul -- twenty ways the coordination quietly breaks.

Fork on GitHub

Download

Setup

Chronos — Distributed Cron Scheduler

git clone <your-fork>
cd GolangDevTraining/chronos
docker compose up --build   # postgres + two app replicas + adminer, hot-reloading on save

Two replicas run on () and (), both against the same database; Adminer is at . Verify with and .