Nobody has started this yet — be first.
Business impact
Anything that needs to know "has every accepted event either landed in Postgres or definitely not landed" — an external drain-detection script, a deployment tool waiting for graceful shutdown to finish before killing a container, a test asserting zero data loss — has to check two signals together: queue_depth == 0 and events_in_flight == 0. During the shutdown drain path specifically, items pulled off the queue are not counted as in-flight, so there's a real window where both signals read zero while an event is still actively being flushed to Postgres. Anything relying on that combined signal to mean "fully drained, safe to proceed" can act too early — exactly the kind of race that shows up as "we lost one or two events on every deploy" and is maddening to reproduce on demand.
Problem
Anything checking "has every accepted event either landed in Postgres or definitely not landed" has to combine queue_depth == 0 with events_in_flight == 0. During the shutdown drain path specifically, items pulled off the queue are not counted as in-flight, so there is a real window where both signals read zero while an event is still actively being flushed.
Current behavior
The normal-operation branch of _collect_batch's while loop calls stats.enter_flight(1) immediately after removing an item from the queue; the stop_event.is_set() drain branch removes items via queue.get_nowait() + queue.task_done() but never calls stats.enter_flight(1).
Expected behavior
An item removed from the queue must be marked in-flight regardless of which branch removed it, and only un-marked once its batch has actually been flushed; queue_depth == 0 and events_in_flight == 0 must never both hold while a dequeued event is still unaccounted for.
Steps to reproduce
pytest practice_tickets/tests/test_ticket_15_shutdown_drain_race.py -v — sets stop_event, calls _collect_batch directly with items already queued, and checks events_in_flight immediately after the items are dequeued but before the (stalled) flush completes.Why this matters
This is a logical race across await points, not a memory-level data race (Python's asyncio has no preemptive threading) — but it's just as real: two branches of the same function are supposed to maintain the same invariant, and one of the two code paths that can remove an item from the queue simply doesn't uphold it. This is exactly the class of bug that a "does the happy path work" test won't catch, because shutdown/drain is, by definition, not the happy path.
Suggested approach
Compare the two branches inside the while loop in _collect_batch (app/flush.py) — the normal-operation branch (using asyncio.wait_for(queue.get(), ...)) and the drain branch (using queue.get_nowait()). Both remove an item from the queue. Which one updates stats, and which one doesn't?
Acceptance criteria
Verification
pytest practice_tickets/tests/test_ticket_15_shutdown_drain_race.py -v
Hints (0/2)
Try it without hints first — the reading is the exercise.
Working on this ticket
Work on a branch named for the ticket — that's what you'll submit.
Branch off your fork
Sign in to create it automatically, or do it yourself:
$git checkout -b fix/shutdown-drain-in-flight-raceFix it and commit
Meet every acceptance criterion, and add a test that would have caught this.
Push the branch
$git push -u origin fix/shutdown-drain-in-flight-raceSubmit it below
Paste your fork URL and the branch name, with a short write-up of the root cause.
Questions
Ask about anything unclear in the ticket — the maintainer and anyone who has solved it can answer. Please don't post full solutions.
Sign in to ask a question or reply.
Sign inhttp://localhost:8000localhost:5432curl localhost:8000/healthzcurl localhost:8000/readyzWork the tickets in practice_tickets/tickets/ (ordered easy → hard); each names one dedicated test in practice_tickets/tests/:
pytest practice_tickets/tests/test_ticket_01_persisted_count.py -v # a single ticket
./practice_tickets/run_tickets.sh # all 20, clean pass/fail summary
TICKET-17 needs a real Postgres to run at all (it checks which column a value lands in, which no fake/in-memory DB backend can meaningfully verify) — set TEST_DATABASE_URL=postgresql://<you>@localhost/ingest_test before running it or the full suite; without it, that one test is skipped, not failed.
Level 1
Fix a bug
Read existing behaviour, correct it.