Nobody has started this yet — be first.
Business impact
Half-open exists specifically to send a trickle of probe traffic (by default, one request) at a just-recovered upstream, instead of slamming it with the full concurrent load the gateway was holding back -- an upstream that is recovering from an outage is often still fragile, and a sudden thundering herd of every previously-blocked request landing on it at once can knock it right back down, defeating the entire purpose of having a gradual recovery probe. Under concurrent traffic, that is exactly what happens today: as soon as the recovery window elapses, a burst of concurrent requests all get admitted as "probes" at once, regardless of half_open_max_probes.
Problem
Breaker.acquire()'s half-open branch checks self._half_open_probes_in_flight >= self._half_open_max_probes and, if there is room, does await asyncio.sleep(0) (yielding to the event loop) before incrementing the counter.
Current behavior
Under concurrent load, many requests each pass the _half_open_probes_in_flight >= _half_open_max_probes check while the counter is still 0 (none of them have incremented it yet), all yield at the await asyncio.sleep(0), and only then each increment it -- so a half_open_max_probes=1 breaker can admit far more than 1 concurrent probe.
Expected behavior
At most half_open_max_probes requests are admitted while a breaker is half-open, even when many requests arrive concurrently at the exact moment the breaker transitions from open to half-open.
Steps to reproduce
Why this matters
This is a genuine, reproducible race, not a theoretical one: the await asyncio.sleep(0) between the read of _half_open_probes_in_flight and the write to it is a real suspension point, so many concurrently scheduled tasks interleave their reads before any of them commits its write -- classic check-then-act-without-a-lock-around-both-steps. Removing the await would not be a fix on its own if the check is still outside the lock scope that covers it; the check-and-increment need to happen as one atomic unit with respect to every other concurrent caller.
Suggested approach
Compare this method to on_success()/on_failure(), which both do their entire read-check-write under a single with no in between the read and the write. 's half-open branch does not have that shape. Whatever fix you land on, the check-and-increment together need to be atomic with respect to every other concurrent call to -- and it should still be possible for a real request's call to happen without holding a lock across the actual network call that follows it (that part of the pipeline is in , not in at all, and is correctly unaffected by this bug).
Acceptance criteria
Verification
pytest practice_tickets/tests/test_ticket_15_breaker_half_open_race.py -v
Hints (0/2)
Try it without hints first — the reading is the exercise.
Working on this ticket
Work on a branch named for the ticket — that's what you'll submit.
Branch off your fork
Sign in to create it automatically, or do it yourself:
$git checkout -b fix/breaker-half-open-raceFix it and commit
Meet every acceptance criterion, and add a test that would have caught this.
Push the branch
$git push -u origin fix/breaker-half-open-raceSubmit it below
Paste your fork URL and the branch name, with a short write-up of the root cause.
Questions
Ask about anything unclear in the ticket — the maintainer and anyone who has solved it can answer. Please don't post full solutions.
Sign in to ask a question or reply.
Sign inasync with self._lock:awaitacquire()acquire()acquire()app/routers/proxy.pyBreakerhttp://localhost:8080fast_echo/echo/.../large/{size_mb}slow_upstream/delay/{s}/flaky/always-downroutes.yamlcurl localhost:8080/healthzcurl localhost:8080/readyz/admin/*Authorization: Bearer <GATEWAY_ADMIN_TOKEN>You do not need Docker running at all to work the tickets — the test suite fakes Redis with fakeredis (plus lupa for real Lua-script support), so everything runs against a plain virtualenv:
python3 -m venv .venv && .venv/bin/pip install -r requirements-dev.txt
.venv/bin/pytest practice_tickets/tests/test_ticket_09_breaker_threshold_off_by_one.py -v # a single ticket
./practice_tickets/run_tickets.sh # all 15, clean pass/fail summary
./practice_tickets/run_tickets.sh 06 09 15 # just a few, by number
All 15 tests fail at the start; as you fix each ticket in app/, its test (and only its test) should flip to passing. .venv/bin/pytest tests/ runs the project's own main suite — as of writing it shows 8 pre-existing failures caused by ticket bugs 02, 04, 06, and 09; fixing those tickets fixes those failures too, not a separate task.
Level 1
Fix a bug
Read existing behaviour, correct it.