A route with multiple upstreams never uses any but the first

Nobody has started this yet — be first.

Business impact

This is the difference between "one upstream, with additional_upstream_base_urls unused" and actually being able to run more than one instance of an upstream behind a single route -- which is the entire point of adding the field in the first place. A route stuck always hitting its first configured upstream cannot be horizontally scaled by adding more upstream instances to its config; every added upstream is dead weight that never receives traffic.

Problem

RouteConfig.additional_upstream_base_urls exists (default []) so a route can list extra upstreams beyond its primary upstream_base_url, and app/proxy/forwarder.py already calls pick_upstream_base_url(route) to decide where each individual request actually goes. But is a stub -- it always returns , completely ignoring .

Current behavior

A route configured with upstream_base_url plus two entries in additional_upstream_base_urls (three upstreams total) still sends every single request to the primary upstream_base_url -- pick_upstream_base_url(route) never reads the extra list.

Expected behavior

When a route has additional_upstream_base_urls set, requests to that route are distributed round-robin across upstream_base_url plus every URL in additional_upstream_base_urls -- roughly evenly, in a consistent rotation, across many requests. Failover / health-aware routing and per-upstream circuit breakers are explicitly out of scope: RouteConfig.breaker_key still resolves to the single primary upstream_base_url for the whole route regardless of how many upstreams it has.

Steps to reproduce

pytest practice_tickets/tests/test_ticket_14_multi_upstream_round_robin.py -v.
Or manually: configure a route with additional_upstream_base_urls set to two extra upstreams, and send 30 requests through it.
Observe: every single request reaches only the primary upstream_base_url; the other two never receive traffic.

Why this matters

pick_upstream_base_url(route) is called once per request, freshly, with only the route object -- it does not currently have anywhere to keep state (like "which upstream did the last request for this route go to") between calls. Round robin needs per-route rotation state that survives across requests but stays scoped to one route.

Suggested approach

Think about where per-route rotation state could live so it survives across requests but stays scoped to one route -- look at how BreakerRegistry keeps one Breaker per upstream key across requests, as a pattern for "state associated with a route/upstream, created lazily, kept somewhere on app.state," not because this ticket needs a class shaped exactly like BreakerRegistry, but because it is already solving the closely related problem of where per-key state that must outlive a single request lives in this codebase.

Acceptance criteria

A route with upstream_base_url plus 2 entries in additional_upstream_base_urls (3 upstreams total), sent 30 requests, results in requests actually reaching all 3 upstreams, not just the first
The distribution is a rotation (round robin), not random -- given a fixed starting state, the sequence of chosen upstreams is deterministic and covers all 3 before repeating any one twice
A route with no additional_upstream_base_urls (the common case) behaves exactly as before, always the primary upstream

Verification

pytest practice_tickets/tests/test_ticket_14_multi_upstream_round_robin.py -v

Hints (0/2)

Try it without hints first — the reading is the exercise.

Working on this ticket

Work on a branch named for the ticket — that's what you'll submit.

Branch off your fork

$git checkout -b feat/multi-upstream-round-robin

Fix it and commit

Meet every acceptance criterion, and add a test that would have caught this.

Push the branch

$git push -u origin feat/multi-upstream-round-robin

Submit it below

Paste your fork URL and the branch name, with a short write-up of the root cause.

Submit your fix

Questions

Stuck on something?

Ask about anything unclear in the ticket — the maintainer and anyone who has solved it can answer. Please don't post full solutions.

No questions yet. If something in this ticket reads ambiguously, you are probably not the only one — ask.

The project

Dynamic API Gateway & Reverse Proxy

A FastAPI reverse proxy that streams every request and response without ever buffering a body in full, with hot-reloadable routing, per-route auth, Redis token-bucket rate limiting, and a per-upstream circuit breaker.

Fork on GitHub

Download

Setup

Dynamic API Gateway & Reverse Proxy

git clone <your-fork>
cd FastAPIDevTraining/gateway_proxy
cp .env.example .env   # adjust GATEWAY_ADMIN_TOKEN etc. if you like
docker compose up --build   # Redis + the gateway (uvicorn --reload) + 2 demo upstreams

The gateway is reachable at . Two demo upstreams come up alongside it — (low-latency JSON echo at , plus for the streaming-passthrough demo) and (, , for timeout and circuit-breaker demos) — and wires five example routes against them, one per feature. Verify the stack with (liveness only) and (pings Redis, which every rate-limited route depends on). Every endpoint (route CRUD, reload, persist, breaker snapshot) requires .