High-Throughput Event Ingestion Service

GET /events/stats reports flush_workers=0 and batch_max_size=0

`GET /events/stats` is the one place an on-call engineer can check a running instance's actual configuration without going and finding its environment variables. Two of the fields that are supposed to answer "how is this instance configured right now" — `flush_workers` and `batch_max_size` — are hardcoded to 0 no matter what the process was actually started with, so the endpoint gives a confidently wrong answer to exactly the question it exists to answer.

Deduped events are counted as "persisted" in GET /events/stats

`events_persisted_total` and `events_deduped_total` on `GET /events/stats` exist specifically so someone can answer "how many events actually became new rows vs. how many were harmless retries that got deduped?" without querying Postgres directly. If deduped events get silently folded into the persisted counter, `events_persisted_total` runs hot (inflated by however many retries happened) while `events_deduped_total` stays permanently at zero — making it look like the idempotency mechanism isn't doing anything, right when someone is trying to verify it is.

/readyz reports "ready" even when most flush workers have crashed

GET /readyz exists so a load balancer / Kubernetes stops routing traffic to an instance that can't actually process what it accepts. If it reports "ready" as long as even one flush worker out of several is still alive, an instance where 3 of 4 flush workers have crashed keeps receiving 100% of its traffic slice, its ingest queue fills up far faster than the one surviving worker can drain, and every request starts getting 503s — all while the readiness probe keeps insisting everything is fine.

A single event's payload can be arbitrarily large

This service's entire memory-bounding guarantee (the README's "Backpressure, for real" section) rests on the assumption that every queued event record is roughly the same, small size. Right now EventIn.payload accepts a JSON object of any size at all, so a single client sending one event with a multi-megabyte payload can blow that assumption open — the per-event size limit the codebase already defines is currently not enforced.

device_id+sequence idempotency keys collide across different devices

This service supports two idempotency schemes: a client-supplied event_id UUID, or a (device_id, sequence) pair for clients that can't mint UUIDs. The second scheme's key is built by concatenating the two fields with no separator between them. Because both fields are variable-length, two completely different, unrelated events from two different devices can produce the identical event_key string — which means the second one silently vanishes, dropped by the ON CONFLICT DO NOTHING upsert as if it were a duplicate. This is common for any pair of devices whose IDs share a prefix relationship (e.g. "device1", "device12"). Events go missing with no error, no log line, and no way to tell from the API response that anything was lost.

POST /events/batch returns 503 even when most events were accepted

ingest_batch's own docstring (still accurate!) explains the intended contract: partial success returns 200 with a per-event accepted/rejected breakdown, and only a complete failure returns 503. The actual condition governing that decision now fires the 503 branch whenever any event in the batch was rejected — even if 999 out of 1000 were accepted. A client watching for 503 as its signal to back off will back off far more aggressively than the real queue state warrants, and a monitoring dashboard counting 503s as "total outage" moments will register one for every batch that merely brushed up against a momentarily-full queue.

A fully successful batch reports rejected_reason="queue_full"

BatchIngestResult.rejected_reason exists to tell a client why some of its events were rejected. A client (or an automated retry/alerting system) that checks rejected_reason to decide whether to back off would see "queue_full" on a batch where nothing was rejected at all — actively wrong information attached to a successful response. Anyone debugging "why does my monitoring say we're seeing queue_full errors when our own dashboards show 0 rejected events?" will lose real time chasing a field that's simply computed from the wrong variable.

One noisy device can fill the entire shared queue

The shared IngestQueue is sized generously, but nothing currently stops one device from claiming all of it. DeviceRateLimiter is already wired into both POST /events and POST /events/batch — both already call rate_limiter.allow(event.device_id) before enqueueing and reject with 429 when it returns False — but allow() always returns True, so a single noisy device can starve every other device sharing the same process's capacity, and there is no fairness protection at all today.

A steady trickle of events never triggers the time-based flush

The whole design of this ingestion service rests on "a batch flushes when EITHER it fills up OR batch_max_wait_seconds elapses, whichever comes first." The time-based half of that guarantee currently only holds if events stop arriving entirely for the full wait window. If events keep trickling in — individually spaced further apart than they'd need to be to fill a batch, but closer together than batch_max_wait_seconds — the wait timer effectively never expires. In production this means any customer whose event rate is below "fills a batch every batch_max_wait_seconds" but above "goes fully silent" can have events sit in the queue far longer than the configured batch_max_wait_seconds promises — potentially indefinitely.

IngestQueue.put() turns ANY error into an innocuous-looking 503

IngestQueue.put() is supposed to convert exactly one condition — the queue staying full for the entire timeout window — into a BackpressureError, which the router turns into a 503. The except clause currently catches any Exception, not just the asyncio.TimeoutError that asyncio.wait_for raises on a genuine timeout. If some other, unrelated bug ever raised an exception from inside this code path, it would be silently reclassified as ordinary backpressure — a 503, indistinguishable from the DB just being slow. Nobody gets paged, no error is logged as an error, and whoever's on call has no way to tell "the queue is genuinely full" apart from "something actually broke" from the outside.

POST /events/batch has no limit on how many events one call can submit

The entire point of the bounded IngestQueue is that memory is capped regardless of traffic — but that guarantee assumes individual requests stay reasonably sized. ingest_batch accepts a list of events with no upper bound on length, so a single POST /events/batch call can force the server to fully parse and validate an arbitrarily large JSON array into memory before the handler even runs, well before backpressure ever gets a chance to apply. Settings.max_batch_request_size already exists specifically to cap this, but nothing reads it.

No early warning before the queue hits hard backpressure

/readyz's binary ready/not-ready signal only tells an operator something is wrong at the same moment clients start getting hard 503s — there is no earlier warning that the queue is getting close to capacity. degraded already exists in /readyz's response body for exactly this purpose, but it's hardcoded False, so there is currently zero lead time between "queue has headroom" and "queue is full and rejecting requests."

Shutdown drain can report "fully drained" while an event is still mid-flush

Anything that needs to know "has every accepted event either landed in Postgres or definitely not landed" — an external drain-detection script, a deployment tool waiting for graceful shutdown to finish before killing a container, a test asserting zero data loss — has to check two signals together: queue_depth == 0 and events_in_flight == 0. During the shutdown drain path specifically, items pulled off the queue are not counted as in-flight, so there's a real window where both signals read zero while an event is still actively being flushed to Postgres. Anything relying on that combined signal to mean "fully drained, safe to proceed" can act too early — exactly the kind of race that shows up as "we lost one or two events on every deploy" and is maddening to reproduce on demand.

occurred_at defaults to a naive (timezone-less) local timestamp

Every event whose client doesn't supply occurred_at explicitly gets timestamped with datetime.now() — a naive datetime with no timezone attached, representing whatever the server process's local clock says. Events that do supply occurred_at get a timezone-aware datetime. These two kinds of timestamps end up mixed together in the same occurred_at::timestamptz[] array sent to Postgres on every batch flush; Postgres silently interprets a naive timestamp using the server's session timezone, not necessarily UTC and not necessarily the same timezone the process's wall clock is actually running in. The result is a systematic, silent timestamp skew for exactly the events a client trusted the server to timestamp correctly — and it's the kind of bug that can pass every test running in a UTC CI environment and only show up once a deployment lands somewhere that isn't UTC.

device_id and event_type are silently swapped on insert

Every event this service has ever ingested since this bug was introduced has its device_id and event_type columns swapped in the database. Nothing crashes: both columns are TEXT, both arrays are the same length, Postgres has no way to know the meaning of the values it's being handed doesn't match the column names they're going into. Any downstream consumer of this table — analytics grouping by device_id, an alert rule filtering on event_type = 'error', a support tool looking up "what has device X sent" — silently gets nonsense results that look superficially plausible, which makes this the kind of bug that can go undetected for a very long time and corrupt a correspondingly large amount of historical data before anyone notices something is off. This ticket requires a real Postgres to run at all — the bug is invisible to any test that only checks "did a row get inserted," it only shows up when you check which column a value landed in.

New events keep getting accepted after shutdown has already begun

Shutdown currently sets a stop event and lets flush workers drain whatever's in the queue, then the process exits — but nothing stops a brand-new HTTP request from being accepted onto the queue at any point during that window, including after the flush workers have already finished draining and are about to exit. An event accepted at exactly that moment sits in the queue with nobody left to pull it out before the process terminates — a narrow but real data-loss window.

Batches that exhaust all flush retries vanish without a trace

When a batch fails every one of its retry attempts, _flush_with_retry gives up and calls record_dropped(batch) — but that function is a no-op stub, so the batch's events are gone with nothing but a log line as any trace they ever existed. This is called out in the project's own README as a known, accepted limitation; closing it means an operator or automated job can inspect what was dropped instead of losing it irrecoverably.

/metrics doesn't aggregate across workers

docker-compose.prod.yml runs this service under 4 separate gunicorn worker processes, each with its own independent Prometheus metric objects — nothing shares them across processes. A single scrape of GET /metrics currently only reflects whichever one worker process happened to answer that connection, not a true sum across all 4. This is called out explicitly in the project README's known limitations, and matters for the same reason GET /events/stats has the same per-process blind spot: a dashboard built on either number under-reports by roughly 4x, depending on which worker answers.