Reliability Layer (Idempotency + Retry + DLQ)

Add an X-Idempotency-Replayed response header

Right now a client (or a support engineer reading a captured request/response pair) cannot tell, from the response alone, whether a charge was just executed or is a replay of a previous execution -- both look identical except for the attempts field, which a caller would have to independently know the "right" value for. A dedicated header turns "did this actually charge the customer again or not?" -- a question that comes up in exactly the stressful moments (a customer dispute, a support escalation, debugging a client's own retry logic) where a clear, explicit answer saves real back-and-forth.

Idempotency fingerprint depends on header key ordering

A false idempotency conflict (409 idempotency_key_reused) for a request that is, semantically, identical to the original is a hard failure a client cannot self-heal from by retrying -- retrying with the same key just gets the same 409 again. If the set of headers folded into the fingerprint ever arrives in a different order across two calls that are otherwise the same request (a very real possibility once headers are collected from a map anywhere upstream -- Go deliberately randomizes map iteration order), a legitimate retry gets permanently rejected instead of replayed, and the caller has no recovery path except picking a brand-new Idempotency-Key, defeating the entire point of supplying one.

DLQ job's first/last failure timestamps are swapped

FirstFailedAt/LastFailedAt are exactly the two fields an on-call engineer glances at first when triaging a dead-lettered job -- "how long has this been broken?" With the fields swapped, that duration (LastFailedAt - FirstFailedAt) comes out negative, which is either confusing (if a human reads it) or silently wrong (if a dashboard/alert computes "age of oldest failure" from it and gets nonsense). Neither is catastrophic on its own, but it's exactly the kind of subtly-wrong observability data that erodes trust in a monitoring dashboard over time.

Add a way to forget a stuck idempotency key

"This customer's charge is permanently stuck in a Conflict/Pending state and support needs it cleared now, not in 24 hours when the TTL expires" is a routine operator request, not an edge case -- a client bug that double-submits a mangled payload under a key it then reuses correctly, or a key that got orphaned by a crashed request that never called Complete/Fail, both leave a key in a state no amount of correct client behavior can recover from before its TTL. Today the only remediation is "wait out the TTL" (up to 24h by default) or "restart the process" (memory backend only, and it clears every key, not just the one that's stuck). An operator-facing way to forget a single key turns a multi-hour-wait escalation into an immediate fix.

In-memory janitor evicts active idempotency keys almost immediately

The entire point of an idempotency key is that it survives long enough for a retried request to find it. If the background janitor deletes keys before they expire instead of after, the at-most-once guarantee this whole project exists to provide breaks down for any request slower than one sweep interval (one minute by default): a retry that lands after the key was wrongly evicted sees Proceed instead of Replay/InProgress and re-executes the operation -- silently reintroducing the exact duplicate-charge risk this library is supposed to eliminate. The other half of the same bug (expired keys are never evicted at all) is a slow, unbounded memory leak in the opposite direction.

X-Request-ID silently missing from every response

The README advertises "every response carries X-Request-ID... so a single request's full retry/idempotency journey is traceable from one log grep." With the header silently dropped, a support engineer handed a customer-reported error has no correlation ID to grep server logs for -- they either have to reconstruct the request from timestamps and hope nothing else happened in that window, or ask the customer to reproduce the issue again. For an on-call engineer during an incident, that's the difference between finding the exact failing request in seconds versus minutes of log spelunking.

Redis-backed DLQ lists jobs newest-first, not oldest-first

GET /v1/dlq is the list an on-call engineer works through during an incident, and the documented (and generally correct) triage order is oldest-first -- the job that's been failing the longest is usually the most SLA-critical, and working newest-first means it gets addressed last, if at all, once the queue empties enough that someone stops looking. This bug only affects the Redis backend, which is specifically the one this project recommends for production (STORE_BACKEND=memory is the zero-setup dev default) -- so it's the textbook "worked fine in every local/dev test, broke specifically in the production configuration" pattern.

Add a Policy.OnRetry observability hook

Right now the only way to observe individual retry attempts from outside this package is to reimplement retry logic around it, or to infer attempt counts after the fact from a Result/Job's attempt history. Teams embedding pkg/retry in a different service (this package is explicitly designed to be go get-able standalone) routinely want per-attempt visibility -- custom metrics with their own labels, a structured log line their own logging pipeline expects, a trace span per attempt -- without forking the retry loop itself. Without a hook, every consumer either duplicates the loop or goes without that visibility.

Add a whole-operation Policy.OverallTimeout

Today the only way to bound how long a Do call can run in total is the caller's own ctx deadline, set independently, or MaxAttempts combined with a worst-case backoff calculation the caller has to do by hand every time a policy changes. For a request-handling path with its own SLA (e.g. an HTTP handler that must respond within N seconds regardless of how many attempts a downstream needs), that mismatch is a real risk: a policy tuned for "retry aggressively" can end up running far longer in the worst case than any caller actually intended, holding a connection/goroutine open the whole time. A Policy-level overall deadline makes "this whole retry operation, no matter how it's configured, will never run past N" an explicit, self-contained guarantee that travels with the policy itself.

MaxAttempts: 0 silently reports success without ever running the operation

This is, arguably, the single worst possible defect a retry-and-DLQ library aimed at payments-style reliability can have: under a specific, plausible misconfiguration, an operation silently never runs at all, and the caller is told it succeeded. A Policy{} zero value reaches Do/DoWithResult any time a caller constructs a policy from partially-populated config and forgets (or a bug elsewhere fails) to set MaxAttempts -- and instead of falling back to a sane default (as every other field already does), the operation is skipped entirely and reported as a clean success. For a charge, that's "the customer was told their payment went through and it never actually touched the payment processor" -- silent, undetectable by the caller, and it will never appear in the DLQ either, because nothing failed. This is precisely the "No silent data loss" guarantee the project's own README explicitly calls out, defeated by a single flipped comparison operator.

Retrying RoundTripper leaks a connection on every retry

An http.Response.Body that's never closed doesn't just leak memory -- Go's net/http transport can't return the underlying TCP connection to its connection pool until the body is closed, so a leaked body means a leaked connection. Under light, ad-hoc testing (a handful of manual requests) this is invisible; under real production load, with every retried request leaking one connection per discarded attempt, this exhausts the transport's connection pool and/or file descriptor limits -- exactly the failure mode most likely to strike during a downstream outage, which is also exactly when this package's retry logic is under the most load and doing the most retrying. This is the same class of failure the project's own RoundTripper doc comment specifically calls out avoiding ("drain and close it so its connection can be reused, rather than leaking it").

Redis-backed idempotency store falsely conflicts every retried request

This breaks the Redis backend's core promise -- the exact thing pkg/idempotency's README section on concurrency correctness spends the most words defending -- for every single request, not just an edge case. Any deployment running STORE_BACKEND=redis (the recommended, multi-replica-safe production configuration) would see every retried request rejected with 409 idempotency_key_reused, even though the retry is byte-for-byte identical to the original. A client that retries after a timeout (the single most common reason a client resends a request with the same Idempotency-Key) gets a hard, unrecoverable conflict instead of the replayed response it's supposed to get -- indistinguishable, from the client's point of view, from a real bug in their own request construction, which is exactly the kind of misleading error that burns hours of the wrong team's time during an incident.

Add a bulk "replay everything" DLQ endpoint

The existing POST /v1/dlq/{id}/replay only replays one job at a time. The single most common real-world reason a batch of jobs lands in the DLQ together is a shared root cause -- the downstream had an outage, a deploy introduced a bug, a dependency's certificate expired -- and once that root cause is fixed, an on-call engineer typically wants to clear the entire backlog it created in one action, not click "replay" N times by hand while babysitting each response. Without a bulk endpoint, recovering from an outage that dead-lettered hundreds of jobs means either writing an ad-hoc script against the existing single-job endpoint, or spending real minutes doing it by hand -- both add avoidable time-to-recovery on top of whatever caused the outage in the first place.