Checks retry, backoff and timeout settings across a call chain to prevent retry storms. Computes exponential delay schedules with full, equal or decorrelated jitter, worst-case latency per layer, expected tries and request amplification. Flags inner timeouts that outlive the caller, retries of non-idempotent calls, retries on 4xx, ignored Retry-After and missing caps or deadlines. Includes a stdlib Python calculator with --self-test.
---
name: retry-backoff-policy-calculator
description: Checks retry, backoff and timeout settings across a call chain to prevent retry storms. Computes exponential delay schedules with full, equal or decorrelated jitter, worst-case latency per layer, expected tries and request amplification. Flags inner timeouts that outlive the caller, retries of non-idempotent calls, retries on 4xx, ignored Retry-After and missing caps or deadlines. Includes a stdlib Python calculator with --self-test.
---
# Retry and Backoff Policy Calculator
You help engineers set retry, backoff and timeout policies that make services more reliable instead of turning a small outage into a big one. You collect the settings of every hop in a call chain (browser, gateway, services, databases, third-party APIs), compute what they really do under failure, and propose a safer policy with numbers.
## Files in this skill
- `scripts/retry_policy.py` - reads a call-chain JSON and prints delay schedules, worst-case latency, expected tries, amplification and flags (Python 3 standard library only)
- `references/backoff-math.md` - formulas for backoff, jitter, worst-case time, expected tries, amplification and timeout nesting
- `references/retry-safety-guide.md` - what is safe to retry, idempotency keys, circuit breakers, retry budgets, common library settings
- `templates/retry-policy-review.md` - review report with current chain, findings and proposed policy
- `examples/checkout-chain.json` - a three-layer checkout chain with typical mistakes
## Workflow
### 1. Map the chain
Ask for each hop, outermost first: attempts (total tries), base delay, multiplier, max delay, jitter type, per-try timeout, overall deadline, what it retries on (status codes, "timeout", "network"), whether the call is idempotent, whether an idempotency key is sent, and whether Retry-After is honored. Read them from code or config (SDK clients, HTTP client retry adapters, Envoy/Istio routes, gRPC service config, job queues). Unknown values: use the library default and mark them as assumed.
### 2. Run the calculator
```
python3 scripts/retry_policy.py examples/checkout-chain.json
python3 scripts/retry_policy.py chain.json --failure-rate 0.5 --max-amplification 8
python3 scripts/retry_policy.py chain.json --json > retry-report.json
python3 scripts/retry_policy.py --self-test
```
Exit code 1 means high-severity findings, 2 means bad input.
### 3. Explain the findings
Order: RETRY_UNSAFE and AMPLIFICATION first (they cause duplicate side effects and outages), then INNER_OUTLIVES_CALLER, HOT_LOOP, NO_PER_TRY_TIMEOUT, then medium and low. Explain each with the numbers from the report, for example "one checkout can become 36 calls to the payments provider".
### 4. Propose a fixed policy
Use `references/backoff-math.md` and `references/retry-safety-guide.md`: retry at one layer, full jitter with a cap, inner worst case below the caller's per-try timeout, deadlines passed down, idempotency keys for writes, no retries on 4xx except 408/429, Retry-After honored, a retry budget or circuit breaker. Re-run the script on the proposed JSON and show before and after.
### 5. Write it up
Fill `templates/retry-policy-review.md` and give config snippets for the user's library.
## Example result
For `examples/checkout-chain.json` at 30% failure rate: worst-case amplification 36 calls per checkout (expected 2.74); the gateway layer's 9.3 s worst case outlives the web app's 8 s per-try timeout; the payments layer retries a non-idempotent charge 4 times without an idempotency key and retries 409; no jitter at the gateway; Retry-After ignored on 429/503.
## Rules
- Never recommend retrying a non-idempotent call without an idempotency key.
- Keep numbers honest: label library defaults and guesses as assumptions.
- Prefer fewer retries with good timeouts over many retries.
- Recommend load tests or fault injection to confirm the policy, not just the math.
FILE:references/backoff-math.md
# Backoff and Timeout Math
## Terms
- **attempts**: total tries including the first. attempts 3 = 1 try + 2 retries.
- **base**: first retry delay. **multiplier**: growth factor (usually 2). **cap**: max single delay.
- **per-try timeout**: how long one try may take before it is abandoned.
- **deadline**: total time budget for the call including all tries and delays.
## Delay before retry k (k = 1 .. attempts-1)
d(k) = min(cap, base x multiplier^(k-1))
| Jitter | Delay | Range | Mean |
|---|---|---|---|
| none | d | d | d |
| full | random(0, d) | 0 .. d | d/2 |
| equal | d/2 + random(0, d/2) | d/2 .. d | 0.75 d |
| decorrelated | min(cap, random(base, 3 x previous)) | base .. min(cap, base x 3^k) | simulated |
Full jitter spreads load best and is the default recommendation. Equal jitter keeps a minimum wait. Decorrelated jitter adapts to the previous delay and works well for long-running clients.
## Worst-case duration of one call
worst = attempts x per_try_timeout + sum of max delays, then cut at the deadline if one is set.
Example: 3 tries x 1 s + 100 ms + 200 ms = 3.3 s.
## Expected tries
With an independent failure probability p per try and n attempts:
E[tries] = 1 + p + p^2 + ... + p^(n-1)
p = 0.3, n = 3 -> 1.39. During a full outage p = 1 and E = n.
## Amplification across layers
If layer A retries 3 times and calls B, which retries 3 times and calls C, which retries 4 times, one user request can become 3 x 3 x 4 = 36 calls to C during an outage. This is a retry storm: the retries keep the failing service overloaded, so it cannot recover.
Rules of thumb:
- Retry at one layer only, usually the one closest to the user or the one that owns the idempotency key.
- Inner layers: attempts 1 or 2 with a short deadline.
- Keep worst-case amplification at or below about 10.
- Add a **retry budget**: retries may be at most 10-20% of requests over a sliding window; past that, fail fast.
## Timeout nesting
Each layer's worst case must fit inside its caller's per-try timeout, minus a margin for network and work:
inner_worst <= outer_per_try_timeout - margin
Otherwise the caller abandons the try and retries while the inner layer is still working, which doubles load. Pass a deadline down the chain (for example a `grpc-timeout` or `X-Request-Deadline` header) and stop when it is used up.
## Retry-After
For 429 and 503, servers may send `Retry-After: <seconds>` or an HTTP date. Wait at least that long (use max(Retry-After, computed backoff)), and give up if it is longer than the remaining deadline.
FILE:references/retry-safety-guide.md
# Retry Safety Guide
## What to retry
| Outcome | Retry? | Notes |
|---|---|---|
| Connection refused / reset before the request was sent | Yes | Safe for any method |
| Timeout after the request was sent | Only if idempotent or with an idempotency key | The server may have done the work |
| 408 Request Timeout | Yes, with backoff | |
| 429 Too Many Requests | Yes, after Retry-After | Never faster than the server asks |
| 500 Internal Server Error | Sometimes | Only if the error is known to be transient |
| 502 / 503 / 504 | Yes, with backoff and jitter | Honor Retry-After on 503 |
| 400 / 401 / 403 / 404 / 405 / 410 / 422 | No | Same request fails the same way; fix the request, refresh auth once at most |
| 409 Conflict | Usually no | Exception: an idempotency key that is still in progress, retry after a short wait |
## Idempotency
- GET, HEAD, OPTIONS, PUT and DELETE are idempotent by HTTP semantics, if the server implements them correctly.
- POST and PATCH are not. To retry them, send an **idempotency key** (a UUID per logical operation, e.g. `Idempotency-Key` header) that the server stores with the result for 24 hours or more. Generate the key once, before the first try, and reuse it for every retry.
- Payments, emails, SMS, order creation and stock reservation must never be retried without a key.
## Client checklist
1. Per-try timeout set and shorter than the caller's.
2. Overall deadline set and passed down the chain.
3. Exponential backoff with a cap and full jitter.
4. attempts <= 3 at the edge, 1-2 inside.
5. Retry only on the table's "Yes" rows.
6. Idempotency key on every retried non-idempotent call.
7. Retry budget or circuit breaker so retries stop during an outage.
8. Metrics: tries per request, retry rate, give-up count, by dependency.
9. Log the final error with the attempt count, not every attempt at error level.
## Circuit breaker basics
- Closed: calls flow; count failures in a rolling window.
- Open: after for example 50% failures over at least 20 calls, fail fast for 30 s.
- Half-open: let a few trial calls through; close if they succeed.
A breaker and a retry budget together stop retry storms from deepening an outage.
## Common library settings
- AWS SDKs: standard mode = 3 attempts, full jitter, base 1 s for throttling; adaptive mode adds client-side rate limiting.
- gRPC retry policy: maxAttempts, initialBackoff, maxBackoff, backoffMultiplier, retryableStatusCodes (UNAVAILABLE is the usual one); retries need hedging/transparent retry rules for non-idempotent methods.
- Envoy / Istio: retry_on: 5xx,reset,connect-failure; num_retries; per_try_timeout; set route timeout >= per_try_timeout x (num_retries + 1).
- Python `urllib3.Retry(total=3, backoff_factor=0.5, status_forcelist=[502,503,504], respect_retry_after_header=True, allowed_methods=[...])`.
Check the library docs for the version you use; defaults change.
FILE:templates/retry-policy-review.md
# Retry Policy Review - <service / call chain> - <date>
**Reviewer:** <name> **Input:** <chain JSON / config files> **Failure rate assumed:** <30%>
## Chain
| # | Caller -> callee | Attempts | Backoff (base / x / cap) | Jitter | Per-try timeout | Deadline | Worst case |
|---|---|---|---|---|---|---|---|
| 1 | | | | | | | |
**Worst-case amplification:** <36> calls per request (target <= 10)
**Expected amplification at <p>:** <2.7>
## Findings
| Severity | Layer | Code | Problem | Fix |
|---|---|---|---|---|
| high | | | | |
## Proposed policy
| # | Caller -> callee | Attempts | Backoff | Jitter | Per-try timeout | Deadline | Retry on |
|---|---|---|---|---|---|---|---|
| 1 | | | | full | | | |
**New worst-case amplification:** <n> **New worst-case latency:** <s>
## Follow-ups
- [ ] Idempotency keys for: <calls>
- [ ] Retry budget / circuit breaker for: <dependency>
- [ ] Dashboards: tries per request, retry rate, give-ups
FILE:examples/checkout-chain.json
{
"name": "checkout: browser -> api gateway -> orders service -> payments provider",
"failure_rate": 0.3,
"max_amplification": 10,
"layers": [
{
"name": "web app -> api gateway",
"attempts": 3,
"base_delay_ms": 200,
"multiplier": 2,
"max_delay_ms": 2000,
"jitter": "full",
"per_try_timeout_ms": 8000,
"deadline_ms": 20000,
"retry_on": ["network", 502, 503, 504],
"idempotent": true,
"respect_retry_after": true
},
{
"name": "api gateway -> orders service",
"attempts": 3,
"base_delay_ms": 100,
"multiplier": 2,
"max_delay_ms": 1000,
"jitter": "none",
"per_try_timeout_ms": 3000,
"retry_on": [500, 502, 503, 504, "timeout"],
"idempotent": true
},
{
"name": "orders service -> payments provider (create charge)",
"attempts": 4,
"base_delay_ms": 250,
"multiplier": 2,
"jitter": "equal",
"per_try_timeout_ms": 2500,
"deadline_ms": 9000,
"retry_on": [409, 429, 500, 503, "timeout"],
"idempotent": false,
"idempotency_key": false,
"respect_retry_after": false
}
]
}
FILE:scripts/retry_policy.py
#!/usr/bin/env python3
"""Retry and backoff policy calculator (Python 3 standard library only).
Input: a JSON file describing a call chain, outermost caller first:
{"name": "checkout", "failure_rate": 0.3, "max_amplification": 10,
"layers": [
{"name": "web -> api", "attempts": 3, "base_delay_ms": 100, "multiplier": 2,
"max_delay_ms": 2000, "jitter": "full", "per_try_timeout_ms": 5000,
"deadline_ms": 12000, "retry_on": [502, 503, 504, "timeout"],
"idempotent": true, "respect_retry_after": true}, ...]}
attempts = total tries including the first one (attempts 1 = no retries).
jitter: none | full | equal | decorrelated. Delay before retry k (k = 1..attempts-1):
none d = min(cap, base * multiplier^(k-1))
full U(0, d) equal d/2 + U(0, d/2)
decorrelated min(cap, U(base, 3 * previous)) (AWS style; expected value simulated, seed 7)
Reports per layer: delay schedule (min / expected / max), worst-case time for one call
(attempts x per-try timeout + max delays, cut at deadline), expected tries at the given
failure rate, and for the chain: worst-case and expected request amplification at the
innermost dependency. Flags:
NO_JITTER, NO_CAP, HOT_LOOP (base 0), TOO_MANY_ATTEMPTS (> 5), NO_PER_TRY_TIMEOUT,
NO_DEADLINE, DEADLINE_TOO_SHORT (deadline < one per-try timeout), INNER_OUTLIVES_CALLER
(worst case > the outer layer's per-try timeout), RETRY_UNSAFE (non-idempotent with
retries and no idempotency key), RETRY_ON_4XX (retrying 400/401/403/404/409/422),
IGNORES_RETRY_AFTER (retries 429/503 without honoring Retry-After), AMPLIFICATION.
Usage:
python3 retry_policy.py chain.json
python3 retry_policy.py chain.json --failure-rate 0.5 --json
python3 retry_policy.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import json
import random
import sys
JITTERS = ("none", "full", "equal", "decorrelated")
BAD_4XX = {400, 401, 403, 404, 405, 409, 410, 422}
SEVERITY = {"INNER_OUTLIVES_CALLER": "high", "RETRY_UNSAFE": "high", "AMPLIFICATION": "high",
"HOT_LOOP": "high", "NO_PER_TRY_TIMEOUT": "high", "RETRY_ON_4XX": "medium",
"NO_JITTER": "medium", "NO_CAP": "medium", "IGNORES_RETRY_AFTER": "medium",
"NO_DEADLINE": "medium", "DEADLINE_TOO_SHORT": "medium", "TOO_MANY_ATTEMPTS": "low"}
class PolicyError(ValueError):
pass
def num(layer, key, default=None, minimum=0):
v = layer.get(key, default)
if v is None:
return None
if isinstance(v, bool) or not isinstance(v, (int, float)):
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be a number")
if v < minimum:
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be >= {minimum}")
return v
def delays(layer):
"""Return [(min, expected, max)] in ms for each retry."""
n = int(layer["attempts"])
base, mult, cap = layer["base_delay_ms"], layer["multiplier"], layer["max_delay_ms"]
capf = (lambda x: x) if cap is None else (lambda x: min(cap, x))
j = layer["jitter"]
out = []
if j == "decorrelated":
rng = random.Random(7)
runs = 20000
sums = [0.0] * (n - 1)
for _ in range(runs):
prev = base
for k in range(n - 1):
prev = capf(rng.uniform(base, prev * 3)) if base > 0 else 0.0
sums[k] += prev
hi = base
for k in range(n - 1):
hi = capf(hi * 3)
out.append((capf(base), sums[k] / runs, hi))
return out
for k in range(n - 1):
d = capf(base * mult ** k)
if j == "none":
out.append((d, d, d))
elif j == "full":
out.append((0.0, d / 2, d))
else:
out.append((d / 2, d * 0.75, d))
return out
def normalize(raw):
layers = raw.get("layers") if isinstance(raw, dict) else None
if not layers or not isinstance(layers, list):
raise PolicyError("JSON must have a non-empty 'layers' list (outermost caller first)")
out = []
for i, l in enumerate(layers):
if not isinstance(l, dict):
raise PolicyError(f"layer {i} must be an object")
l = dict(l)
l.setdefault("name", f"layer {i + 1}")
a = num(l, "attempts", 1, 1)
if int(a) != a:
raise PolicyError(f"layer {l['name']!r}: attempts must be a whole number")
l["attempts"] = int(a)
l["base_delay_ms"] = num(l, "base_delay_ms", 0)
l["multiplier"] = num(l, "multiplier", 2, 1)
l["max_delay_ms"] = num(l, "max_delay_ms", None)
l["per_try_timeout_ms"] = num(l, "per_try_timeout_ms", None, 1)
l["deadline_ms"] = num(l, "deadline_ms", None, 1)
j = str(l.get("jitter", "none")).lower()
if j not in JITTERS:
raise PolicyError(f"layer {l['name']!r}: jitter must be one of {', '.join(JITTERS)}")
l["jitter"] = j
l["retry_on"] = list(l.get("retry_on", []))
l["idempotent"] = bool(l.get("idempotent", True))
l["idempotency_key"] = bool(l.get("idempotency_key", False))
l["respect_retry_after"] = bool(l.get("respect_retry_after", False))
out.append(l)
return out
def expected_tries(n, p):
if p >= 1:
return float(n)
return sum(p ** k for k in range(n))
def analyze(raw, failure_rate=None, max_amp=None):
layers = normalize(raw)
p = failure_rate if failure_rate is not None else raw.get("failure_rate", 0.2)
if not 0 <= p <= 1:
raise PolicyError("failure rate must be between 0 and 1")
max_amp = max_amp if max_amp is not None else raw.get("max_amplification", 10)
res = {"name": raw.get("name", "retry policy"), "failure_rate": p, "max_amplification": max_amp, "layers": []}
worst_amp, exp_amp = 1, 1.0
for i, l in enumerate(layers):
n = l["attempts"]
ds = delays(l)
flags = []
retries = n > 1
if retries and l["jitter"] == "none":
flags.append(("NO_JITTER", "all clients retry at the same moments; use full or decorrelated jitter"))
if retries and l["max_delay_ms"] is None and n > 3:
flags.append(("NO_CAP", "no max_delay_ms; exponential delays grow without bound"))
if retries and l["base_delay_ms"] == 0:
flags.append(("HOT_LOOP", "base_delay_ms is 0; retries fire immediately and hammer the dependency"))
if n > 5:
flags.append(("TOO_MANY_ATTEMPTS", f"{n} attempts; 3 (2 retries) is usually enough"))
if l["per_try_timeout_ms"] is None:
flags.append(("NO_PER_TRY_TIMEOUT", "no per-try timeout; one hung call can block every retry"))
if l["deadline_ms"] is None and retries:
flags.append(("NO_DEADLINE", "no overall deadline across attempts"))
if l["deadline_ms"] is not None and l["per_try_timeout_ms"] and l["deadline_ms"] < l["per_try_timeout_ms"]:
flags.append(("DEADLINE_TOO_SHORT", "deadline is shorter than a single try"))
if retries and not l["idempotent"] and not l["idempotency_key"]:
flags.append(("RETRY_UNSAFE", "retries a non-idempotent call without an idempotency key (double charge / duplicate write risk)"))
bad = sorted(c for c in l["retry_on"] if isinstance(c, int) and c in BAD_4XX)
if retries and bad:
flags.append(("RETRY_ON_4XX", f"retries client errors {bad}; they will fail the same way again"))
if retries and any(c in (429, 503) for c in l["retry_on"]) and not l["respect_retry_after"]:
flags.append(("IGNORES_RETRY_AFTER", "retries 429/503 but ignores the Retry-After header"))
max_delay_total = sum(d[2] for d in ds)
exp_delay_total = sum(d[1] for d in ds)
worst = None
if l["per_try_timeout_ms"] is not None:
worst = n * l["per_try_timeout_ms"] + max_delay_total
if l["deadline_ms"] is not None:
worst = min(worst, l["deadline_ms"])
elif l["deadline_ms"] is not None:
worst = l["deadline_ms"]
if i > 0:
parent = layers[i - 1]
pt = parent["per_try_timeout_ms"]
if pt is not None and (worst is None or worst > pt):
flags.append(("INNER_OUTLIVES_CALLER", f"worst case {fmt_ms(worst)} > caller per-try timeout {fmt_ms(pt)}; the caller gives up and retries while this layer is still working"))
et = expected_tries(n, p)
worst_amp *= n
exp_amp *= et
res["layers"].append({
"name": l["name"], "attempts": n, "jitter": l["jitter"],
"delays_ms": [[round(a, 1), round(b, 1), round(c, 1)] for a, b, c in ds],
"max_delay_total_ms": round(max_delay_total, 1), "expected_delay_total_ms": round(exp_delay_total, 1),
"per_try_timeout_ms": l["per_try_timeout_ms"], "deadline_ms": l["deadline_ms"],
"worst_case_ms": None if worst is None else round(worst, 1),
"expected_tries": round(et, 3), "cumulative_worst_amplification": worst_amp,
"cumulative_expected_amplification": round(exp_amp, 3),
"flags": [{"code": c, "severity": SEVERITY[c], "detail": d} for c, d in flags]})
if worst_amp > max_amp:
res["layers"][-1]["flags"].append({"code": "AMPLIFICATION", "severity": "high",
"detail": f"one top-level request can become {worst_amp} calls to the innermost dependency during an outage (limit {max_amp}); retry at one layer only or add a retry budget"})
res["worst_amplification"] = worst_amp
res["expected_amplification"] = round(exp_amp, 3)
res["high_flags"] = sum(1 for l in res["layers"] for f in l["flags"] if f["severity"] == "high")
return res
def fmt_ms(v):
if v is None:
return "unbounded"
if v >= 1000:
s = f"{v / 1000:.2f}".rstrip("0").rstrip(".")
return s + " s"
return f"{v:g} ms"
def to_markdown(r):
out = [f"# Retry policy report: {r['name']}", "",
f"Failure rate used for expected values: {r['failure_rate']:.0%}", "",
"| Layer | Attempts | Jitter | Max total delay | Per-try timeout | Deadline | Worst case | Expected tries |",
"|---|---|---|---|---|---|---|---|"]
for l in r["layers"]:
out.append(f"| {l['name']} | {l['attempts']} | {l['jitter']} | {fmt_ms(l['max_delay_total_ms'])} | "
f"{fmt_ms(l['per_try_timeout_ms'])} | {fmt_ms(l['deadline_ms'])} | {fmt_ms(l['worst_case_ms'])} | {l['expected_tries']:g} |")
out += ["", "## Delay schedule (min / expected / max)"]
for l in r["layers"]:
if not l["delays_ms"]:
out.append(f"- {l['name']}: no retries")
continue
parts = [f"retry {k + 1}: {fmt_ms(a)} / {fmt_ms(b)} / {fmt_ms(c)}" for k, (a, b, c) in enumerate(l["delays_ms"])]
out.append(f"- {l['name']}: " + "; ".join(parts))
out += ["", "## Amplification at the innermost dependency",
f"- Worst case (every try fails): {r['worst_amplification']} calls per top-level request",
f"- Expected at {r['failure_rate']:.0%} failure rate: {r['expected_amplification']:g} calls per top-level request", "",
"## Findings"]
n = 0
for sev in ("high", "medium", "low"):
for l in r["layers"]:
for f in l["flags"]:
if f["severity"] == sev:
n += 1
out.append(f"- **{sev.upper()} {f['code']}** ({l['name']}): {f['detail']}")
if not n:
out.append("- No findings.")
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
def one(**kw):
base = {"name": "x", "attempts": 3, "base_delay_ms": 100, "multiplier": 2, "max_delay_ms": 1000,
"jitter": "full", "per_try_timeout_ms": 1000, "deadline_ms": 5000, "retry_on": [503],
"respect_retry_after": True}
base.update(kw)
return base
def codes(r, i=0):
return {f["code"] for f in r["layers"][i]["flags"]}
d = delays(normalize({"layers": [one(jitter="none", attempts=5)]})[0])
ok("exponential no jitter 100,200,400,800", [x[2] for x in d] == [100, 200, 400, 800])
d = delays(normalize({"layers": [one(jitter="none", attempts=6)]})[0])
ok("cap applied at 1000", d[-1][2] == 1000 and d[-2][2] == 800)
d = delays(normalize({"layers": [one(jitter="full")]})[0])
ok("full jitter mean half", d[0] == (0.0, 50.0, 100))
d = delays(normalize({"layers": [one(jitter="equal")]})[0])
ok("equal jitter 50..100 mean 75", d[0] == (50.0, 75.0, 100))
d = delays(normalize({"layers": [one(jitter="decorrelated", attempts=4)]})[0])
ok("decorrelated within bounds", all(a <= b <= c for a, b, c in d) and d[0][2] == 300 and d[2][2] == 1000)
r = analyze({"layers": [one()]})
ok("clean policy has no flags", not codes(r))
ok("worst case = 3x1000 + 100 + 200", r["layers"][0]["worst_case_ms"] == 3300)
r = analyze({"layers": [one(deadline_ms=2000)]})
ok("deadline caps worst case", r["layers"][0]["worst_case_ms"] == 2000)
ok("expected tries p=0.5 n=3 is 1.75", expected_tries(3, 0.5) == 1.75 and expected_tries(3, 1) == 3)
ok("no jitter flagged", "NO_JITTER" in codes(analyze({"layers": [one(jitter="none")]})))
ok("hot loop flagged", "HOT_LOOP" in codes(analyze({"layers": [one(base_delay_ms=0)]})))
ok("no cap flagged with 4+ attempts", "NO_CAP" in codes(analyze({"layers": [one(max_delay_ms=None, attempts=4)]})))
ok("unsafe retry flagged", "RETRY_UNSAFE" in codes(analyze({"layers": [one(idempotent=False)]})))
ok("idempotency key makes it safe", "RETRY_UNSAFE" not in codes(analyze({"layers": [one(idempotent=False, idempotency_key=True)]})))
ok("4xx retry flagged", "RETRY_ON_4XX" in codes(analyze({"layers": [one(retry_on=[400, 503])]})))
ok("Retry-After ignored flagged", "IGNORES_RETRY_AFTER" in codes(analyze({"layers": [one(retry_on=[429], respect_retry_after=False)]})))
ok("single attempt is not a retry policy", not codes(analyze({"layers": [one(attempts=1, jitter="none", deadline_ms=None)]})))
r = analyze({"layers": [one(per_try_timeout_ms=2000, deadline_ms=None), one(per_try_timeout_ms=1000, deadline_ms=None)]})
ok("inner outlives caller flagged", "INNER_OUTLIVES_CALLER" in codes(r, 1))
r = analyze({"layers": [one(per_try_timeout_ms=4000), one(per_try_timeout_ms=1000, deadline_ms=3500)]})
ok("inner within caller timeout ok", "INNER_OUTLIVES_CALLER" not in codes(r, 1))
r = analyze({"layers": [one(), one(), one()], "max_amplification": 10})
ok("3x3x3 amplification 27 flagged", r["worst_amplification"] == 27 and "AMPLIFICATION" in codes(r, 2))
r = analyze({"layers": [one(), one()]}, failure_rate=0.5)
ok("expected amplification 1.75^2", r["expected_amplification"] == 3.062)
for bad, label in ((({"layers": []}), "empty layers rejected"), (({"layers": [one(jitter="random")]}), "bad jitter rejected"),
(({"layers": [one(attempts=0)]}), "zero attempts rejected"), (({"layers": [one(attempts=2.5)]}), "fractional attempts rejected")):
try:
analyze(bad)
ok(label, False)
except PolicyError:
ok(label, True)
ok("fmt_ms", fmt_ms(1500) == "1.5 s" and fmt_ms(250) == "250 ms" and fmt_ms(None) == "unbounded")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Check retry/backoff/timeout policies across a call chain.")
ap.add_argument("file", nargs="?", help="chain JSON ('-' for stdin)")
ap.add_argument("--failure-rate", type=float, help="probability that one try fails (default from file or 0.2)")
ap.add_argument("--max-amplification", type=float, help="flag chains above this worst-case amplification (default 10)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("chain JSON required (or --self-test)")
try:
raw = json.load(sys.stdin if a.file == "-" else open(a.file, encoding="utf-8"))
r = analyze(raw, a.failure_rate, a.max_amplification)
except (OSError, PolicyError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["high_flags"] else 0
if __name__ == "__main__":
sys.exit(main())