Model Track Record

Weekly Projection Calibration

As of 2026-07-18 · source: _bmad-output/planning-artifacts/projection-band-calibration-backtest-1877.md

Every week we publish a projected floor-to-ceiling range plus a single point projection for every skill-position player, built walk-forward — the projection for week 8 has only ever seen weeks 1–7. We checked how often the published range actually covered what happened, on two independent held-out replays.

Pooled RMSE, our leak-free point projection vs a naive season-to-date trailing average (lower is better). Lower RMSE is better — our projection beats the naive baseline at every position.

QB bands under-cover the most (59.1% vs the 65% target) — the residual miss is mild and concentrated in the low tail (bench/DNP weeks), not a width bug. Effective sample size is roughly 68 independent week×position clusters per leg, not the raw row count — read the second decimal as suggestive, not a tight season-level claim.

Guillotine Survival Mechanism

As of 2026-07-19 · source: data/survival_skill_scoreboard.json

In guillotine leagues, the last-place team is eliminated every week — the format punishes a shallow roster harder than any other. Our guillotine draft board orders players by a projected floor (a realistic worst week), not average projected points. We backtested that ordering against the market's own ADP as the baseline to beat, across four seasons (2022, 2023, 2024, 2025), with both sides matched to the same information vintage — no lookahead on either side.

Positive in 2 of 4 backtested seasons (one-sided sign-test p=0.6875 — not significant at this sample size). Weakest week in the backtest: week 11, where the market baseline slightly out-lasted our board (-0.3088 weeks of alive-rate edge).

The entire claim rests on n_eff = 4 independent seasons. The 18 draft slots inside each season collapse to a few distinct survival outcomes (shared score realisation), so they add resolution, not independence. Survival edge was positive in 2/4 seasons (exact one-sided sign-test p = 0.6875); even a perfect sweep at this n is not significant at 0.05. The season-clustered 95% CI on the survival edge is [-3.8684, 2.1768] weeks and CROSSES zero; with so few clusters this CI is wide and should be read as directional, not a certified magnitude. These numbers are the board_order='published' (served, #2114) headline; the 'floor' board scores -1.0147wk vs this one's -0.7794wk on the same seasons/market — a reminder that floor-only evidence does not certify a served change.

We are not claiming our guillotine board currently outperforms the market: the primary survival-edge interval crosses zero, and the finish-rank-skill metric is currently negative and statistically significant. This is a mechanism we are testing in the open, including the metrics that don't clear our own bar yet.

Tested and closed

As of 2026-07-18 · source: data/qb_floor_v2_evidence_1798.json

Not every idea we test ships. Here's one we closed rather than arm, with the numbers that closed it.

How to read this page

• Everything above is leak-free by construction: every backtest here is checked to confirm the model never used information from the game, week, or season it is being graded on.
• Small samples are flagged, not hidden. The guillotine backtest spans four NFL seasons — four independent data points for anything season-level, not thousands. We report effective sample size, not just row count, and mark a result “not yet demonstrated” whenever its confidence interval crosses zero.
• No raw third-party proprietary values appear on this page. Every accuracy metric is our own model's output, graded against publicly available actual results.

For our always-on, week-by-week graded hit-rate ledger (start/sit rankings and streaming picks), see our Track Record page.