How well does the model match reality?
Every EDHbots number is a model output. This page compares the model's predictions against real games logged with the free tracker, and it is published whether or not the numbers are flattering. If predicted win shares drift more than 15 percentage points from logged reality, that is a bug in the model, not a marketing problem.
validation corpus
10 under engine rev32 — the Calibration 01 physical series, below.
0 games under engine rev58 — the current engine's predictions are unscored.
Full corpus by engine version: 11 under rev47 · 10 under rev32 · 8 under rev48 · 4 under rev45 · 3 under rev44 · 1 under rev51 · 1 under rev57. Rows from earlier versions are kept and scored separately — never pooled with current predictions.
Below the 500-game threshold we consider honest to score against. Until the corpus is large enough, no accuracy claim is made here — log your games to build it.
After you log, we show the residual on the spot — winner match and clock — at the precision the row earned.
observed win rate by seat (4-player games)
mean logged game length: 8.1 turns
what the corpus has taught
Logged games validate; they never fit. No parameter moves to shrink error — games rank the ledger and name the escalations, humans diagnose, mechanics get built from card text, versions bump, and new games judge. Each closed loop so far:
10 physical games (TMNT–Omo, logged under rev32)
Found: the table went 10–0 at mean 7.1 turns against a registered 71.3% ±3.7 / mean 12.0 — right winner, right route, wrong clock.
Diagnosed (by a person): team-wide trigger payoffs (a counter on EACH connecting attacker) were compressed into one body's growth — named at the table, confirmed against card text.
Built: engine cal01 — team-wide grow rows for 62 cards; the input-artifact half got the snapshot freshness pipeline and the five-artifact registration rule.
Loop closed: re-registered 75.9% ±2.7 / mean 9.3 in advance of the next series (movement reported, not targeted — the mean moved 0.1 toward 7 and the residual was NOT tuned away).
Log a game to add to the corpus — attach your deck for a sharper score; commander-only rows count forever.
Calibration Report 01
TMNT vs. Tricky Terrain — ten physical games against a pre-registered prediction. Filed 2026-08-29 — the first external data in the project's history. Ten games is a story with numbers, not the scored aggregate; the aggregate starts at 500 games, below.
the registered prediction
Head-to-head, exact precon lists: Heroes in a Half Shell vs. Omo, Queen of Vesuva. Seed 302065220, 1,000 games, engine rev32, registered in conversation before game 3.
The registration as made stands and is scored as made. Its input artifacts are part of the story: the deployed card snapshot was stale (63.6% TMNT coverage shown, vs 93.3% on a fresh catalogue) and the reported ±3.7 implies ~575 decisive games, not 1,000. Registrations now pin five artifacts — lists as matched, iterations as run, coverage as shown, snapshot date, engine version — or they cannot be scored later.
the ten games
| # | winner | turn | ending | first player | status |
|---|---|---|---|---|---|
| 1 | TMNT | T7 | combat (commander damage would land T8) | TMNT | observed |
| 2 | TMNT | T8 | combat | TMNT | observed |
| 3 | TMNT | T6 | combat | TMNT | pre-registered |
| 4 | TMNT | T7 | combat (commander damage would land T8) | TMNT | pre-registered |
| 5 | TMNT | T7 | combat | TMNT | pre-registered |
| 6 | TMNT | T8 | combat | Omo | pre-registered |
| 7 | TMNT | T8 | combat | Omo | pre-registered |
| 8 | TMNT | T7 | combat | Omo | pre-registered |
| 9 | TMNT | T6 | combat | Omo | pre-registered |
| 10 | TMNT | T7 | commander damage | Omo | pre-registered |
Games 1–2 were played against an earlier registration that simulated the wrong opponent list; they are retained as observed rows. Games 3–10 are the clean pre-registered test. One pilot on both seats, straight rules, 40 life — a stated distortion the engine deliberately refuses to model. (2 observed, 8 pre-registered — the results are indistinguishable.)
observed vs. predicted
TMNT 10–0 · mean length 7.1 vs. 12.0 predicted · range T6–T8 · 9 combat endings, 1 commander damage.
Right winner. Right route — combat overwhelmingly, with commander damage live behind it. Wrong clock.
The product miss, ×1.7, decomposes into two named pieces, each with its fix or its guard:
- Input artifact: 12.0 → 9.4 turns on the true lists — the registration was made against a stale deployed card snapshot (63.6% TMNT coverage shown vs. 93.3% fresh). Fixed by the snapshot freshness pipeline and the import-loudness banner; guarded by the five-artifact registration rule.
- Engine miss, ×1.3: 9.4 predicted vs. 7.1 observed. Named candidate confirmed at the table: Menace on multiple bodies guaranteed connection through a blocker-light board, and every connection fired Heroes in a Half Shell's trigger — a +1/+1 counter on each connecting Mutant, Ninja, or Turtle, plus a draw. The engine expressed that payoff as growth on a single stand-in body; the real card compounds across the whole connecting team while refilling the hand. Addressed by the team-wide trigger rows (engine cal01); the solo-pilot confound is held open for the next series.
- “a route it may not actually take at a table” — the win-condition warning shipped on TMNT's result before game one — commander damage was flagged as a route behind the combat kill, and it finished one turn behind the combat kill in three of ten games.
- “grow the host's own power as a stand-in army” — the stated approximation for token and counter payoffs, on the results page before game one — the clock miss sits exactly inside it.
the loop, closed
The fix shipped as engine cal01: team-wide trigger growth for 62 cards in the snapshot. On this exact matchup and seed it moved the win share 70.5% → 75.9% (+5.4pp) and the mean 9.4 → 9.3 turns. Reported, not targeted: the mean moved 0.1 turns toward 7. The team-wide payoff is real — it is exactly the compounding the table saw — but it is not the clock's governor. No lever was adjusted to close the residual.
Re-registered, in advance, for the next physical series: 75.9% ±2.7 TMNT · mean 9.3 · median win T9 · seed 302065220 · waiting for the games.
re-registration artifacts: 1,000 (all decisive; 0 draws) · coverage TMNT 93.3% · Omo 81.8% · card data 2026-08-27 · engine cal01 · lists data/reference-decks/precon-tmnt-turtle-power.txt · precon-omo-tricky-terrain.txt, both 99+1 fully matched
rendered from the filed record (docs/calibration-report-01.md · docs/team-triggers.md, filed 2026-08-29) — aggregates computed from the pinned game rows, never hand-copied
Latest scored report
No scored report yet. The scoring pipeline runs once the corpus passes 500 games with commander-identified seats: for each logged pod we simulate the matchup with the production engine and score predicted win shares against outcomes (mean absolute error and Brier score, segmented by bracket). The pipeline and its thresholds are in the repository — nothing about this page is hand-tuned.
Method: predictions come from the same pod engine users run, with the adaptive interaction policy and default settings — no per-game tuning. Games whose decks the engine cannot identify are excluded and the exclusion count reported. Draws in simulation are excluded from predicted win shares exactly as they are in the product.
live engine rev58 · corpus stats live from the game log · case study from the filed record