NotionBench: making 890 agent rollouts legible

Notion shipped a developer platform in May 2026 and said it was built for coding agents. I wanted to know whether that was true, so I wrote 38 tasks across every programmable surface, ran eight agent setups through each task three times, and had a program grade the lot. The benchmark was the easy half. The hard half was the page.

Most eval sites ship a leaderboard sorted by score. On this run the top six configurations finished within five points of each other, and 21 of the 38 tasks were solved by every setup on every trial. Ranking by score would have told a reader almost nothing. The page had to answer four different questions: which setups differ, how consistently, at what cost, and where they got stuck.

There was a second audience with a second problem. The grid is 912 cells and took about three days to run across subscription accounts with usage windows. While it ran, I needed to see at a glance whether a config was working, cooling down, quietly broken, or halted by a bad verifier. So the same page had to work as a private monitoring dashboard before it became a public report.

I audited the four benchmark sites I was borrowing from: Supabase Evals, Terminal-Bench, MCPMark, and tau-bench. Each one publishes a score column first. Almost none put cost beside score, and none of them separated "can solve it" from "solves it every time". Single-trial scores hide consistency by construction, and that is exactly the dimension where these eight setups turned out to differ.

The first user was me, for three days. I designed the page against seeded mock fixtures before any real data existed, then watched the live version while the run executed. Every change in the live section came from a moment when the dashboard failed to tell me something: a config sitting in a rate-limit cooldown that read as stalled, two real verifier bugs that each silently invalidated every cell that touched the task, and two OpenCode setups that ran short when their account hit a weekly usage limit.

1. Jul 31: seeded mock fixtures and a mock status server, so the page was designed and reviewed before a single real cell existed.
2. Jul 31: the runner grew a serve command that hosts the page and a status endpoint. Live mode, agent lanes, and the failures feed followed the same day.
3. Jul 31: folding the docs axis into one row per task turned out to change the meaning of pass^3. I wrote that cost down in the commit and in the page's footnote instead of hiding it.
4. Aug 1 to 3: the real run. Two verifier bugs presented as several configs failing the same task with the same diagnostic. The deterministic watchdog, the halt and acknowledged alert levels, and the redo flow that retires stale rows to a superseded file came out of those two incidents.
5. Aug 3: the two OpenCode setups ran short. Starred totals, denominators on reliability, and the excluded-cell rule shipped with the final results.

890 of the 912 cells were run and scored; the 22 cells a usage limit prevented are recorded with their reason rather than counted as zero. The site is pure static: no build step, no external requests, system fonts, and charts drawn as inline SVG so the page works from a file on disk and deploys with one command.

The shipped page answers config-level questions well. The question I kept wanting while writing the findings was task-level: on which tasks do two setups disagree, and what did the failing trial actually say? I built the explorer below for this case study, in React, on the real run data. It adds the two patterns that experiment-tracking tools converged on: a differences-only toggle that hides every task all eight setups solved on every trial (20 of the 38), and a compare-two mode. Every cell opens its trials and the verifier's diagnostic, and the matrix is fully keyboard navigable.

I used Claude Code to scaffold the component from a written spec that fixed the state shape, the metric definitions, and the required states first. I wrote the metric functions to match the shipped site's numbers, reviewed every render against the leaderboard, and rejected two of its defaults: zebra striping (rows carry state) and a spinner-only loading state (replaced with a skeleton that holds the layout).

• The mobile leaderboard clips its right-hand columns. The table needs a column picker or a card layout under 640px; the explorer prototype solves this with a sticky first column and horizontal scroll.
• The scatter's direct labels crowd at the top right where five setups sit within a point of each other. A hover-to-lift or a small inset zoom would help more than smaller type.
• The page header says 912 runs where the writeup says 890 finished. The grid size and the scored count should both be on the page, in one sentence.
• Reliability at k=3 is a coarse instrument. Extending to k=5 through the resume flow is already supported by the runner; the column should say which k it is at, which it now does.