How the board refreshes itself, and what happens when a refresh fails
No human sits in the ranking loop and no scheduler lives in the source repository. A data plane discovers, admits, reads, scores, validates and publishes on a fixed cadence, keeps the last good board when a cycle fails, and reports its own health. The contract and the failures we have hit.
People sometimes assume a board that updates every few minutes has someone watching it. It does not, and the design goal was that it never should: every rank on ModelCap is produced by a loop that runs without anybody in it, and the site is only allowed to show a board that loop has validated. This note is the operating contract, written for readers rather than for the code.
The loop
- Discover. About every five minutes: OpenRouter, the Hugging Face router, created-at and trending feeds, and exact author queries for every reviewed official namespace.
- Admit. An official, complete, original language artifact becomes a catalogue row in the next full refresh, or within five minutes if it is brand new.
- Read. Model cards at pinned revisions, the benchmark boards, price lists and provider catalogues.
- Score. The whole graph is recomputed: identity folds, evidence freshness, scores, confidence, intervals, ranks. Nothing is incremental, so a rule change re-scores everyone consistently.
- Validate. Contiguous current ranks across canonical identities, a separate contiguous archive rank, every lineage bound equal to its parent bound minus the published penalty, every floored row backed by a real predecessor, every sealed board reproducible from its inputs.
- Publish. The validated snapshot is sealed with its source revision and served hot; the site reads it on the next request. A full cycle targets fifteen minutes.
Last good wins
The single most important property of the loop is what it does when a step fails: nothing. A refresh that fails validation leaves the previous accepted snapshot untouched, logs why, and tries again next cycle. The site never shows a half-computed board, never blanks, and never regresses to an older method version because a fetch timed out. When a source is down, the models it measured keep their rows and the source is marked as an issue on the health endpoint.
| State | Meaning |
|---|---|
| healthy | every source ok, snapshot fresh, method version matches the deployed code |
| degraded | serving normally; at least one source is partial or standing by (a private Hub repo, a fallback in standby) |
| stale | no accepted snapshot within the expected window; last good is still served |
| not production-ready | discovery itself is stale or the loop has stopped; an operator is paged |
“Degraded” is the everyday state and it is not an apology. On the day this was written the board was degraded because one of 357 Hugging Face repositories is private and the Arena live fallback was on standby, both by design. What pages an operator is the last row.
Git is not the scheduler
The source repository holds the code and a bundled snapshot so that a fresh checkout can build and run. It does not hold the live data, and no commit or workflow drives a refresh. This is a deliberate split: a board that updated by committing data would have its history polluted with a thousand robot commits a day, and a board that could only be refreshed by a deploy would be down whenever the deploy pipeline was. A release ships the code; the first cycle after a release is always a full one, forced by a mismatch between the running revision and the published one, so a new method never scores against an old dataset.
Failure modes we have actually hit
- An evidence ledger that stopped growing. The ledger that records every admitted observation hit a ceiling and the loop stalled without crashing. The site stayed up serving last good while the external dead-man check paged on the health endpoint. The fix was a guard on the ledger and a first-look rule for operators: read the refresh journal before touching anything.
- A packed file the installer did not expect. The refresh packs a set of artifacts and the installer accepts an allowlist. When the two rosters disagreed, publish rejected every cycle as “unexpected member” while each cycle logged success on its own side. Now a test asserts the rosters agree.
- A board version changing under a fixed URL. Terminal-Bench began serving its 4.0 board at its 2.1 address; the fetcher returned no usable rows and went stale rather than admit a different board's numbers. Correct, and written up in the dormancy note.
- A rule that was right and a result that was wrong. The launch-day retirement of a measured #1, described in its own note, and the FP8 seating problem in one seat per product. Both were caught by audits of the live board against outside sources, not by the loop, which is why the loop now has alerts for “a measured row disappeared”.
The public health endpoint is at /api/health; the operating documentation and the refresh scheduler are in the repository under operations and the data-plane folder, linked from the about page.