From a model card to a start rank: how a new open-weight model gets its day-one position

A leaderboard that waits for independent benchmarks before it will show a rank is a leaderboard that is always a week late. ModelCap's product rule is the opposite: every rankable model gets a starting rank on launch, and later independent evidence moves it. For an open-weight release, the start comes from the publisher's own release comparison, which is nearly always a table on the Hugging Face model card that names this model next to models we already rank.

That sentence hides a lot of engineering and one hard rule: the card's numbers are the publisher's claims, so they can locate a model, and they can never become measured evidence.

The loop

1. See it. Discovery reads the OpenRouter and Hugging Face router feeds, the created-at and trending feeds, and fans out exact author queries for every reviewed official namespace. A first-party release must not depend on winning the global top 200.
2. Admit it. An official, complete, original language artifact becomes a catalogue row in the same full refresh. Admission is identity, not quality; official status mints no score.
3. Read the card. The README is fetched at the pinned revision. Comparison surfaces are read in order of preference: embedded HTML tables, markdown tables, the Hub's structured evaluation results, tight prose claims, and finally evaluation chart images through OCR. Peer columns are kept; a table that only yields the target's own numbers is an incomplete extract.
4. Place it. Each peer column is resolved to at most one catalogue identity. On every public benchmark row, the target's number is placed among the peer numbers, that relative position is mapped onto the peers' current ModelCap scores, and the start is the conservative lower bound of the combined placements.
5. Move it. Later refreshes re-read the card when its revision or our extractor changes, ingest any admitted independent observation for the exact identity, and rebuild the index. Measured evidence outranks the card start.

Worked example: Qwen3.8-27B

The acceptance test for this rule is Qwen's 27B release. Its “Text Performance” table names Qwen3.6-27B, Qwen3.7-Plus, Muse Glimmer-30B and Opus 4.6 Max. Dropping the in-house benchmarks it also lists (QwenSWEBench, CoWorkBench, JobBench, Agents' Last Exam) leaves the public rows, and on those the card reads:

So the required opening sandwich is Glimmer below Qwen3.6-27B below Qwen3.8-27B, roughly level with Qwen3.7-Plus, below Opus 4.6 Max. With the scores those peers held when the rule was written, that means opening in the mid sixties, not near Glimmer in the twenties and not near Opus in the eighties, and certainly not on a parameter-count prior.

The placement rules that matter

• Measured peers anchor. Qwen3.7-Plus and Opus 4.6 Max are measured, so they carry the placement. Modeled peers may bound the sandwich but may not be the only load-bearing anchors when measured ones exist.
• Same-lab columns are order evidence only. A newer official sibling should not open below an older family member when the card says it is ahead, but a lab's claim about its own model is not proof of absolute score.
• Cross-lab columns get a haircut. A publisher's numbers for a competitor are compared with what we already know about that competitor, and the card's optimism is discounted accordingly before the target is placed.
• Variant markers must agree. “Thinking” beside a mode word or in a parenthetical names a runtime configuration of the catalogue product; a bare suffix names a different artifact and fails closed. Dotted versions stay atomic: 3.5 is not 5.
• Ambiguity abstains. If the card cannot be parsed or the peers cannot be resolved, the row publishes as unranked with a written reason. No number is invented and no size prior is substituted for a table we failed to read.

The bug that almost sent it to the prior

Building this, the 27B kept falling through to the corpus placement with the reason that its peers could not be resolved. The cause was one dropped label. Peer resolution matched header text to catalogue names by token subset, so the coarse product “Qwen Plus” also matched the header “Qwen3.7-Plus”, the resolver saw two candidates, and the fail-closed rule threw the correct one away. The fix scores each candidate by how much of its alias vocabulary the header covers and takes the unique maximum; a tie at the top still fails closed. Any future matcher on this board has to use the score, not the boolean.

Closed API products have no card, so they open on the boards that have already measured them, or on the family-succession start described in the launch-day incident note. The full contract for day-one placement lives in the methodology.