What the Index refuses to count

Most of the questions we get about the ModelCap Index are about what it does. The more revealing list is what it does not do. Every item below is a signal we have, could use, and have chosen not to, and in each case the board would look more complete if we did. This note is the reasoning, so the choices can be argued with rather than guessed at.

Popularity

Hugging Face downloads and likes, OpenRouter route volume, Vercel gateway spend share and trending feeds are all shown on the site and none of them touch a score. Popularity measures distribution, pricing, marketing and habit. It lags capability by months for a good model and leads it by months for a well-promoted one, and it rewards whichever checkpoint happens to be the default download. The discoveries page exists precisely so that popularity has somewhere to live that is not the ranking.

Price, providers and context

A model served by six providers at a low price with a million-token window is more useful than the same model served by one provider at a high price. It is not more capable. Those fields are on every row because they decide what you can actually use; they are never inputs to the rank because the rank is trying to answer a different question. The ranking lists (cheapest, largest context, free tier) are where those fields order things.

Parameter counts

Total and active parameters are metadata. The Index uses size only to pick nearby anchors when it estimates a model from its architecture and era, and even then size is never a positive quality signal: a larger model is not presumed better. The product rule for launches goes further and says that when a comparison table exists, a size prior is not an acceptable opening rank at all. Opening a model on its parameter count is the failure mode the whole start-rank loop was built to prevent.

The publisher's own numbers

A model card's benchmark table is the most information-dense document a launch produces, and the Index reads every one it can. What it does with them is narrow. Card rows are stored with an admission status of quarantined and are never copied into the benchmark evidence a measured rank is built from. They may locate a new model relative to peers already on the board, which is how a launch gets a day-one position, and the moment an independent board scores the same identity that observation takes over. In-house benchmarks that nobody else can run are dropped entirely, and so are stacked cells with two metrics where we would have to guess which number to keep. Hub structured evaluation results get the same quarantine as the card they came from.

A language model's opinion

No LLM sits in the ranking loop. One does not choose a score, match a benchmark name, resolve a peer, or read a card into numbers. Extraction is deterministic: a value that cannot be traced to a table cell, a structured evaluation row, or a tight prose pattern is not emitted, and unknown labels are dropped rather than interpreted. This is the rule we are asked to relax most often, usually with a reasonable-sounding case, and it is the one we are least willing to, because a board built by a model has no audit trail a reader can follow.

Anything it cannot bind to one identity

• A moving latest alias, an ambiguous id, or a lineage edge that resolves to more than one repository admits no evidence and receives no rank.
• A Hugging Face revision identifies the artifact we inspected; it does not establish which revision an external leaderboard evaluated, and the row never claims that it does.
• Lineage can establish ancestry; it cannot prove a fine-tune improved or preserved capability, so an inherited score always carries a penalty and always ranks below its parent.
• Quantization penalties are conservative policy, larger for lower bit widths, not measured loss curves. The methodology says so in those words.

The one label we had to make vaguer

Sometimes honesty about a source means saying less. When a repository on Hugging Face stops resolving, the Hub returns the same response whether the repository is private, gated, or deleted, and it does so both to anonymous requests and to credentialed ones that are not members. We verified this from two networks with a made-up organisation name. An outsider therefore cannot tell “the publisher hid it” from “the publisher removed it”, and a label that said “private” would be a guess dressed as a fact. The row now says the artifact is not publicly available (private, gated, or removed), the source stays healthy, and a test pins the wording so nobody sharpens it later without the evidence to do so.

The full list, with the executable rules behind each item, is in the limits section of the methodology.