FAQ
Quick answers. For the full reasoning behind any of these, follow the link to SPEC.md, which is the governing document if anything here is unclear or seems to conflict with it.
What does LLM-Archive actually measure? Whether a commercial AI model's decision changes when the same question is reworded in a way that shouldn't change the answer — measured as a gap between two response distributions, not a per-question "did it flip" count. See glossary.md for the exact terms and for-researchers.md for what the number does and doesn't prove.
What is LLM-Archive for? Research. It keeps an open, dated, checkable record of one narrow property — whether a model keeps the same decision when a question is reworded without changing its meaning — so that other researchers can reproduce, cite and extend it. It is not a product evaluation or a ranking, it is not run to rate or promote any model, and no number is meant as a basis for deciding whether to buy, use or trust one. The scenarios are fictional dilemmas; no personal data is sent to or collected from a model. See SPEC.md §1.2.
Why isn't a perfectly consistent model reported as 100% stable? Because with a finite number of responses, two samples from the identical distribution still produce some measured gap by chance alone. This "noise floor" is published next to every measurement (null_floor_pct) so a reader can see how much of any gap is just sampling noise. At the project's chosen sample size (n=30 per version), the ceiling is about 89.9% in the worst case (a binary decision at 50/50) — see SPEC.md §6 for why n is this small: real calls against the real API come back deterministic without a client-controllable sampling parameter, so this project no longer pays for repeat samples that would just repeat the same answer.
Is there an AI "judge" grading the model's answers? No. Extraction is entirely deterministic: a response must end with a DECISION: <token> line, with token from a fixed, closed list of options — checked by ordinary code, not by another model. Anything outside that exact line is never read, and a response without one, or with conflicting tokens, is scored unparseable rather than interpreted.
Why measure a second, frozen model every day? To catch the project's own pipeline breaking before mistaking that for a change in the model being studied. If a fixed, never-changing reference model suddenly answers differently, something in the project's own setup changed — not the commercial model. This is disclosed as a real limit, not a complete guarantee: if the reference model stays flat, that rules out the project's own pipeline as a cause, but it does not rule out the commercial provider silently changing something upstream (a router, a hidden system instruction, quantization) — no local check can see behind a hosted API. See SPEC.md §7.
Why track a downloadable reference model at all, separately from the commercial one? As a control experiment proving the method itself reproduces, run once per model generation rather than monthly, since a downloadable model can't be silently changed by a provider — it's re-run under the project's own control instead. The commercial series and the open_weights series are never drawn on the same chart or averaged together; they answer different questions.
Why not just compare the model to how humans answer the same question? Any such comparison (human_model_gap) is published, but only as a secondary, per-protocol note where a directly comparable multi-lab or meta-analytic human study exists — never as a headline, and never aggregated across protocols. The reason: proving a human number and a model number measure "the same underlying thing" is a much harder claim than measuring whether a model is consistent with itself, and the project's headline metric is built to avoid depending on that harder, contested claim.
What's the difference between open, guard, and sealed protocols? How much of a protocol's exact wording is published, and when:
open— every raw response immediately, plus the wording of the panel's first scenario. The other 14 scenarios' wording is not published yet: a disclosed gap, not the design (glossary.md,SPEC.md§1.1).guard— only outcome categories now; the wording after four model generations. That opening is a commitment the project keeps by hand — no code counts generations or opens anything on a timer.sealed— not even that a specific scenario exists in detail; only that the protocol exists and how many trials it has. Never opens.
Full explanation, including why this matters for detecting contamination: glossary.md.
Why keep most protocols hidden at all — isn't that against the spirit of an open project? Because a protocol whose exact wording is public will eventually be read by a model during training, and once that happens, "the model's answer moved" and "the model has now seen the test" become indistinguishable from the outside. Keeping most protocols private for a while, and pairing each public one with a hidden "twin" measuring the same underlying phenomenon, is what makes it possible to tell those two explanations apart later. The wording does eventually become public for guard protocols (after four generations, opened by hand rather than automatically) — sealed protocols are the deliberate exception, kept back permanently as an uncontaminated instrument for the very long term.
Can I run the same test against a different model and get it published? Yes — see reproducing.md for exactly what's required for a submission to count as a valid comparison_point, and what such a point does and doesn't mean.
What happens if the project goes quiet — is the data lost? No. Two independent mechanisms exist for this specifically: the project publishes a disclosed degradation ladder rather than going silent (reduced scope before reduced rigor — n and the statistical checks are never quietly loosened to save money, SPEC.md §9), and every scheduled measurement that doesn't happen is logged with a specific cause rather than left unexplained (SPEC.md §10). A gap the project names itself is a disclosed limitation; a gap discovered by someone else later would be a much bigger problem — which is exactly why it's designed not to happen that way.
Does the metric say anything about model capability, alignment, or "how human" a model is? No, deliberately. See the "What it does not claim" section of for-researchers.md.
I'm a researcher whose work is cited (or miscited) here — what do I do? Open an issue or write to mkalognomos@gmail.com, naming the run_id or file and what you expected instead. Nothing is edited or deleted silently: a correction is disclosed publicly in the next release notes.
Is this live yet? Yes. The public site (llm-archive.github.io) and its repository (github.com/LLM-Archive/llm-archive.github.io) are both live. See the root README.md and the release notes on GitHub for the current build state.