We check whether an AI changes its mind when a question says the same thing in different words and we publish the results

Results

What we've measured so far.

The current picture

model: claude-sonnet-5

When we change how a protocol is measured, we release the change as a new version (v0, v1, ...) and keep the old version's published numbers too — so you can always check a number against what came before, instead of it quietly changing meaning.

A second check — the reference model

We also run the same protocols on a model that anyone can download and re-run, so that the method itself can be shown to actually work · Download qwen2.5

model: qwen2.5-1.5b-instruct-q4_k_m

When we change how a protocol is measured, we release the change as a new version (v0, v1, ...) and keep the old version's published numbers too — so you can always check a number against what came before, instead of it quietly changing meaning.

Downloads

Everything downloads freely. No sign-up, no form, no request.

The most important file: one row for every measurement we've made.

All the files

Exactly what is measured, how, and what every field means: the full specification (SPEC.md).

Every file comes with a sha256 — a digital fingerprint that lets you check you downloaded exactly the same file. What each column inside each csv means is explained in full in the data dictionary, part of the separate guide.

Need help

Everything comes from the public GitHub repo. No account, no API key.

How to cite it

@software{llm_archive_2026_22881127,
  author       = {LLM-Archive},
  title        = {LLM-Archive: a long-running stability archive for commercial language models},
  month        = sep,
  year         = 2026,
  publisher    = {Zenodo},
  version      = {v0.4},
  doi          = {10.5281/zenodo.22881127},
  url          = {https://doi.org/10.5281/zenodo.22881127}
}

How to verify it hasn't been tampered with

Download the file, compute its own sha256, and compare it with the one we publish. If they match, the file is authentic. verify.py does this check automatically — you don't need to install anything else.

What isn't downloadable yet

The questions and answers for protocols that are still private stay hidden — if we published them now, the next model could "learn" them and the measurement would lose its meaning. We open them once the corresponding model is four generations old (four newer model generations have been measured after it) — by hand, as a commitment we keep, not as something the code does by itself. Their number and existence are always disclosed.

License

Design

We ask the AI the same question in different words, and see if it changes its mind.

Every month, the same model answers 15 fixed questions, each one worded in 4 different ways. A program — not a human, and not another AI — reads its decision, so that no subjective judgment enters the measurement. We don't care whether the answer is "correct" — only whether it stays the same when the wording changes. Everything someone needs to re-run the same measurement is public.

One example — 2 of the 4 versions (see below why two aren't enough). We ask the model to pick an answer:

This is the exact same thing. A stable model picks the same option both times. Real models often don't.

Why two versions aren't enough

If we only compared two wordings and found a difference, we wouldn't know whether the change in wording was to blame, or whether the model simply isn't reading carefully. That's why we use four versions of each question.

How the number is calculated

We ask the model the same question many times with one wording and many times with an equivalent wording, and measure how different the answers are. The closer the two groups of answers are, the higher the score.

Even a perfectly stable model never scores exactly 100 — there's always a bit of randomness in the answers, so a truly stable model measures around 95%. That's normal, not a flaw in the measurement.

How we know a result is trustworthy

Before a measurement makes it onto the official chart, it goes through three checks:

• that we didn't lose too many answers (e.g. the model refusing to answer)
• that whatever change we see isn't just random variation
• that the model is actually reading the question — we check this by changing something meaningful and seeing whether the answer changes too

If any check fails, the result is still published separately, with a note explaining why it didn't make it onto the main curve — no measurement ever disappears silently.

The metric

It measures Stability: a number from 0 to 100. 100 means "the AI never changed its mind when only the wording changed." The higher, the more stable.

How it calculates

Every number on this site is built from the same three parts. A family (e.g. sunk-cost fallacy) is one kind of decision problem, the scenarios are 15 different variations of it, and the versions (A/A′/B/C) are four ways of wording the same scenario, to see whether the answer changes just because the wording did.

• 12 protocols = 3 families × 4 rewording types. Each protocol is one family, tested one way (wording, an anchor number, order, or which option is marked "default"). Not 12 different problems — 3 problems, each tested 4 different ways.
• 45 scenarios = 15 × 3, not 15 × 4 × 3. The 15 Questions in each family are written once and reused across all 4 rewording types, never duplicated.
• 720 version-texts = 12 × 15 × 4. The total number of written prompts, ready to send to a model: one for every protocol, scenario, and version.
• 120 trials per version, 480 per full measurement. Asking a model once isn't enough — it can answer differently by chance. So each scenario is asked 8 times per version (8 × 15 = 120), and with 4 versions that's 480 calls for one measurement.

In one sentence: 3 families × 4 rewording types = 12 protocols; each protocol asks the model 480 times (15 scenarios × 4 versions × 8 repeats) to produce one measurement.

In plain words

1. Tests stability under framing. The model is given the same dilemma phrased in equivalent ways.
   Example — Version A: "Out of a group of 600 people, 200 will be saved." Version B: "Out of a group of 600 people, 400 will die." The meaning is identical: if the model's answer differs between the two, the measurement records that its decision was swayed by wording alone.
2. Uses four versions of every question. A, the baseline wording. A′, a null change (spacing, punctuation) that measures noise. B, an equivalent rewording — what's actually being measured. C, a real change in meaning, where the answer should change, confirming the model is actually reading the question.
3. Tracks two different categories of model. Commercial models (Claude, GPT-class) are tracked over time to record how their behavior shifts with each update. Open models (e.g. Qwen2.5) serve as a control group — anyone can download the same weights and verify the method themselves.
4. Publishes fully open data. Every measurement, the CSV files (stability.csv, outcomes.csv) and the JSONL logs are free to download — data under CC-BY-4.0, code under AGPL-3.0.

Goal

Its goal is to measure the stability of LLM decisions — whether a model changes its answer when the same question is asked in different words, even though the meaning stays exactly the same. We compare four wordings of each question across two kinds of models, commercial and open, and publish everything we measure as open data under CC-BY-4.0, with the measurement pipeline itself open-source under AGPL-3.0 (see Data for the full license breakdown).

Two models, two different roles

A commercial model (e.g. Claude, GPT) is really a product: it changes behind the scenes, without notice. It's valuable to track it over time, but once it's retired, no one else can re-ask the same model to check our measurement.

That's why we run the same method on a frozen reference model too. Its weights are open, so it stays available forever, and anyone who wants to can download it and re-run what we're doing, today.

The two series are never compared against each other — each answers a different question: "what changed" and "does the method work".

Recent activity

Methodology

An automated process that runs every month, with no human intervention in the measurement itself.

No answer is judged by a human or another AI

We ask the model to end its answer with one fixed line:

DECISION: <choice>

Only this line is read, mechanically. If it's missing, or ambiguous, the answer doesn't count — we don't interpret anything else in the text.

What we don't prove

Three things we disclose ourselves, upfront:

The 12 protocols, one by one

3 families (see "How it calculates") × 4 rewording types = 12 protocols. Each row below is one protocol: the family it belongs to, and the one thing that particular protocol changes. Each protocol has a series: the latest (v1) is the one being measured now. Earlier series (v0) will likely open once a newer series with new scenarios replaces them.

What each protocol actually asks (the 15 scenarios themselves) stays hidden for guard-lane protocols while the model being measured is still active — see the callout above. Two protocols (the wording type of each of two families) are open-lane: every answer they ever produced is public in open-lane/2026.jsonl, and the wording of their first scenario — all four versions — is on this page, in the row's own panel. The other 14 scenarios' wording isn't published yet; that file carries a prompt_sha256 for each trial, which lets you confirm a question you already hold, not read one you don't.

Twins

A twin is a pair of protocols — one open, one guard — from the same family, testing a different type. Only 2 of the 12 protocols are open-lane (every answer public, and the first scenario's wording): risky_choice_framing__wording and base_rate_neglect__wording. Each has a twin — the anchoring protocol of the same family, which stays guard (hidden while its model is still active):

risky_choice_framing__wording  (open)   ↔  risky_choice_framing__anchoring  (guard)
base_rate_neglect__wording     (open)   ↔  base_rate_neglect__anchoring     (guard)

Why it exists. Open-lane text will end up in some future model's training data — that's expected, not a flaw. But then a later shift in the number has two possible causes that look identical: the model changed, or the model has now read the panel. The guard twin — same phenomenon, same declared lotteries, never published — is the discriminator. Only the difference between the two time series separates contamination from a real change.

Open and guard lanes

Only 2 of the 12 protocols are open-lane overall (the two wording protocols above) — everything else is guard.

2 open + 10 guard = 12, and only 4 of the 10 guard protocols (the two anchoring ones) carry a twin_id. The other 8 are simply hidden, with no public counterpart checking them — only the anchoring type (alongside wording) has "full" status; order and default are excluded from twin pairing by design, since pairing a headline number with a type that doesn't even enter the same aggregate wouldn't be a real comparison.

How we get to the result

From the model's raw answers to the number in the results table, in order:

Why it matters

The archive doesn't answer "how smart is an AI." It answers "did something change, when, and for whom."

Models are updated without notice and retired without a trace. Here, how they decide stays on the record.

A few words about the system behind it

A scientific approach. This isn't an attempt at a general "intelligence score," the way classic benchmarks like MMLU or GSM8K work. It focuses on one specific, critical property instead: cognitive bias and decision stability in LLMs.

Transparency. Failed runs and data gaps are published too, not hidden — alongside a full explanation of the methodology and the quality checks behind every number.

How you can use it

You can use LLM-Archive according to your own role — researcher, developer, or someone simply evaluating AI solutions. Here are the main ways to put the site and its data to use.

1. As a benchmarking tool

Choosing which AI model to build into an application, product or company process?

• Check the stability score. See which model is least affected by framing — how a question happens to be worded. Building something for legal, financial or medical use? You want a model that doesn't change its mind just because a user phrased the question differently.
• Watch for drift over time. Track whether a model (say, a particular Claude or GPT version) became less stable after its maker shipped an update.

2. Data analysis and research

All of the project's data is open and downloadable directly from Downloads:

• stability.csv — the aggregate results and stability score for every measurement.
• open-lane/2026.jsonl — the full trial-by-trial raw data: the exact questions, wordings and answers each model gave.

Download the CSV/JSONL files and load them into Python (pandas, Jupyter), R or Excel to run your own statistical analysis of LLM bias — framing effects, anchoring bias, and so on.

3. Replication and control experiments

Researcher or AI engineer who wants to run the same protocols on your own models?

• Verify the files. Download verify.py from the site and confirm the data's authenticity against its published SHA-256 checksums.
• Run the reference model yourself. Download it (its weights are open, e.g. Qwen2.5-1.5B) and use LLM-Archive's own methodology — 12 protocols, 4 versions (A/A′/B/C) — to reproduce the results on your own machine or server.

4. Academic citation

Writing a paper, article or study about LLM reliability and behavior? Cite LLM-Archive directly using the BibTeX citation on the Downloads page.

In short: if you don't write code, use it as a buying/selection guide for a model. If you do, download the CSV/JSONL data and dig into where and why language models break.

Glossary

You don't need to know about benchmarks to read the results. Here we explain, in plain terms, the codes and words that show up in the tables.

Protocols and versions

Rewording types

The codes are stable identifiers. The friendly title makes it easier to read, while the code preserves the record's exact identity.

Metrics and indicators

Run settings

Things you see on the page