Tuesday, Oct 27, 2026 The claims desk. Receipts included. POWERED BY LENZ ↗
IsThis

Hallucination rates vary by model, benchmark, and confidence-elicitation method.

The Claim

When asked to state their confidence, frontier large language models hallucinate in more than half of their answers.

Verified by Lenz ·

The Short Version

Available evidence does not establish that frontier models hallucinate in most answers when asked for confidence. Some models exceed 50% on particular benchmarks, but others—including reported evaluations of GPT-4o and Llama 3.1 405B—remain below it. Many cited measurements also omit confidence elicitation or examine only high-confidence subsets, so they cannot support the claim’s broad causal framing.

Caveats

  • The claim generalizes from selected models and benchmarks to frontier models as a class.
  • Several cited hallucination rates were measured without asking models to state confidence.
  • Rates are not directly comparable because studies handle refusals and define hallucinations differently.

The Receipts

  1. AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models

    arxiv.org

  2. [2506.07309] ConfQA: Answer Only If You Are Confident

    ar5iv.labs.arxiv.org

  3. A Unified Benchmark for Hallucination Detection across ...

    arxiv.org

  4. arxiv.org

    arxiv.org

  5. : A Decision-Theoretic Approach to Evaluating Large Language Model Confidence

    arxiv.org

  6. arxiv.org

    arxiv.org

  7. The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models

    p.rst.im

  8. UNCERTAINTY?

    proceedings.iclr.cc

  9. Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators

    arxiv.org

  10. Calibrated Language Models Must Hallucinate

    arxiv.org

+ 35 more sources — see the full list on Lenz

Filed Under

AI HallucinationConfidence ElicitationFrontier Large Language Models

More Fact Checks