Substantive disagreements between AI models on fact-checking outcomes are common.
NOT BSTOTAL BS
NOT BS — Verdict: True
Verified by Lenz ·
The Short Version
Evidence from multiple studies shows that AI fact-checking models often reach materially different verdicts on the same claim, with reported substantive conflicts commonly in the roughly 15% to 30% range on challenging datasets. That is frequent enough to count as common in real-world use. Rates do vary by claim difficulty, ambiguity, prompting, and evidence quality.
Caveats
'Common' should not be read as 'most claims' in every domain; disagreement is concentrated on harder, ambiguous, or politically contentious claims.
Some studies showing different overall label distributions do not, by themselves, prove claim-by-claim disagreement; the strongest evidence comes from direct pairwise conflict analyses.
Disagreement rates are sensitive to experimental setup, including prompt design, provided evidence, language, and benchmark composition.