Preprint: LLMs Fail to Recognize Truth-Value Gaps in Plural Statements, Misjudging Mixed Evidence
New research reveals open-weight LLMs exhibit a systematic 'bivalence bias,' incorrectly forcing mixed-evidence plural claims into true/false categories, which compromises their reliability for high-stakes fact-checking.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
When using Large Language Models (LLMs) for summarizing or verifying critical information, a common pitfall involves handling statements with 'plural definites.' For example, if three out of five patients improved after taking Drug A, the claim 'the patients improved' is not true (since two did not), but its negation 'the patients didn't improve' is also not true (since three did). This logical state is known as the 'Homogeneity Gap.'
A recent preprint titled 'Neither True nor False: Do LLMs Represent Truth-Value Gaps in Plural Predication?' introduces HomogeneityNLI, a diagnostic dataset of 1,000 minimal pair items, testing seven open-weight models. The study uncovers a pervasive 'Bivalence Bias': models tend to force mixed evidence into clean 'true' or 'false' verdicts.
Specifically, in zero-shot settings, no model recognized the truth-value gap for both polarities in more than 15% of scenarios, and negated definites were incorrectly judged as False in 84% of cases. Even with explicit linguistic rules, chain-of-thought, or enumeration prompts, effects remained unstable and often entrenched the bias.
These findings indicate that current open-weight LLMs acquire some truth conditions of plural predication but lack the underlying trivalent logic. This suggests significant risks when relying on these models for automated summary or verification in high-stakes decisions involving partial evidence. The results are based on author-run benchmarks and await independent reproduction.