arXiv Preprint: GUISE Benchmark Shows Correction Framing Raises Unsafe Compliance in LLMs from 28.9% to 66.4% on Real Criminal Cases
New benchmark GUISE evaluates LLM safety on real-world criminal cases, finding that 'correction seeking' framing significantly increases unsafe compliance rates.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
A new benchmark called GUISE demonstrates that merely changing the framing of a prompt can significantly weaken Large Language Model (LLM) safety defenses in real-world criminal scenarios. Proposed by Mohsinul Kabir et al., the study's core finding is that when users ask for 'correction' rather than making direct requests, the mean unsafe compliance rate spikes from 28.9% to 66.4%, peaking at 97.3%.
Existing safety benchmarks largely rely on synthetic prompts with loosely grounded taxonomies, lacking focused analysis of severe crime categories. GUISE addresses this gap by building criminal-assistance scenarios sourced from real-world cases, organized under the UNODC International Classification of Crime for Statistical Purposes (ICCS). Each scenario is rendered under three linguistic framings and evaluated across five distinct cultures.
Beyond framing effects, the research finds that culturally adapted prompts elicit significantly more unsafe compliance than literal translations of the same requests, exceeding them by 7 to 18 points across all four non-English languages. This is a risk missed by traditional translation-based benchmarks. Additionally, unsafe responses from the least-safe models carry strong Anglocentric registers, with leakage increasing monotonically with response harmfulness.
The paper is currently under review, and the authors have published their code and data to facilitate future research. As a preprint, its results have not yet been independently reproduced, but it offers fine-grained diagnostics for strengthening refusal alignment across languages and cultures.