French University Study: Mainstream AIs Fail to Reliably Detect Other AIs in Reverse Turing Tests
Research from Université Paris Cité shows major LLMs like ChatGPT and Claude have a success rate of only about 3% in identifying other AI agents, undermining their use as identity verification tools.
According to a report by TechNews, a new study from Université Paris Cité in France indicates that when seven mainstream chatbots were placed in the position of "interrogators" attempting to distinguish whether their conversation partner was a human or another AI, their performance was almost completely inaccurate.
The experiment involved models including ChatGPT 4.5, Claude 3.7 Sonnet, Gemini 2.5 Pro, and Grok 3 questioning hidden-identity subjects for up to ten rounds. The results showed that out of 98 tests against AI subjects, the AI judges successfully identified only 3; if the subject AI used more complete persona settings, the identification rate dropped to zero. In contrast, when facing human subjects, these models correctly judged 123 out of 140 tests.
The research team analyzed that the errors stemmed primarily from expression style rather than content itself. The few AIs that were detected often revealed themselves through overly polished and structurally complete answers; when AIs responded in a more natural, concise manner (with average word counts close to humans), these cues nearly disappeared. Furthermore, AI judges rated the "human-likeness" of AI subjects higher than that of actual humans, showing that machines not only struggle to catch machines but are even more likely to mistake them for humans.
This result highlights that using AI to detect AI is currently not a reliable tool for real identity verification. The study reflects limitations in mimicry capabilities within specific conversational contexts, rather than whether AI possesses true understanding.