Study Finds New Safety Judge Jev Highly Robust in Rule-Based Tasks but Less Accurate in Phishing Detection Than GLM Models
A first controlled comparison reveals that while the non-generative safety judge Jev resists attacks effectively in clear-rule tasks like input moderation, its baseline detection rate for phishing emails (62%) is significantly lower than that of GLM models (77-81%), and it remains vulnerable to specific forged inputs in judgment-heavy contexts.
ImportanceMaterialEvidenceE2 unreplicatedWrite-upQuick
Researchers from Nankai University and Zhongguancun Laboratory conducted the first controlled adversarial evaluation of TypeSafe AI's new safety judge, Jev, comparing it against large language model judges (GPT, GLM) and purpose-built classifiers (Llama Guard). The study found that Jev exhibits exceptional robustness in rule-based tasks such as input moderation: eight types of forgery attacks targeting its structured input failed to bypass any of its decisions. A simple query asking "why someone might make a harmful request" bypassed only 9% of Jev's blocks, compared to a 56% bypass rate for GPT-6 Astra.
However, the results reversed in judgment-based tasks like phishing email detection. Even without adversarial attacks, Jev detected only 62% of phishing emails, significantly below the 77-81% detection rates of the GLM judges. Furthermore, the forgery attacks that failed in input moderation succeeded here, bypassing 19-29% of the emails Jev flagged. While Jev was the fastest judge, returning decisions in an average of 0.7 seconds versus 4-24 seconds for LLM judges, it also produced the highest false positive rate, flagging 27% of benign requests as harmful.
The findings are based on an arXiv preprint (2610.17820) and have not yet been independently reproduced. The research suggests that Jev is well-suited for security filtering scenarios with clear rules, but suffers from lower accuracy and vulnerability in tasks requiring holistic contextual judgment.