Amazon Invites Third-Party Review of Its Own AI Safety Tests: Passing Doesn't Mean Reliable
Amazon asked the independent nonprofit AVERI to review the quality of its internal AI safety evaluation datasets rather than model outputs. The review confirmed several strengths while also identifying improvement areas, one of which is that in some cases, models may refuse based on superficial cues in a prompt, meaning that passing those tests doesn't necessarily indicate reliable safety behavior in new situations.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
AI safety evaluations typically focus on whether a model outputs harmful content during testing. A less frequently examined question is: how good are the tests themselves? If test cases are too easy, unevenly distributed across risk categories, or contaminated by pretraining data, a model's 'pass' may reflect the test's weaknesses rather than the model's safety. In October 2026, Amazon published an account of one attempt to address this gap: inviting the independent nonprofit AVERI (AI Verification and Evaluation Research Institute) to review the quality of its internal safety evaluation datasets themselves, not model performance on those tests.
AVERI's team, led by executive director Miles Brundage, drew a random sample of prompts from Amazon's internal safety-evaluation suite and assessed them independently, without input from Amazon's evaluation team, examining diversity, difficulty calibration, and methodological soundness. The review confirmed strengths in topical coverage, diversity of testing approaches, and low contamination risk, but also surfaced several improvement areas. One concerns evaluation signal quality: in some cases, models may refuse based on superficial cues in a prompt, so passing those tests doesn't necessarily indicate reliable safety behavior in new situations.