Anthropic pays Accenture to embed evaluators inside the company
Embedded evaluation gets its first concrete form, but the company being evaluated is paying the evaluator, so independence and reporting standards remain open questions.
ImportanceMaterialEvidenceE3 inspectableWrite-upStandard
On September 18, Anthropic announced a partnership with Accenture for "embedded evaluation": outside evaluators will work inside the company with access comparable to an employee's, doing red-teaming, alignment assessments and testing model safeguards.
Anthropic and Accenture each expect to invest at least $1 billion in building capacity in this area over the next five years, with the work led by Faculty, Accenture's specialist AI business. Anthropic says it will fund Accenture's work directly, and that it is in dialogue with METR and other nonprofit evaluators to pilot elements using their own funding; the partnership is non-exclusive.
The announcement concedes there are as yet no standards for what information embedded evaluators can access or how they should report findings, and no settled funding system. The company being evaluated paying its own evaluator is the core tension, and whether independently funded evaluators such as METR actually come in is what to watch.