ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsTopic · AWS AI指标之争Material3 min readDisconfirmAI Editorial

Colleen Aubrey, Head of Applied AI at AWS: Don't Invent New Metrics for AI, Measure It with Your Existing Business Metrics

Colleen Aubrey, Senior Vice President at Amazon Web Services (AWS), argues that enterprises should not create new measurement systems for artificial intelligence. She uses Wells Vehicle Electronics as an example to illustrate that truly effective AI projects change business metrics that the Chief Financial Officer is already tracking—such as inventory levels, resolution rates, and cycle times—rather than token consumption or prompt usage.

2026-11-07 02:56 GMT+8

Colleen Aubrey has worked at Amazon for over twenty years, having built customer-facing products for millions of sellers and advertisers. In 2026, in her capacity as Senior Vice President of Applied AI Solutions at Amazon Web Services (AWS), she offered a rather direct suggestion for corporate AI practices: stop inventing new AI metrics.

Her core argument is that measuring whether AI is successful does not require a completely new system of metrics.

"I worry that people start trying to come up with new metrics to measure the impact of AI, but it’s the same metrics," she explained. "It’s your business metrics that matter."

This statement sounds simple, but it points to a widespread practical issue: after investing in AI, many companies build AI-specific dashboards for token consumption, prompt usage frequency, and model call tracking, only to find that while these numbers rise, business results remain stagnant.

The proof point Aubrey provides is Wells Vehicle Electronics. According to AWS reports, the company reduced inventory by 7% within three months while expanding the number of SKUs actively monitored from 20 to 2,000. She emphasized that no one at Wells talks about tokens; they talk about working capital. If a project cannot be tied to a metric the Chief Financial Officer is already tracking, it is an experiment, not an investment.

Sign in / Sign up
ReadingResearchRadarInvestment framework

What does this distinction mean in practice? Aubrey suggests that before any AI project receives funding, bind it to a business metric the company is already tracking—inventory levels, problem resolution rates, hiring cycle time, production cycle time—and set a baseline. If this metric does not change within a defined time window, the project should not scale, regardless of how impressive the demonstration was.

Her argument also touches on a deeper mechanism: AI usually does not eliminate bottlenecks but moves them. Internal experience at Amazon shows that when AI coding tools were widely deployed, teams first discovered insufficient test coverage; after fixing tests, they found pipeline issues; after fixing pipelines, they found manual approval steps slowing down deployment. The result for Aubrey's team this year was a doubling of the rate of pushes to production, but she explicitly stated that this speed increase did not come from the coding assistant itself, but from fixing all the downstream problems exposed by the assistant.

The implication for enterprise procurement of AI tools is this: if you buy the tool but skip pipeline repair, activity will rise but output will remain flat. She recommends reserving budget and personnel for downstream fixes rather than discovering new constraints six months later.

Aubrey also proposed a lesson at the product strategy level: feature parity is a trap. When she took over the AWS supply chain product, the team was building five modules attempting to catch up with ERP vendors that had decades of head start. When she rebooted the product as Connect Decisions, she asked a different question: where is the pain most acute, and where are existing tools weakest? The answer was planners—who spend their days firefighting, overwhelmed by a flood of signals, by the time they make a decision the moment has passed. This principle of picking a wedge rather than a platform is what she sees as key to moving enterprise AI projects out of pilot mode. A narrow, painful problem yields quick wins, clear metrics, and a credible foundation for expansion. Projects that attempt to reimagine an entire function at once often stall.

It must be noted that the data for Wells Vehicle Electronics comes from AWS's retelling and has not been independently verified. Aubrey's own narrative also indicates that actual productivity gains from AI at Amazon largely came from infrastructure repairs rather than AI tools themselves, making it complex to attribute results solely to AI investment as a single variable. Her advice serves better as an evaluation discipline—using existing metrics to verify whether a project actually changes business outcomes—rather than as causal proof.

Sources

  • siliconangle.comSiliconANGLE↗
  • cnbc.comCNBC↗
Topic

AWS AI指标之争

【背景】随着企业AI投入激增，如何证明其投资回报率（ROI）成为难题。亚马逊云科技（AWS）应用AI解决方案高级副总裁Colleen Aubrey提出反直觉主张：企业不应为AI发明新指标，而应沿用库存周转、解决率等传统业务指标。

4 steps1 piecesUpdated 2026-11-07 23:22

How it got here

  1. 2026-05-01亚马逊发现员工通过滥用AI工具刷取Token用量以攀升内部排行榜，随即关闭该内部排行榜。cnbc.com ↗
  2. 2026-09-01亚马逊向员工分发内部备忘录，明确反对仅以Token使用量或代码生成量作为AI采用率的衡量标准，称其为“严重缺陷”的指标。cnbc.com ↗
  3. 2026-10-27CNBC公开报道亚马逊上述内部备忘录，披露公司鼓励工程师谨慎使用AI，强调长期业务成果而非短期活动量。cnbc.com ↗

What is still unsettled

  • 除了Wells Vehicle Electronics，是否有其他大型传统企业公开证实了‘不新增AI指标’策略的成功？
  • 亚马逊内部备忘录提到的‘特性速度’（feature velocity）具体如何计算，是否已推广至外部客户？
Read the full topic →
Continue researching →