Colleen Aubrey, Head of Applied AI at AWS: Don't Invent New Metrics for AI, Measure It with Your Existing Business Metrics
Colleen Aubrey has worked at Amazon for over twenty years, having built customer-facing products for millions of sellers and advertisers. In 2026, in her capacity as Senior Vice President of Applied AI Solutions at Amazon Web Services (AWS), she offered a rather direct suggestion for corporate AI practices: stop inventing new AI metrics.
Her core argument is that measuring whether AI is successful does not require a completely new system of metrics.
"I worry that people start trying to come up with new metrics to measure the impact of AI, but it’s the same metrics," she explained. "It’s your business metrics that matter."
This statement sounds simple, but it points to a widespread practical issue: after investing in AI, many companies build AI-specific dashboards for token consumption, prompt usage frequency, and model call tracking, only to find that while these numbers rise, business results remain stagnant.
The proof point Aubrey provides is Wells Vehicle Electronics. According to AWS reports, the company reduced inventory by 7% within three months while expanding the number of SKUs actively monitored from 20 to 2,000. She emphasized that no one at Wells talks about tokens; they talk about working capital. If a project cannot be tied to a metric the Chief Financial Officer is already tracking, it is an experiment, not an investment.
What does this distinction mean in practice? Aubrey suggests that before any AI project receives funding, bind it to a business metric the company is already tracking—inventory levels, problem resolution rates, hiring cycle time, production cycle time—and set a baseline. If this metric does not change within a defined time window, the project should not scale, regardless of how impressive the demonstration was.
Her argument also touches on a deeper mechanism: AI usually does not eliminate bottlenecks but moves them. Internal experience at Amazon shows that when AI coding tools were widely deployed, teams first discovered insufficient test coverage; after fixing tests, they found pipeline issues; after fixing pipelines, they found manual approval steps slowing down deployment. The result for Aubrey's team this year was a doubling of the rate of pushes to production, but she explicitly stated that this speed increase did not come from the coding assistant itself, but from fixing all the downstream problems exposed by the assistant.
The implication for enterprise procurement of AI tools is this: if you buy the tool but skip pipeline repair, activity will rise but output will remain flat. She recommends reserving budget and personnel for downstream fixes rather than discovering new constraints six months later.
Aubrey also proposed a lesson at the product strategy level: feature parity is a trap. When she took over the AWS supply chain product, the team was building five modules attempting to catch up with ERP vendors that had decades of head start. When she rebooted the product as Connect Decisions, she asked a different question: where is the pain most acute, and where are existing tools weakest? The answer was planners—who spend their days firefighting, overwhelmed by a flood of signals, by the time they make a decision the moment has passed. This principle of picking a wedge rather than a platform is what she sees as key to moving enterprise AI projects out of pilot mode. A narrow, painful problem yields quick wins, clear metrics, and a credible foundation for expansion. Projects that attempt to reimagine an entire function at once often stall.
ソース:SiliconANGLE