Nearly 4,000 employees: gains from formal AI training do not persist
Analysis of 158,496 conversations from nearly 4,000 back-office employees over eight months shows mature-usage metrics rose in the month of formal training but did not keep improving afterward.
After a single round of formal AI training, employees' mature-usage behavior improved only in the month the training was completed, with no sustained improvement in later months — the conclusion of an analysis of nearly 4,000 back-office employees at a large professional services firm across eight months of 2025 AI usage data (158,496 conversations and 713,564 prompts). Companies have commonly followed a deploy path of "hand out accounts, run one training, wait for productivity," but behavioral evidence for it was lacking.
The researchers combined whether goals were clear, whether tasks covered multiple use types, and whether stepwise or verification strategies were used into a "mature usage" metric. Senior employees and the strategy, digital innovation, and project management teams scored higher; longer initial prompts and more iterations also correlated with these metrics. High usage frequency did not reliably correspond to more mature behavior. The authors accordingly propose that domain experience, task structure, and continuous feedback may matter more than a one-off course.
But the result does not prove training ineffective: trainees were not randomly assigned, and the study did not observe actual output quality or productivity. The sample comes from a single firm's back-office roles, and the main data window is still 2025 chat-style tools, so it cannot be directly extrapolated to today's agentic workflows; the study measures usage behavior, not final performance, and long prompts and multi-turn exchanges may simply reflect harder tasks. Next steps require randomized or staged training trials that place task outcomes, review errors, time spent, and sustained behavior on the same timeline. Compared with counting active users, firms can more verifiably check whether employees can define goals, provide context, check results, and embed these behaviors into concrete workflows.
The study was released as a working paper (arXiv:2608.27364, v1) and has not yet been replicated by independent teams.