Study finds LLM agents fail to load relevant skills at scale, with recall dropping from 73.6% to 27.4%

A newly released preprint indicates that Large Language Model (LLM) agents relying on implicit routing over metadata suffer severe "routing collapse" as their skill libraries expand. Researchers held tasks and golden skills fixed in the SkillsBench benchmark while adding irrelevant skills, observing a significant decline in the agents' ability to load relevant instructions.

Specifically, when the number of irrelevant skills reached 1,000, loaded-skill recall dropped sharply from 73.6% to 27.4%, and the rate of loading no skills at all surged from 9.3% to 51.6%. In contrast, a control group using explicit selection mechanisms maintained a 73.9% recall rate within the same large-scale library.

The analysis suggests this performance degradation is not due to agents mistakenly loading plausible alternative skills, but rather stems from semantic interference caused by irrelevant skill descriptions and conflicts between skill-selection instructions and task-execution directives. The study found that masking only the descriptions of irrelevant skills largely restored loading capabilities, whereas asking agents to plan skill use themselves still yielded lower recall than preselection.

Proposed by authors Jinlin Liu et al., this finding is currently an arXiv preprint (submitted October 13, 2026) and has not yet been independently reproduced. It signals that as agent toolsets grow, implicit retrieval may no longer be a reliable scaling path, suggesting that explicit pre-filtering or stricter metadata isolation might be necessary engineering trade-offs.