When agents can execute complex tasks lasting hours, a flawed initial hypothesis can lead them down rabbit holes involving hundreds of tool calls. Rafal Wilinski, founding engineer at Runlayer, argues that traditional improvements in model capability or prompt engineering do not fully address this because they lack company-specific "procedural knowledge." His solution involves building "skills"—playbooks that agents can read when relevant—and using the Model Context Protocol (MCP) as a unified distribution layer for these skills.
Wilinski identifies three main barriers in the current skill ecosystem: excessive developer-centricity making it hard for non-technical staff; the time cost of writing documentation after difficult work; and uneven client support where many LLM applications cannot load local skill files. To solve this, Runlayer uses an MCP server as a central knowledge base. Clients query the remote server for relevant skills instead of installing local copies. This architecture resolves compatibility issues and allows enterprises to dynamically manage access permissions based on roles and policies.
The key innovation lies in how skills are generated. Drawing inspiration from the Voyager Minecraft agent, Runlayer enables agents to accumulate knowledge automatically during real work. The system analyzes successful traces, distilling them into reusable code or instructions via frontier LLMs. Failed traces are also treated as valuable signals, revealing missing guardrails, dependencies, or edge cases. This process is asynchronous and autonomous, eliminating the need for manual documentation.
Wilinski presented a concrete case: In a task attempting to run a Chromium fork inside AWS Lambda using the Sonnet 4.6 model, the initial success rate was only 36% (9 successes out of 25 attempts). After enabling the self-improvement mechanism, one successful run was distilled into a skill and injected into all subsequent attempts, raising the success rate to 100% (25/25). While this is a single-task internal result, it demonstrates that preserving and reusing "hard-won lessons" can significantly reduce repeated exploration costs and improve reliability.
The ultimate vision is an "organizational flywheel": even if underlying models change or degrade, the company's governed library of accumulated skills remains available. This is not just a technical architecture but a strategic asset aimed at converting scattered individual experiences into persistent corporate capabilities. However, readers should note that this automated loop relies on strict safety checks (against prompt injection and PII leakage), and its effectiveness has primarily been validated in controlled environments, not yet proven universally applicable across all complex business scenarios.
Sources :youtube.com