ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

Systematic Trading Author Robert Carver: AI Can Only Participate in Idea Generation and Small Code Writing in Quantitative Investment Research; Testing Frameworks Must Be Written by Humans

Carver proposes that in systematic trading research, AI should be limited to two stages: idea generation and writing small, parameterizable predictive code. The robust testing framework responsible for evaluating these predictions must be written and controlled by humans, because the opacity of AI makes three types of overfitting undetectable.

2026-11-17 01:49 GMT+8

Robert Carver has proposed a clear boundary for the use of AI in systematic trading research: allow AI to participate in idea generation and the writing of small, parameterizable predictive code, but require that the robust testing framework used to evaluate these predictions be written and controlled by humans. The core logic of this suggestion is not simply that AI is unreliable, but rather that the opacity of AI renders overfitting undetectable.

Carver categorizes overfitting into three types: explicit, implicit, and tacit. Explicit overfitting involves directly tuning parameters within the sample data. Implicit overfitting occurs through indirect fitting via repeated research loops. Tacit overfitting happens when researchers already know what works and unconsciously encode known answers into the code during strategy design. He argues that while these three types of overfitting persist after AI intervention, they are wrapped in an opaque block, making it impossible even to know if they are occurring.

In Carver's framework, AI is permitted to participate in only two stages. First is idea generation, data scoping, and initial research. The output of this stage must be trustworthy data sourced directly from original sources, not data paraphrased by AI, because AI may introduce hallucinations or hidden look-ahead bias. Second is writing a small piece of parameterizable Python code that takes data as input and outputs a forecast, incorporating parameters that require calibration. This code must be small enough for its correctness to be verified by humans, though Carver notes it likely contains tacit fitting.

Sign in / Sign up
ReadingResearchRadarInvestment framework

From this point onward, AI is completely excluded. Carver emphasizes that the robust testing framework wrapping this predictive code—the system responsible for generating parameter fitting spaces, optimizing rule combinations, and testing statistical significance and transaction costs—cannot be written by AI or run by AI agents. He allows a minor exception: small parts of the framework code may be assisted by AI, AI may suggest improvements, and tests may be written with AI assistance, but the overall architecture and logic must remain under human control.

This framework includes two specific mechanisms. The first is an 'add-only' system: once a predictive rule has been tested, it cannot be removed from the backtest. This directly addresses implicit overfitting—if researchers can continuously remove poorly performing rules based on backtest results, they are essentially selecting within the sample. Carver believes that with this mechanism, multiple testing corrections are unnecessary, and there is no risk of incorrectly discarding rules that work but have natural profitability levels below an arbitrary statistical threshold. The second mechanism is testing both the positive and negative versions of each strategy simultaneously to counteract tacit overfitting—if AI or humans already know that trend following is effective, testing only the positive version amounts to encoding known answers into the design.

Carver himself acknowledges a significant unfinished aspect of this framework: the specific implementation of the robust fitting engine is currently under development and will be detailed in future blog posts and his sixth book. Additionally, he points out an inherent limitation: while the correctness of small predictive code can be verified, the code itself may already carry tacit fitting, which is invisible at the code level. Carver offers no solution for this, leaving it as a known boundary of the framework.

Sources

  • qoppac.blogspot.com↗
Continue researching →