Systematic Trading Author Robert Carver: AI Can Only Participate in Idea Generation and Small Code Writing in Quantitative Investment Research; Testing Frameworks Must Be Written by Humans
Carver proposes that in systematic trading research, AI should be limited to two stages: idea generation and writing small, parameterizable predictive code. The robust testing framework responsible for evaluating these predictions must be written and controlled by humans, because the opacity of AI makes three types of overfitting undetectable.
Robert Carver has proposed a clear boundary for the use of AI in systematic trading research: allow AI to participate in idea generation and the writing of small, parameterizable predictive code, but require that the robust testing framework used to evaluate these predictions be written and controlled by humans. The core logic of this suggestion is not simply that AI is unreliable, but rather that the opacity of AI renders overfitting undetectable.
Carver categorizes overfitting into three types: explicit, implicit, and tacit. Explicit overfitting involves directly tuning parameters within the sample data. Implicit overfitting occurs through indirect fitting via repeated research loops. Tacit overfitting happens when researchers already know what works and unconsciously encode known answers into the code during strategy design. He argues that while these three types of overfitting persist after AI intervention, they are wrapped in an opaque block, making it impossible even to know if they are occurring.
In Carver's framework, AI is permitted to participate in only two stages. First is idea generation, data scoping, and initial research. The output of this stage must be trustworthy data sourced directly from original sources, not data paraphrased by AI, because AI may introduce hallucinations or hidden look-ahead bias. Second is writing a small piece of parameterizable Python code that takes data as input and outputs a forecast, incorporating parameters that require calibration. This code must be small enough for its correctness to be verified by humans, though Carver notes it likely contains tacit fitting.