The earliest motivation for building EH was simply laziness. There are too many repetitive operations in a browser: viewing pages, finding content, clicking controls, and switching pages—I wanted the computer to do it for me. When V2EX user niuheishou built the web perception system EH, the first engineering decision he faced was: should the model look, or should the code look?
He initially chose the model. He built a complete set of YOLO annotation tools for himself, designed annotation rules, and manually annotated them one by one. During his most intense three days, he slept only twelve hours, accumulating approximately 2,300 basic training samples. The training results looked good, but problems arose when actually applied to pages: the samples were mostly from fixed browser resolutions and zoom levels, and the bias from his manual annotations was seriously learned by the model. Large elements could be handled roughly, but the boxes for small elements often deviated wildly—a seemingly small error landing on a button could determine exactly what gets clicked.
For the next three full months, he struggled against this jitter in the boxes. He added geometric priors, tried GNN correction, performed morphological dilation; re-annotated, re-trained, and supplemented data under different zooms and resolutions. Each time he felt closer to solving the problem, changing the page state would reveal new biases.
The turning point came from the DOM. He later described his feelings at the time: the browser already knew some things, yet he was still making the model guess again from pixels. After the DOM entered the system, the first thing that changed was annotation—the boxes that used to require painstaking manual drawing could now be generated by code according to unified rules. Inconsistencies in manual annotation decreased, and the work of patching around annotations became much lighter.
But the inherent problems of the model were not solved. The trained model still jittered; increasing the model size and input resolution did not completely eliminate deviations on the page. The question gradually shifted: how much longer must I patch along this path to achieve the stable results I want?
So the prototype began to shift. He broke down his own judgments about the page, writing them out as code line by line—which parts are content, which are controls, under what conditions action can be taken, and under what conditions it should stop. Subjective judgments that previously existed only in his mind gradually became rules that could be executed repeatedly and checked.
This process was slow, but for the first time it showed him another possibility: instead of constantly chasing and correcting deviations in the model's output boxes, he could first clearly articulate his process of understanding the page, and then let the code execute it. After the single-site version finally ran successfully, he faced the next problem—generalizability. One site required about a 200KB JSON configuration file; switching to another site meant continuing adaptation. It took about a month to analyze what could be reused across sites versus what was just site-specific experience, and two more months before the first cross-site reusable engine finally took shape.