ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Web Perception: Three Months of Tinkering with Model Bounding Boxes Was Less Effective Than Coding My Own Judgment

An independent developer documents the complete detours taken while building a web perception system called EH. Initially using YOLO for visual localization, manual annotation bias and resolution dependency caused bounding box jitter, requiring three months of patching. Introducing the DOM reduced annotation inconsistency, but the model still jittered; ultimately, subjective judgments about pages were decomposed into executable code rules to create a cross-site reusable engine. The article includes throughput and coefficient of variation data across six sites, five sets of window condition tests, and the author's self-defined boundaries regarding generalizability.

2026-11-07 21:30 GMT+8

The earliest motivation for building EH was simply laziness. There are too many repetitive operations in a browser: viewing pages, finding content, clicking controls, and switching pages—I wanted the computer to do it for me. When V2EX user niuheishou built the web perception system EH, the first engineering decision he faced was: should the model look, or should the code look?

He initially chose the model. He built a complete set of YOLO annotation tools for himself, designed annotation rules, and manually annotated them one by one. During his most intense three days, he slept only twelve hours, accumulating approximately 2,300 basic training samples. The training results looked good, but problems arose when actually applied to pages: the samples were mostly from fixed browser resolutions and zoom levels, and the bias from his manual annotations was seriously learned by the model. Large elements could be handled roughly, but the boxes for small elements often deviated wildly—a seemingly small error landing on a button could determine exactly what gets clicked.

For the next three full months, he struggled against this jitter in the boxes. He added geometric priors, tried GNN correction, performed morphological dilation; re-annotated, re-trained, and supplemented data under different zooms and resolutions. Each time he felt closer to solving the problem, changing the page state would reveal new biases.

Sign in / Sign up
ReadingResearchRadarInvestment framework

The turning point came from the DOM. He later described his feelings at the time: the browser already knew some things, yet he was still making the model guess again from pixels. After the DOM entered the system, the first thing that changed was annotation—the boxes that used to require painstaking manual drawing could now be generated by code according to unified rules. Inconsistencies in manual annotation decreased, and the work of patching around annotations became much lighter.

But the inherent problems of the model were not solved. The trained model still jittered; increasing the model size and input resolution did not completely eliminate deviations on the page. The question gradually shifted: how much longer must I patch along this path to achieve the stable results I want?

So the prototype began to shift. He broke down his own judgments about the page, writing them out as code line by line—which parts are content, which are controls, under what conditions action can be taken, and under what conditions it should stop. Subjective judgments that previously existed only in his mind gradually became rules that could be executed repeatedly and checked.

This process was slow, but for the first time it showed him another possibility: instead of constantly chasing and correcting deviations in the model's output boxes, he could first clearly articulate his process of understanding the page, and then let the code execute it. After the single-site version finally ran successfully, he faced the next problem—generalizability. One site required about a 200KB JSON configuration file; switching to another site meant continuing adaptation. It took about a month to analyze what could be reused across sites versus what was just site-specific experience, and two more months before the first cross-site reusable engine finally took shape.

By April 2026, he conducted systematic tests. Six sites (Xiaohongshu, Bilibili, Taobao, Pinterest, Reddit, TikTok), totaling 21 runs, focused on collection throughput and coefficient of variation (CV). Results ranged from 1.42% to 8.34%: Pinterest at 78.90 Pins per minute with a CV of 1.42%; Bilibili at 50.9 content cards per minute with a CV of 2.70%; TikTok at 5.70 videos per minute with a CV of 3.10%; Xiaohongshu at 40.0 content cards per minute with a CV of 5.83%; Taobao at 28.03 product cards per minute with a CV of 7.85%; Reddit at 15.63 posts per minute with a CV of 8.34%.

Regarding resolution, he used the TikTok video switching path to conduct five sets of window and zoom condition tests, covering laptop baseline windows, 125% browser zoom, narrower windows, and native 4K and 2K modes. Across all five sets, 150 video sources were recorded, and all 145 switches succeeded without getting stuck. For someone tortured by model box jitter for three months, this result was indeed reassuring.

However, the author drew clear boundaries: this mainly validated only the video switching path, and did not prove that all operations on all sites could adapt to all resolutions. When Xiaohongshu switched layouts in narrow windows, configurations relying on fixed areas encountered issues early on. Each site had only three to six runs, representing small-sample observations; CV measures throughput fluctuation and cannot be treated as recognition accuracy or task success rate; since each site collects different objects, throughputs are not suitable for direct horizontal ranking.

These tests belong to early engine experiments from April 2026, still using site configurations. That the same framework could run on several sites was progress; stability after changing content or windows is the next step. Reducing site configurations and lowering maintenance burdens requires further advancement. The author said that to make the computer rely less on him, he first had to clarify his accustomed judgments bit by bit.

Sources

  • v2ex.comV2EX↗
Continue researching →