V2EX user limuyang2: In an AI Agent memory system, switching the vector model requires rebuilding the entire database, while switching the intent analysis model only needs a single configuration change
A V2EX user shared the complete selection process for migrating the memory foundation of an AI Coding Agent to a self-built OpenViking. The core finding is that the vector embedding model determines the underlying spatial geometry of the database; once selected, it cannot be changed without requiring a full rebuild of the index. In contrast, reasoning nodes such as Vision-Language Models (VLMs), intent analysis, and reranking are stateless and can be swapped at any time by changing a line of configuration. Based on this, the author chose the Volcano Engine Doubao multimodal vector model (128K context) as the permanent foundation and unexpectedly discovered that Alibaba Cloud Bailian's Tongyi Lingque dialogue analysis model could reduce intent analysis latency from 7 seconds to 0.8 seconds, solving the frontend lag issues encountered by most users.
V2EX user limuyang2 migrated the memory foundation of their AI Coding Agent to OpenViking, an open-source project from ByteDance's Volcano Engine. They ran it on an entry-level Alibaba Cloud ECS instance costing 99 RMB per year with 2 cores and 2GB of RAM. With no local GPU memory available, all computation was offloaded to external APIs, leaving the machine to handle only storage and retrieval locally.
They summarized a core principle: components in an AI Agent memory system fall into two categories with completely different selection logic. Vector embedding models determine the spatial geometric features of the database's underlying layer. If you want to switch them later, even if the output dimensions remain identical, all previous vectors become invalid, necessitating a full-scale rebuild of the index across the entire database. As memories accumulate, the cost and downtime risks of secondary reconstruction are extremely high. Conversely, VLMs, intent analysis, and reranking involve immediate inference; if they prove unsatisfactory today, you can simply change a line of configuration and restart to switch them out.
Based on this asymmetry, they opted for Volcano Engine's Doubao doubao-embedding-vision-251215 (priced at 0.7 RMB per million tokens, with a 128K context window) for the vector model, rather than Alibaba Cloud Bailian's qwen3-vl-embedding, which costs the same 0.7 RMB but offers only a 32K context window. They abandoned the commonly recommended free BGE-M3 from SiliconFlow because real-world tests showed its recall quality lagged significantly behind newer generation models. Additionally, the public free pool frequently suffered from packet loss and timeouts.
The pitfall that truly caught most users off guard was the intent analysis node. OpenViking's official default example only populates the vlm and embedding configuration items. However, if query_planner is not explicitly declared, the system defaults to calling the main VLM to analyze intent for every round of conversation. If the chosen VLM is a large model with chain-of-thought capabilities (such as Zhipu's glm-5.3-flash), a single preliminary analysis consumes 6 to 7 seconds, causing the frontend interaction to freeze. Many mistakenly attribute this to insufficient machine specifications or inherent system bloat.
They tested a range of general-purpose large models as alternatives with unsatisfactory results: Alibaba Cloud Bailian's Qwen3.7-Flash required 1.1 to 1.4 seconds even with thinking mode disabled; SiliconFlow's Qwen3.5-9B had a throughput of only 30 tok/s, taking 4 seconds; and Qwen2.5-7B often failed to adhere to formatting standards, outputting raw arrays that caused parsing errors. Finally, they discovered tongyi-xiaomi-analysis-flash (Lingque - Dialogue Analysis Flash) in the Alibaba Cloud Bailian model marketplace. This model is officially positioned as a domain-specific model specializing in dialogue information extraction, scenario classification, and low-latency analysis.
Real-world test results showed a first-token delay of 0.31 seconds, a generation throughput of over 200 tok/s, and a total planning time of approximately 0.8 seconds, fitting well within sub-second budgets. More importantly regarding format strictness: OpenViking's underlying layer requires models to return standard JSON objects in the form of {"queries": [...]}. General-purpose large models tend to add polite conversational filler or wrap outputs in markdown fences, whereas Tongyi Lingque achieved 100% strict adherence, outputting clean, raw JSON that seamlessly matched the deserialization contract. In scenarios where chitchat is rejected from search, it decisively outputs an empty array within 0.4 seconds, avoiding meaningless vector database queries.
The author believes this model's success was not due to luck, but rather its vertical fine-tuning focus on dialogue extraction, scenario classification, and format compliance, which precisely aligned with the engineering requirements of OpenViking's pre-routing. Their final takeaway is that specialized small models dominate general-purpose large models in vertical scenarios. For tasks like intent analysis that require classification, extraction, strict JSON formatting, and extreme speed, fine-tuned dialogue analysis small models are vastly superior to general-purpose large models that habitually write lengthy essays or engage in chain-of-thought reasoning.
It should be noted that all data regarding latency, throughput, and format compliance mentioned above are based on the author's self-testing. Publicly available test scripts, sample sizes, and specific hardware configurations have not been disclosed. Conclusions may vary under different network environments, concurrent loads, and input complexities.