ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsTopic · AI告警调查拆分3 min read·2026-10-17

LINE Yahoo Intern: AI Alert Watcher Separates Deterministic and Exploratory Tasks

Huo Yonghao, a student at Hosei University, developed 'AI Alert Watcher' during his internship at LINE Yahoo. The article details the initial failure of letting an LLM handle all Kafka alert investigations (leading to timeouts, hallucinations, and format errors) and the subsequent architectural refactor that assigned 'four core metrics checked every time' to fixed code queries while leaving 'flexible clue mining' to the LLM. This design ensures basic facts reach Slack even if the AI part fails, offering a concrete example of responsibility separation for LLMs in operations.

ImportanceMaterialEvidenceE3 inspectableWrite-upQuick

2026-09-29 10:00 GMT+8

A common pitfall when using Large Language Models (LLMs) for system operations is assigning them both "deterministic data retrieval" and "uncertain logical reasoning." Huo Yonghao, a third-year student at the Faculty of Information Science, Hosei University, experienced this firsthand while developing "AI Alert Watcher" during a four-week internship at LINE Yahoo. The tool was designed to receive Kafka alerts from Grafana, perform preliminary investigations, and push results to Slack.

The initial version used a typical Agent pattern: the LLM autonomously decided which data sources to query via Grafana MCP (Model Context Protocol), wrote PromQL statements, interpreted logs, and generated final reports. However, real-world integration revealed severe fragility. When the LLM guessed incorrect data source IDs, messed up MCP tool parameters, or got stuck in long metadata searches, the entire HTTP request would fail due to timeouts. Worse, if the LLM's JSON output deviated slightly from the schema, the parser would error out, resulting in no notification in Slack—operators wouldn't even know an investigation had started. Additionally, the LLM sometimes misinterpreted "No Data" as "Normal," or claimed it had "investigated" despite execution logs showing no actual data retrieval.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Facing only five days of remaining development time, Huo and his mentor re-evaluated the operators' true needs: regardless of the alert type, there were four core metrics that always needed checking—Traffic (inflow volume), HOST (server response status), Critical Cluster Issues (e.g., Offline Partitions), and Consumer Lag. Since this information was fixed, there was no need for the LLM to "think" about how to retrieve it.

Consequently, Watcher underwent a key architectural refactor, splitting the investigation into two independent stages. Stage 1 was executed by the Watcher code itself, using predefined PromQL queries to fetch the four core metrics. This process bypassed the LLM entirely, making it immune to model hallucinations, format errors, or API timeouts. Stage 2 then handed off to the LLM, which used Grafana MCP to read Loki logs, correlate other metrics, or look for specific clues like OOM/GC events based on the facts retrieved in Stage 1.

This "Deterministic + Exploratory" separation significantly improved robustness. First, even if the Stage 2 LLM call failed completely, the four core facts from Stage 1 would still be formatted and sent to Slack, ensuring operators saw the basic status. Second, troubleshooting became clear: no notification meant a Webhook issue; basic data without AI analysis meant an LLM stage issue; missing data meant a Prometheus query issue.

In implementation details, Huo introduced several critical improvements. For instance, Webhook processing was made asynchronous (returning HTTP 202 immediately) to avoid gateway timeouts from waiting for the LLM. He strictly distinguished four states of Prometheus return values: observed (value present), no data (empty result), failed (query error), and conflicting, preventing monitoring blind spots from being reported as healthy. Furthermore, he stopped trusting the LLM's natural language descriptions and instead checked OpenCode execution history logs to verify whether Grafana tools were actually called successfully and returned non-empty results.

Regarding cost and efficiency, Huo estimated expenses based on average token consumption from 20 alert samples. Assuming 10 alerts per day, each consuming ~21k input tokens and ~78k cache tokens, with 20 working days per month, the annual cost remained manageable. More importantly, citing team feedback, he noted that manual completion of the same preliminary investigation typically took 5-7 minutes. While Watcher doesn't replace all decision-making, it eliminates the time spent repeatedly looking up fixed metrics and allows operators to browse summaries during non-urgent moments (like meetings) before deciding whether to interrupt their work for deep dives.

The true value of this internship report lies not in showcasing a perfect AI Agent, but in honestly documenting the shift from "over-reliance on LLM flexibility" to "returning to engineering determinism." For teams building Operations Copilots, this is a clear signal: don't let the LLM handle database connections and fixed-field extraction; save its compute power for the ambiguous clues that truly require cross-modal correlation and semantic understanding.

Sources

  • 原始出处techblog.lycorp.co.jp↗
Topic

AI告警调查拆分

LINEヤフー实习生霍永豪在开发AI Alert Watcher时，确立了将Kafka告警调查拆分为“确定性固定查询”与“探索性LLM分析”的架构范式。初期让LLM全权负责数据源识别、指标获取及报告生成，因超时、幻觉及格式错误导致调查整体失…

3 steps1 piecesUpdated 2026-10-17 22:43

How it got here

  1. 2026-08-17霍永豪开始实习，接手基于Webhook触发LLM通过Grafana MCP进行全权调查的Watcher雏形。
  2. 2026-09-05结合测试发现LLM推测数据源ID、误用工具参数及JSON格式崩溃等问题，导致调查链条断裂。
  3. 2026-09-11剩余5天实稼动日时，确立新架构：Watcher固定执行4项核心指标查询，LLM仅负责追加调查与日志关联。

What is still unsettled

  • 固定查询的4项指标是否适用于所有类型的Kafka集群告警场景？
  • LLM在第二阶段的追加调查中，如何量化其对最终优先度判定的实际贡献权重？
Read the full topic →
Continue researching →