LINE Yahoo Intern: AI Alert Watcher Separates Deterministic and Exploratory Tasks
Huo Yonghao, a student at Hosei University, developed 'AI Alert Watcher' during his internship at LINE Yahoo. The article details the initial failure of letting an LLM handle all Kafka alert investigations (leading to timeouts, hallucinations, and format errors) and the subsequent architectural refactor that assigned 'four core metrics checked every time' to fixed code queries while leaving 'flexible clue mining' to the LLM. This design ensures basic facts reach Slack even if the AI part fails, offering a concrete example of responsibility separation for LLMs in operations.
ImportanceMaterialEvidenceE3 inspectableWrite-upQuick
A common pitfall when using Large Language Models (LLMs) for system operations is assigning them both "deterministic data retrieval" and "uncertain logical reasoning." Huo Yonghao, a third-year student at the Faculty of Information Science, Hosei University, experienced this firsthand while developing "AI Alert Watcher" during a four-week internship at LINE Yahoo. The tool was designed to receive Kafka alerts from Grafana, perform preliminary investigations, and push results to Slack.
The initial version used a typical Agent pattern: the LLM autonomously decided which data sources to query via Grafana MCP (Model Context Protocol), wrote PromQL statements, interpreted logs, and generated final reports. However, real-world integration revealed severe fragility. When the LLM guessed incorrect data source IDs, messed up MCP tool parameters, or got stuck in long metadata searches, the entire HTTP request would fail due to timeouts. Worse, if the LLM's JSON output deviated slightly from the schema, the parser would error out, resulting in no notification in Slack—operators wouldn't even know an investigation had started. Additionally, the LLM sometimes misinterpreted "No Data" as "Normal," or claimed it had "investigated" despite execution logs showing no actual data retrieval.