LecturaInvestigaciónRadarMarco de inversión
Iniciar sesión / Registrarse
← Volver a la lectura
NoticiasTema · AI告警调查拆分Material3 min de lecturaEditorial de DisconfirmAI

LINE Yahoo Intern: AI Alert Watcher Separates Deterministic and Exploratory Tasks

Huo Yonghao, a student at Hosei University, developed 'AI Alert Watcher' during his internship at LINE Yahoo. The article details the initial failure of letting an LLM handle all Kafka alert investigations (leading to timeouts, hallucinations, and format errors) and the subsequent architectural refactor that assigned 'four core metrics checked every time' to fixed code queries while leaving 'flexible clue mining' to the LLM. This design ensures basic facts reach Slack even if the AI part fails, offering a concrete example of responsibility separation for LLMs in operations.

2026-09-29 10:00 GMT+8

A common pitfall when using Large Language Models (LLMs) for system operations is assigning them both "deterministic data retrieval" and "uncertain logical reasoning." Huo Yonghao, a third-year student at the Faculty of Information Science, Hosei University, experienced this firsthand while developing "AI Alert Watcher" during a four-week internship at LINE Yahoo. The tool was designed to receive Kafka alerts from Grafana, perform preliminary investigations, and push results to Slack.

The initial version used a typical Agent pattern: the LLM autonomously decided which data sources to query via Grafana MCP (Model Context Protocol), wrote PromQL statements, interpreted logs, and generated final reports. However, real-world integration revealed severe fragility. When the LLM guessed incorrect data source IDs, messed up MCP tool parameters, or got stuck in long metadata searches, the entire HTTP request would fail due to timeouts. Worse, if the LLM's JSON output deviated slightly from the schema, the parser would error out, resulting in no notification in Slack—operators wouldn't even know an investigation had started. Additionally, the LLM sometimes misinterpreted "No Data" as "Normal," or claimed it had "investigated" despite execution logs showing no actual data retrieval.

Iniciar sesión / Registrarse
LecturaInvestigaciónRadarMarco de inversión

Facing only five days of remaining development time, Huo and his mentor re-evaluated the operators' true needs: regardless of the alert type, there were four core metrics that always needed checking—Traffic (inflow volume), HOST (server response status), Critical Cluster Issues (e.g., Offline Partitions), and Consumer Lag. Since this information was fixed, there was no need for the LLM to "think" about how to retrieve it.

Consequently, Watcher underwent a key architectural refactor, splitting the investigation into two independent stages. Stage 1 was executed by the Watcher code itself, using predefined PromQL queries to fetch the four core metrics. This process bypassed the LLM entirely, making it immune to model hallucinations, format errors, or API timeouts. Stage 2 then handed off to the LLM, which used Grafana MCP to read Loki logs, correlate other metrics, or look for specific clues like OOM/GC events based on the facts retrieved in Stage 1.

This "Deterministic + Exploratory" separation significantly improved robustness. First, even if the Stage 2 LLM call failed completely, the four core facts from Stage 1 would still be formatted and sent to Slack, ensuring operators saw the basic status. Second, troubleshooting became clear: no notification meant a Webhook issue; basic data without AI analysis meant an LLM stage issue; missing data meant a Prometheus query issue.

In implementation details, Huo introduced several critical improvements. For instance, Webhook processing was made asynchronous (returning HTTP 202 immediately) to avoid gateway timeouts from waiting for the LLM. He strictly distinguished four states of Prometheus return values: observed (value present), no data (empty result), failed (query error), and conflicting, preventing monitoring blind spots from being reported as healthy. Furthermore, he stopped trusting the LLM's natural language descriptions and instead checked OpenCode execution history logs to verify whether Grafana tools were actually called successfully and returned non-empty results.

Regarding cost and efficiency, Huo estimated expenses based on average token consumption from 20 alert samples. Assuming 10 alerts per day, each consuming ~21k input tokens and ~78k cache tokens, with 20 working days per month, the annual cost remained manageable. More importantly, citing team feedback, he noted that manual completion of the same preliminary investigation typically took 5-7 minutes. While Watcher doesn't replace all decision-making, it eliminates the time spent repeatedly looking up fixed metrics and allows operators to browse summaries during non-urgent moments (like meetings) before deciding whether to interrupt their work for deep dives.

The true value of this internship report lies not in showcasing a perfect AI Agent, but in honestly documenting the shift from "over-reliance on LLM flexibility" to "returning to engineering determinism." For teams building Operations Copilots, this is a clear signal: don't let the LLM handle database connections and fixed-field extraction; save its compute power for the ambiguous clues that truly require cross-modal correlation and semantic understanding.

Fuentes

  • techblog.lycorp.co.jp↗
Tema

AI告警调查拆分

日本LINE雅虎(LINEヤフー株式会社)的两名实习生在2026年秋季的技术报告中,揭示了大语言模型(LLM)直接用于基础设施告警调查时的典型失败模式:超时、幻觉及格式错误导致关键信息丢失。针对此问题,团队确立了“职责分离”架构:由固定脚本…

2 pasos2 piezasActualizado 2026-10-23 17:04

Cómo llegó hasta aquí

  1. 2026-10-17法政大学学生霍永豪发布报告,详述在LINE雅虎Kafka团队开发AI Alert Watcher的过程。初期全权委托LLM导致连锁失败,后重构为“Watcher固定查询4项核心指标 + LLM灵活挖掘线索”的双阶段架构。techblog.lycorp.co.jp ↗
  2. 2026-10-19高知大学学生坂下康信发布报告,介绍在LINE雅虎LINE礼物SRE团队构建的事件驱动AI自主调查护栏系统。通过容器隔离、权限分离和执行步数上限,解决了AI无限循环和误操作风险,并选定了轻量级Agent框架flue。techblog.lycorp.co.jp ↗

Lo que aún está pendiente

  • 该“固定+探索”分离架构在其他非Kafka或更复杂的微服务监控场景中是否具有普适性?
  • 随着LLM推理能力的提升,原本需要硬编码的“确定性查询”边界是否会动态缩小?

También en este hilo

  • Kochi University Student Sakashita Yasunobu Built an Event-Driven AI Investigation Harness During an Internship at Japan's LINE Yahoo Corporation2026-10-19
Leer el tema completo →
Continuar investigando →