Meta Engineer Aditya Gautam Details Three-Agent Architecture for Short-Form Video Misalignment and Attribution

Meta engineer Aditya Gautam presented a talk on the AI Engineer channel detailing two core problems he works on at scale: cross-modal temporal misalignment within single videos (e.g., sports content shifting to political topics) and the difficulty of attributing originality amidst AI-generated content.

Gautam outlined a three-agent architecture to tackle these issues. The Perceiver splits videos at points of temporal change, emitting clip-level embeddings, tags, and OCR. The Reviewer analyzes this JSON output alongside live comments and sentiment. The Retriever indexes topics, embeddings, and entities into inverted, vector, and graph stores to pull similar clips and authors during inference.

Each agent runs on a small, specialized Vision-Language Model (VLM) rather than a frontier model. The team pretrained the vision encoder on internal images, instruction-tuned it to a JSON schema, and optimized it via a DPO loop using production samples. Evaluation metrics extended beyond precision and recall to include per-node latency, reasoning budgets, token cost, and judge drift.