AWS Multi-Agent System for Aerospace Component Substitution Assessment: Deterministic Code Makes Verdicts, Large Models Handle Retrieval, Routing, and Synthesis
An Amazon Web Services machine learning blog presents a multi-agent system designed for aerospace component substitution assessment. Its core design principle strictly reserves decision-making authority for deterministic Python code, while large language models are limited to retrieval, routing, and document synthesis. The system employs a three-tier model division of labor (Haiku for routing, Sonnet for tool execution, Opus for synthesis) and utilizes Nova multimodal embeddings to calculate cosine similarity between part photographs as supplementary evidence. All assessment results are derived from self-reported tests on a fictional dataset.
When an aerospace engineer faces a qualified component that is obsolete or has an excessively long lead time, they must spend days cross-referencing vendor datasheets, interface control documents, and qualification records to draft a substitution justification for review. An incorrect substitution on a flight-critical component can lead to mission failure, yet the evidence chain supporting each decision is often scattered across emails and undocumented institutional knowledge, lacking a formal, repeatable package for audit.
The Amazon Web Services machine learning blog describes a multi-agent engineering assistant aimed at compressing this manual cross-referencing process. Its core design rule is singular: engineers make every decision, and the system assembles the evidence supporting that decision. This rule dictates where large language models are used and where deterministic code is employed.
The architecture is defined by three pillars. First, deterministic specification-fit evaluation: Python-written tolerance-rule code compares each candidate component against the reference component and the engineer’s stated requirements item by item. Because identical inputs consistently yield identical pass/flag conclusions, the results can enter formal engineering review. Second, multi-agent orchestration: a lightweight planner routes queries to three specialist agents (Catalog, Evaluation, Knowledge), and a supervisor synthesizes findings only when more than one specialist contributes. Third, human-in-the-loop decision-making: the system drafts justification memos and Integration Impact Assessments for engineering review but does not recommend or approve substitutions. Every evaluation also writes an audit entry aligned with AS9100 sections 8.4 and 8.5.6.
Model tiering represents a pragmatic cost-latency choice within this architecture. Routing is a classification task involving no tool calls, so it does not require a large reasoning model; a Haiku-based planner returns a numbered routing plan in seconds. Specialist agents use Sonnet to execute tools. The Opus supervisor runs only when two or more specialists contribute, because synthesizing a single specialist’s already complete answer adds approximately 8 seconds of latency without quality gain.
The deterministic scoring engine is the foundation of the system’s auditability. Specification-fit evaluation runs as standard Python code against hand-written classification requirement patterns. These patterns define critical specifications and tolerance rules for each component category. The score_substitution_fit function compares items, counts passes and flags, and presents lifecycle risk. Because conclusions must be reproducible, this step involves no large language models. Specialist agents present result tables and explain flags but cannot alter the verdict.
Amazon Nova Multimodal Embeddings assumes two distinct roles in this scheme. In multimodal retrieval, datasheets, interface control documents, and performance charts are embedded into the same semantic space; engineers can input text queries or upload chart images, and retrieval returns matching charts with source attribution. In visual similarity scoring, when two candidate components are compared, Nova Multimodal Embeddings calculates the cosine similarity of their photographs, after which Claude analyzes specific observable differences (mounting hole positions, connector locations, housing geometry). This score provides auxiliary evidence for deterministic specification comparisons, but the verdict itself is based on specification matching.
The authors share three practical lessons learned while building retrieval paths. First, crop charts before indexing: entire datasheet pages contain roughly 90% boilerplate content (headers, specification tables, footers) shared by all components, which dominates embedding vectors. After cropping chart regions, image-upload matching improved from 3 out of 7 correct parts to 7 out of 7. Second, filter results when an engineer specifies a part name: cross-modal text-to-image similarity cannot reliably return the correct part’s chart because torque curve images do not display visible part names. Third, log every silent fallback on the image path: missing s3vectors:QueryVectors permissions do not raise errors; image queries silently degrade to text retrieval, causing the model to subsequently name the wrong part with high confidence.
Regarding assessment results, all trajectory-level quality metrics scored above 0.95, and tool parameter extraction was near-perfect at 0.980. Tool selection accuracy improved from 0.809 to 0.861, primarily attributed to consolidating into a single managed knowledge base, fixing missing permissions on the image query path, and tightening planner routing to avoid chaining unnecessary specialists. The target success rate of 0.892 is lower because the query set includes safety-class prompts; agents correctly refuse to recommend or approve substitutions, but the evaluator counts these correct refusals as unmet goals.
It must be clarified that all assessment figures come from self-reported tests on a synthetic dataset from a fictional manufacturer, AnyCompany, without real engineering review or independent reproducibility verification. The solution demonstrates a reusable architectural pattern: in domains where verdict conclusions require auditability and reproducibility, strictly separate deterministic code from LLM reasoning. LLMs handle fuzzy retrieval and narrative synthesis, while code handles precise comparison and final determination.
This pattern is transferable to other component categories. Adapting the domain requires three changes: replace component data, edit requirement patterns to define critical specifications and tolerance rules for the category, and point the knowledge base to your own S3 corpus. Agents, tools, and the scoring engine require no modification. Because deterministic tools are standard Python functions, PLM or ERP integrations can call the scoring function directly without agent involvement.