ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

The First Cost Lever for Generative AI: Determine if a Step Needs a Model Before Deciding Which One

An Amazon Web Services (AWS) machine learning blog proposes a workflow decision framework: for each step, sequentially answer three questions—can it be validated by rules, can the model write the rule once, or is it selecting from fixed options—to route steps to deterministic code, constrained decision calls, or appropriately sized models, thereby reducing unnecessary model invocations.

2026-11-18 05:14 GMT+8

The most overlooked cost lever in generative AI systems is not which model you choose, but whether a step actually requires calling a model at all. The Amazon Web Services (AWS) machine learning blog notes that teams often find a model call effective for one step during prototyping and then adopt it as the default for subsequent steps—including input validation, request body formatting, and routing selection. While costs go unnoticed at prototype volume, at production scale these low-value model calls become one of the largest expense items in the architecture: costs grow with request count, latency stacks up layer by layer in chained calls, and identical inputs may yield different results, making testing and auditing more difficult.

The blog proposes a three-question decision framework requiring sequential answers for each workflow step: First, can success be verified by a rule, such as format, pattern, keyword, or numeric range? Second, can the model write the rule once, such as a regular expression or JSON schema? Third, does the step select one answer from a fixed list, such as routing, classification, or a yes/no judgment? If the answer to all three questions is no, the step is open-ended and should be handed off to an appropriately sized model.

The core insight of the framework is that most routing and classification steps fall between rules and open-ended generation. Rules cannot cover every case, but these steps also do not require generating text—they only need to provide one answer from a predefined list of fixed options. The blog categorizes these steps as 'decision rather than generation,' advocating for a component that restricts output to allowed options and provides an escalation signal: confident answers proceed directly, while uncertain ones escalate to a larger model. On AWS, this can be implemented in four ways: using a small Amazon Bedrock model with structured outputs to limit answers to enumerated options plus an 'unsure' option; using embedding similarity to match the nearest labeled example, using the similarity score as an escalation signal; deploying the experimental open-source decision model Strands Decider, which returns calibrated confidence; or training a classifier on Amazon SageMaker AI to return category probabilities with low latency and overhead once labeled data is available. Regardless of the method chosen, the escalation signal must be validated against your own labeled data and thresholds set accordingly.

Sign in / Sign up
ReadingResearchRadarInvestment framework

Before the decision framework lies an even simpler optimization: a deterministic fast path. Many messages carry unambiguous signals, such as 'refund' or 'reset password,' which keyword matching can resolve in single-digit milliseconds. If rules handle 95% of requests, only one in twenty requests reaches the model—this is before any savings from model selection are even realized. The blog recommends using Choice states in AWS Step Functions to branch before invoking any model, using Lambda to host rules requiring more logic, and using API Gateway request validation to reject malformed input before computation begins, sending only the residual cases that rules cannot resolve to constrained decision calls or models.

When a model is indeed needed, choosing the cheapest and fastest model that meets the quality floor is often not the newest or largest one. The blog suggests defining four requirements before comparing models: quality floor (the minimum acceptable output quality on a golden evaluation set), cost ceiling (the maximum paid per request or per successful outcome), latency target (the p95 or p99 response time the experience can tolerate), and compliance constraints (such as data residency requirements). Amazon Bedrock offers multiple model families and sizes behind a single API, supporting intelligent prompt routing and model distillation, allowing you to select the right size for each task without changing your integration.

The blog concludes with a four-step practical path: first, tag every invocation (deterministic code, constrained decision call, or model call) and track the cost per successful outcome; second, move rule-able steps to run on Lambda or Step Functions, which usually yields the largest and safest gains; third, add constrained decision calls with escalation signals for fixed-option steps; and finally, run the golden evaluation set before switching models, keeping models behind configuration switches for easy rollback. The core principle is simple: ask whether a step needs a model first, then ask which model to use.

Sources

  • aws.amazon.comAmazon Web Services↗
Continue researching →