The First Cost Lever for Generative AI: Determine if a Step Needs a Model Before Deciding Which One
An Amazon Web Services (AWS) machine learning blog proposes a workflow decision framework: for each step, sequentially answer three questions—can it be validated by rules, can the model write the rule once, or is it selecting from fixed options—to route steps to deterministic code, constrained decision calls, or appropriately sized models, thereby reducing unnecessary model invocations.
The most overlooked cost lever in generative AI systems is not which model you choose, but whether a step actually requires calling a model at all. The Amazon Web Services (AWS) machine learning blog notes that teams often find a model call effective for one step during prototyping and then adopt it as the default for subsequent steps—including input validation, request body formatting, and routing selection. While costs go unnoticed at prototype volume, at production scale these low-value model calls become one of the largest expense items in the architecture: costs grow with request count, latency stacks up layer by layer in chained calls, and identical inputs may yield different results, making testing and auditing more difficult.
The blog proposes a three-question decision framework requiring sequential answers for each workflow step: First, can success be verified by a rule, such as format, pattern, keyword, or numeric range? Second, can the model write the rule once, such as a regular expression or JSON schema? Third, does the step select one answer from a fixed list, such as routing, classification, or a yes/no judgment? If the answer to all three questions is no, the step is open-ended and should be handed off to an appropriately sized model.
The core insight of the framework is that most routing and classification steps fall between rules and open-ended generation. Rules cannot cover every case, but these steps also do not require generating text—they only need to provide one answer from a predefined list of fixed options. The blog categorizes these steps as 'decision rather than generation,' advocating for a component that restricts output to allowed options and provides an escalation signal: confident answers proceed directly, while uncertain ones escalate to a larger model. On AWS, this can be implemented in four ways: using a small Amazon Bedrock model with structured outputs to limit answers to enumerated options plus an 'unsure' option; using embedding similarity to match the nearest labeled example, using the similarity score as an escalation signal; deploying the experimental open-source decision model Strands Decider, which returns calibrated confidence; or training a classifier on Amazon SageMaker AI to return category probabilities with low latency and overhead once labeled data is available. Regardless of the method chosen, the escalation signal must be validated against your own labeled data and thresholds set accordingly.