Reading

Models deployed into production often encounter inputs that were entirely absent from their training sets: proprietary terminology for new products, shifts in user behavior, or unexpected formatting. These issues remain invisible in offline benchmark tests because those benchmarks are frozen in the past. The core argument of this AWS blog is direct: without recording production inference data, teams cannot detect drift, identify failure patterns, or improve models using real interactions. The new Data Capture feature in Amazon SageMaker HyperPod writes every inference input and output to Amazon Simple Storage Service (Amazon S3) across three independent tiers (endpoint, load balancer, and model pod). The authors state that the latency impact is minimal and does not affect model availability.

The article walks through a concrete scenario to demonstrate this closed loop: a home-appliance retailer deployed a support-ticket router using vLLM with guided JSON decoding to classify tickets into six product categories. After going live, the retailer launched its own Arctis series of air conditioners. Because the router lacked a category for these units, it either routed them to 'Other' or incorrectly sent them to ProductFridge. The authors simulated pre- and post-launch traffic using 900 synthetic tickets. By capturing all records via Tier 3 to S3 and querying them with Athena, they found that the share of tickets in the 'Other' category jumped from 4.8% before launch to 36.8% after. A word-frequency analysis showed the term 'arctis' appeared 141 times, precisely pinpointing the blind spot.

After identifying the issue, the authors translated the diagnosis into a SQL labeling policy: tickets matching keywords like 'arctis', 'air condition', 'ac unit', or 'mini split' were relabeled as ProductAC, while others kept their original predictions; mixed signals were flagged for human review. This policy generated 900 training samples from the captured data. Using a second GPU on the same cluster, the HyperPod training operator ran a 60-second LoRA fine-tuning job (rank 16, approximately 29.9 million trainable parameters) and deployed the merged model as a second endpoint. On a held-out evaluation set of 280 tickets, recall for the new ProductAC category improved from 0/40 to 40/40, with no regression observed in the five established categories.

However, the authors highlighted several important limitations. First, the model's self-reported 'confidence' field is not a calibrated probability: among 14 air-conditioner tickets misrouted to ProductFridge, 10 reported confidence scores above 0.9, indicating that filtering by confidence would retain many incorrect labels. Second, because the evaluation and training sets came from the same generator, the 100% accuracy on the held-out set reflected template consistency rather than generalization capability. The authors wrote 24 manual probe tickets to test further; after fine-tuning, the model scored 20/24, exposing two shortcuts learned from the data: equating 'cooling' with air conditioners, and labeling any ticket mentioning Arctis as ProductAC (including stock and price inquiries). Third, the keyword strategy only works when new category boundaries are clear; ambiguous or overlapping categories require routing to Amazon SageMaker Ground Truth for human labeling. Ultimately, this is a vendor-conducted walkthrough using synthetic data, not an independently verified production result.

Non-profit research institute Allen Institute for AI (Ai2), in collaboration with researchers from Carnegie Mellon University, NVIDIA, and Johns Hopkins University, has released XYBench, a new benchmark comprising over 8,000 questions designed to test how AI assistants respond to requests based on misconceptions. The benchmark targets the 'XY problem': a user wants to achieve goal X but assumes approach Y will work, thus asking for help with Y while ignoring the actual issue. For example, a user might ask how to keep frosting from sliding off cupcakes, when the real problem is that the cupcakes are too warm to frost.

The research team evaluated models including Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro. The results showed that while these models often suggested solutions, they tended to keep their answers focused on the user's original, mistaken request rather than steering them toward their underlying goal. Compared to human reference answers, all five models were less likely to steer users away from the mistaken approach and toward a solution suited to their goal.

XYBench measures model responses across three dimensions: pragmatic solution (whether it addresses both the stated request and the intended goal), pragmatic emphasis (whether the answer focuses on the stated request or the intended goal), and misconception identification (whether it discourages the mistaken approach and guides toward the correct one). Experiments indicated that even when the user's potential goal was explicitly provided, models still underperformed humans in misconception identification, and sometimes performed worse, suggesting that detecting intent is only part of the challenge; models must also act on it when it diverges from the literal request.

The benchmark won the Best Resource Paper Award at EMNLP 2026. Data, code, and the paper are publicly available for researchers to evaluate whether prompting or training improvements can help models better identify misconceptions and offer more effective alternatives.

Humanoid robot researchers face a problem that seems simple but is actually tricky: human demonstrations capture the coordination and rhythm of natural movement, but robots cannot directly copy the same joint motions because their limb proportions, joint structures, and ranges of motion differ. John Malaska, Senior Product Manager at NVIDIA, provides a detailed explanation in a technical blog post about the adaptation pipeline of the open-source tool SOMA Retargeter, describing which stages make up the pipeline, what conflicts each stage addresses, and how configuration options affect the final output.

The pipeline consists of three stages. The first stage is scaling: mapping motion targets on the human skeleton proportionally to the size of the target robot. The second stage is inverse kinematics (IK) solving: using the Newton physics engine, built on NVIDIA Warp, to find a set of joint angles for each frame that allows the robot to get as close as possible to the scaled target positions. The third stage is post-processing correction, most typically foot-plant correction—detecting contact intervals and adjusting foot trajectories to eliminate sliding artifacts. These three stages address problems at different levels: scaling resolves size differences, inverse kinematics resolves joint reachability constraints, and post-processing correction resolves contact consistency.

Malaska uses a source action of a one-handed wave to illustrate the role of reference pose alignment: if the Booster T1's reference pose is not aligned with the SOMA human reference, the robot will extend both arms outward; after alignment, one hand waves beside its head while the other arm rests naturally at its side. The input motion and IK parameters are identical; the only variable is the alignment of the reference pose. This example isolates the effect of the pose-mapping variable, demonstrating that while scaling solves size differences, incorrect pose mapping causes the robot to perform semantically incorrect actions.

When a robot cannot simultaneously satisfy the targets for all body parts, IK weights determine which parts take priority. SOMA Retargeter provides an experimental IK weight optimizer that automatically adjusts weights using a short segment of motion representative of the target task—it repeatedly retargets these motions, checking the tightness of the robot's following of the targets and the smoothness of the motion, and saves the selected weights as a new configuration. Malaska recommends tuning the configuration with representative motions first, then processing larger motion collections through headless batch retargeting.

The output is a CSV file containing the robot's root position, orientation, and joint angles for each frame. These trajectories can be converted into motion library formats such as SONIC or ProtoMotions for use in reinforcement learning policy training. During training, the policy learns how to convert retargeted references into actual control actions; during simulation, balance, ground contact, actuator limits, and tracking accuracy are evaluated.