Microsoft and Hugging Face: Database-State Rewards Boost Qwen3.8-27B Agent Consistency Nearly Threefold
Microsoft and Hugging Face released the ThinkingBox training method, using final database states as reward signals for RL fine-tuning. Results show Qwen3.8-27B's rate of consistently completing tasks across 20 attempts rose from 7.5% to 21.1% on the ThinkingBox-Bench benchmark, significantly improving agent reliability.
Microsoft and Hugging Face jointly released the ThinkingBox training method and its experimental results on October 22, 2026. The method uses the final database state after an AI agent executes a task as an executable reward signal for reinforcement learning (RL) post-training, aiming to address the reliability issue where agents claim completion but fail to correctly modify data.
The research team used this method to fine-tune the open-source model Qwen3.8-27B for 50 steps using GRPO (Group Relative Policy Optimization). On the ThinkingBox-Bench benchmark, which contains 507 tasks, the trained model showed significant reliability improvements: the number of tasks successfully completed in all 20 attempts rose from 38 (7.5%) to 107 (21.1%), nearly tripling. The average single-attempt success rate (pass@1) also improved from 51.70% to 60.78%.
Notably, this gain in consistency came with a slight decrease in 'at-least-once' coverage (pass@20), dropping from 89.35% to 87.97%. This suggests that the RL process may have 'sharpened' the model's behavior, focusing on stably solving mastered tasks while slightly reducing its ability to explore new solutions.
For comparison, on this benchmark, Claude Opus 5.5 achieved a 47.53% consistency rate, GPT-5.4 reached 25.25%, and Kimi-K3 was at 13.41%. Although the fine-tuned Qwen3.8-27B still trails these frontier closed-source models in absolute performance, it demonstrates an effective path for optimizing open-source model reliability through environmental feedback. Related code and datasets have been open-sourced on GitHub and Hugging Face.