ReadingResearchRadarInvestment framework
Sign in / Sign up
← Back to reading
NewsMaterial4 min readDisconfirmAI Editorial

Square Enix Engineer Shota Fujii Tests Local LLMs: Basic Tasks Feasible, Complex Tool Orchestration Still Requires Frontier Models

Shota Fujii, an AI engineer at Square Enix, reported on results from deploying Nemotron 3.5 Super VL in an internal H100 environment during an NVIDIA event, validating it against four categories of game development tasks. While basic Q&A and script generation performed well with controlled costs, local models were completely unable to handle complex tasks involving free-form tool combinations (such as building a game from scratch), which must be routed to cloud-based frontier models. Although PDCA iteration improved action quality, a key finding was that developers must define evaluation criteria and verification methods themselves.

2026-11-11 19:29 GMT+8

Development costs for AAA projects at game studios continue to rise annually, a trend confirmed by data released by the Japanese Ministry of Economy, Trade and Industry in February 2026 showing extended development cycles and rising labor costs. At the Agentic AI Day Japan 2026 event hosted by NVIDIA Japan, Shota Fujii, an engineer in the AI and Engine Development Division at Square Enix, presented a potential path to reduce costs: delegating certain tasks to locally run large language models rather than relying entirely on pay-per-token cloud-based frontier models.

His assessment is not a simple replacement of all cloud calls with local models, but a tiered approach based on task difficulty: basic Q&A and script generation are handled locally, while high-freedom complex orchestration is still routed to frontier models like ChatGPT. This conclusion comes from systematic validation conducted across four types of game development tasks after he deployed Nemotron 3.5 Super VL on the company's H100 GPU cluster.

The first layer of validation focused on basic capabilities. In 'chat' tests, Shota Fujii evaluated the model across three dimensions: basic knowledge and short Q&A (e.g., explaining FPS terminology or answering what the world's highest peak is), calculation problems with definite answers (e.g., how many seconds are 180 frames at 60fps), and creative output requiring format constraints (e.g., providing three forest village names within eight characters each). All three types of tasks were completed correctly. Response time comparisons showed that Nemotron led other models in basic knowledge questions at 0.79 seconds (others exceeded one second), but was actually the slowest in calculation and creative tasks, indicating that its response speed fluctuated more significantly than cloud models as difficulty increased.

Sign in / Sign up
ReadingResearchRadarInvestment framework

In 'script generation' tests, the model successfully generated a correct Python image inversion script in one go, taking 22 seconds; it also output HTML mini-game code in one shot, including left/right movement, jumping, sword collision detection, and win conditions. Shota Fujii considered these basic validation results consistent with expectations, proving that Nemotron has practical value for structured coding tasks. However, he noted that for lower-difficulty script tasks, ChatGPT and Claude still hold a slight advantage in speed.

The second layer of validation involved multimodal capabilities and toolchains. As a vision-language model, Nemotron accurately described the content of a video of a dragon eating in about 10.6 seconds, a result Shota Fujii called personally surprising. The more complex 'action generation' test introduced NVIDIA's motion diffusion model Kimodo: Nemotron planned keyframes and foot positions, passing them to Kimodo in JSON format, which then generated full-body motions. In an Unreal Engine environment, the character successfully completed actions such as clearing obstacles, carrying items, jumping over gaps, climbing 1.2-meter steps, and crawling under a 1.1-meter bar.

The most insightful discovery in action generation came from iterative optimization using the PDCA cycle. In initial walking motions, the character's feet slid approximately 16.4 centimeters when crossing a bar. After having Nemotron execute its own Plan-Do-Check-Act cycle, the slide was reduced to 1.7 centimeters. However, the process exposed a counterintuitive conclusion: more keyframes are not necessarily better. The final solution with the highest completion rate used only six keyframes; adding more frames resulted in extra movements, foot slipping, or stiff hands.

The third layer, 'MCP integration,' testing revealed the hard limits of local models. The Model Context Protocol allows AI to directly call tools: Blender version 5.1 and above and Unreal Engine 5.8 both provide experimental MCP servers. Simple cross-tool tasks—generating a tree in Blender and placing it in an Unreal scene—were completed smoothly. However, when the task escalated to building a complete treasure-hunting game from scratch, Nemotron running alone stalled at an intermediate stage and could not produce a playable game. Introducing NVIDIA's model routing library NeMo Switchyard, which assigned simple steps to Nemotron and routed difficult steps to ChatGPT, allowed some objects to respond, but the game remained incomplete.

In his summary, Shota Fujii emphasized two practical points: first, local LLMs are suitable for basic operations, but high-freedom tasks currently require hybrid routing to frontier models; second, for difficult problems, the PDCA cycle is effective only if developers predefine evaluation criteria and correction strategies—the model will not design acceptance standards for you. He also suggested that when using video input to assist PDCA, capturing only the relevant problem segments rather than full-screen footage significantly improves improvement efficiency.

Sources

  • pc.watch.impress.co.jpPC Watch↗
  • siliconangle.comSiliconANGLE↗
Continue researching →