ReadingResearchRadarInvestment frameworkAI Company Primary Sources
Sign in / Sign up
← Back to reading
NewsMaterial3 min readDisconfirmAI Editorial

InfluxData Developer Content Writer Jessica Wachtel Independent Test: GPT-6.1 Terra's Numerical Aggregation Accuracy Is Only Half That of Astra, But It Returns Uncomputed Queries to the User

OpenAI claims that GPT-6.1 Terra reaches parity with its flagship Astra model on the Agents' Last Exam benchmark at one-tenth the price. Developer content writer Jessica Wachtel independently tested three synthetic task sets and found that while both models tied in event triage, Terra scored only 26/50 on SQL aggregation tasks compared to Astra's 48/50. Terra's failure mode was not fabricating numbers, but rather returning the most difficult calculation parts to the user, assigning the same confidence level to incorrect answers as it did to correct ones.

2026-10-30 22:59 GMT+8

When OpenAI released GPT-6.1 Terra on October 26, it claimed the model performed comparably to its flagship GPT-6 Astra on the Agents' Last Exam benchmark, while costing only one-tenth per token. Jessica Wachtel, a developer content writer at InfluxData, decided to independently verify this claim using three sets of synthetic tasks. She ran each model once through the OpenAI API with high reasoning effort settings and found that the claim does not hold true across all task types.

The widest divergence appeared in SQL reasoning tests: she constructed a fake database containing 500 software build records and posed 25 questions covering soft deletes, retries, three currencies, and time zone dependencies. Astra scored 48/50, while Terra scored 26/50. Terra processed all 500 rows and correctly handled soft deletes and retries, but on six questions requiring cost or duration aggregation across all rows, it did not attempt to provide an answer. Instead, it wrote out SQL queries for the user to run themselves, marking them as 'not calculated.' Wachtel noted this was the first time she had seen AI hand work back to the user.

More concerning is another failure mode exhibited by Terra. In the SQL test, it answered six questions incorrectly, four of which involved count deviations of only one or two. Terra assigned the same confidence levels to these incorrect answers as it did to its correct ones, providing no signal that they might be wrong. If these figures were used directly in reports, users would have no way of knowing which data points required verification.

Sign in / Sign up
ReadingResearchRadarInvestment frameworkAI Company Primary Sources

However, Terra did not lag behind in all tasks. In the incident log triage test—which involved approximately 7,000 lines of fault logs across more than 20 services, including distractors such as drifting clocks, misattributed errors, and irrelevant alerts—both models achieved perfect scores of 50/50. Both successfully traced changes to the ledger service settings and identified three servers with incorrect clocks, without being misled by false information. In the data validation test, Terra scored 45/50 and Astra scored 50/50, a relatively small gap.

Across all three tests combined, Terra took a total of 258.8 seconds with an estimated token cost of $0.61, scoring 121/150; Astra took 709.2 seconds with a cost of $7.88, scoring 148/150. Terra’s cost was approximately 7.8% of Astra’s, and its speed was about 2.74 times faster. Wachtel pointed out that these runs demonstrate the trade-offs between cost, time, and score, but they do not prove that OpenAI deliberately reduced reasoning capabilities to achieve these results.

Wachtel also highlighted important limitations: these are single-run results on synthetic tasks, not estimates of repeatability or performance across all workloads. She iterated through three versions of the tests because both models consistently returned identical scores on simple and medium-difficulty versions, prompting her to increase the difficulty until the weaker model, GPT-5 Mini, scored very low. Because these high-difficulty versions were selected after observing earlier results, they should not be treated as representative samples of everyday work. Her recommendation is to remain cautious regarding unverified numerical aggregation or exhaustive data validation, but Terra may be an acceptable low-cost alternative for event triage-type tasks, provided the results are verified.

Sources

  • thenewstack.ioThe New Stack↗
  • openai.comOpenAI↗
Continue researching →