SemiAnalysis publishes first long-context, multi-turn AgentX benchmarks for TPUv7 across four models, November 5, 2026

SemiAnalysis said Google has taken up the challenge Nvidia CEO Jensen Huang issued on the Dwarkesh Podcast, when he noted that no other chip vendor was showing results on the InferenceX benchmark. The firm published what it calls the first official long-context, multi-turn, realistic agentic inference results for TPUv7 (Ironwood). The results cover four models. TPUv7 is either competitive on performance per dollar for external customers or, in some configurations, the best result submitted. SemiAnalysis frames this as the software half of the TPU story. Nvidia's moat has historically rested on its developer ecosystem as much as on silicon.

Qwen3.5 Results Prompted a Counter-Submission From Nvidia

The most consequential result is Qwen3.5 397B on vLLM, Google's first AgentX submission, made two weeks ago. SemiAnalysis says TPUv7 running FP8 delivered 1.67x better performance per dollar than GB300 NVL72 running FP4, a lower-precision format that normally carries a large throughput advantage. With llm-d disaggregated serving, TPUv7 FP8 beat GB300 FP8 at every interactivity level, by more than 2x at high throughput. It also exceeded 1,000 tok/s/user in the ultrafast region.

SemiAnalysis says Nvidia reacted aggressively. It "tasked a strike force of human engineers and agent swarms to work 997, late into the night until 2am local time and on weekends, across USA and mainland China." Nvidia's first GB300 FP4 counter-submission does beat TPUv7 FP8 on decode performance. The report says it was overtuned for decode at the expense of prefill, which hurt end-to-end results. On an end-to-end basis, TPUv7 FP8 still leads, and the counter-submission remains 1.64x worse per dollar than TPUv7 FP8 across most of the curve. SemiAnalysis believes Nvidia could regain a slight lead on GB300 FP4 versus TPUv7 FP8 with enough optimization. It does not think Nvidia can regain the lead in FP8-to-FP8 comparisons for certain models.

Nvidia may argue that Qwen3.5 does not matter to it. SemiAnalysis rejects that, citing its submission analytics. Nvidia has spent more than 20% of sweep GPU hours on the model for certain SKUs, and it has assigned some of its strongest inference engineers to it. SemiAnalysis adds that certain hyperscalers and AI labs use Qwen3.5 397B, and multi-trillion-parameter derivatives of its architecture, as proxy models for hardware purchasing and capacity planning.

GLM-5.2 and Kimi K3: Cheapest at FP8, but FP4 GPUs Still Win Outright

On GLM-5.2, a 744B-parameter mixture-of-experts model with roughly 39B active parameters, TPUv7 is the cheapest FP8 option. From 60 to 300 tok/s/user it delivers 1.2x to 2.7x the tokens per dollar of B200, B300 and MI355X, and 4.4x that of H200 at 150 tok/s/user. The report is blunt about the limit of that claim. Nvidia's FP4 GPUs still lead on cost at most speeds. At 100 tok/s/user, GB300 and GB200 FP4 deliver 3.6x and 3.5x the tokens per dollar of TPUv7. TPUv7 has no native FP4 support, and SemiAnalysis does not expect that to change until TPUv8 in Q1 2027. FP4 does carry a slight accuracy penalty on this model, because GLM-5.3 was not natively quantization-aware trained for block-scaled FP4. That blunts but does not erase Nvidia's advantage.

The Kimi K3 result, submitted with vLLM developer Inferact using MXFP4 weights, shows how quickly the competitive picture moves. At launch on October 17, TPUv7 delivered 0.77x to 0.91x of GB300's tokens per dollar from 50 to 200 tok/s/user. It was also 1.6x faster for a single user, at 391 tok/s/user against 251. On Google's internal cost basis it beat GB300 at 100 and 200 tok/s/user. After four Nvidia software refreshes, GB300 FP4 now leads by 1.5x to 2.6x. TPUv7 still delivers 1.15x to 2.5x the tokens per dollar of GB200, B200, B300 and MI355X. Preview Rubin VR200 submissions show 4.3x to 5.6x TPUv7, and up to 6x in some comparisons. SemiAnalysis calls that an unfair matchup, because Rubin should be compared with TPUv8i, which it plans to benchmark soon. It also stresses that the Rubin figures remain preview submissions it has not independently reproduced.

Meta's Muse Spark Open Shows TPU Competitiveness on a GPU-Shaped Model

The most technically interesting result may be Meta's Muse Spark Open, an 816B-parameter model run on SGLang with TorchTPU and submitted by RadixArk, a lead maintainer of SGLang. SemiAnalysis says Meta is already deploying TPUv7 in production through this stack. It uses SGLang for production inference and vLLM for reinforcement learning rollouts.

The model was designed with GPUs in mind. Its 64 head dimension means that, for long-context attention, 25% of the TPU's systolic array is padded. GPU matrix cores consume smaller tiles, so a head dimension of 64 instead of 128 costs almost nothing on an H100 or B200 while making the attention layer cheaper. SemiAnalysis notes that labs with large TPU fleets, such as Anthropic and Google DeepMind, and soon OpenAI, typically co-design models so attention shapes fit the hardware. TPUv7 still posted competitive decode performance per dollar, even on Meta's cost basis for buying TPUs. End-to-end latency is not yet competitive, and the report says prefill kernels need further optimization.

Where the Gains Come From

SemiAnalysis attributes much of the progress to software work by Google and open-source maintainers. Decode megakernels run an entire decode step in a single kernel and prefetch the next layer's weights while the current layer computes. That removes the idle gaps between the hundreds to thousands of kernel launches in a conventional step. AMD reports that its own version cuts a GLM-5.2 decode step from about 2,459 launches to 210 and runs 2.31x faster at batch 1.

On the Qwen3.5 stack, Google also open-sourced TPU Sync, a framework that spills KV cache and linear attention state to host DRAM. Cache-aware routing in llm-d, which sends each turn to the host holding its prefix, raised decode cache hit rates from 56% to 76% according to Google. For GLM-5.3, optimizations include decode context parallelism, use of TPUv7's SparseCores to gather the 2,048 selected cache entries, a repacked indexer layout that cuts wasted bytes, and additional small-batch kernel shapes. The disaggregated approach has costs. At the fastest point, p90 time to first token is 8.6 seconds against 11.1 seconds for GB300 FP4. Turn-based routing still leaves TPUs behind GB300 TRT-LLM FP4 end to end, though well ahead of GB300 SGLang at both FP8 and FP4.

Nvidia's Bureaucracy Problem

SemiAnalysis devotes a notable portion of its note to criticizing Nvidia's culture, calling bureaucracy "one of the biggest risks to NVIDIA Moat that Jensen has to tackle." It says open-source maintainers and customers report that simple driver features needed for PyTorch CUDA graph capture have sat in "internal NVIDIA legal review" for three months. Requests to improve DCGM power telemetry on ConnectX, NVSwitch and HGX boards have been similarly slow. In its experience, Nvidia now responds to external suggestions, commits and feedback more slowly than AMD, Google and Amazon.

Cluster access is part of the complaint. SemiAnalysis says Nvidia's contributions of compute to community maintainers have become "wrapped in DMV style bureaucracy and stingy at best," which it finds hard to square with a $5 trillion company. Google, by contrast, is using the Nvidia playbook of giving maintainers plenty of chips. Day 0 TPU support has now been delivered for Muse Spark Open and Kimi K3.1 Pro. AMD has also become more generous with upstream maintainers. The report credits some Nvidia teams, notably the ex-CentML vLLM group, which matched the AMD and TPU megakernel work within a week.

Caveats on the Evidence

Investors should weigh several points. The benchmark, InferenceX, is SemiAnalysis's own, though it is backed by a broad group of buyers, labs and open-source projects. The report separates Google's internal cost basis from the pricing external customers would pay, and the TPU advantage narrows or reverses on some of those comparisons. Rubin results are unverified previews, and Nvidia has so far shown it can improve its results quickly. The report concedes that TPUv7 lacks FP4 and trails on end-to-end latency in some configurations. Even so, SemiAnalysis argues that credible TPU price-performance on agentic workloads, delivered through vLLM, SGLang, llm-d and PyTorch, weakens the case that CUDA is the only viable production path. It also notes that Oracle, OpenAI, Anthropic and Meta have all become TPU customers, as its Accelerator Model projected. Further AgentX submissions from RadixArk and the first TPUv8i benchmarks are expected soon.