700 tokens per second: Google TPU performance targets Nvidia dominance

The artificial intelligence chip market is witnessing a significant shift in vendor power as Google’s tensor processing units begin to serve major external clients. The move directly challenges the established dominance of Nvidia in the data center hardware space. This development marks a pivotal moment for enterprise buyers who have long relied on a single supplier for their computational needs.

Meta, the developer of the PyTorch framework, has integrated Google’s TPUs into its production environment through new, PyTorch-native versions of the SGLang and vLLM inference engines. These tools were previously dependent on Google’s proprietary JAX framework, creating a barrier for companies that preferred the flexibility of PyTorch. The new releases allow for a more seamless integration without requiring extensive code rewrites, making the hardware a viable option for a broader range of AI developers.

Zongwei Zhou, Meta’s lead for PyTorch and TPU integration, confirmed that the SGLang-TorchTPU engine is currently live in production. He attributed the success to a tight collaboration between Meta, Google, and the SGLang team at RadixArk. This partnership provides Meta with a mature serving stack that aligns with its existing software infrastructure. Jana van Greunen, director of PyTorch engineering at Meta, emphasized that the company cannot be limited by a single hardware architecture. She noted that the new tooling enables the deployment of PyTorch workloads directly onto Google’s chips.

This is not the first time Meta has experimented with Google’s hardware. In the past, the company used the PyTorch/XLA bridge to run models on TPUs between 2020 and 2023 before canceling those specific contracts. The current effort represents a more native approach to solving the compatibility issues that previously hindered adoption. Reports suggest that Meta signed a multi-year rental agreement with Google, with talks last year indicating rentals starting in 2026 and potential purchases in 2027.

Performance metrics indicate that the new configuration is competitive with leading Nvidia hardware. Inferact, a startup involved in building the vLLM integration, reported that a tuned setup serving the Kimi K3 model exceeded 700 tokens per second on the seventh-generation Ironwood TPU. This test utilized 16 chips with speculative decoding. In comparison, a similar setup on 16 Nvidia GB200s reached 452 tokens per second. Without speculative decoding, the new kernels for specific models deliver 1.4 to 2 times the throughput of the Nvidia baseline at small batch sizes.

The timing of this announcement coincides with Amazon’s decision to open-source its TorchNeuron backend for Trainium chips. Both moves appear designed to break the software lock-in that has favored Nvidia. The SGLang team plans to add support for new models, including DeepSeek V4.1 Flash, and aims to provide day-zero support for new open models by the first quarter of 2027. Stakeholders will share further details at the SGLang Summit in San Francisco in November 2026.