700 Tokens Per Second: Google's AI Chips Break Into Meta's Infrastructure

Meta’s production AI infrastructure is now processing data at speeds exceeding 700 tokens per second on Google’s seventh-generation Ironwood tensor processing units. This performance milestone, achieved using a tuned configuration of the vLLM inference engine, marks a significant shift in the landscape of artificial hardware. The benchmark, which involved 16 Ironwood chips utilizing speculative decoding, outperformed a comparable setup of 16 Nvidia GB200s that managed 452 tokens per second. This specific test focused on serving Moonshot AI’s open-weight Kimi K3 model, which features 2.8 trillion parameters.

The breakthrough relies on the launch of TorchTPU, a new native backend that allows Google’s TPUs to run directly on the PyTorch framework. Previously, running PyTorch workloads on TPUs required using JAX, Google’s own framework, which created friction for companies like Meta that are deeply invested in PyTorch. The new SGLang-TorchTPU and vLLM-TorchTPU versions eliminate the need for extensive code rewriting, allowing the hardware to function as a native option for PyTorch users. This development is being framed as a critical step to loosen the software lock-in that has long favored Nvidia’s ecosystem.

Meta’s integration with this new stack is immediate and operational. Zongwei Zhou, Meta’s PyTorch-TPU lead, confirmed that SGLang-TorchTPU is already running in production at Meta. He described the achievement as the result of a close three-way collaboration between Meta, Google, and the SGLang team. Zhou noted that the engine provides Meta with a remarkably mature, PyTorch-native serving stack. This team operates at RadixArk, a startup founded by the creators of SGLang, which built the engine in partnership with the two tech giants.

Jana van Greunen, director of PyTorch engineering at Meta, emphasized the strategic importance of hardware flexibility. In a post on Google’s developer blog, she stated that Meta cannot be constrained by a single hardware architecture. She added that TorchTPU enables the company to deploy its PyTorch workloads directly to Google TPUs. Google confirmed that it co-developed the TorchTPU backend specifically with Meta’s PyTorch team to address this need.

This is not Meta’s first foray into Google’s hardware, but it represents a more integrated approach. Google refers to PyTorch/XLA, the older bridge between PyTorch and TPUs, as a predecessor to the new TorchTPU. Reports from SemiAnalysis indicated last year that Meta research teams had used TPUs via that bridge between 2020 and 2023 before canceling those contracts. The current push appears to be a renewed effort to make TPUs a viable, long-term option for Meta, which had previously resisted migrating to JAX.

The timing of this announcement is notable for its proximity to competing moves in the industry. It came just one day after Amazon announced it would open-source TorchNeuron, its own native PyTorch backend for its Trainium2 and Trainium3 chips. Both Google’s and Amazon’s initiatives appear designed to offer alternatives to Nvidia’s dominant position by ensuring that major AI frameworks can run natively on their respective custom silicon.

While the exact location of Meta’s production TPUs remains unclear, industry talks last year suggested a path toward Google Cloud TPU rentals in 2026, followed by hardware purchases in 2027. Meta is reported to have since signed a multi-year rental agreement. The new kernels for models like Kimi K3 and Qwen 3.8 27B reportedly deliver roughly 1.4 to 2 times the throughput of the GB200 baseline at small batch sizes, even without speculative decoding.

Looking ahead, the SGLang-TorchTPU project plans to expand its model support, starting with DeepSeek V4.1 Flash and Reflection Beam. RadixArk is also working toward Day 0 support for new open models by the first quarter of 2027. Google, Meta, and RadixArk are scheduled to share further details at the SGLang Summit 2026, taking place in San Francisco on November 12 and 13.