Google and Meta Open-source TorchTPU to Bring Native PyTorch Support to Google TPUs

TorchTPU is an open-source project developed by Google in collaboration with Meta, designed to provide a native PyTorch backend for Google TPUs. This initiative enables PyTorch workloads to run on TPU hardware with minimal code modifications, supporting eager and compiled execution, distributed computing, and hardware-specific optimizations.

Google began facilitating PyTorch workloads on TPUs in 2020 through PyTorch/XLA, which utilized JAX, XLA, and lazy tensors. However, Google identified significant limitations with this approach, including the need for intrusive source-code modifications, severe compile-time latency, and inadequate support for PyTorch’s native eager execution mode.

TorchTPU adopts a fundamentally different strategy, prioritizing eager execution and allowing existing PyTorch workloads to run with minor adjustments. Transitioning from GPUs to TPUs involves simply replacing the `cuda` device with `tpu` and the `nccl` distributed backend with `tpu_dist`. Developers can also utilize the new `torch.accelerator.current_accelerator` API to identify the available accelerator.

TorchTPU supports three eager execution modes: strict, debug, and DeferAndFuse. The DeferAndFuse mode is a hybrid eager/compiled execution approach that Google claims can be 50% or more faster than Strict Eager while reducing peak HBM memory usage.

To evaluate TorchTPU’s capability to run existing PyTorch models without modifications, Google established an integration pipeline that assessed approximately 5,000 Hugging Face models across over 500 architectures in both eager and compiled modes on various TPU hardware generations.

For performance validation, Google tested a 30-billion parameter Muse Glimmer model implemented in standard PyTorch. On a single TPUv7 chip, inference throughput reached 25.7k token/s at a sequence length of 5120 and 22.4k token/s at 10,240. For training, using four TPUv7 chips, throughput was 5.6k token/s and 4.6k token/s at the same sequence lengths.

Google also conducted experiments with the Qwen3 MoE 235B model for training workloads, scaling from 1,024 to 8,192 TPUv7 chips, reporting competitive performance metrics.

Source: InfoQ