OpenAI Develops Pando for Enhanced Git Scalability
OpenAI has developed Pando, a new backend for Git repositories, to address the increasing size and demands of its monorepo. This initiative was driven by the need to support the scale of concurrent work enabled by Codex, without disrupting ongoing engineering efforts or altering the team’s Git usage.
Pando is designed to achieve two primary objectives: enabling horizontally scalable writes optimized for concurrent pushes to multiple branches, and maximizing read throughput to ensure clients can download data quickly enough to utilize their available network bandwidth. The implementation of Pando has resulted in a nearly fivefold reduction in median push latency and allows for the delivery of nearly a petabyte of Git data daily.
The decision to build a new source control system stemmed from the rapid changes in development pace at OpenAI, largely due to Codex. This evolution led to an increase in the number of branches, pushes, merges, and fetches, which in turn strained the existing systems supporting the monorepo. By the end of last year, the team experienced slow fetches and pushes, alongside failed merges, with reliability dipping below 90%.
OpenAI recognized that the limitations they faced might be due to Git itself or the way their monorepo was served. Git has historically been adapted by developers over the past two decades to accommodate evolving software development practices. OpenAI aimed to maintain a familiar workflow while enhancing the backend to manage parallel pushes to different branches and serve the same history to multiple readers simultaneously.
Pando retains compatibility with the Git protocol and client, allowing engineers and agents to integrate it into their existing workflows seamlessly. This compatibility also means that improvements made to Pando could benefit the broader community, as OpenAI plans to contribute enhancements back to the upstream Git repository.
Engineering for Git at Scale
Scaling a source control system involves two key dimensions: accommodating many writers pushing to different branches and many readers fetching the same history. Pando optimizes for both dimensions by coordinating updates to each branch independently and reusing packfiles across fetches.
By allowing unrelated branches to be updated in parallel, Pando ensures that readers fetching a branch receive the necessary history for that revision. Additionally, by reusing packfiles, Pando minimizes the computational overhead associated with generating individual packfiles for each client, enabling efficient scaling of writes and reads.
Git Object Storage Primer
Git repositories maintain two critical components: objects and references. Objects represent snapshots of files and their history, while references point to commits. Each object is identified by a hash of its type and contents, and they are immutable, meaning that changes create new blobs rather than altering existing ones.
Branches and tags serve as references that point to commits, and the structure of a Git repository allows for efficient storage and transfer of these objects. When a client fetches data, it specifies which commits it wants, and the server responds with the necessary objects while excluding those already held by the client.
Existing Distributed Git Service Patterns
In designing Pando, OpenAI evaluated conventional approaches to scaling Git, including three-phase commit (3PC) and write-ahead log (WAL) systems. The 3PC model requires coordination across multiple replicas, which can introduce latency and performance bottlenecks as write volumes increase.
Conversely, the WAL-based design utilizes a shared manifest in blob storage to order updates across the repository. While this approach allows for parallel uploads, it still faces challenges related to managing updates to the shared manifest, which can limit overall performance.
Architecting Pando
Pando rethinks some traditional constraints associated with Git systems. It adopts a relaxed consistency model, known as causal consistency, to scale writes and reuses packfiles to enhance read performance. This design allows for efficient handling of concurrent pushes while providing a consistent view of individual references.
With Pando, when a client pushes to a branch, the incoming packfile is stored as an immutable blob, and the system records a ref version that includes the new tip and any dependencies. This approach enables efficient fetching and minimizes conflicts during concurrent updates.
OpenAI’s simulation of Pando against traditional scaling methods demonstrated that it supports thousands of clients pushing concurrently without performance degradation. Pando’s architecture allows it to validate incoming packfiles while streaming them to blob storage, ensuring efficient handling of both pushes and fetches.
As OpenAI continues to enhance Pando, it aims to further contribute to the Git community, ensuring that the improvements made can benefit developers beyond its own organization.
Source: OpenAI News