GitHub Unveils Next-Generation Git Storage Architecture to Power Agentic Development

At last week’s GitHub Universe, the company announced a fundamental shift in its underlying Git storage architecture, a move designed to accommodate the rapidly evolving landscape of agentic software development. As AI agents increasingly take over the heavy lifting of coding, testing, and deployment, the traffic patterns on GitHub repositories have shifted from human-centric bursts to sustained, high-frequency machine interactions. The new architecture is built to support this, enabling thousands of concurrent agents to push to a single repository simultaneously while maintaining consistently low latency for read traffic.

Internal testing has yielded impressive results: the system has successfully supported 5,000 concurrent agents operating on a single repository, processing 7,200 pushes per second. This milestone marks the culmination of a year-long effort to transform GitHub’s infrastructure while keeping the platform live for hundreds of millions of developers. For the end-user, the experience remains unchanged—developers will continue to push, fetch, and branch as they always have—but the engine beneath the hood has been entirely reimagined.

The Shift Toward Agentic Scale

Over the past year, GitHub engineers observed a profound change in what constitutes a "busy" repository. Traditionally, even the most active repositories on GitHub.com experienced at most a few dozen writes per second, usually occurring in short, sporadic bursts. However, the rise of autonomous agents—which operate around the clock and in parallel—has turned these bursts into sustained, high-throughput traffic. This influx of activity creates an amplification effect: each push from an agent triggers a chain reaction of reads from CI systems, deployment pipelines, and other agents seeking context.

To address this, GitHub’s engineering team identified two foundational principles for their next-generation storage: minimizing coordination and decoupling storage from compute. The company’s previous efforts, such as migrating nearly all repository data into Azure and establishing virtual replicas backed by Azure Blob Storage, served as the essential bridge to this new architecture. By observing that Git objects are immutable and uniquely identified by their contents, engineers realized that the vast majority of a push does not require global synchronization. Only reference updates—the small pointers that track the current state of a branch—require consensus. This realization allowed GitHub to move away from tightly coupled systems where storing data, serving requests, and maintaining repositories were inseparable tasks.

Introducing "Project Rubicon"

In May 2026, following the successful migration to Azure, GitHub launched "Project Rubicon," a design initiative aimed at maximizing throughput while upholding the platform’s strict requirements for correctness, durability, and availability. Rubicon represents a departure from traditional Git hosting, where requests were often bottlenecked by the need to coordinate entire repository states.

Under the new model, incoming requests from Git clients, the API, and the web interface arrive at "Spokes," GitHub’s long-standing routing and load-balancing service. Spokes maintains its critical role in partition management and ensuring tenant fairness. However, the data plane behind Spokes is entirely new. Traffic is now directed to a fleet of virtual replicas—disposable compute nodes that serve reads and writes. Because these replicas treat local disk as a cache that can be rebuilt on demand, the system is resilient; if a replica fails, the repository continues to function as other replicas absorb the traffic and refill their caches.

For long-term durability, the system relies on two primary pillars: Azure Blob Storage for the immutable pack data, and a specialized service known as "Ref-CAS" for reference management.

Minimizing Coordination with Ref-CAS

Managing reference updates at scale is perhaps the most significant challenge in high-throughput Git environments. Because multiple agents may attempt to update a branch simultaneously, the system requires a mechanism to manage these "compare-and-swap" operations without creating performance bottlenecks. GitHub’s early experiments with object storage for reference updates showed that relying on simple blobs resulted in limited throughput, as only one conditional write could be in flight at a time. Conversely, sharding references across multiple blobs increased read latency, as the system had to query multiple locations to ensure it had the latest state.

The solution was the creation of Ref-CAS, a dedicated replicated log service. Ref-CAS operates as a small, high-performance consensus group that handles reference updates through a quorum. When an agent pushes to a repository, the update is committed to the Ref-CAS log, providing a sequence number that acts as the repository’s "version." This allows for incredibly efficient reads: if a virtual replica checks the version and finds it is current, it avoids the overhead of fetching any data at all. If the replica is behind, it receives only the necessary updates. Critically, Ref-CAS never handles object data itself; it merely stores pointers to the pack data already safely residing in Azure Blob Storage. This separation allows the system to process tens of thousands of reference operations per second.

Decoupling Storage and Compute for Compaction

Maintaining repository performance while handling thousands of writes per second necessitates an efficient "compaction" strategy. In legacy architectures, compaction—the process of merging Git packs and removing unreachable objects—often ran on the same nodes serving user traffic, creating performance degradation. In the Rubicon architecture, this task is entirely offloaded to dedicated compaction nodes.

These nodes operate independently in the background, interacting directly with Blob Storage and the Ref-CAS service. By using a leasing system, multiple compaction nodes can work on different "shards" of a repository simultaneously, allowing the system to scale horizontally. As these nodes merge packs and update the pack manifest, the virtual replicas are notified of the changes, ensuring that the repository remains performant without sacrificing CPU or memory on the nodes that handle user requests. This creates a "flat" pack count, where the efficiency of the repository remains stable even as the volume of pushes continues to climb.

The Path Ahead

The Rubicon architecture is not just a solution for today’s traffic; it is a foundation for a future where agentic development is the norm. GitHub’s ultimate goal is to eliminate the need for special handling or manual migrations for repositories as they scale. Whether a project begins as a solo side-effort or explodes into a massive monorepo with thousands of contributing agents, the underlying infrastructure is designed to adapt seamlessly.

Looking forward, the team is focused on three key areas: further increasing the scale of the architecture, reducing the cost of agent-driven operations by minimizing data fetches, and contributing back to the Git project to evolve the protocol itself for better compatibility with autonomous workflows. This new architecture, currently in a limited private preview with a broader rollout scheduled for early next year, represents a pivotal moment for GitHub. As the nature of software development changes, the tools that support it are evolving to ensure that the only limit on innovation is the breadth of the developer’s ideas.