At the annual GitHub Universe conference, the engineering team behind the world’s largest code hosting platform unveiled a fundamental shift in its underlying Git storage architecture. This new system, designed to handle the explosive growth of "agentic" development, is capable of supporting thousands of concurrent writes per second while maintaining scalable, low-latency read performance. The breakthrough represents a pivotal moment for GitHub, as the platform transitions to support a future where thousands of automated agents—rather than just individual developers—collaborate simultaneously on the same repositories.
Internal benchmarks have demonstrated the system’s robustness, with the architecture successfully supporting 5,000 concurrent agents pushing to a single repository at a rate of 7,200 commits per second. Throughout these high-intensity operations, the platform maintained consistent, low-latency performance for read traffic. This transition, which GitHub has been executing while the service remains fully operational for its millions of users, is set to become the bedrock of the platform’s infrastructure as AI-driven coding becomes the industry standard.
The Challenge of Agent-Scale Development
Over the past year, the nature of a "busy" repository has been redefined by the rise of automated agents. Traditionally, GitHub’s infrastructure was optimized for human-led workflows, characterized by bursts of activity followed by periods of relative calm. However, as organizations increasingly deploy fleets of AI agents to handle refactoring, testing, and continuous integration, these bursts are evolving into sustained, high-volume traffic.

This shift creates a compounding effect on infrastructure. Every push from an agent triggers a waterfall of subsequent read operations, as CI/CD pipelines, deployment systems, and other agents fetch the latest commits for context. Data from the platform indicates that the busiest repositories on GitHub.com have seen traffic quadruple in a matter of months, with growth trajectories continuing on an exponential path. To stay ahead of this curve, GitHub’s engineering team determined they needed an architecture capable of scaling by at least two orders of magnitude beyond current requirements, all while ensuring that Git—as a protocol—remains unchanged for the end user.
Introducing Project Rubicon
The development of the new architecture, internally dubbed "Project Rubicon," followed a set of core engineering principles established over the last year: minimizing coordination and decoupling storage from compute. Following the initial migration of the vast majority of GitHub repositories into Azure, the team sought a design that could maximize throughput while preserving the strict correctness and durability guarantees expected of a platform housing the world’s source code.
In the Rubicon architecture, the flow of data remains familiar to developers, but the engine under the hood has been completely overhauled. Git, API, and web requests are routed through the existing "Spokes" layer, which manages load balancing and repository placement. However, the data plane is entirely new. Spokes now routes traffic to a fleet of virtual replicas—disposable, transient compute instances that serve reads and writes. Because these replicas are essentially caches that can be rebuilt on demand, the system is inherently resilient; if a node fails, others seamlessly pick up the traffic, and a new node fills its cache as requests arrive.

The durable state of the repository is now split into two specialized components: Azure Blob Storage for immutable pack data, and a new service called Ref-CAS for reference updates.
Minimizing Coordination: Mutating Reference State
The most significant bottleneck in high-throughput Git environments is the need for coordination. When multiple writers attempt to update the same branch or tag, they must reach a consensus. Historically, this coupling of storage and consensus made scaling difficult. GitHub’s team observed that while Git objects themselves are immutable and can be stored in parallel without agreement, reference updates—the pointers that tell Git where a branch ends—require strict ordering.
The team initially experimented with using object storage directly for these updates. While intuitive, they found that approaches like write-ahead logs or segmented blobs created either a "write ceiling" or unacceptable read latency. To solve this, they built Ref-CAS, a specialized, replicated log service. By using a quorum-based consensus model, Ref-CAS allows for thousands of operations per second by only locking the specific references touched by a transaction.

Crucially, Ref-CAS does not store the Git objects themselves; it only stores pointers to them. A virtual replica must ensure the pack data is already safely committed to Azure Blob Storage before it commits a transaction to Ref-CAS. This separation ensures that reference state is always pointing to durable, immutable data, allowing the system to scale its throughput for reference updates independently of the repository’s total size.
Decoupling Storage from Compute: Optimized Compaction
As a repository receives thousands of pushes, it accumulates a large number of "packs," which can degrade performance if left unmanaged. In the legacy architecture, compaction—the process of merging these packs—was a resource-heavy task performed on the same replicas serving traffic. Under the new architecture, this process is fully decoupled.
Dedicated compaction nodes now operate in the background, interacting directly with Azure Blob Storage. These nodes claim ownership of specific repository shards via leases, allowing GitHub to scale maintenance horizontally. As a compaction node merges packs, it updates the repository’s manifest in Ref-CAS, ensuring that virtual replicas are always aware of the most efficient way to access data. This process ensures that, regardless of how many thousands of writes occur, the pack count remains effectively flat, preventing the performance degradation that would otherwise occur as a repository ages.

Garbage collection follows a similar philosophy, offloading the compute-intensive task of removing unreachable objects to specialized workers. By moving these maintenance tasks off the critical path, GitHub has achieved a flat latency curve even as write volumes climb, providing a level of stability that was previously unattainable.
The Path Forward
The completion of this architecture is not the end of the project, but rather the opening of a new chapter. By building a system where reads, writes, and maintenance tasks scale independently, GitHub has created a framework that treats every repository—from a small personal project to the largest enterprise monorepo—with the same underlying efficiency.
Looking ahead, the engineering team is focused on three primary goals: scaling the architecture further to handle unpredictable surges, making agent operations more efficient by reducing the volume of data fetched during standard workflows, and continuing to contribute to the core Git protocol to better support the demands of automated development.

This infrastructure is currently in a limited private preview, with a broader rollout planned for early next year. For developers, the transition will be invisible. They will continue to push, fetch, and branch as they always have, but with the confidence that the platform can handle the weight of an increasingly automated, agent-led development lifecycle. As the industry crosses this threshold, GitHub’s goal remains to ensure that the only limit on what teams can build is the scale of their own ideas.

