At GitHub Universe last week, the platform officially unveiled its next-generation Git storage architecture, a massive engineering undertaking designed to meet the demands of a new era of software development. As artificial intelligence-driven agents become increasingly integrated into the coding workflow, repositories are facing unprecedented traffic patterns. GitHub’s new system, codenamed "Project Rubicon," is built to handle thousands of writes per second while maintaining scalable, low-latency reads. Internal benchmarks have demonstrated the architecture’s capability to support 5,000 concurrent agents interacting with a single repository, processing 7,200 pushes every second without compromising performance.
This shift marks a significant milestone in how GitHub supports the massive, automated workflows of the future. The project has been developed and deployed under the hood while the platform remained fully operational, ensuring that for the hundreds of millions of developers on GitHub, the fundamental experience of pushing, fetching, and branching remains unchanged.
Building for the Agent-Scale Future
The rise of agentic development has fundamentally altered the definition of a "busy" repository. Over the past year, GitHub engineers observed that while traditional developer-driven traffic usually consists of short, intermittent bursts, agent-based activity is shifting toward sustained, high-frequency interaction. Each automated push triggers a cascade of secondary operations, including CI/CD pipeline executions, deployment system requests, and context-fetching by other agents. This creates an amplification effect that scales in tandem with write traffic.
GitHub’s data illustrates this trend clearly: some of the most active repositories on the platform have seen their daily request volumes grow fourfold in just a few months, with no sign of plateauing. To stay ahead of this exponential trajectory, the engineering team realized they could not simply patch existing systems. They needed an architecture designed for at least two orders of magnitude more scale.
The design goals for Project Rubicon were stringent. The system needed to support massive concurrency, provide robust durability, maintain low latency even under heavy load, and offer the ability to scale globally. Most importantly, it had to be flexible enough to handle these workloads without requiring special configuration for individual repositories.
The Philosophy of Rubicon: Decoupling and Minimalism
The architecture of Project Rubicon is built on two core principles: minimizing coordination and decoupling storage from compute.
Historically, Git infrastructure tightly coupled data storage, request serving, and repository maintenance. Rubicon breaks this mold by recognizing a fundamental truth about Git: Git objects are immutable and content-addressed. Because they are named by their contents, multiple writers can store these objects simultaneously without needing to coordinate or agree on a global state. The only part of a push that requires synchronization is the reference update—the pointer that tells Git which commit is the current head of a branch.
By isolating these reference updates, GitHub has allowed the various components of its infrastructure to scale independently. The architecture introduces "Spokes," a routing and load-balancing service that directs incoming requests to a fleet of disposable "virtual replicas." These replicas are responsible for serving reads and writes, but because their local disks function as caches, they can be replaced at any time without impacting system correctness. If a replica is removed, its replacement simply rebuilds its cache from the durable data store as new traffic arrives.
Reimagining Reference State and Compaction
To handle the high-throughput requirements of reference updates, GitHub introduced a specialized service called Ref-CAS (Content-Addressable Storage). Unlike legacy systems that struggled with the latency of global locks, Ref-CAS functions as a replicated log service. Each reference update is treated as a compare-and-swap operation committed by a quorum. This ensures that when multiple agents attempt to update the same reference, only one succeeds at a time, while non-contending updates proceed in parallel.
This design effectively eliminates the bottleneck of traditional locking. Because Ref-CAS never handles the actual object data—only the pointers to that data—it remains extremely lightweight and fast. The actual pack data resides in Azure Blob Storage, which provides the massive aggregate throughput necessary to handle the high volume of incoming writes.
The second major challenge in this new architecture was maintaining repository performance despite the rapid accumulation of data. Every push adds a new "pack" to storage, and a growing number of packs can degrade performance. In the legacy system, compaction—the process of merging these packs—ran on the same replicas that served traffic, creating a resource drain.
Rubicon moves this process entirely to the background. Dedicated compaction nodes operate directly against the durable storage, using the Ref-CAS service to notify virtual replicas as the underlying data structure evolves. By utilizing leases, these nodes can scale horizontally; as a repository grows, GitHub can simply add more compaction nodes. This ensures that the number of packs remains effectively flat, providing a consistent experience for developers even as the volume of pushes continues to climb.
A Foundation for the Future
Project Rubicon represents more than just a performance upgrade; it is a foundational shift that prepares GitHub for the next generation of software engineering. By moving toward an architecture where every repository—regardless of size or activity—runs on the same high-performance system, GitHub is removing the artificial ceilings that once constrained large-scale projects.
The engineering team has already identified several areas for future improvement. These include further reducing the latency of garbage collection, making agent operations more efficient by minimizing the amount of data transferred, and evolving the Git protocol itself to better suit the needs of automated agents.
As GitHub continues to roll out this architecture—currently in a limited private preview with a broader release planned for early next year—the platform aims to ensure that the only limit on developer productivity is the scope of their ideas. By automating the complexities of storage and scaling, GitHub is positioning itself to remain the backbone of the software industry, even as the nature of building software undergoes a radical transformation. The transition to Rubicon signals that GitHub is not just adapting to the era of agents, but actively building the infrastructure that will define how they work.
