As the global development community increasingly relies on GitHub as the backbone of its software supply chain, the imperative for rock-solid infrastructure has never been higher. Following a period of intense operational focus, GitHub has released a comprehensive update detailing the progress made throughout August and September. This report highlights a dual-track strategy: a massive, ongoing migration of production workloads to Azure, balanced against a rigorous effort to bolster the resilience of existing systems. While significant strides have been made in reducing database pressure and improving service reliability, recent incidents underscore that the journey toward absolute platform stability is an iterative, often challenging, process.
A Period of Transformation and Trial
The past two months have been defined by a concerted push to modernize GitHub’s backend architecture. The engineering team has been laser-focused on reducing shared-database pressure, implementing sophisticated overload controls, and expanding the containment mechanisms that prevent isolated failures from cascading across the platform. A primary pillar of this effort is the migration of production services to Microsoft Azure. By October 7, the scope of this transition reached a significant milestone, with 120 out of 160 production services now deployed in the Central US region, alongside the migration of 13 primary MySQL databases.
The transition to Azure is not merely a change in hosting; it is a fundamental reconfiguration of how traffic is managed. Current metrics indicate that Azure now handles approximately 65.4% of read traffic for GitHub’s core application, 48.3% of Git read traffic, and a substantial 80.2% of stream-processing traffic. However, these figures represent a work in progress rather than a finished project. The complexity of such a migration was highlighted in September when a planned database failover triggered an automated emergency process that inadvertently disconnected database copies in the Central US region, leading to service disruptions. This incident served as a stark reminder of the risks inherent in large-scale infrastructure changes. In response, GitHub has temporarily disabled that specific automated process during planned handoffs while refining its recovery protocols.
Balancing Capacity and Stability
The migration strategy faced a deliberate pause on September 11, when the engineering team identified uneven capacity distribution across availability zones. This forced a temporary halt to deployments and traffic increases, demonstrating GitHub’s commitment to prioritizing stability over aggressive speed. Although permanent ramps resumed on September 18 following temporary relief, the team missed its ambitious goal of having every production service migrated to the Central US region by the end of September. The focus has since shifted to durable capacity improvements, with the objective of unlocking all available regional capacity by the end of October.
This cautious approach is guided by a clear internal philosophy: availability must always precede capacity, and capacity must precede new feature development. This hierarchy of values is essential for a platform that serves millions of developers. Outside of the regional migration, the engineering team achieved notable technical victories. By moving 24 core authentication tables off the platform’s oldest shared database, they successfully offloaded approximately one million queries per second from those replicas. Similarly, the database cluster responsible for managing issues and pull requests underwent a massive cleanup, removing 980 million unnecessary rows from the edit history. This not only recovered roughly 1.8 terabytes of storage but also contributed to a 77% reduction in weekly deadlocks compared to their peak in June.
Strengthening System Resilience
The goal of these infrastructure investments is to ensure that when a failure does occur, it remains contained. GitHub has made meaningful progress in this area. For instance, half of the sampled pull request traffic can now maintain limited functionality even when a shared dependency suffers a failure. Furthermore, the platform’s database traffic routing has become more dynamic; during a surge in pull request activity on September 23, one service group successfully scaled from 14 to 19 instances automatically.
In addition, the implementation of new production overload controls has provided responders with a vital tool to protect high-priority API and web workflows. By proactively rejecting lower-priority requests during periods of extreme saturation, the system can preserve the integrity of the most critical operations. These technical improvements are supported by updated operator documentation and intensive training, ensuring that the human element of incident response remains as sharp as the automated systems. GitHub Actions, in particular, has benefited from this renewed focus, seeing significant gains in operating headroom. Reductions in Redis operations and CPU usage, combined with a new infrastructure cache, have drastically improved response times and decreased the number of queued requests, providing a smoother experience for users relying on automation.
Analyzing Recent Operational Incidents
Despite these gains, the reality of managing a platform of GitHub’s scale is that incidents remain inevitable. Two specific events in September illustrate the complexity of maintaining system-wide uptime. The first occurred on September 13, lasting one hour and 28 minutes, and was characterized by a fluctuation in the failure rate at the platform’s edge. The root cause was a background data-cleanup job that placed excessive load on a shared database. While the job included a safeguard intended to slow down write operations if replicas fell behind, it failed to account for the incoming load on the primary server. Consequently, the job continued to execute until the primary database hit its connection limit, stalling other critical requests. The impact was compounded by a lack of aggressive timeouts on database calls and a retry loop that inadvertently hammered the already saturated database.
The second incident, occurring on September 28 and lasting 52 minutes, involved Copilot code reviews failing to deliver results. The disruption was triggered by a service update that inadvertently caused the system to send only "limited" versions of review results to some customers, while others received the full version. Because the service incorrectly reported that jobs were completed successfully, the missing data went undetected by automated monitoring until users began reporting incomplete reviews. The issue was resolved by reverting the service change, but it highlighted a critical gap in observability regarding how different versions of data are handled across the platform.
These incidents, alongside the disruptions seen in August, underscore a fundamental truth: continuous improvement is not merely about adding features, but about rigorously auditing and hardening the existing ecosystem. For the developers who depend on GitHub, the message is clear—the platform is undergoing a significant transformation aimed at long-term reliability. As the team continues to move workloads to Azure and refine its failure containment strategies, the focus remains on reducing customer risk and ensuring that the platform remains a reliable, high-performance environment for the global software community. By integrating the lessons learned from recent outages, GitHub is working to ensure that the infrastructure of tomorrow is significantly more resilient than that of today.
