In the fast-paced world of global software development, the stability of core infrastructure is the foundation upon which millions of developers build their projects. Throughout August and September, GitHub’s engineering teams have been intensely focused on a multi-pronged strategy to enhance platform reliability, optimize database performance, and execute a complex migration to Azure. While the platform continues to make significant strides in scalability and resilience, the period was not without its challenges, including two notable service disruptions in September that highlighted the ongoing nature of this large-scale transformation.
A Period of Infrastructure Evolution
The last few months have been defined by a concerted effort to modernize GitHub’s backend architecture. The primary objective is to move away from legacy configurations that have historically created performance bottlenecks. By reducing pressure on shared databases, implementing sophisticated overload controls, and creating necessary headroom for GitHub Actions, the team is working toward a more stable future. Furthermore, the expansion of failure containment mechanisms—specifically designed to isolate issues within pull request workflows—and the continued migration of production workloads to Azure remain central to the platform’s long-term roadmap.
As of October 7, the migration metrics illustrate the sheer scale of the operation. Currently, 120 out of 160 production services have been successfully deployed to the Central US region, with 13 of 51 primary MySQL databases now operating within the new environment. The transition to Azure is already yielding tangible results in traffic distribution: the cloud provider is now handling 65.4% of read traffic for the core application, 48.3% of Git read traffic, and an impressive 80.2% of stream-processing traffic. These figures reflect a significant shift in where and how GitHub’s traffic is processed, signaling that the platform is well on its way to a more cloud-native architecture.
Navigating Capacity and Migration Challenges
The path to modernization is rarely linear, and the past two months have served as a reminder of the complexities involved in shifting such a massive, distributed system. On September 11, the engineering team made the strategic decision to pause deployments and traffic increases in the Central US region. The pause was prompted by the discovery of uneven capacity distribution across various availability zones, a condition that threatened to compromise the stability of the new environment.
After implementing temporary measures to stabilize the region, permanent traffic ramps were resumed on September 18. Despite this progress, the team missed an ambitious goal set for September 30, which aimed to have every production service fully deployed in the new region. This delay was compounded by an incident during a planned database failover, where an automated emergency process triggered a premature disconnection of database copies in the Central US region, inadvertently disrupting user access. In response, the team has disabled that specific automated process during planned handoffs and is currently focusing on improving recovery workflows to prevent a recurrence. The objective remains clear: unlocking full available capacity by the end of October.
Reducing Technical Debt and Optimizing Performance
Beyond the regional migration, GitHub has made substantial progress in reducing technical debt within its existing systems. One of the most significant achievements has been the migration of 24 core authentication tables away from one of the platform’s oldest shared databases. This move has effectively offloaded approximately one million queries per second from the database’s replicas, significantly easing the strain on a critical piece of infrastructure.
The database cluster responsible for supporting issues and pull requests has also undergone a rigorous cleanup. By purging roughly 980 million unnecessary rows of edit history, the team has reclaimed an estimated 1.8 terabytes of storage. This initiative is complemented by a new, continuous cleanup process that prevents unbounded growth from accumulating in the future. These efforts are already paying off in terms of platform stability; weekly deadlocks have fallen by approximately 77% compared to the peak observed in June.
Furthermore, the team has successfully implemented new resilience patterns. Half of all sampled pull request traffic can now continue to function in a limited capacity even when a shared dependency fails. This "fail-soft" approach ensures that even during internal outages, critical developer workflows remain accessible. Additionally, the implementation of dynamic database traffic routing allows the platform to automatically scale capacity during sudden traffic spikes. A notable example occurred on September 23, when a surge in pull request activity prompted the system to scale a specific service group from 14 to 19 instances automatically. New overload controls have also been deployed, allowing responders to prioritize critical API and web workflows by rejecting lower-priority requests when the system approaches saturation.
Advancements in GitHub Actions
GitHub Actions, a critical tool for modern CI/CD, has also seen significant performance improvements. Through a series of production optimizations, the engineering team has reduced Redis operations by approximately 28% and relative CPU utilization by 30%. A newly introduced infrastructure cache has further optimized the service, leading to an 84% reduction in Redis bandwidth. These improvements have translated into a better experience for users: average response times for Actions have dropped by 66%, median response times for primary inventory requests have improved by 96%, and the volume of queued requests has decreased by 95%.
September Incident Reports: Lessons in Resilience
Despite these systemic improvements, September saw two significant incidents that underscored the importance of proactive monitoring and fault tolerance. On September 13, at 09:16 UTC, GitHub experienced an incident lasting one hour and 28 minutes. The disruption was triggered by a background data-cleanup job that placed excessive load on a shared database. While the job included a safeguard to throttle writes if replicas fell behind, the mechanism failed to account for the incoming load on the primary database server.
As the primary database reached its connection limit, requests dependent on it began to fail. The impact was exacerbated by two factors: the absence of a quick timeout on database calls—which caused request handlers to hang—and a retry loop in the token creation process that kept resending failed writes, further saturating the database. The team has since prioritized implementing faster timeouts and refining retry logic to ensure that systems fail gracefully rather than locking up.
Later in the month, on September 28, a 52-minute incident affected Copilot code reviews. The issue stemmed from a change in the service responsible for delivering review results. The system began sending "limited" copies of reviews to some customers, while others received the full version. Because the service was configured to only post reviews from the full copy, it effectively failed to process reviews for a segment of users while incorrectly reporting the job as successful. The lack of granular monitoring meant the discrepancy went undetected until users began reporting incomplete reviews. The incident was mitigated by reverting the recent code change, and the team is now working to improve observability and monitoring for review delivery to ensure that failures are caught immediately.
Looking Ahead: Availability, Capacity, and Features
The events of August and September, including the major outages on August 6 and 17, serve as a constant reminder that technical improvements are only valuable when they translate into reduced risk for the user. The platform’s engineering philosophy remains steadfast: prioritize availability first, followed by capacity, and finally, new features. As GitHub continues its transition to a more modern, cloud-based infrastructure, the focus remains on building a platform that is not only faster and more capable but, above all, reliable enough to support the global developer community.
For real-time status updates and detailed post-incident reports, users are encouraged to monitor the official GitHub status page. As the team pushes forward into the final quarter of the year, the ongoing work to stabilize the Central US region and refine internal service dependencies will continue to be the primary focus of the engineering organization.
