In the fast-paced world of global software development, the stability of core infrastructure is paramount. GitHub, as a central hub for millions of developers, continuously balances the demands of feature delivery with the rigorous requirements of system availability. Over the course of August and September, the engineering team at GitHub focused on a deliberate, phased strategy to modernize its infrastructure, even as it navigated a series of performance challenges and service incidents that highlighted the complexities of scaling a platform of this magnitude.
A Period of Infrastructure Evolution
The past two months have been defined by a concerted effort to optimize GitHub’s backend architecture. The engineering team has been working toward several key objectives: reducing the pressure on shared databases, implementing more granular overload controls, creating additional headroom for GitHub Actions, and refining failure containment strategies for pull requests. Furthermore, the ongoing migration of production workloads to Azure remains a centerpiece of the platform’s long-term infrastructure roadmap.
By October 7, the progress of this regional migration was significant. Approximately 120 of GitHub’s 160 production services had been successfully deployed to the Central US region. Additionally, 13 of the 51 primary MySQL databases were transitioned to the new environment. The reliance on Azure for core traffic has grown steadily; the cloud provider now handles 65.4% of read traffic for GitHub’s core application, 48.3% of Git read traffic, and 80.2% of stream-processing traffic. While these metrics indicate a successful transition for many components, the path has not been without its hurdles.
On September 11, the team made the decision to pause deployments and traffic increases in the Central US region due to observed uneven capacity across availability zones. This "capacity pause" was a necessary step to ensure that the infrastructure could reliably support the anticipated load. Following a period of temporary relief, permanent traffic ramps resumed on September 18. The team continues to work on durable capacity adjustments, with the goal of unlocking all available regional capacity by the end of October.
The transition process also highlighted the dangers of aggressive timelines. An ambitious goal to have every production service deployed in the Central US by September 30 was ultimately missed. During a planned database failover, an automated emergency process inadvertently disconnected Central US database copies, causing service disruptions for customers in that region. In response, GitHub has disabled this specific automated process during planned handoffs and is actively improving recovery procedures to prevent a recurrence.
Database Optimization and System Hardening
Beyond the regional migration, GitHub has made substantial gains in database performance. The engineering team successfully migrated 24 authentication-core tables off of GitHub’s oldest shared database, effectively removing roughly one million queries per second from its replicas. Similarly, the database cluster supporting issues and pull requests underwent a major cleanup; approximately 980 million unnecessary rows from edit history were removed, freeing up an estimated 1.8 terabytes of storage. These efforts have yielded tangible results, with weekly deadlocks falling by approximately 77% from their peak in June. Continuous cleanup processes are now in place to prevent the unbounded growth of such data in the future.
Resilience is also being built directly into the application logic. Half of the sampled traffic for pull requests can now operate with limited functionality even when a shared dependency fails, a testament to the success of failure-containment efforts. During a September 23 spike in pull request activity, the platform demonstrated the value of dynamic scaling, as one database group automatically expanded from 14 to 19 instances to meet the increased load. Furthermore, new production overload controls now allow responders to prioritize critical API and web workflows by rejecting lower-priority requests before they can saturate the system. Operator training and documentation for these new controls have been finalized, ensuring the team is well-prepared to manage future traffic surges.
GitHub Actions, a critical tool for automated workflows, also saw significant improvements in operating headroom. Production refinements resulted in a 28% reduction in measured Redis operations and a 30% decrease in relative CPU usage. A newly implemented infrastructure cache proved particularly effective, reducing Redis bandwidth by 84%, average response times by 66%, median response time for primary inventory requests by 96%, and queued requests by 95%.
September Incident Analysis
Despite these advancements, the August 6 Actions outage, the August 17 GitHub.com incident, and the September failover disruptions serve as stark reminders that technological improvements must be validated under real-world stress. Two specific incidents in September provide insight into the challenges of maintaining a complex, interconnected ecosystem.
On September 13, at 09:16 UTC, GitHub experienced an incident lasting one hour and 28 minutes. The root cause was a background data-cleanup job that placed excessive load on a shared database. While the job included a safeguard to slow down writes if replicas fell behind, the mechanism failed to account for incoming load on the primary database server. Consequently, the job continued to execute until the primary server reached its connection limit, rendering dependent services unreliable. The impact was exacerbated by the lack of quick timeouts on database calls, which caused request handlers to hang, and a retry loop for token creation that flooded the already saturated database. To prevent future incidents, the team is focusing on implementing more robust connection management and "fail-fast" patterns for background tasks.
Later in the month, on September 28, a 52-minute incident affected Copilot code reviews. The issue stemmed from a change in a service that processes review results. Copilot code review logic expected a full copy of the results, but a recent update caused the service to return only a limited, partial copy for certain users. Because the system reported the job as completed successfully despite the missing data, the failure went undetected until customers began reporting incomplete reviews. The team mitigated the issue by reverting the service change and is now working on better monitoring to detect discrepancies between expected and received service outputs.
The Path Forward
The overarching philosophy guiding these efforts remains unchanged: availability must take precedence over capacity, and capacity must be secured before new features are introduced. The engineering team’s commitment to this principle is evident in the systematic way they have approached the last two months. By focusing on the structural integrity of the platform, reducing technical debt in the database layers, and improving the observability of background processes, GitHub aims to mitigate the risks inherent in its scale.
While the missed migration targets and the service interruptions of September demonstrate the complexity of these infrastructure upgrades, the progress made in database throughput, cache efficiency, and automated failover capabilities suggests that the foundation of the platform is becoming increasingly resilient. As the team works to complete the Central US deployment and finalize its capacity goals, the focus will remain on ensuring that these improvements translate into a more stable and reliable experience for the global developer community. Ongoing updates regarding service status and engineering progress remain available through GitHub’s official status page and the company’s engineering blog.
