In the rapidly evolving landscape of global software development, the stability of core infrastructure remains the most critical priority. For GitHub, maintaining the availability of the platform that powers millions of developers is an ongoing, complex engineering challenge. Following an intensive two-month period, the company has released a comprehensive progress report detailing its infrastructure evolution, the ongoing migration to Microsoft Azure, and the lessons learned from two significant service degradations that occurred in September.
This update serves as a status report on GitHub’s broader strategic initiative to prioritize availability and capacity above new feature development. The narrative for August and September is one of measured progress punctuated by the reality of operating at a massive, global scale.
The Strategic Shift: Capacity and Resilience
Over the past two months, GitHub has focused on a series of technical initiatives designed to fortify its backbone. These efforts include reducing the immense pressure on shared databases, implementing new overload controls, creating necessary headroom for GitHub Actions, and refining the failure containment strategies for pull requests. A critical component of this work is the ongoing migration of production workloads to Azure.
As of October 7, the migration metrics illustrate a substantial transition: 120 out of 160 production services have been successfully deployed in the Central US region, and 13 out of 51 primary MySQL databases have completed their migration. The shift in traffic is equally notable. Azure now handles 65.4% of read traffic for GitHub’s core application, 48.3% of Git read traffic, and 80.2% of stream-processing traffic. While these figures represent distinct workloads and cannot be aggregated, they underscore the scale of the transition.
However, the migration has not been without its setbacks. On September 11, the engineering team made the difficult decision to pause deployments and traffic increases in the Central US region. This was prompted by the discovery of uneven capacity across availability zones. While temporary measures allowed for a resumption of permanent ramps by September 18, the team is still working toward the goal of unlocking all available capacity in the region by the end of October. The project also faced a missed milestone: an aggressive target to have every production service deployed in the region by September 30 was not met.
The complexity of these migrations was highlighted during a planned database failover, where an automated emergency process inadvertently disconnected Central US database copies, causing service disruptions for customers in that region. In response, GitHub has disabled that specific automated process during planned handoffs while simultaneously working to improve the reliability and speed of recovery protocols.
Optimizing Database and Application Performance
Beyond the regional migration efforts, GitHub has made significant strides in optimizing its existing infrastructure. A major achievement involved moving 24 authentication-core tables away from the platform’s oldest shared database. This transition effectively removed approximately one million queries per second from the database’s replicas, significantly alleviating the strain on legacy systems.
Simultaneously, the team performed a massive cleanup of the database cluster supporting issues and pull requests. By purging approximately 980 million unnecessary rows of edit history, engineers recovered an estimated 1.8 terabytes of storage. This effort has already yielded measurable performance gains; weekly deadlocks have dropped by approximately 77% from their June peak. To prevent the recurrence of such unbounded growth, the team has implemented continuous cleanup processes.
Reliability improvements have also been integrated into the platform’s logic. For instance, half of the sampled pull request traffic is now designed to maintain limited functionality even when a shared dependency fails. This was validated by production results where several read operations remained available during a shared-database incident. Furthermore, the platform’s database traffic routing has become more dynamic; during a surge in pull request activity on September 23, the system automatically scaled one service group from 14 to 19 instances, demonstrating the effectiveness of automated capacity management.
To protect the platform during high-traffic events, GitHub has deployed new production overload controls. These mechanisms allow responders to prioritize essential API and web workflows by rejecting lower-priority requests before the system reaches saturation. The implementation has been supported by comprehensive operator documentation and training.
GitHub Actions has also received significant attention. The team successfully reduced measured Redis operations by roughly 28% and relative CPU usage by approximately 30%. The introduction of a dedicated infrastructure cache has further optimized the service, reducing Redis bandwidth by 84%, decreasing average response times by 66%, and cutting the median response time for primary inventory requests by 96%. These optimizations have substantially improved the headroom for automated workflows.
September Incident Reports: Analyzing Failures
Despite these engineering improvements, the reality of complex distributed systems is that incidents are inevitable. September saw two notable events that impacted service delivery, serving as reminders that technical optimizations are only effective when they hold up under real-world stress.
The first incident occurred on September 13, beginning at 09:16 UTC and lasting one hour and 28 minutes. During this window, users experienced a spike in failures at the platform’s edge. The root cause was traced to a background data-cleanup job that placed excessive load on a shared database. While the job included a safeguard to throttle writes when replicas lagged, the mechanism failed to account for incoming load on the primary database server. Consequently, the job continued to execute until the primary server reached its connection limit, causing requests that relied on that database to stall.
Recovery was hampered by two secondary issues: a lack of aggressive timeouts on database calls—which caused request handlers to wait indefinitely—and a retry loop that continuously re-sent failed writes, further saturating the database. GitHub’s response focused on immediate mitigation through connection management and service restarts. To prevent future occurrences, the team is implementing more granular safeguards that account for primary database load and refining timeout policies to ensure that services fail fast rather than clogging resources during a database bottleneck.
The second incident took place on September 28, lasting 52 minutes and impacting the completion of Copilot code reviews. The issue stemmed from a change in the service responsible for delivering review results. The service began sending only a "limited" copy of the review to certain customers, excluding the necessary full content. Because the system mistakenly identified these partial results as successful completions, the missing data was not immediately flagged.
The incident was identified only after users reported that reviews remained incomplete for over 30 minutes, triggering alerts. GitHub mitigated the impact by reverting the service change. Moving forward, the team is focusing on improving monitoring for asynchronous jobs to ensure that partial or missing data is detected immediately, rather than waiting for user reports or long-running timeouts.
Looking Ahead: The Principle of Availability
These incidents and the associated remediation efforts underscore a core philosophy that GitHub continues to emphasize: the hierarchy of "availability, then capacity, then features." While the engineering team remains committed to delivering new capabilities for developers, the events of August and September confirm that the path to a more resilient platform requires a constant focus on the underlying infrastructure.
As GitHub continues its migration to Azure and the refinement of its database and application logic, the lessons learned from these outages will be integrated into the architecture. The goal remains consistent: to minimize customer risk through proactive capacity planning, automated overload protection, and a commitment to the continuous improvement of system stability. For the developers who rely on GitHub every day, these updates are a transparent look at the difficult, often invisible work required to keep the world’s software supply chain running smoothly.
