In an ongoing effort to maintain transparency regarding platform reliability, GitHub has released a comprehensive update detailing its infrastructure performance throughout August and September. While the company continues to make significant strides in its migration to Azure and the optimization of its core database systems, the period was marked by two notable service disruptions that underscored the challenges of scaling one of the world’s most critical developer platforms.
The past two months have been defined by a rigorous internal mandate: "availability, then capacity, then features." This philosophy remains the cornerstone of GitHub’s engineering strategy, emphasizing that technical innovation is secondary to the fundamental requirement of a stable, performant platform. Despite these efforts, the company acknowledged that its work is far from complete, pointing to a series of significant incidents, a temporary pause in capacity expansion, and a missed migration target as evidence that the journey toward a fully optimized architecture is ongoing.
A Period of Infrastructure Transformation
The central focus of GitHub’s recent technical roadmap has been the aggressive migration of production services to Azure. By October 7, the transition had reached a critical milestone, with 120 out of 160 production services successfully deployed to the Central US region. Additionally, 13 of the 51 primary MySQL databases have been migrated to the new environment. These migrations are yielding measurable improvements in traffic handling, with Azure now managing 65.4% of read traffic for GitHub’s core application, 48.3% of Git read traffic, and an impressive 80.2% of stream-processing traffic.
However, the path to full regional migration has not been without obstacles. On September 11, GitHub was forced to hit the pause button on its Central US deployments and traffic increases. The decision was prompted by the discovery of uneven capacity distribution across availability zones—a technical hurdle that threatened to degrade performance rather than enhance it. After implementing temporary relief measures, the team resumed permanent ramps on September 18. Despite this progress, the company missed its ambitious September 30 deadline to have every production service fully deployed in the Central US.
A further complication arose during a planned database failover, where an automated emergency process triggered a premature disconnection of Central US database copies. This unexpected action resulted in service disruptions for customers in the region. In response, GitHub has disabled the automated process during planned handoffs and is currently refining its recovery protocols to ensure that future failovers do not lead to similar customer-facing issues. The engineering team remains committed to unlocking the full potential of its available capacity by the end of October.
Optimizing Databases and Enhancing Resilience
Beyond the migration effort, GitHub has achieved significant success in reducing pressure on its shared-database infrastructure. A major milestone was the migration of 24 core authentication tables away from the platform’s oldest shared database. This transition effectively removed approximately one million queries per second from the database’s replicas, providing much-needed breathing room for the system.
Simultaneously, the team performed a massive cleanup of the database cluster supporting issues and pull requests, purging roughly 980 million unnecessary rows from the edit history. This operation recovered an estimated 1.8 terabytes of storage. The impact on performance has been tangible, with weekly deadlocks falling by approximately 77% compared to their June peak. To ensure these gains are not lost to the natural growth of the platform, GitHub has implemented continuous cleanup processes to prevent unbounded data accumulation.
Resilience engineering has also been a priority. GitHub has successfully implemented a system where half of all sampled pull request traffic can continue to function, albeit with limited capabilities, even when a shared dependency fails. This "graceful degradation" was validated during real-world incidents, where several read operations remained operational despite broader database issues. Furthermore, the platform has introduced automated database traffic routing, which can scale capacity dynamically during traffic spikes. This was demonstrated on September 23, when a specific group of instances automatically expanded from 14 to 19 to accommodate a sudden surge in pull request activity.
To protect the platform from future saturation, GitHub has deployed new overload controls. These mechanisms allow responders to prioritize high-value API and web workflows by rejecting lower-priority requests during peak stress. To support these changes, the engineering team has completed extensive training and updated operator documentation to ensure rapid, effective responses during future incidents.
Advancements in GitHub Actions
GitHub Actions, a critical component of the modern developer workflow, has also seen substantial improvements in operating headroom. By optimizing the underlying infrastructure, the team reduced Redis operations by approximately 28% and decreased relative CPU utilization by about 30%.
The introduction of a dedicated infrastructure cache has yielded even more dramatic results. This cache reduced Redis bandwidth usage by 84% and improved average response times by 66%. Perhaps most notably, the median response time for primary inventory requests improved by 96%, while queued requests dropped by 95%. These optimizations ensure that the CI/CD pipelines that developers rely on daily are faster, more efficient, and significantly more resilient to load.
Anatomy of Recent Incidents
Despite the overarching progress, the incidents of September served as sobering reminders of the fragility inherent in complex systems. On September 13, GitHub experienced a service disruption lasting one hour and 28 minutes. The root cause was traced to a background data-cleanup job that placed excessive load on a shared database. While the job was equipped with safeguards to throttle writes when replicas fell behind, it failed to account for the incoming load on the primary database server.
The situation was exacerbated by two technical oversights. First, the request handlers lacked a "fail-fast" timeout mechanism, causing them to wait indefinitely for the saturated database rather than gracefully terminating. Second, a retry loop designed for token creation continued to send write requests even as the system was failing, effectively preventing the database from recovering. GitHub has since improved its timeout configurations and optimized the retry logic to ensure that systems can recover independently under load.
A second incident occurred on September 28, lasting 52 minutes, which affected Copilot code reviews. This disruption was caused by a change in a service responsible for processing review results. The service began sending a "limited" version of results to some users—which excluded the actual comments—while successfully sending the "full" version to others. Because the system incorrectly reported that these incomplete jobs had finished successfully, the issue went undetected by automated monitoring until users began reporting missing reviews. The incident was resolved by reverting the offending code change.
In response, GitHub is strengthening its observability and monitoring systems. The team is implementing more granular health checks that can distinguish between successful job completion and the delivery of incomplete data. By closing these monitoring gaps, GitHub aims to ensure that such regressions are caught before they reach the end user.
As GitHub moves forward, the lessons learned from these outages are being integrated into the platform’s evolving architecture. The balance between rapid feature deployment and the necessity of a stable, reliable environment remains the primary challenge for the engineering team. By continuing to prioritize infrastructure robustness and rigorous testing, GitHub aims to maintain its position as the reliable foundation for the global software development community. Users are encouraged to monitor the official status page for real-time updates and to follow the GitHub engineering blog for ongoing, in-depth technical reports on the platform’s development.
