GitHub has released its latest monthly transparency report, detailing the operational challenges faced throughout October. The report outlines eight distinct incidents that resulted in degraded performance across the platform, underscoring the ongoing tension between rapid infrastructure expansion and the requirement for rock-solid reliability. As GitHub continues its aggressive push to migrate services to Microsoft Azure, the engineering team is finding that scaling capacity is only half the battle; the other half is ensuring that those growth efforts do not introduce new, unforeseen vulnerabilities.
The October Landscape: Growth Amidst Growing Pains
For the GitHub engineering team, October was characterized by a push toward increased capacity. The company successfully integrated significant, independently scalable Azure capacity, while simultaneously working to dismantle complex failure points that could cause isolated errors to escalate into platform-wide outages. Despite these improvements, the month was marked by a series of write outages, monitoring gaps during sensitive data migrations, and a notable regional network disruption. These events served as a stark reminder that as GitHub expands, it must pair capacity growth with safer operational practices, stronger service isolation, and a more robust approach to regional recovery.
By November 9, the progress of the migration was substantial: 140 of 151 production services and 32 of 50 MySQL primaries were successfully operating within Azure’s Central US region. The numbers reflect a platform in transition, with Azure now handling 63% of migrated-service reads, 66% of monolith reads, and 52% of Git reads. Furthermore, a compute expansion across all nine Central US Kubernetes clusters has more than doubled the available capacity.
This investment is not merely precautionary; it is currently supporting tangible growth. Since March, traffic from migrated services has surged by 82%. By leveraging the Central US region, GitHub has been able to absorb this traffic without placing an undue burden on its legacy datacenters. Beyond raw capacity, the team implemented targeted reliability upgrades. Adjustments to GitHub Actions improved capacity for critical Azure control-plane requests, while a new NAT-gateway mitigation reduced network usage by 85%. In a testament to the effectiveness of these granular improvements, only four production jobs failed during the deployment of that mitigation—a significant improvement over the roughly 800 failures that would have been expected under the previous configuration.
Database efficiency also saw a major focus. By migrating a high-traffic GitHub Actions endpoint that processes over 250 million daily requests to a more efficient service path, the team reduced peak database reads by 85%. Additionally, a simple UI change—requiring a minimum of three characters for repository topic searches—reduced slow or terminated queries by 98%. Despite these successes, the core philosophy remains unchanged: availability must take precedence over capacity, and capacity must be prioritized over new features.
October 05: Actions and Database Connectivity
On October 5, starting at 19:11 UTC, users experienced a 3-hour and 38-minute disruption where GitHub Actions jobs on hosted runners remained queued for extended periods. During the peak of the incident, which lasted roughly 90 minutes, the vast majority of jobs failed to start within the five-minute threshold. The root cause was a partial networking failure within the cloud provider’s infrastructure, which severed communication between one of GitHub’s datacenter regions and its cloud-hosted databases. Rather than failing gracefully, the service entered a state of waiting for timeouts, leading to a memory exhaustion event that triggered a restart. As jobs shifted to healthy regions, the sudden spike in demand overwhelmed those areas as well.
October 06: Migration Overload
A separate incident on October 6 involved a 6-minute window of elevated request failures across GitHub.com, followed by a three-hour period where enterprise migrations were paused. The issue was triggered by a surge in migration activity, which caused the GitHub Enterprise Importer to push a massive volume of complex queries to a shared database. These queries effectively exhausted the database’s connection pool. Because the importer lacked a mechanism to throttle its load during congestion, it inadvertently stifled other core features relying on the same database. The situation was resolved by pausing migrations and deploying a fix to process queries in smaller, more manageable batches.
October 07: Write Outages and Routing Errors
October 7 saw two distinct write outages, each lasting roughly ten minutes, caused by an error in database orchestration. During routine maintenance, a newly replaced machine initialized with outdated configuration records that erroneously listed retired database servers as active. This caused the machine to broadcast routing instructions that directed database writes for several core services to non-existent or retired servers. When an automated maintenance task failed to respect a pause command, it triggered a secondary election that re-promoted the misconfigured machine, causing a repeat of the initial outage.
October 19: Network Saturation
On October 19, an internal actions service lost connectivity to cache servers in a single cloud region. The subsequent retry storm, where every service instance attempted to re-establish connections simultaneously, saturated the network links between the datacenter and the cloud region. This ripple effect degraded performance for other services sharing that same network path, leading to a 1-hour and 28-minute period of delayed workflow status updates and log retrieval.
October 21: Event Streaming and Memory Limits
A 4-hour and 30-minute incident on October 21 impacted the platform’s event streaming infrastructure. As memory usage on the event brokers grew with the volume of client connections, the servers began to reach their limits. When the initial nodes were taken out of service to alleviate the pressure, the traffic shifted to the remaining brokers, causing a cascading failure that spread across the cluster. Audit log services were particularly affected, taking nearly seven hours to fully recover due to the need to process the accumulated backlog of events after the platform stabilized.
October 28: Copilot Authentication Failure
On October 28, users of GitHub Copilot experienced a 1-hour and 32-minute disruption affecting several GPT-family models. The issue stemmed from an invalidation of the credentials used to authenticate with the upstream model provider. As the provider rejected requests as unauthenticated, the error was propagated to users. While auto-model selection mitigated the issue for some, features dependent on specific, non-redundant models—such as code scanning autofix—remained unavailable until the credentials were refreshed.
October 30: Database Connection Pooling
The final major incident of the month occurred on October 30, when a database connection pool reached its capacity limit, causing a 9-minute outage for various GitHub.com services. The exhaustion was driven by specific update requests that held onto connections for longer than expected while performing secondary lookups. With the pool operating near its maximum capacity, these stalled connections left no room for new requests, which then timed out and failed. The incident hit Git operations and pull request merges the hardest.
As GitHub moves into the final months of the year, the construction of the East US 3 region continues, albeit about a week behind the original schedule. The team plans to begin a validated traffic ramp in early December. For GitHub, the path forward is clear: the focus remains on hardening the infrastructure through better dependency isolation, more rigorous migration safety, and the continued pursuit of regional recovery capabilities. By refining these systems, the engineering team hopes to ensure that as the platform continues to scale, it does so with the reliability that its global user base expects.
