GitHub障害の原因はオートスケールの誤設定、復旧に7時間47分

"We are investigating reports of impacted performance for some GitHub services."（Posted Aug 17, 2026 - 13:40 UTC）／"This incident affected: Git Operations, Webhooks, API Requests, Issues, Pull Requests, Actions, Pages, and Copilot."— GitHub Status より

"On August 17, 2026, from 13:28–21:15 UTC (7h 47m), GitHub.com experienced elevated errors and latency across Issues, Pull Requests, APIs, Actions, and Copilot. At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. SAML/OIDC authentication, SCIM, and Team Sync were also affected, as well as Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on GitHub.com. Most services recovered by 16:36 UTC as our Central US datacenter recovered; Actions was degraded until approximately 18:03 UTC; and Copilot Token Service fully recovered by 21:02."— GitHub Status より

"Some of the failing traffic was moved from Central US to Northern Virginia where it was served successfully until the network failure in Central US was debugged and resolved."／"Pausing HAProxy on those nodes simultaneously produced immediate broad recovery."／"The degradation affecting API Requests, Actions, Git Operations, Issues, Pages, Pull Requests and Webhooks has been mitigated. We are monitoring to ensure stability."（16:59 UTC）／"Git Operations is experiencing degraded performance. We are continuing to investigate."（17:30 UTC）— GitHub Status より

"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. That growth explains the pressure on our systems, but it does not excuse these outages."— GitHub Blog より

"The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits."— GitHub Status より

"One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers."／"Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints."— GitHub Status より

"Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service."／"The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR and 2) blocking inbound Copilot Token Service token requests at the load balancers with a 403, and then gradually ramping back up traffic per-site to allow callers to succeed."／"Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS. Reducing gateway authentication retries and blocking retry-triggering responses stabilized Copilot Token Service and completed recovery."／"We have partially disabled authentication token retries and have seen improvement, and we are monitoring impact before fully applying this mitigation."（19:13 UTC）／"Copilot usage via the GitHub CLI and GitHub App are unaffected."（20:08 UTC）— GitHub Status より

"- Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity."／"- Auditing Istio request, concurrency, and scaling limits across affected services."／"- Reviewing retry limits and backoff behavior across gateways and clients."／"- Addressing the VS Code retry behavior that amplified Copilot token traffic."／"- Improving load-balancer capacity monitoring and regional failover safeguards."— GitHub Status より

"The August 6 and August 17 incidents led to two immediate changes. First, we are applying consistent retry limits, retry budgets, and variable timeouts across service-to-service interactions to prevent retry storms and cascading load. Second, we are reviewing lower-priority CPU and memory alerts to identify components that could fail during sudden traffic spikes."— GitHub Blog より