GitHub outage: sidecar limit, VS Code retry storm

Aug. 17 GitHub downtime: 7h 47m, with Copilot down longest

GitHub.com was degraded for 7 hours 47 minutes on August 17, and Copilot was the last part to fully recover, by 21:02 UTC. From 13:28 UTC, error rates rose and responses slowed across Issues, Pull Requests, the APIs, Actions and Copilot. Most of that cleared by 16:36 UTC, once GitHub's Central US data center recovered.

GitHub CTO Vlad Fedorov's blog post of August 20 gives the same 7 hours 47 minutes. He calls it GitHub's second major incident of the month, after an Actions failure on August 6, and says GitHub let down anyone trying to ship software that day.

Sign-in and GitHub Enterprise Cloud data residency were hit

Sign-in was hit along with the other services, and single sign-on through SAML/OIDC, SCIM user provisioning and Team Sync were all affected. At the peak, web and API error rates ran at about 20%, and archive and raw-content downloads at about 50%. GitHub had already posted both figures in its 14:04 UTC status update, before it had found the root cause.

GitHub Enterprise Cloud with data residency lets a company choose where its code and data are stored, and hosts the enterprise on a dedicated GHE.com subdomain. Its Actions workflows were affected too if they depended on public workflow step definitions hosted on GitHub.com.

GitHub's August availability report measures the damage another way: at the peak, 56.07% of incoming requests to the affected services failed or ran slow at GitHub's edge. Across the whole incident window, about 29,000 organizations saw at least one failed or slow request, with roughly 4.8 million such requests in total. The same report says no data was lost.

GitHub incident timeline: GitHub Actions down until ~18:03

Actions stayed degraded until about 18:03 UTC, and Copilot Token Service did not fully recover until 21:02, more than four hours after most services. The table uses the incident report's times plus the status updates GitHub posted during the day.

GitHub's own documents do not agree on the start time and length. The August availability report heads this entry "13:40 UTC (lasting 7 hours and 35 minutes)", and its impact summary gives about 6 hours 44 minutes.

The incident report and the August 20 post both use the longer 7h 47m. None of the three documents explains the difference.

The cause: an Istio sidecar limit that autoscaling missed

The outage began when a new traffic peak pushed an Istio sidecar to its concurrency limit, and autoscaling did not add capacity because its policy watched the host service instead. GitHub's incident report names the immediate cause as network saturation on the load balancers in Central US. In the August 20 post, Fedorov adds that neither the August 6 outage nor this one was caused by a code or configuration change, and he calls both capacity failures.

Autoscaling watched the host service, not the sidecar proxy

Autoscaling was keyed to the host service, so when the sidecar hit its concurrency limit, nothing scaled up. In Istio's sidecar mode, every workload runs next to a sidecar proxy, a process that carries its traffic to and from the rest of the mesh. When that proxy is the bottleneck, the workload behind it can look healthy by its own metrics while requests pile up in front of it.

GitHub's incident report calls this a misconfigured policy. The first fix listed in the August availability report names the gap: autoscaling should account for sidecar concurrency and capacity, "not just the host service."

Load balancer saturation spread to four HAProxy nodes

From the sidecar, the failure spread until four HAProxy nodes ran out of flow capacity and degraded the gateway authentication path. A load balancer distributes incoming requests across servers, and HAProxy is open-source software that does that job. Many services route through that data center and share its authentication gateway, so authentication slowed or failed across all of them.

Optimistic retry logic made the overload worse by piling more requests onto internal load balancers. Pausing HAProxy on those four nodes at once brought broad recovery right away. Scraping attacks on codeload endpoints slowed recovery, and the report files them under complicating factors, not causes.

Why the Copilot outage lasted longest: a retry storm

A retry storm, where large numbers of clients keep resending requests that failed, is the reason Copilot came back last. It hit Copilot Token Service, which handles Copilot's authentication tokens, and kept Copilot authentication failing sporadically in some apps for hours after most of GitHub had recovered by 16:36 UTC. Fedorov's post calls it a client-side retry loop that added load during recovery, one GitHub had to contain before it could safely let traffic back in.

VS Code retry logic drove Copilot token traffic up about 10x

A latent retry bug in VS Code multiplied the traffic about 10 times: Copilot Token Service went from a normal 7–9K requests per second (RPS) to 70–100K RPS. What set it off was a slowdown in replies from a single internal endpoint. From there, each failed token operation could spawn many more requests and leave the client retrying in a loop.

The incident report puts the bug in VS Code, and the availability report calls it a client retry bug. Neither one names the Copilot extension, although some coverage did. Working out what a client is doing from its traffic alone is a separate problem, explained in AI Agent Detection: Binary Bot Filters Miss 39%.

Late in the incident, the 20:08 and 20:45 UTC status updates said Copilot through the GitHub CLI and the GitHub App was unaffected. By then the failures were sporadic Copilot authentication errors in some applications.

GitHub stopped the retry storm with load balancer 403s

GitHub ended the storm by cutting retries at its gateway and having its load balancers answer Copilot token requests with a 403, then gradually letting traffic back in site by site. A 403 is the HTTP "forbidden" status, and since the load balancers returned it themselves, the blocked requests stopped there instead of reaching the token service. The storm hit Northern Virginia, the site that had taken over some failing traffic from Central US while GitHub debugged the network failure there.

The retry cut shipped as a pull request that temporarily scaled back gateway retries. At 19:13 UTC, GitHub said it had partially disabled authentication token retries and was seeing improvement. The incident report credits those retry cuts, together with blocking the responses that triggered retries, for steadying Copilot Token Service and finishing the recovery.

GitHub postmortem: fixing Istio sidecar limits and retries

GitHub's fixes target the two failures: make scaling see the sidecar's limits, and keep retries from turning a partial failure into a wider one. The incident report lists five follow-up actions, and the August availability report repeats them in slightly different words. The August 20 post adds changes meant to reach past this one incident.

On capacity, Fedorov's post puts monthly commits at 2.9 billion, up from 1.4 billion in April, and says that growth explains the pressure but does not excuse the outages. Since making reliability commitments earlier in 2026, he writes, GitHub has added more than 3 million CPU cores and 120 petabytes of fast storage. As of the post, Azure carried roughly 58% of GitHub's platform load, up from 12% in May.

Growth on that scale invites guesses about its cause, with AI coding tools a common one. As of September 29, 2026, none of GitHub's three documents on this outage says what drove the growth or names AI as a cause.

Retry logic and failover fixes in the GitHub incident report

The incident report's five follow-up actions cover sidecar-aware autoscaling, Istio limits, retry behavior, the VS Code bug, and load-balancer monitoring and regional failover. Put plainly, GitHub says it will:

• Make autoscaling react to the sidecar's concurrency and capacity as well
• Recheck the request, concurrency and scaling limits Istio applies to the affected services
• Revisit how many retries gateways and clients allow and how they back off between attempts
• Fix the VS Code retry behavior that multiplied Copilot token traffic
• Watch load-balancer headroom more closely and add safeguards to regional failover

Fedorov's post adds two changes drawn from the August 6 and August 17 incidents. The first is to apply consistent retry limits, retry budgets and variable timeouts between services to keep storms like this one from forming. The second is to review lower-priority CPU and memory alerts for components that could fail when traffic jumps.

Beyond those two, the post says GitHub is separating its critical systems and cutting the dependencies they share. The aim is fewer outages, and smaller ones when one does happen.

GitHub availability report: five GitHub outages in August

GitHub's August availability report lists five incidents, and the August 17 outage was neither the first nor the longest of them. The report, by Jakub Oleksy, is dated September 9, 2026.

GitHub publishes these reports monthly, and the June edition counted six incidents, the July edition eight. The August edition says the repairs from these incidents have already improved capacity monitoring and management, retry policies and the resilience of core services. It restates GitHub's order of priorities as availability, then capacity, then features.

Apart from the start time and length, GitHub's three documents agree on how the August 17 outage unfolded, though only the incident report names Istio, HAProxy and VS Code. A sidecar limit the autoscaler did not watch led to saturated load balancers and HAProxy nodes, and a VS Code retry bug delayed Copilot Token Service's full recovery until 21:02 UTC.