A routine maintenance error caused a partial outage in Google Cloud’s us‑central1‑b region when an engineer unplugged all fiber‑optic cables, leading to severe network degradation.
An internal Google Cloud maintenance mishap caused a brief but noticeable outage in the us‑central1‑b region after an engineer unintentionally unplugged multiple fiber‑optic cables.
What happened
During a scheduled network upgrade, a Google engineer was instructed to verify that all fiber connections were properly seated. Instead of checking, the engineer removed every visible fiber‑optic cable from the patch panel, effectively disconnecting a large portion of the region’s backbone.
The accidental disconnection triggered a cascade of routing failures, causing severe latency spikes and packet loss for customers running workloads in the affected zone.
Impact on services
Clients reported intermittent connectivity to Compute Engine instances, Cloud SQL databases, and Kubernetes clusters. While the outage was limited to the us‑central1‑b region, the ripple effect was felt across services that rely on cross‑region replication.
- Increased latency for API calls
- Temporary loss of access to storage buckets
- Brief downtime for load‑balanced web applications
Google’s response
Google’s Site Reliability Engineering (SRE) team restored the fiber connections within 45 minutes and issued a post‑mortem outlining the procedural gaps that allowed a single individual to disconnect critical infrastructure without additional checks.
The company announced new safeguards, including mandatory peer review of any physical cable manipulation and automated alerts for sudden loss of link state.
This incident underscores the importance of rigorous change‑management processes, even for seemingly simple hardware actions.
The outage serves as a reminder that human error can still affect highly automated cloud environments, prompting both providers and customers to revisit their resilience strategies.
The Register coverage of Google engineer’s fiber‑optic mishap