Microsoft repaired the impacted links, restoring networking capacity between datacenters. Many internal and external services saw signs of recovery from 15:30 UTC. All services fully restored and showing no more signs of higher-than-normal latency.
Posted Jul 06, 2023 - 04:19 CEST
Update
Crews from Microsoft their fiber partners have been able to access the area, and have partially restored the impacted links. They are also accelerating efforts to augment networking capacity, with onsite teams preparing relevant cabling to bring additional capacity between datacenters online. They expect that their onsite repairs and augmentations will be able to bring sufficient additional capacity online in the next few hours, which will provide headroom before the Thursday business day starts in EMEA. The next update will be provided in 2 hours, or as events warrant.
We as Ultimo see some additional latency but also already improvements in the last hours. We expect business as usual in the coming hours regarding performance.
Posted Jul 05, 2023 - 22:55 CEST
Identified
A subset of customers in West Europe may experience packet drops, timeouts and increased latencies in responses. We determined a number of fiber links between datacenters have become unavailable due to a fiber cut. Automated systems are balancing load and rerouting traffic, but customers may still experience degradation.
Current Status: Our investigations are showing that service-to-customer communications over Internet/Express route are not experiencing impact currently, with impact predominantly to internal service-to-service communications from one datacenter to another datacenter in the West Europe region.
We are exploring multiple mitigation options. The next update will be provided in 60 minutes, or as events warrant.
Posted Jul 05, 2023 - 12:39 CEST
Investigating
Starting at 07:22 on 05 Jul 2023, Azure customers with resources in West Europe may experience packet drops and increased latency in application response. Requests timeouts may also be experienced. Microsoft is currently investigating traffic ingestion issues as the preliminary root cause. Microsoft keeps continue to collaborate with relevant teams to identify mitigation options. The next update will be provided in 60 minutes, or as events warrant.