Azure Services - Australia East - Customer on this region might experience intermittent service degradations due to cooling issue in a section of a data centrer
Incident
Report for JFrog Cloud
Resolved
Azure Successfully recovered all standard storage tenants
Monitoring
Current Status: Having successfully recovered 99% of storage services and 99% of impacted Virtual Machines, we are actively investigating individual downstream services to confirm their recovery status and mitigate remaining issues. At this stage, we believe most downstream services that are still experiencing impact are the result of dependencies on one of three services with investigations ongoing.
Power has been restored to all clusters hosting Cosmos DB. Accounts that were not failed over should see recovery in Australia East. Accounts failed over to alternate regions will continue to operate in these alternate regions; the Cosmos DB team has begun onlining the Australia East region for accounts that were failed over earlier.
Over 99% of impacted Azure storage accounts are now fully recovered, a small number resources are currently being replaced or repaired due to thermal damage in order to bring the remaining impacted storage accounts back online.
Azure SQL is continuing to recover impact to a single backend cluster, impacting a small subset of databases instances. Additionally, we are taking steps to reduce the overall zonal load to facilitate improved resource allocation and recovery.
Further updates on these remaining recovery efforts will be provided in 60 minutes, or as events warrant.
Power has been restored to all clusters hosting Cosmos DB. Accounts that were not failed over should see recovery in Australia East. Accounts failed over to alternate regions will continue to operate in these alternate regions; the Cosmos DB team has begun onlining the Australia East region for accounts that were failed over earlier.
Over 99% of impacted Azure storage accounts are now fully recovered, a small number resources are currently being replaced or repaired due to thermal damage in order to bring the remaining impacted storage accounts back online.
Azure SQL is continuing to recover impact to a single backend cluster, impacting a small subset of databases instances. Additionally, we are taking steps to reduce the overall zonal load to facilitate improved resource allocation and recovery.
Further updates on these remaining recovery efforts will be provided in 60 minutes, or as events warrant.
Update
Update from Azure (region azprod-aue-202009)
Current Status: We do not have an exact ETA at this time, but temperature in the impacted datacenter have been stabilized. The Azure service recovery process has commenced and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require additional recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units. If your workloads are protected by Azure Site Recovery or Azure Backup, we recommend to either initiate a failover to the recovery region or recover using Cross Region Restore.
Current Status: We do not have an exact ETA at this time, but temperature in the impacted datacenter have been stabilized. The Azure service recovery process has commenced and is expected to progressively return over a number of hours. Due to the nature of this issue our storage scale units are expected to require additional recovery efforts to ensure all resources return in a consistent state. Note that any new allocations for resources will automatically avoid the impacted scale units. If your workloads are protected by Azure Site Recovery or Azure Backup, we recommend to either initiate a failover to the recovery region or recover using Cross Region Restore.
Identified
The issue has been identified and a fix is being implemented.
Investigating
From Azure:
Summary of Impact: Starting at 08:30 UTC on 30 August 2023, a subset of customers with workloads hosted in the Australia East may be experiencing difficulties accessing and managing resources deployed in this region.
Current Status: Azure are actively working on a cooling event which has impacted a specific datacenter in the Australia East region. The result of this event is that resources might experience unexpected restarts. Further updates will be provided in an hour or as events warrant.
Summary of Impact: Starting at 08:30 UTC on 30 August 2023, a subset of customers with workloads hosted in the Australia East may be experiencing difficulties accessing and managing resources deployed in this region.
Current Status: Azure are actively working on a cooling event which has impacted a specific datacenter in the Australia East region. The result of this event is that resources might experience unexpected restarts. Further updates will be provided in an hour or as events warrant.
This incident affected: APAC - SouthEast2 (Sydney) - AWS (Artifactory, Xray, Pipelines, Distribution, Mission Control).