The block storage problems in EWR1 are now resolved. Additional analysis and cleanup will occur behind the scenes but no customers should be further impacted.
Posted 3 months ago. Jul 08, 2019 - 02:00 UTC
There is still one set of spans that are being recovered
it is likely that all but 1-2% of volumes are good now.
Posted 3 months ago. Jul 07, 2019 - 23:17 UTC
The cluster is now stable and we are continuing to monitor as the cluster completes a full recovery and health check.
Posted 3 months ago. Jul 07, 2019 - 21:45 UTC
Block Storage volumes in ewr1 are still recovering. We are 11% recovered at this time.
Posted 3 months ago. Jul 07, 2019 - 15:33 UTC
Recovery of block storage volumes in EWR1 continues but is painfully slow and careful work. Volumes are slowly coming back online and servers utilizing that storage resuming normal operation. We will notify again when all volumes are fully online.
Posted 3 months ago. Jul 07, 2019 - 06:58 UTC
Issue have been identified and recovery is still under way
Posted 3 months ago. Jul 07, 2019 - 04:08 UTC
We have identified the issue affecting our block storage volumes. At this point we are working to find a way to correct the problem.
Posted 3 months ago. Jul 06, 2019 - 23:10 UTC
We are investigating the cause of block storage volumes going into read only mode in our ewr1 datacenter. We have also seen that new block storage volumes cannot be attached to servers.