The block storage problems in EWR1 are now resolved. Additional analysis and cleanup will occur behind the scenes but no customers should be further impacted.
Posted Jul 08, 2019 - 02:00 UTC
Update
There is still one set of spans that are being recovered
it is likely that all but 1-2% of volumes are good now.
Posted Jul 07, 2019 - 23:17 UTC
Monitoring
The cluster is now stable and we are continuing to monitor as the cluster completes a full recovery and health check.
Posted Jul 07, 2019 - 21:45 UTC
Update
Block Storage volumes in ewr1 are still recovering. We are 11% recovered at this time.
Posted Jul 07, 2019 - 15:33 UTC
Update
Recovery of block storage volumes in EWR1 continues but is painfully slow and careful work. Volumes are slowly coming back online and servers utilizing that storage resuming normal operation. We will notify again when all volumes are fully online.
Posted Jul 07, 2019 - 06:58 UTC
Update
Issue have been identified and recovery is still under way
Posted Jul 07, 2019 - 04:08 UTC
Identified
We have identified the issue affecting our block storage volumes. At this point we are working to find a way to correct the problem.
Posted Jul 06, 2019 - 23:10 UTC
Investigating
We are investigating the cause of block storage volumes going into read only mode in our ewr1 datacenter. We have also seen that new block storage volumes cannot be attached to servers.