The block storage problems in EWR1 are now resolved. Additional analysis and cleanup will occur behind the scenes but no customers should be further impacted.
Jul 8, 02:00 UTC
There is still one set of spans that are being recovered
it is likely that all but 1-2% of volumes are good now.
Jul 7, 23:17 UTC
The cluster is now stable and we are continuing to monitor as the cluster completes a full recovery and health check.
Jul 7, 21:45 UTC
Block Storage volumes in ewr1 are still recovering. We are 11% recovered at this time.
Jul 7, 15:33 UTC
Recovery of block storage volumes in EWR1 continues but is painfully slow and careful work. Volumes are slowly coming back online and servers utilizing that storage resuming normal operation. We will notify again when all volumes are fully online.
Jul 7, 06:58 UTC
Issue have been identified and recovery is still under way
Jul 7, 04:08 UTC
We have identified the issue affecting our block storage volumes. At this point we are working to find a way to correct the problem.
Jul 6, 23:10 UTC
We are investigating the cause of block storage volumes going into read only mode in our ewr1 datacenter. We have also seen that new block storage volumes cannot be attached to servers.
Jul 6, 21:55 UTC