Degraded Block Storage Services In EWR1
Incident Report for Packet
The block storage problems in EWR1 are now resolved. Additional analysis and cleanup will occur behind the scenes but no customers should be further impacted.
Posted Jul 08, 2019 - 02:00 UTC
There is still one set of spans that are being recovered

it is likely that all but 1-2% of volumes are good now.
Posted Jul 07, 2019 - 23:17 UTC
The cluster is now stable and we are continuing to monitor as the cluster completes a full recovery and health check.
Posted Jul 07, 2019 - 21:45 UTC
Block Storage volumes in ewr1 are still recovering. We are 11% recovered at this time.
Posted Jul 07, 2019 - 15:33 UTC
Recovery of block storage volumes in EWR1 continues but is painfully slow and careful work. Volumes are slowly coming back online and servers utilizing that storage resuming normal operation. We will notify again when all volumes are fully online.
Posted Jul 07, 2019 - 06:58 UTC
Issue have been identified and recovery is still under way
Posted Jul 07, 2019 - 04:08 UTC
We have identified the issue affecting our block storage volumes. At this point we are working to find a way to correct the problem.
Posted Jul 06, 2019 - 23:10 UTC
We are investigating the cause of block storage volumes going into read only mode in our ewr1 datacenter. We have also seen that new block storage volumes cannot be attached to servers.
Posted Jul 06, 2019 - 21:55 UTC
This incident affected: North America (EWR1 Block Storage).