Degraded Block Storage Services In EWR1
Incident Report for Packet
Resolved
The block storage problems in EWR1 are now resolved. Additional analysis and cleanup will occur behind the scenes but no customers should be further impacted.
Posted 3 months ago. Jul 08, 2019 - 02:00 UTC
Update
There is still one set of spans that are being recovered

it is likely that all but 1-2% of volumes are good now.
Posted 3 months ago. Jul 07, 2019 - 23:17 UTC
Monitoring
The cluster is now stable and we are continuing to monitor as the cluster completes a full recovery and health check.
Posted 3 months ago. Jul 07, 2019 - 21:45 UTC
Update
Block Storage volumes in ewr1 are still recovering. We are 11% recovered at this time.
Posted 3 months ago. Jul 07, 2019 - 15:33 UTC
Update
Recovery of block storage volumes in EWR1 continues but is painfully slow and careful work. Volumes are slowly coming back online and servers utilizing that storage resuming normal operation. We will notify again when all volumes are fully online.
Posted 3 months ago. Jul 07, 2019 - 06:58 UTC
Update
Issue have been identified and recovery is still under way
Posted 3 months ago. Jul 07, 2019 - 04:08 UTC
Identified
We have identified the issue affecting our block storage volumes. At this point we are working to find a way to correct the problem.
Posted 3 months ago. Jul 06, 2019 - 23:10 UTC
Investigating
We are investigating the cause of block storage volumes going into read only mode in our ewr1 datacenter. We have also seen that new block storage volumes cannot be attached to servers.
Posted 3 months ago. Jul 06, 2019 - 21:55 UTC
This incident affected: EWR1 Block Storage.