Customer Portal
Incident Report for Packet
Postmortem

Sequence of Events

At 1624 UTC a number of TLS certificates for our core Consul, Vault, and Nomad infrastructure failed to regenerate, causing a communication failure in those services which trickled down to the running services and the load balancers directing traffic. The problem was quickly identified but due to the nature of service discovery recovery took several hours as we brought up each layer and deployed services.

Root Cause

TLS cert expiration caused by failure to auto-regenerate the certs for our core service discovery layer which impacted our orchestration layer and our load balancers.

Customer Impact

All access to API and Portal was down until 20:20 UTC when the API came back online. Portal remained impacted until 21:25. PacketConnect’s portal interface was fully restored at 23:03.

Timeline

All times are in UTC.

16:24 TLS Cert expires

16:30 Response begins

17:01 Problem identified and resolution begins

19:10 Nomad infrastructure is back online, jobs can now be resubmitted

20:20 API Infrastructure is back online and being served

21:25 Portal is back online

21:36 Staff Portal is back online

23:03 Packet Connect fully restored

23:24 Incident fully resolved

Posted about 2 months ago. Jun 28, 2019 - 00:23 UTC

Resolved
This incident has been resolved.
Posted 2 months ago. Jun 26, 2019 - 23:24 UTC
Monitoring
We are thrilled to report that, our customer portal & API are back up and running. We will continue to monitor for any residual errors/issues. Should you encounter a hiccup along the way, do reach out via support@packet.com.
Posted 2 months ago. Jun 26, 2019 - 21:28 UTC
Update
We are continuing to work on a fix for this issue.
Posted 2 months ago. Jun 26, 2019 - 19:50 UTC
Update
We have found and fixed the issue that we had previously identified. In the course of bringing that system back online, a second issue was identified, and we are working on that second problem now. We appreciate your continued patience.
Posted 2 months ago. Jun 26, 2019 - 19:11 UTC
Identified
We have identified the issue and currently implementing a fix.
Posted 2 months ago. Jun 26, 2019 - 17:27 UTC
Investigating
We are currently investigating an influx of error messages with our customer portal. Please reach out to support@packet.com should you have questions, or otherwise need assistance.
Posted 2 months ago. Jun 26, 2019 - 16:46 UTC
This incident affected: Packet API and Packet Customer Portal.