The issue is resolved as of 10:05am PST.

Incident Report

Impact

Between 9:10–10:05 AM, 37,806 calls were dropped due to call worker failures. All Daily users were impacted.

Timeline (all times in PST)

9:02 AM — On-call engineer notices pods crashing in the Daily cluster.
9:11 AM — Black box probe alert fires; acknowledged by on-call engineer, triggering investigation.
9:27 AM — Issue escalates to the point of impacting all calls on Daily.
9:30 AM — Status page created to inform users of impact and request they switch to the Weekly channel.
9:34 AM — Incident team assembles.
9:37 AM — Rollback to previous deployment is initiated. Due to a large backlog of unprocessed jobs, rollback is delayed waiting for an excessive number of pods to become ready.
10:05 AM — Forceful cutover is initiated and service is restored.

What Went Well

Monitoring detected the issue before it became widespread.
On-call engineer assembled the incident team quickly.

Action Items

Improve emergency rollback procedure to bypass or relax pod readiness checks during incidents, enabling faster cutover.
Continue ongoing observability improvements to reduce MTTD.

Note

A full root cause analysis is underway and available upon request. This report is intended as a summary of the incident timeline, impact, and immediate action items.

Note that this issue impacted the Daily cluster only. Customers desiring increased stability (at the cost of delayed access to features) should switch to the Weekly channel by navigating to Organization Settings on the Vapi Dashboard and changing the Channel to "weekly".

We are rolling back to an earlier deployment. We are seeing calls getting picked up, but will update here once resolved.

We are seeing decreased success rate in calls on the daily channel. The team is investigating and we will post updates here. In the meantime, we highly recommend switching to weekly channel to mitigate service disruption.