Elevenlabs released a fix and is fully operational now. We are also seeing normal levels, but will continue to monitor.

For impacted users, we recommend implementing Vapi Fallback Plan to automatically failover in the future
https://docs.vapi.ai/voice-fallback-plan

IR August 4th: Call Degradation due to Pod Evictions

TL;DR

On August 4th, an incident occurred due to aggressive pod consolidation by Karpenter, which caused Redis pods to be evicted and restarted. This led to API pod failures, triggering a failover to an outdated networking component, resulting in dropped calls. The incident caused a total of 393 calls to be dropped.

Timeline (PST)

August 4th

• 11:02-11:27 AM - Core team identifies Karpenter pods in CrashLoopBackOff (OOMKi...

We are no longer seeing signs of connection issues. The issue should be resolved now, but we will continue to monitor. We apologize for any inconvenience caused.

Incident Report:
Increased call failures on July 25 (PST)

Summary (TL;DR)
On July 25 between 7:00–7:15am PST, a spike in call volume caused some calls to fail with a worker-not-available error. The fallback service for short calls (our serverless workers) could not start because its image architecture didn’t match the configured runtime (image built for ARM64, runtime set to x86). We stabilized the platform by scaling primary workers and then corrected the configuration. Service is o...

Elevenlabs released a hotfix and is fully operational now.

Dashboard calls should be back to normal.

This was resolved

We have identified and resolved the issue. Apologies for the disruption.

Resolved now.

TLDR: A temporary slowdown caused by saturation in our API gateway layer increased response times until they exceeded the edge-network timeout, causing a 524 HTTP response for some API requests.

Timeline in PST
01:00 AM First elevated 524 error responses detected
06:35 AM Rolled back recent backend release (no improvement)
07:19 AM Rolled back related network changes (no improvement).
08:22 AM Scaled up API gateway
09:36 AM Scaled up API gateway further
10:00 AM Reverted the previous night's...

The issue has been resolved: https://neonstatus.com/aws-us-west-oregon/incidents/01JYM23FB7HR82VPZR9DBKVPP8#01JYM4F8Y8MZARWJWRFKV01AP5.

Our database provider has reported this issue as resolved from their end

TL;DR

In response to hallucinations reports in the Success Evaluation feature, we updated our integration with Gemini LLM to use Structured Output. This inadvertently changed the type of the call.analysis.successEvaluation field from string | null to string | number | boolean | null, introducing a breaking change for customers with strict type validation and those using Vapi Server SDKs.

Timeline (all in PT)

• June 12, 11:32pm: Enterprise and Startup users report hallucinations in Su...

It is resolved.

Summary:
We experienced an issue related to API key validation within our WebSockets implementation when sending the API key more than once.

Details:
The issue arose during API key validation within our WebSockets implementation. Our system validates that the API key provided during the initial message is the same as proceeding messages.

A recent change introduced during a release caused a mismatch in how API keys were compared. Specifically, the system was comparing hashed API keys agains...

API was down due to user error in routine maintenance. Service has since been restored