We experienced intermittent spikes in 5xx errors on our APIs in the weekly cluster. The root cause was identified, and a fix has already been implemented.

During this period, both inbound and outbound calls may have been affected, as they rely on the APIs for data, resulting in potential service degradation.

Services are restored.

Problem has been resolved now. All services are healthy again.

We have fixed the issue. Call Logs are returned correctly by API

Cartesia has resolved the issue, and is full operational.

We have identified the root cause. The problem has been fixed by rolling back a recent deployment on daily.

We have scaled up our telephony infrastructure resources and bumped our rate limits. We haven't seen any more issues in the last 20 minutes, and call transfers are working as expected now. We are closely monitoring.

Deepgram is investigating an issue where a subset of requests may return elevated rates of 5XX errors or experience significantly higher time to first byte

This incident has been resolved.

The issue was pinpointed and reverted quickly

Elevenlabs released a fix and is fully operational now. We are also seeing normal levels, but will continue to monitor.

For impacted users, we recommend implementing Vapi Fallback Plan to automatically failover in the future
https://docs.vapi.ai/voice-fallback-plan

IR August 4th: Call Degradation due to Pod Evictions

TL;DR

On August 4th, an incident occurred due to aggressive pod consolidation by Karpenter, which caused Redis pods to be evicted and restarted. This led to API pod failures, triggering a failover to an outdated networking component, resulting in dropped calls. The incident caused a total of 393 calls to be dropped.

Timeline (PST)

August 4th

• 11:02-11:27 AM - Core team identifies Karpenter pods in CrashLoopBackOff (OOMKi...

We are no longer seeing signs of connection issues. The issue should be resolved now, but we will continue to monitor. We apologize for any inconvenience caused.

Incident Report:
Increased call failures on July 25 (PST)

Summary (TL;DR)
On July 25 between 7:00–7:15am PST, a spike in call volume caused some calls to fail with a worker-not-available error. The fallback service for short calls (our serverless workers) could not start because its image architecture didn’t match the configured runtime (image built for ARM64, runtime set to x86). We stabilized the platform by scaling primary workers and then corrected the configuration. Service is o...

Elevenlabs released a hotfix and is fully operational now.

Dashboard calls should be back to normal.

This was resolved

We have identified and resolved the issue. Apologies for the disruption.

Resolved now.