We are seeing increased latency and requests timing out from API and DB degradation. We are working with our DB provider to resolve this, and have made a change.

Now monitoring to ensure improvement.

There was a DB restart and things are looking normal now. The issues have been resolved.

we experienced elevated errors in our api (for phone calls create) at 9:35 AM PT for few minutes. This has been resolved.

We had a restart on our database endpoint leading to a small blip in 500s for api endpoints.

We had a small blip on daily channel with call.in-progress.error-vapifault-worker-died errors due to a new daily deployment. We have rolled it back.

The issue with twilio inbound calls failing on daily has been resolved. The root cause was connection timeouts on a new egress proxy service.

We detected calls logs not reflecting in the dashboard for some time. This was due to an error while attaching a partition and has been resolved now. Call logs will be populated soon if they were missing.

After further investigation with our WebRTC provider, this does not seem to be platform issue. We will follow up with impacted users directly.

We experienced intermittent spikes in 5xx errors on our APIs in the weekly cluster. The root cause was identified, and a fix has already been implemented.

During this period, both inbound and outbound calls may have been affected, as they rely on the APIs for data, resulting in potential service degradation.

Services are restored.

Problem has been resolved now. All services are healthy again.

We have fixed the issue. Call Logs are returned correctly by API

Cartesia has resolved the issue, and is full operational.

We have identified the root cause. The problem has been fixed by rolling back a recent deployment on daily.

We have scaled up our telephony infrastructure resources and bumped our rate limits. We haven't seen any more issues in the last 20 minutes, and call transfers are working as expected now. We are closely monitoring.

Deepgram is investigating an issue where a subset of requests may return elevated rates of 5XX errors or experience significantly higher time to first byte

This incident has been resolved.

The issue was pinpointed and reverted quickly

Elevenlabs released a fix and is fully operational now. We are also seeing normal levels, but will continue to monitor.

For impacted users, we recommend implementing Vapi Fallback Plan to automatically failover in the future
https://docs.vapi.ai/voice-fallback-plan

IR August 4th: Call Degradation due to Pod Evictions

TL;DR

On August 4th, an incident occurred due to aggressive pod consolidation by Karpenter, which caused Redis pods to be evicted and restarted. This led to API pod failures, triggering a failover to an outdated networking component, resulting in dropped calls. The incident caused a total of 393 calls to be dropped.

Timeline (PST)

August 4th

• 11:02-11:27 AM - Core team identifies Karpenter pods in CrashLoopBackOff (OOMKi...