Previous incidents
Degradation in connecting SIP calls.
Resolved Sep 12 at 03:47pm PDT
Services are restored.
1 previous update
High errors with connecting to our assistant Db. Calls and API are affected.
Resolved Sep 10 at 12:27pm PDT
Problem has been resolved now. All services are healthy again.
1 previous update
Call Logs Since 09/05 00:00 UTC not loading
Resolved Sep 04 at 06:43pm PDT
We have fixed the issue. Call Logs are returned correctly by API
1 previous update
Cartesia Voices Degraded
Resolved Sep 04 at 01:55pm PDT
Cartesia has resolved the issue, and is full operational.
1 previous update
Elevated cases of assistant not responding in calls on daily cluster
Resolved Sep 03 at 03:06am PDT
We have identified the root cause. The problem has been fixed by rolling back a recent deployment on daily.
1 previous update
Call Transfers Degradation on Vapi Phone Number
Resolved Sep 02 at 12:24pm PDT
We have scaled up our telephony infrastructure resources and bumped our rate limits. We haven't seen any more issues in the last 20 minutes, and call transfers are working as expected now. We are closely monitoring.
1 previous update
Deepgram Aura-2 TTS Performance Degradation
Resolved Aug 28 at 12:04pm PDT
Deepgram is investigating an issue where a subset of requests may return elevated rates of 5XX errors or experience significantly higher time to first byte
Aura-2 TTS Performance Degradation
Resolved Aug 27 at 02:00pm PDT
This incident has been resolved.
1 previous update
Intermittent issues with the dashboard loading
Resolved Aug 25 at 05:04pm PDT
The issue was pinpointed and reverted quickly
2 previous updates
Elevated error rates on ElevenLabs Voice requests
Resolved Aug 05 at 09:14pm PDT
Elevenlabs released a fix and is fully operational now. We are also seeing normal levels, but will continue to monitor.
For impacted users, we recommend implementing Vapi Fallback Plan to automatically failover in the future
https://docs.vapi.ai/voice-fallback-plan
1 previous update
API and calls were momentarily disrupted
Resolved Aug 05 at 12:12pm PDT
IR August 4th: Call Degradation due to Pod Evictions
TL;DR
On August 4th, an incident occurred due to aggressive pod consolidation by Karpenter, which caused Redis pods to be evicted and restarted. This led to API pod failures, triggering a failover to an outdated networking component, resulting in dropped calls. The incident caused a total of 393 calls to be dropped.
Timeline (PST)
August 4th
- 11:02-11:27 AM - Core team identifies Karpenter pods in CrashLoopBackOff (OOMKi...
1 previous update
Deepgram is experiencing connection issues
Resolved Jul 28 at 10:19pm PDT
We are no longer seeing signs of connection issues. The issue should be resolved now, but we will continue to monitor. We apologize for any inconvenience caused.
1 previous update
Incident Report: Increased call failures on July 25 (PST)
Resolved Jul 25 at 02:46pm PDT
Incident Report:
Increased call failures on July 25 (PST)
Summary (TL;DR)
On July 25 between 7:00–7:15am PST, a spike in call volume caused some calls to fail with a worker-not-available error. The fallback service for short calls (our serverless workers) could not start because its image architecture didn’t match the configured runtime (image built for ARM64, runtime set to x86). We stabilized the platform by scaling primary workers and then corrected the configuration. Service is o...
Elevenlabs is experiencing increased Text-to-Speech latency
Resolved Jul 24 at 11:30am PDT
Elevenlabs released a hotfix and is fully operational now.
1 previous update
Web Calls from Assistant Page are unavailable
Resolved Jul 17 at 10:50am PDT
Dashboard calls should be back to normal.
1 previous update
Deepgram transcription is degraded
Resolved Jul 17 at 10:34am PDT
This was resolved
1 previous update
Up to date calls not returned in API or dashboard
Resolved Jul 09 at 09:24pm PDT
We have identified and resolved the issue. Apologies for the disruption.
1 previous update
Playht voices are degraded
Resolved Jul 07 at 11:18pm PDT
Resolved now.
1 previous update