Everything is functional. We're still working with Cartesia to get to bottom. We'll change back to degraded if the issue raises again during investigation.

RCA: Vapifault Worker Timeouts

TL;DR

On May 12, approximately 335 concurrent calls were either web-based or exceeded 15 minutes in duration, surpassing the prescaled worker limit of 250 on the weekly environment. Due to infrastructure constraints, Lambda functions could not supplement the increased call load. Kubernetes call-worker pods could not scale quickly enough to meet demand, resulting in worker timeout issues. The following day, this issue reoccurred due to the prescaling limit...

RCA: Providerfault-transport-never-connected

Summary

During a surge in inbound call traffic, two distinct errors were observed: "vapifault-transport-worker-not-available" and "providerfault-transport-never-connected." This report focuses on the root cause analysis of the "providerfault-transport-never-connected" errors occurring during the increased call volume.

Timeline of Events (PT)

• 10:26 AM: Significant spike in inbound call volume.
• 10:26 – 10:40 AM: Intermittent H...

RCA: SIP Calls Ending Abruptly

TL;DR

A SIP node was rotated, and the associated Elastic IP (EIP) was reassigned to the new node. However, the SIP service was not restarted afterward, causing the SIP service to use an incorrect (private) IP address when sending SIP requests. Consequently, users receiving these SIP requests attempted to respond to the wrong IP address, resulting in ACK timeouts.

Timeline (PT)

• May 12, ~9:00 pm: SIP node rotated and Elastic IP reassigned, but SI...

RCA: Phone Number Caching Error in Weekly Environment

TL;DR

Certain code paths allowed caching functions to execute without an associated organization ID, preventing correct lookup of the organization's channel. This unintentionally enabled caching for the weekly environment, specifically affecting inbound phone call paths. Users consequently received outdated server URLs after updating phone numbers.

Timeline (PT)

• May 10, 1:26 am: Caching re-enabled for users in daily envir...

RCA: 11Labs Voice Issue

TL;DR

Calls began failing due to exceeding the 11Labs voice service quota, resulting in errors (vapifault-eleven-labs-quota-exceeded).

Timeline of Events (PT)

• 12:04 PM: Calls begin failing due to 11Labs quota being exceeded.
• 12:16 PM: Customer reports the issue as a production outage.
• 12:24 PM: Contacted 11Labs support regarding quota exhaustion.
• 12:25 PM: 11Labs support recommends enabling usage-based billing.
• 12:26 PM: Usag...

RCA for May 2nd User error in manual rollout

Root cause:

• User error in kicking off a manual rollout, driven by unblocking a release
• Due to this, load balancer was pointed at an invalid backend cluster

Timeline

• 5:24pm PT: Engineer flagged blocked rollout, Infra engineer identified transient error that auto-blocked rollout
• 5:31pm PT: Infra engineer triggered manual rollout on behalf of engineer, to unblock release
• 5:43pm PT: On-call was paged with issue in rollout manager, e...

We've applied a fix to the dashboard web call feature. Root cause was misconfiguration on the authentication strategy used for this feature.

The team has applied a fix to frontend issue. Root cause was a change related to authentication handlers. Additionally, our KYC vendors are extending observability features in their SDK

We have resolved the issue. Will upload RCA 04/30 noon PST.

TL;DR: Recordings weren't uploaded to object storage due to some invalid credentials. We generated and applied new keys.

We have determined the issue and resolved. We will update by noon PST with an RCA.

TL;DR: Adding a new CIDR range to our SIP cluster caused issues where the servers were unable to discover each other.

Resolved the issue, blocked offending user and reviewed rate limits

API rollback completed and errors subsided

Improvements shipped reliably fixed the issue. Team has commenced medium-term, and is investigating long-term scalability improvements

Anthropic rate limiting is resolved after raising quota

For RCA please checkout https://status.vapi.ai/incident/528384?mp=true

For RCA please check https://status.vapi.ai/incident/528384?mp=true

Between 2025/03/27 8:40 PST and 9:35 PST, a small portion of SIP calls had their call durations initially inflated due to an internal system hang. The call duration information has been fixed retroactively.

TL;DR

After deploying recent infrastructure changes to backend-production1, Redis Sentinel pods began restarting due to failing liveness checks (/health/ping_sentinel.sh). These infra changes included adding a new IP range, causing all cluster nodes to cycle. When Redis pods restarted, they continually failed health checks, resulting in repeated restarts. A rollback restored API functionality. The entire cluster is being re-created to address DNS resolution failures before rolling forwar...

Issue was mitigated via rollback. We're investigating and will update with an RCA

Recording upload errors are recovered. We are continuing to monitor

TL;DR

It was decided that we should make Google Voicemail Detection the default option. On 16th March 2025, a PR was merged which implemented this change. This PR was released into production on 18th March 2025. On the morning of 19th March 2025, it was discovered that customers were experiencing call failures due to this change. Specifically: Google VMD was turned on by default, with no obvious way to disable it via the dashboard. Google VMD generated false positives when the bot identifi...

Resolved now.

RCA:
Timeline (in PT)
4:10pm New release went out for a small percentage of users.
4:15pm Our monitoring picked up increased errors in ending calls.
4:34pm Release was auto rolled back due to increased errors and incident was resolved.

Impact
Calls to end with unknown-error
End of call report was missing

Root cause:
A missing DB migration caused issues in fetching data during end of call.

Remediation:
Add CI check to make sure we don't release code when ...

The issue has subsided, we experienced a brief spike in call initiations and didn't scale up fast enough.

Immediate term, we're vertically scaling our call worker instances. Near term, we're rolling out our new call worker architecture for rapid scaling

RCA for SIP Degradation for sip.vapi.ai

TLDR;
Vapi sip service (sip.vapi.ai) was intermittently throwing errors and not able to connect to calls. We had some major flaws in our SIP infrastructure which was resolved by rearchitecting the whole thing from scratch.

Impact
- Call to Vapi SIP uri or Vapi phone numbers were failing to connect with 480/487/503 errors
- Inbound calls to Vapi getting connected but audio not coming out, eventually causing silence timeouts or customer-did-not...

TL;DR

Weekly Cluster customers saw vapifault-transport-never-connected errors due to workers not scaling fast enough to meet demand

Timeline in PST

• 7:00am - Customers report an increased number of vapifault-transport-never-connected errors. A degradation incident is posted on BetterStack
• 7:30am - The issue is resolved as call workers scaled to meet demand

Root Cause

• Call workers did not scale fast enough on the weekly cluster

Impact

There were 34 instances of vapifault-tr...

We are working with impacted customers to investigate but have not seen this issue occurring regularly.

TL;DR

Calls ended abruptly due to call-workers restarting themselves caused by high memory usage (OOMKilled).

Timeline in PST

• March 13th 3:47am: Issue raised regarding calls ending without a call-ended-reason.
• 1:57pm: High memory usage identified on call-workers exceeding the 2GB limit.
• 3:29pm: Confirmation received that another customer experienced the same issue.
• 4:30pm: Changes implemented to increase memory request and limit on call-workers.
• March 14th 12:27pm: Changes dep...

RCA: SIP 480 Failures (March 13-14)

Summary
Between March 13-14, SIP calls intermittently failed due to recurring 480 errors. This issue was traced to our SIP SBC service failing to communicate with the SIP inbound service. As a temporary mitigation, restarting the SBC service resolved the issue. However, a long-term fix is planned, involving a transition to a more stable Auto Scaling Group (ASG) deployment.

Incident Timeline
(All times in PT)

March 13, 2025
07:00 AM – SIP...

TL;DR

During the response to the concurrent Deepgram bug, it was noticed that an unstable Lodash change committed to main was leaked into the prod dashboard.

Timeline in PST

• 12:10am - Breaking changes were introduced to the main branch
• 12:42am - Afterwards, another commit was merged to main
  • This merge incorrectly triggered a deployment of the production dashboard
• 12:45am - A rollback in Cloudflare Pages was completed, restoring service
• 12:47am - Shortly afterward, a fix w...

TL;DR

An application-level bug was leaked into production, causing a spike in pipeline-error-deepgram-returning-502-network-error errors. This resulted in roughly 1.48K failed calls.

Timeline in PST

• 12:03am - Rollout to prod1 containing the offending change is started
• 12:13am - Rollout to prod1 is complete
• 12:25am - A huddle in #eng-scale is started
• 12:43am - Rollback to prod3 is started
• 12:55am - Rollback to prod3 is complete

Root Cause

• An application-level bug related...

RCA: vapifault-transport-never-connected errors caused call failures
Date: 03/10/2025

Summary:
A recent update to our production environment increased the memory usage of one of our core
call-processing services. This led to an unintended triggering of our automated process restart
mechanism, resulting in a brief period of call failures. The issue was resolved by adjusting the
memory threshold for these restarts.

Timeline:
1. 5:50am A few calls start facing issues in starting due to
vapifau...

We have rolled back the faulty release which caused this issue. We are monitoring the situation now.

The issue was caused by Vonage sending an unexpected payload schema, causing validation to fail at the API level. We deployed a fix to accommodate for the schema.

The API bug was reverted and we confirmed service restoration

We are seeing calls go through fine now, and are still keeping an eye out