API was down due to user error in routine maintenance. Service has since been restored

Summary
Users experienced login issues with our dashboard due to an unintended deployment of a staging version to the production environment.

Timeline (in PST):

• 3:17 PM: Internal engineers identified issues affecting developer workflows.
• 4:19 PM: Breaking change is introduced and unintentionally deployed to production
• 4:38 PM: First customer reports surfaced; engineering team immediately escalated internally.
• 4:43 PM: Public status page updated to notify customers.
• 4:54 PM: Correc...

Everything is functional. We're still working with Cartesia to get to bottom. We'll change back to degraded if the issue raises again during investigation.

RCA: Vapifault Worker Timeouts

TL;DR

On May 12, approximately 335 concurrent calls were either web-based or exceeded 15 minutes in duration, surpassing the prescaled worker limit of 250 on the weekly environment. Due to infrastructure constraints, Lambda functions could not supplement the increased call load. Kubernetes call-worker pods could not scale quickly enough to meet demand, resulting in worker timeout issues. The following day, this issue reoccurred due to the prescaling limit...

RCA: Providerfault-transport-never-connected

Summary

During a surge in inbound call traffic, two distinct errors were observed: "vapifault-transport-worker-not-available" and "providerfault-transport-never-connected." This report focuses on the root cause analysis of the "providerfault-transport-never-connected" errors occurring during the increased call volume.

Timeline of Events (PT)

• 10:26 AM: Significant spike in inbound call volume.
• 10:26 – 10:40 AM: Intermittent H...

RCA: SIP Calls Ending Abruptly

TL;DR

A SIP node was rotated, and the associated Elastic IP (EIP) was reassigned to the new node. However, the SIP service was not restarted afterward, causing the SIP service to use an incorrect (private) IP address when sending SIP requests. Consequently, users receiving these SIP requests attempted to respond to the wrong IP address, resulting in ACK timeouts.

Timeline (PT)

• May 12, ~9:00 pm: SIP node rotated and Elastic IP reassigned, but SI...

RCA: Phone Number Caching Error in Weekly Environment

TL;DR

Certain code paths allowed caching functions to execute without an associated organization ID, preventing correct lookup of the organization's channel. This unintentionally enabled caching for the weekly environment, specifically affecting inbound phone call paths. Users consequently received outdated server URLs after updating phone numbers.

Timeline (PT)

• May 10, 1:26 am: Caching re-enabled for users in daily envir...

RCA: 11Labs Voice Issue

TL;DR

Calls began failing due to exceeding the 11Labs voice service quota, resulting in errors (vapifault-eleven-labs-quota-exceeded).

Timeline of Events (PT)

• 12:04 PM: Calls begin failing due to 11Labs quota being exceeded.
• 12:16 PM: Customer reports the issue as a production outage.
• 12:24 PM: Contacted 11Labs support regarding quota exhaustion.
• 12:25 PM: 11Labs support recommends enabling usage-based billing.
• 12:26 PM: Usag...

RCA for May 2nd User error in manual rollout

Root cause:

• User error in kicking off a manual rollout, driven by unblocking a release
• Due to this, load balancer was pointed at an invalid backend cluster

Timeline

• 5:24pm PT: Engineer flagged blocked rollout, Infra engineer identified transient error that auto-blocked rollout
• 5:31pm PT: Infra engineer triggered manual rollout on behalf of engineer, to unblock release
• 5:43pm PT: On-call was paged with issue in rollout manager, e...

We've applied a fix to the dashboard web call feature. Root cause was misconfiguration on the authentication strategy used for this feature.

The team has applied a fix to frontend issue. Root cause was a change related to authentication handlers. Additionally, our KYC vendors are extending observability features in their SDK

We have resolved the issue. Will upload RCA 04/30 noon PST.

TL;DR: Recordings weren't uploaded to object storage due to some invalid credentials. We generated and applied new keys.

We have determined the issue and resolved. We will update by noon PST with an RCA.

TL;DR: Adding a new CIDR range to our SIP cluster caused issues where the servers were unable to discover each other.

Resolved the issue, blocked offending user and reviewed rate limits

API rollback completed and errors subsided

Improvements shipped reliably fixed the issue. Team has commenced medium-term, and is investigating long-term scalability improvements

Anthropic rate limiting is resolved after raising quota