RCA: 11Labs Voice Issue

TL;DR

Calls began failing due to exceeding the 11Labs voice service quota, resulting in errors (vapifault-eleven-labs-quota-exceeded).

Timeline of Events (PT)

• 12:04 PM: Calls begin failing due to 11Labs quota being exceeded.
• 12:16 PM: Customer reports the issue as a production outage.
• 12:24 PM: Contacted 11Labs support regarding quota exhaustion.
• 12:25 PM: 11Labs support recommends enabling usage-based billing.
• 12:26 PM: Usage-based billing activated; issue resolved immediately.

Root Cause Analysis

• The incident occurred because the monthly quota limit for 11Labs voice services was reached.
• Example error log: { "message": "This request exceeds your quota of 2000000000. You have 4 credits remaining, while 23 credits are required for this request.", "error": "quota_exceeded", "code": 1008 }

What Went Wrong?

• Lack of proactive alerting: No paging occurred because logs were being sampled and adequate monitors were not in place in the new logging system.
• Initial difficulty diagnosing the issue quickly due to limited familiarity with the new logging tool (Axiom).

What Went Well?

• Rapid response and effective support provided by the external vendor (11Labs).
• Swift resolution once the problem was clearly identified.