RCA: Phone Number Caching Error in Weekly Environment

TL;DR

Certain code paths allowed caching functions to execute without an associated organization ID, preventing correct lookup of the organization's channel. This unintentionally enabled caching for the weekly environment, specifically affecting inbound phone call paths. Users consequently received outdated server URLs after updating phone numbers.

Timeline (PT)

• May 10, 1:26 am: Caching re-enabled for users in daily environment using the feature flag.
• May 13, 10:42 am: Customer reports phone calls referencing outdated server URLs after updates.
• May 13, 11:18 am: Caching disabled globally; service fully restored.
• May 13, ~10:00 pm: Fix deployed to weekly environment; caching globally re-enabled.

Impact

• Customers experienced degraded service; updates to server URLs or assistant configurations for phone numbers did not immediately reflect during calls.
• Issue previously identified and resolved in daily environment resurfaced in weekly due to incomplete implementation of the feature flag.

What Went Wrong?

• Inadequate testing of the feature flag allowed unintended caching on some paths.
  • Lack of proper failure handling when organization ID was missing.
• Issue surfaced through customer reporting, not internal monitoring.
• Fix deployed to daily environment was not applied to weekly environment in time.

What Went Well?

• Feature flag system allowed rapid disabling of caching globally once identified.