RCA: SIP Calls Ending Abruptly

TL;DR

A SIP node was rotated, and the associated Elastic IP (EIP) was reassigned to the new node. However, the SIP service was not restarted afterward, causing the SIP service to use an incorrect (private) IP address when sending SIP requests. Consequently, users receiving these SIP requests attempted to respond to the wrong IP address, resulting in ACK timeouts.

Timeline (PT)

• May 12, ~9:00 pm: SIP node rotated and Elastic IP reassigned, but SIP service was not restarted.
  • Calls appeared to succeed initially because they were routed through a healthy SIP node.
• May 13, 12:44 pm: Customer reports SIP calls consistently failing after approximately 30-31 seconds.
• May 13, 12:49 pm: SIP service restarted; customer confirms issue resolved.

Impact

• 35 calls experienced "ACK timeout" failures, corresponding directly to failed customer calls.

What Went Wrong?

• Lack of monitoring and alerting for SIP-related failures.
  • Issue persisted unnoticed for approximately 3 hours.
  • Customer reported issue first, not internal systems.
• Absence of documented runbooks for SIP node rotation process.
• No load test conducted following node rotation to verify successful SIP routing.

What Went Well?

• Rapid issue remediation following customer escalation.