Back to overview
Degraded

API is degraded

Nov 12 at 02:12pm PST
Affected services
Vapi API

Resolved
Nov 12 at 02:15pm PST

TL;DR: API pods were choked. Our probes missed it.

Root Cause

Our API experienced DB contention. Recent monitoring system changes meant our probes didn't pick up this contention and restart the pods.

Timeline

  • November 12th 2:00pm PT - Customer reports of API failures
  • November 12th 2:05pm PT - Oncall team determined cause and scaled and restarted pods
  • November 12th 2:10pm PT - Full functionality restored.

Changes we've implemented

  1. Restored higher sensitivity thresholds for our monitoring systems
  2. Currently investigating underlying database connection management

If working on realtime distributed systems excites you, consider applying: https://jobs.ashbyhq.com/vapi/295f5269-1bb5-4740-81fa-9716adc32ad5

Created
Nov 12 at 02:12pm PST

Seeing long connection times. Investigating.