API is degraded
Resolved
Oct 09 at 02:27am PDT
Everything is back up for now.
Here's what happened:
* At 2:05am PT: We were alerted of the cannot execute UPDATE in a read-only transaction
errors by Datadog.
* By 2:15am: We determined it was unhealthy pooler state and restarted the DB to force reset all the connections.
* By 2:25am: We are back up.
We have several hypothesis on how the pooler session state got mangled. We're tracking them down right now.
UPDATE: We spent several days going back and forth with Supabase on why our DB was put in read-only mode. They didn't have a concrete answer either, our collective best guess is transaction wraparound.
Affected services
Vapi API
Created
Oct 09 at 02:11am PDT
We're investigating and will have more to share soon.
For now, write paths seem to be completely down with the error cannot execute INSERT in a read-only transaction
and cannot execute UPDATE in a read-only transaction
while read paths are going through.
Affected services
Vapi API