Back to overview
Downtime

API is degraded

Oct 09 at 02:11am PDT
Affected services
Vapi API

Resolved
Oct 09 at 02:27am PDT

Everything is back up for now.

Here's what happened:
* At 2:05am PT: We were alerted of the cannot execute UPDATE in a read-only transaction errors by Datadog.
* By 2:15am: We determined it was unhealthy pooler state and restarted the DB to force reset all the connections.
* By 2:25am: We are back up.

We have several hypothesis on how the pooler session state got mangled. We're tracking them down right now.

UPDATE: We spent several days going back and forth with Supabase on why our DB was put in read-only mode. They didn't have a concrete answer either, our collective best guess is transaction wraparound.

Created
Oct 09 at 02:11am PDT

We're investigating and will have more to share soon.

For now, write paths seem to be completely down with the error cannot execute INSERT in a read-only transaction and cannot execute UPDATE in a read-only transaction while read paths are going through.