Root Cause Analysis (RCA) for the Incident – Timeline in PT

TL;DR

A recent security fix by Supabase impacted database projects using pg_net 0.8.0, causing failures in the POST /credential endpoint and new user signups.

Timeline

March 5th, 3:00 AM PT: Failures in POST /credential endpoint and new user signups begin.

3:26 AM PT: On-call engineer observes a surge in errors related to POST /credential, including an unusual PostgresError.

3:32 AM PT: Team reproduces the issue locally and declares an incident, escalating it internally.

3:34 AM PT: Error logs indicate a Postgres-level issue: permission denied for table httprequestqueue.

3:44 AM PT: High-priority support ticket created with Supabase; investigation begins.

3:59 – 4:58 AM PT:

- Various queries attempted to fix the permission issue on httprequestqueue.

- Issue reproduced using a simple pg_net test query:

sql
SELECT net.httpget('https://news.ycombinator.com') AS requestid;


- Supabase confirms the issue is on their end and requires their intervention.

- Team devises a plan to remove dependency on pg_net for signups and credentials, partially restoring services.

4:58 AM PT: Supabase resolves the root cause; full service restoration.

Impact

During the incident, POST /credential requests and new user signups failed.

Root Cause

A security fix by Supabase affected database projects using pg_net 0.8.0, including our production database.

Changes We've Made

• Enhanced Monitoring: Added a dedicated monitor to detect similar permission or security issues quickly.
  

Changes We Will Make

• Migration Strategy: Plan to migrate away from pg_net triggers where possible to reduce similar risks.
  

What Went Well

• The issue was quickly identified and escalated internally.
  
• A workaround was devised to restore partial functionality while awaiting Supabase’s resolution.
  

What Went Poorly

• Service was impacted until external intervention was completed.
  
• The dependency on pg_net introduced an external risk factor beyond our immediate control.
  

For more details, refer to the Supabase incident: https://status.supabase.com/incidents/skb7jbxwfm47

Incident has been resolved. Signups are working again and credentials can now be added.

Supabase has raised a public incident on this and are working on resolving it.
https://status.supabase.com/incidents/skb7jbxwfm47

We are working with supabase team to resolve it.

Signups and credential creation are not working. We are investigating.