TL;DR

During the response to the concurrent Deepgram bug, it was noticed that an unstable Lodash change committed to main was leaked into the prod dashboard.

Timeline in PST

• 12:10am - Breaking changes were introduced to the main branch
• 12:42am - Afterwards, another commit was merged to main
  • This merge incorrectly triggered a deployment of the production dashboard
• 12:45am - A rollback in Cloudflare Pages was completed, restoring service
• 12:47am - Shortly afterward, a fix was authored and pushed to main
• 12:58am - The fix was deployed to the prod dashboard

Root Cause

• The root cause was a seemingly unrelated commit that inadvertently triggered a production dashboard deployment.

Impact

The prod dashboard was unstable/broken for approximately 16 minutes.

What went poorly?

• We spent extra time determining how the breaking changes made their way into production.

What went well?

• The issue was fixed promptly after it was identified.

If working on realtime distributed systems excites you, consider applying: https://jobs.ashbyhq.com/vapi/295f5269-1bb5-4740-81fa-9716adc32ad5

Dashboard is not loading for some percent of users. We are looking into it.