Summary
Users experienced login issues with our dashboard due to an unintended deployment of a staging version to the production environment.

Timeline (in PST):

• 3:17 PM: Internal engineers identified issues affecting developer workflows.
• 4:19 PM: Breaking change is introduced and unintentionally deployed to production
• 4:38 PM: First customer reports surfaced; engineering team immediately escalated internally.
• 4:43 PM: Public status page updated to notify customers.
• 4:54 PM: Corrective actions deployed.
• 5:08 PM: Additional steps taken to accelerate resolution for users.
• 5:17 PM: Issue fully resolved and status page updated accordingly.

Impact:

• Users were temporarily unable to log into the dashboard.
• The issue was promptly reported and escalated by affected users.

Root Cause:
A configuration change intended to streamline internal development processes unintentionally led to the deployment of a staging version of our dashboard to the production environment. This occurred because the system did not adequately distinguish between environments in the deployment workflows, resulting in incorrect settings being applied in production.

What Went Well:

• Internal escalation was rapid, and the status page effectively informed users quickly.

What Went Poorly:

• Limited tooling for rapid rollbacks led to extended resolution time.
• Insufficient clarity around deployment workflows contributed to the incident.

Corrective Actions Taken:

• Immediately reverted the unintended deployment and restored the correct production configuration.
• Purged caches to expedite the resolution.

Future Preventative Measures:

• Enhance deployment configuration to clearly separate staging and production environments.
• Improve tools and processes for more rapid rollback capabilities in future deployments.

The sign-in issue has been resolved, and a fix has been successfully deployed. Users should now be able to access the dashboard as expected. We are currently preparing a RCA and will share it soon.

We are currently investigating an issue preventing some users from signing in to the dashboard. The team is actively working on a fix. We will provide updates as progress is made. Thank you for your patience.