Summary:
We experienced an issue related to API key validation within our WebSockets implementation when sending the API key more than once.

Details:
The issue arose during API key validation within our WebSockets implementation. Our system validates that the API key provided during the initial message is the same as proceeding messages.

A recent change introduced during a release caused a mismatch in how API keys were compared. Specifically, the system was comparing hashed API keys against non-hashed API keys. This comparison would always fail, as hashed and non-hashed keys are inherently different. The impacted API keys were legacy API keys, which were not being hashed.

Timeline (GMT +2):
Release Ready: 9:23 AM
Full Deployment: 9:52 AM
Reported by Vapi: 12:28 PM
Rollback Initiated: 12:53 PM

Impact:

This issue impacted a small number of clients using non-legacy API keys who also provided the API key multiple times during the WebSocket connection. Specifically, if the API key was provided during the initial connection and then again in subsequent messages, our system performs a validation check. Due to a flawed comparison between hashed and non-hashed API keys, this validation check failed for those clients sending API keys multiple times, resulting in the error you saw.

Resolution:
- The engineering team has implemented a fix to ensure API keys are compared correctly, regardless of whether they are hashed or non-hashed. The fix has been deployed.

Preventative Measures:
- To prevent similar issues in the future, the following steps are being taken:
We already had tests for this, but unfortunately, we found issues with the tests that clearly didn't catch this because of a race condition. That race condition has since been solved.
- We’ve also made sure the tests now block merges.

Services are back up now. Elevenlabs rolled back a change, errors have come down now, resolving it. We will keep monitoring the situation further for some time.

We are working with 11labs team to resolve an issue wherein 11labs are not working when users bring their own key on Vapi.