TL;DR

It was decided that we should make Google Voicemail Detection the default option. On 16th March 2025, a PR was merged which implemented this change. This PR was released into production on 18th March 2025. On the morning of 19th March 2025, it was discovered that customers were experiencing call failures due to this change. Specifically: Google VMD was turned on by default, with no obvious way to disable it via the dashboard. Google VMD generated false positives when the bot identified itself as a bot.

Timeline in PST

• 16th March 2025: the offending PR is merged.
• 18th March 2025, 3:08 PM: the offending PR is released to production.
• 19th March 2025, 8:52 AM: Vapi Eng bot reports an incident: https://vapi-ai.slack.com/archives/C06GT64R399/p1742399522864239
• 19th March 2025, 9:18 AM: It is determined that the issue is likely caused by Gemini VMD.
• 19th March 2025, 10:04 AM: Production is rolled back, immediately resolving the issue.
• 19th March 2025, 11:00 AM: Hotfix is committed to production.

Root Cause

Several issues were identified:

• Google VMD should not have been set as the default option. Any non-essential feature should be disabled by default.
• From a dashboard perspective, "undefined" should always imply "off".

Additionally:

• Google VMD produced false positives whenever the bot revealed itself as an AI or otherwise implied it was non-human. Examples:
  • "Thank you for calling Jim Adler and Associates! I’m Kendall, an AI assistant. This call may be recorded for quality and training purposes as well as to help direct your information to the right person. I’m here to answer questions or book appointments—how may I assist you?"
  • "Thank you for calling Max Electric! This call is being recorded for quality and training purposes. You are calling outside of our business hours. This is Matthew. Please let me know how I can help!"

This appears to be an edge case identifiable primarily through actual usage.

What went poorly?

• A non-essential feature was set as a default option.

What went well?

• The issue was taken seriously as soon as it was identified.
• The root cause was quickly discovered.

Remediation

• Production was rolled back promptly.
• A hotfix was implemented to stabilize production (ensuring Google VMD is no longer the default).
• A longer-term fix has been developed to mitigate false positives.

If working on realtime distributed systems excites you, consider applying: https://jobs.ashbyhq.com/vapi/295f5269-1bb5-4740-81fa-9716adc32ad5

We have released a fix for this issue

We have identified the root cause and rolled back. We are working on fix.

Google VMD is intermittently flagging on-going calls as "voicemail" and causing them to end with customer-did-not-answer. We are investigating and will have an update by 12pm PST latest.

Users can resolve this by using an alternate VMD provider (Twilio or OpenAI).