TL;DR

In response to hallucinations reports in the Success Evaluation feature, we updated our integration with Gemini LLM to use Structured Output. This inadvertently changed the type of the call.analysis.successEvaluation field from string | null to string | number | boolean | null, introducing a breaking change for customers with strict type validation and those using Vapi Server SDKs.

Timeline (all in PT)

• June 12, 11:32pm: Enterprise and Startup users report hallucinations in Success Evaluation field. Engineer acknowledges reports and begins work in a solution by migrating to Gemini Structured Output.
• June 16, 11:35pm: Migration to Structured Output is completed. Update passes automated code tests and is merged into main branch.
• June 17, 1:24pm: Update is released, inadvertently introducing changes in the type of call.analysis.successEvaluation property.
• June 18, 11:15am: Enterprise users reports breaking change in webhook message; investigation begins.
• June 18, 1:51pm: Vapi team decides to retain the new type change and communicates to affected users, requesting updates to their servers to accept string | number | boolean | null.
• June 18, 3:43pm: Enterprise users reports Go SDK-specific issue; investigation begins.
• June 18, 4:08pm: Team identifies broader SDK impact and start work on a patch to revert API to string-only output while keeping Structured Output.
• June 18, 7:42pm: Patch reverting API output to string-only is released.

Impact

Between June 17th 1:24 pm and June 18th 7:42 pm, organizations in daily channel, using strict type validation on their servers or using Vapi Server SDKs experienced issues when processing post call analysis events.

What went wrong?

• Automated tests failed to catch the breaking change in API response.
• Poor communication of internal changes to core platform features.
• Underestimated the impact, leading to a late rollback (+24hrs)

What went well?

• Organizations in weekly channel were not affected.
• Calls were not affected on any of the channels.
• Hallucination issue appears resolved.

Action Items

• Testing: Build comprehensive integration tests to catch response type changes.
• Communication: Design better notifications and public changelog protocols for potential breaking changes.
• Support: Support affected customers and requested server updates. Follow ups to confirm no further issues and assist with any remaining fixes.

Organizations in daily channel report breaking change end of call report.
Property call.analysis.successEvaluation was migrated from string | null to string | number | boolean | null.

Organizations in weekly channel are not affected.