Google introduced Gemini 3.5 Transcribe on August 26 with reported word error rates of 4.0% for streaming audio and 2.6% for recorded audio. The release also calls out order IDs, postal codes, custom vocabulary, and live corrections. Those details matter more to support teams than a single average accuracy number.
A transcript can be 99% right and still send the wrong order number to billing. Before shipping voice input, use this compact test: collect 40 production-shaped utterances, mark every critical field, run each through the complete customer interface, require exact matches for those fields, and verify that every wrong value is caught before the chatbot sends or acts on it. The scorecard below shows how.
Start with the fields that can break the workflow
Ordinary words and critical fields do not carry the same risk. If speech-to-text changes "I'd like to check" to "I want to check," the intent survives. If it changes order AK-0819 to AK-0890, the support case points at the wrong record.
List the values your chatbot collects or repeats. Mark a field critical when one wrong character could expose data, misroute work, or cause an unwanted action.
| Field | Test variations | Passing behavior |
|---|---|---|
| Order and ticket IDs | Spoken letters, leading zeroes, repeated digits, weak signal | Exact value or a targeted request to repeat it |
| Names | Uncommon spelling, hyphenated name, correction after first attempt | Shows the captured spelling and lets the visitor edit it |
| Email addresses | Dots, plus signs, similar-sounding letters, custom domains | Confirms the full address before follow-up |
| Postal addresses | Apartment numbers, street suffixes, city names, postcode spacing | Separates fields and confirms any uncertain part |
| Dates and times | Relative dates, time zones, "fifteen" versus "fifty" | Converts only after resolving ambiguity |
| Money and quantities | Cents, decimals, negatives, item counts | Repeats the normalized value before a consequential action |
Do not pad the set with easy studio speech. Use the conditions your visitors create: laptop microphones, phone speakers, background conversation, short pauses, accents, code-switching, product names, and tired people reading identifiers from another screen.
A useful 40-utterance first pass might contain 12 identifiers and contact fields, eight self-corrections, eight noisy clips, six domain-specific terms, and six multilingual or code-switched messages. Reuse each clip after a model, prompt, browser, or audio pipeline change. Consistent inputs make a regression visible.
Measure the field, not only the words
Word error rate is useful for broad transcription quality. It counts substitutions, deletions, and insertions against the number of words in the reference transcript:
word_error_rate = (substitutions + deletions + insertions) / reference_words
It does not express business consequence. A 100-word message with one wrong digit has a 1% word error rate and may still fail the task. Add three product-level measures:
critical_field_exact_match = correct_critical_values / critical_values_tested confirmation_catch_rate = wrong_critical_values_caught_before_send / wrong_critical_values_produced silent_commit_rate = wrong_critical_values_accepted_by_the_workflow / critical_values_tested
The exact-match measure gives no partial credit. AK-0819 and AK-819 may refer to different records. The catch rate tests the interface around the model. The silent commit rate reveals the failures that can reach a queue, CRM, account lookup, or action.
Set gates by consequence. A support note can tolerate a harmless punctuation difference. An authenticated account lookup should require an exact identifier or a fresh confirmation. A refund, booking, address change, or outbound message should never proceed from an unconfirmed value that the transcript marked as uncertain.
This makes voice testing a complement to chatbot model evaluation. Model evaluation asks which system answers better. Transcription testing asks whether the system heard the request well enough to answer or act at all.
Test corrections as their own feature
People repair speech while they talk:
"My order is A K zero eight one four. Sorry, make that A K zero eight one nine."
A polished transcription model may remove filler and rewrite the sentence into clean prose. That cleanup is useful until it removes evidence of which value was corrected. Your expected result should be the final identifier, AK-0819, with no action taken on the abandoned one.
Include several correction shapes:
- a replacement in the same sentence;
- a correction after the chatbot repeats the value;
- a letter-by-letter spelling repair;
- a changed date after the user checks a calendar;
- a negation such as "not Tuesday, Thursday."
Then inspect more than the final transcript. Did the interface briefly submit the first value? Did an account lookup start before the correction finished? Did the transcript keep both values and leave the chatbot to guess? Did the confirmation repeat the right value?
OpenAI's account of its continuous voice system explains why this boundary is getting harder. Streaming systems can listen, respond, and delegate work on separate paths. That responsiveness feels natural, but it also creates more chances for a partial utterance to escape into application logic. Treat end-of-speech detection and the final transcript event as product controls, not invisible plumbing.
Run the test through the real interface
A clean audio file sent directly to a model API tests the model. It does not test your voice chatbot.
Run the corpus through the same browser, microphone permission flow, codec, network path, streaming settings, transcript editor, and send button a visitor uses. Inject a slow connection. Deny microphone permission once. Stop recording immediately after a digit. Switch audio devices midway through a session. Put the tab in the background and return.
Record the events in order:
recording_started partial_transcript: "order A K zero eight one four" partial_transcript: "sorry make that A K zero eight one nine" final_transcript: "Order AK-0819" visitor_reviewed: true message_sent: "Order AK-0819" workflow_lookup_started: after_message_sent
The ordering is the proof. A perfect final transcript does not repair a lookup that started on the first partial result.
Keep partial transcripts out of business actions. Use them to render responsive text if needed, then wait for a final result and the required user confirmation. If the connection drops before finalization, preserve the audio state only as long as your privacy policy allows and ask the visitor to try again. Do not silently convert the last partial fragment into a submitted request.
Match confirmation to consequence
Confirmation should be specific enough to catch an error without turning every message into a tedious read-back exercise.
| Consequence | Example | Confirmation rule |
|---|---|---|
| Low | General product question | Show editable text before send |
| Moderate | Email for a support reply | Display the address and ask the visitor to approve it |
| High | Account lookup or address change | Repeat the critical value and require an explicit yes or edit |
| Irreversible or chargeable | Refund, cancellation, purchase | Confirm the complete action payload under the normal approval policy |
Ask about the uncertain field, not the whole sentence. "Was that order AK-0819?" is easier to check than replaying a paragraph. For email addresses and codes, a visual confirmation is often clearer than speech because the visitor can inspect every character.
Do not let confidence scores waive a high-consequence confirmation. A model can be confidently wrong on a name it has never seen. Confidence can decide whether to ask for repetition sooner, but the workflow should decide which fields always need approval.
The same rule applies after handoff. Put the confirmed value in a structured field and keep the original user message beside it. A support agent should be able to see what the visitor approved without replaying audio or guessing which of two transcript values won.
A worked release decision
Suppose a retailer tests 40 utterances containing 28 critical values. The new transcription setup produces four ordinary wording errors and one wrong order ID. The interface spots the low-confidence ID, asks the visitor to confirm it, and the visitor fixes it before sending.
The results are:
| Measure | Result | Release reading |
|---|---|---|
| Word error rate | 3.1% | Good broad accuracy, but not a release gate by itself |
| Critical-field exact match | 27 of 28, or 96.4% | One identifier still needs a recovery path |
| Confirmation catch rate | 1 of 1, or 100% | The interface caught the model's critical error |
| Silent commit rate | 0 of 28, or 0% | No wrong critical value reached the lookup workflow |
This build can be reasonable for editable dictation because the visitor saw and corrected the only critical error. It is not ready for automatic account lookup without confirmation. The same model result supports two different release decisions because the interfaces create different risk.
Save the clip, reference transcript, field labels, raw output, final submitted text, environment, model version, and decision. When the transcription provider changes, rerun the exact set before adding new cases. A moving benchmark cannot distinguish improvement from a friendlier sample.
Review what users correct in production
Offline clips will miss new products, local street names, seasonal noise, and the strange way customers pronounce internal acronyms. Production corrections are the fastest source of the next test cases.
Compare the final transcript with the text the visitor actually sends. A changed identifier, email address, date, or product name should enter a review queue after redaction. Do not treat every edit as a transcription defect. Visitors also rewrite intent and soften tone. Label only changes that repair what was spoken.
Sample both corrected and uncorrected dictation. An unedited message is not proof of accuracy because visitors may not notice the mistake. Review conversations that failed an account lookup, caused a repeated question, or arrived at a human with conflicting fields. The chatbot QA sampling guide explains how to keep targeted defect hunting separate from an unbiased baseline.
Retain audio only when you have a stated purpose, consent, access control, and deletion window. Many teams can improve the product with redacted text pairs and structured error labels. If audio is necessary for accent, noise, or timing analysis, restrict it more tightly than ordinary conversation logs.
Ship the confirmation path, not the benchmark
New speech models will keep lowering average error rates. That is welcome, but support reliability depends on the one wrong character that an average hides. Mark critical fields, test real correction patterns through the full interface, and block silent commits. The best voice input is not the one that never mishears. It is the one that makes a consequential mistake easy to catch before it leaves the composer.
In Agentkit, widget voice dictation puts transcribed text in the composer so visitors can review and edit it before sending, while conversation logs preserve the sent text for later QA.
No credit card required.
