AIE Talks

Valeria Wu Fon and Tom Ouyang describe Google's move from separate speech components toward a natively multimodal speech-to-speech model. Older systems chained feature extraction, acoustic modeling, pronunciation modeling, language modeling, and rescoring before producing text. End-to-end speech recognition simplified that chain, but it still handled transcription as one isolated task. Gemini models are trained on interleaved audio, video, and text,...

Bohan Li explains how EliseAI makes voice agents respond quickly while still using slow, intelligent language models. He compares the system with a self-driving car: transcription turns audio into usable data, the language model plans a response, and text-to-speech turns that response into action. The transcription layer combines a fast streaming engine with a slower, more accurate engine that can...

Charlie Guo argues that voice agents do not need to answer with speech. Voice interaction has three modes: speech to speech, speech to action, and event to speech. All existed before current models, but better models make them easier to combine. Guo spends most of the talk on speech to action, where a model uses tools to fill forms, operate...

Sumanyu Sharma compares his past work listening to police radio at Citizen with his current concern about voice agents. Crime usually affects a limited local group, while a voice agent can apply one bad prompt or architecture change to many users. The failures are often ordinary: skipped verification, an unauthorized discount, incorrect information, or an appointment that was never booked....

Paula Dozsa explains why Tolan treats voice as a different engineering problem from text chat. Users expect the system to respond within a couple of seconds, interrupt it, change subjects, and speak while doing something else. A half-second latency increase caused broad product damage, so Tolan measures every pipeline stage and accepts extra latency when it prevents bad interruptions. The...