alt.hn

5/4/2026 at 7:42:47 PM

How OpenAI delivers low-latency voice AI at scale

 https://openai.com/index/delivering-low-latency-voice-ai-at-scale/

by Sean-Der

5/4/2026 at 8:52:33 PM

Very grateful that OpenAI published the article/publicized their usage of Pion[0] a library I work on. If you aren't familiar with WebRTC it's a super fun space. I work on a book WebRTC for the Curious [1] that details how it works.

[0] https://github.com/pion/webrtc

[1] https://webrtcforthecurious.com

by Sean-Der

5/4/2026 at 10:54:17 PM

I use pion thanks for making it!

Curious if you thought their approach was necessary, it seemed like a ton of complexity to reduce one of the faster parts of a voice AI setup. Having a fast model and accurate VAD seems way more important than fine tuning WebRTC transit times.

by ericmcer

5/4/2026 at 11:24:37 PM

Thanks for using it :)

I think It’s a case of you improve what you own. The owners of WebRTC servers were aggressively improving their part. They don’t own the inference servers.

by Sean-Der

5/4/2026 at 9:28:38 PM

Appreciate you putting the entire book online!

I read parts of it a while ago when I had an idea on using webRTC data channels to pass data from databases to browser clients via a CLI. Your book made me understand that it's probably not a great fit for my use case. I just used a centralized control plane and websockets instead.

I still feel like there is something fun that we can do with webRTC data channels + zero copy Apache Arrow arraybuffers + duckdb WASM, but haven't figured it out yet

by aleda145

5/4/2026 at 11:33:00 PM

Thanks for reading it!

You can't beat Websockets :) Especially since you have so much tooling/existing stuff that works with HTTP.

I have been trying to get a website off the ground that does Datachannels + SQlite in the browser and then users sync between each other. I have gotten distracted so many times though.

by Sean-Der

5/5/2026 at 8:07:22 AM

Super cool! Please give me a ping if you ever launch that (my email is on my website (in my profile))

by aleda145

5/4/2026 at 10:36:53 PM

What is preventing the fun is that even though we now have IPv6 widely enough available we still can't have p2p connections in the browser without a cumbersome control plane of servers. If you could join a federation in the browser from some bootstrap IPs then I think we could have some real distributed fun.

by oezi

5/4/2026 at 9:17:21 PM

Thanks for WebRTC for the Curious and for Pion! Not using the latter directly, but have used both to better understand WebRTC

by dtran

5/5/2026 at 1:07:53 PM

For those unfamiliar with WebRTC, the Pion FAQ page has a good description:

> WebRTC is a standardized protocol for P2P communication. It allows two peers to exchange media and data. It is encrypted by default, and handles connectivity establishment in many different network conditions. It is supported in browsers, and has multiple out of browser implementations.[0]

[0]: https://github.com/pion/webrtc/wiki/FAQ#what-is-webrtc

by ssgodderidge

5/4/2026 at 9:01:29 PM

slightly unrelated but what’s with storing the entire codebase in the root directory instead of a nested src folder? It makes getting to the README a lot more difficult

by thatxliner

5/4/2026 at 9:13:23 PM

Thats the default for go projects. Go imports are repository strings (e.g.):

     import ("github.com/go-sql-driver/mysql")
so it's standard to have the library files in the root directory.

by nemothekid

5/5/2026 at 5:36:12 PM

FWIW I usually don't structure my Go projects this way unless they're very very small. This is what I usually do for anything larger than 2-3 files:

  ├── cmd
  │   └── binary-name
  │       └── main.go (may subpackage for things like CLI porcelain, etc)
  ├── go.mod
  └── internal
      └── app.go (and subpackages, etc)

by dgunay

5/5/2026 at 11:58:31 AM

I assume this is why GitHub has the annoying #readme-ov-file slug

by saagarjha

5/4/2026 at 9:16:51 PM

This is valid criticism. Go fanbois don't like listening to any go criticism. They were all like who needs templates in go. and now go has templates.

To me go code looks like somebody vomitted stuff in the root dir and i have to wade through that every time. No namespacing. nothing

by a456463

5/4/2026 at 9:44:07 PM

I don't like go as a personal preference but reducing them to "fanboys" is a bit reductive. I'm sure the same could be said about your own favorite language.

by junon

5/7/2026 at 8:35:58 PM

Also as far as being civil: please watch literally any Rob Pike vibe chastizing Python and Java communities to no end. It is that rhetoric. That is why Go has fanbois. They can only see what the Gopher says and what Rob Pike says. Nothing else.

Other languages have fanbois but they didn't disparage and denigrate other communities to prove themselves.

by a456463

5/4/2026 at 10:00:35 PM

Is it reductive when its describing a group of people that like something and refusing to hear any ill of it? The comment wasn't shade at people using the language in general.

And you're right, fanboys are in every language. But resorting to changing the argument by whataboutism is a bit reductive.

by altmanaltman

5/4/2026 at 11:09:26 PM

I’m not a go fanboy, but I do know from other contexts that so-called “fanboy“ behaviour is frequently associated with level-headed supporters getting defensive in the face of imprecise criticism.

There’s an oft-repeated pattern where valid specific criticisms morph into broad criticism, which morphs into judgement, which breeds defensiveness, which feeds the criticism. Once you recognise this pattern, you see it everywhere.

by simondotau

5/5/2026 at 1:18:04 AM

Sure, and there's the near-identical pattern where valid specific criticisms are taken as broad criticism even though they aren't, etc., etc..

by twodave

5/5/2026 at 5:04:42 PM

Thats the defensive step outlined above.

by xboxnolifes

5/5/2026 at 12:38:22 AM

Ok... The question was why is it like that. The answer is because it's in go. Nobody was anything other than civil before you neckbearded in here. Chill. There's a sane way to say what you said.

by idiotsecant

5/7/2026 at 8:34:04 PM

I wish I could grow a beard. Also as far as being civil: please watch literally any Rob Pike vibe chastizing Python and Java communities to no end. It is that rhetoric. That is why Go has fanbois. They can only see what the Gopher says and what Rob Pike says. Nothing else.

by a456463

5/5/2026 at 5:32:06 PM

I believe it was "They were all like who needs generics in go. and now go has go generate and templates."

I guess I qualify as a Go fanboi -- it is not perfect but gets the job done for a lot of us, sorry it doesn't work for you.

But back to your point about "vomit in the root dir", Go does have namespacing of sorts via packages, and the pattern you criticized is not the only way -- often just a simple main.go at the root bringing in packaged functionality.

by pstuart

5/5/2026 at 2:07:01 AM

WebRTC is great and so is Pion, thanks for help making and maintaining it! I loved learning about WebRTC from WebRTC for the Curious!!!

by willmeyers

5/5/2026 at 12:17:35 AM

I used pion and it was fantastic. Most of the article seems pretty standard webrtc techniques for performant voice.

by ryanar

5/4/2026 at 10:10:27 PM

Only a software dev would start their referencing at 0 lol

by haaz

5/4/2026 at 10:11:53 PM

I do this too I never made the connection.

by vorticalbox

5/4/2026 at 8:52:00 PM

The low latency is more of a pain point than a good thing, the way they have it implemented. Trying to have a casual conversation with it, as humans we naturally pause, and GPT will take this as you are "done" and start blabbing away.

I also suffer from finding the appropriate word I want as I've gotten older and slower, and this fast-voice-gpt just ends up frustrating me more than helping. I have to sit there and think out the whole sentence in my head before I say anything -- not very natural.

by legohead

5/4/2026 at 9:04:44 PM

I think these are 2 different layers of "latency". The latency in the article is referring to the transport of the audio stream itself while the latency in your scenario is about how quickly to start responding inside the audio stream.

by zamadatix

5/4/2026 at 11:53:00 PM

They are orthogonal.

Suppose you have 100ms audio latency and no wait time. Then, natural pause will trigger response immediately but you won't notice it has started until after ~200ms (round-trip time). Twice as annoying.

by hun3

5/4/2026 at 11:06:20 PM

I think he’s saying they are doing an insane level of complexity to shave ~100ms off response times in a scenario where that isn’t important and might even be a negative

by ericmcer

5/4/2026 at 11:27:13 PM

When GP mentioned reducing conversational latency as a negative that made sense (and should probably be done IMO), it just wasn't the same category of latency the article talks about reducing. I.e. increasing "network latency" just makes the conversation feel more and more out of sync, it doesn't change the rate at which the AI will interrupt ("turn latency") because the latter is based on the duration of the pause in the audio stream, not the duration it took to deliver that audio stream.

If you meant there is a case where reducing the network latency at the same delivery reliability for a given audio stream is actually a negative then I'd love to hear more about it as I'm a network guy always in search of an excuse for latency :D.

by zamadatix

5/5/2026 at 4:13:24 PM

> If you meant there is a case where reducing the network latency […] is actually a negative then I'd love to hear more about it

That is exactly what parent did:

> they are doing an insane level of complexity to shave off ~100 ms

The downside is everything they had to do to achieve it, and maintaining that capability going forward, when the product can tolerate much more. It is the definition of premature optimization.

It just maybe isn’t at a level where it is relevant in your argument/decision space in IT.

by svnt

5/5/2026 at 3:58:23 AM

By you want to be able to interject “hold on…” and have it immediately stop talking, when it goes off the rails.

And GP is correctly pointing out that the only negative here (silence waiting latency maybe being too low) is tunable separately from the network latency number.

by theptip

5/5/2026 at 4:04:04 AM

I want to be able to click the "Stop" button on my earphones remote. I want to be able to interject "woah" or "stop!" or "wait!" or that it would detect that I've inhaled a breath, or that my eyes glazed over. I want the LLM to figure out that every speed setting for its voice output is in "auctioneer" territory rather than "lecturing university professor with tenure and a pension" pacing.

But we won't get any of that, because the prime directive of LLMs is to burn tokens like there's no tomorrow. Burn tokens on a naïve answer without asking clarifying questions. Burn tokens on writing, debugging, and running a Python script or accessing and parsing 10 websites without asking for consent. Burn tokens on half-baked images with misspellings and 31 fingers. Burn tokens arguing "how many 'r's in strawberry?". Burn tokens asking a followup question at the end of every single answer, begging the user to re-engage and burn more tokens.

There is a little red "Stop" control when text output is being produced, at least, but does "Stop" halt everything and throw away the context? Re-prompt from the beginning?

The "maximize tokens burnt" prime directive is not to be found in any system prompt or user documentation. It is seemingly a common feature of the training for any consumer model.

Currently, if I'm using voice for an LLM, I use the voice dictation in the keyboard feature, because then the response is in text. There is no way to prevent "responding in kind" if I query the thing with audio. Or in Swahili.

by ButlerianJihad

5/5/2026 at 9:59:13 AM

newer models tend to use fewer thinking tokens to solve the same problems, and is a strong counterexample to your entire comment

by spongebobstoes

5/5/2026 at 4:45:46 PM

you actually don’t want it to immediately stop because people say things like “hm” “yeh” during machine output. Maybe you say “no” to someone next to you and don’t want to interrupt output.

To confidently interrupt I would want to assert that the user has been speaking for > N time. You could do other things like parse a streaming transcription for keywords but generally it feels like bad UX to me to cut output the second input is detected. Letting the user talk for 1-2s gives a much stronger signal and it isn’t too weird for someone to keep talking for 1.5s after you start.

by ericmcer

5/4/2026 at 9:29:38 PM

I’ve also experienced this and it’s really annoying. There is this pressure to keep talking if I’m not done with my thought that feels pretty unnatural at least for me. If I’m searching for the right word, I want the opportunity to find it.

I think the solution is to handle pauses more intelligently rather than having a higher latency protocol. With low latency you can interrupt and the bot can immediately stop rambling.

by janalsncm

5/4/2026 at 11:18:18 PM

100%. I have to hold the floor by filling the space with "ummmmmmmm.... uhhhh...." which inevitably distracts me from my point altogether. Poor user experience.

by wnmurphy

5/5/2026 at 1:35:41 AM

Seems like there's a big risk of having that habit leak into human conversation. A lot of people try really hard to train themselves not to add those fillers.

by Gracana

5/5/2026 at 1:21:09 AM

I find this is a problem even with human conversations. Some people just aren’t very good at telegraphing when they’ve finished ‘their turn’ talking. Or worse yet, aren’t willing to take turns in the first place.

by taneq

5/5/2026 at 12:12:10 AM

Have you tried telling it to pause to let you think?

I often use it while I’m walking and tell it to not respond until I initiate a conversation.

by discordance

5/5/2026 at 12:40:19 AM

I’ve tried this and it says it will but just keeps cutting in. I hate this feature so much.

by pottertheotter

5/5/2026 at 4:27:44 AM

If anyone has an alternative I’m all ears.

This would be a killer feature for me and something I’ve tried to use on cross-country road trips.

by 650REDHAIR

5/5/2026 at 6:58:31 AM

I know it's not the perfect solution for you, but I use a voice recorder and send the LLM the transcript. And my god is it working great.

Usually I just explain the things I want it to do. The longest was 30 minutes rambling of explaining the methods section of a paper in non chronological order. It worked unbelievable good for me.

by notrealyme123

5/5/2026 at 12:12:23 PM

If you're setting this up yourself instead of using a lab's built-it speech functionality, you can run a small LLM in parallel, on a local model or small model like Haiku, that acts as a gate for either doing TTS on the response or not. Its only job is to decide if the transcription it receives is of someone being done talking or if that person is likely to still be mid-thought or mid-sentence.

by momotheritz

5/4/2026 at 9:25:26 PM

With higher latency this would be even more of an issue. When you pause and start talking again, the model wouldn't catch that until it has already interrupted you.

The actual implementation is at fault. I had some luck with instructing the model to only respond with "Mhm" until I've explicitly finished my thought and asked it a question. Makes this much less of an issue.

But I've decided that their voice mode is completely unusable for a different reason: the model feels incredibly dumb to interact with, keeps repeating and re-phrasing what I said, ends every single answer with a "hook" making the entire interaction idiotically robotic, completely ignores instructions when you ask it to stop that, and - most importantly - doesn't feel helpful for brainstorming. I was completely surprised how bad it is in practice; this should be their killer app but the model feels incredibly badly tuned.

by throwuxiytayq

5/4/2026 at 9:58:25 PM

This has more to do with Voice Activity Detection (VAD) than the latency described in the article

by dtran

5/4/2026 at 11:08:14 PM

That seems to be the issue: VAD is insufficient here.

Knowing when to respond requires semantic understanding, which probably only the model itself is capable enough.

Maybe it’s hard for them to train it to only respond once it seems appropriate to do so?

by lxgr

5/4/2026 at 11:35:24 PM

I am excited for VAD to go away. PersonaPlex totally seems like the future.

However things like 'Call center helpline' turn based actually seems better! You don't want to be interrupted when giving information back and forth (I think?)

by Sean-Der

5/4/2026 at 11:18:52 PM

Exactly. It's a tangent, but clearly a pain point for enough users.

by wnmurphy

5/5/2026 at 6:06:50 AM

There's a really interesting project in Japanese natural language processing called J-Moshi that had a novel approach and in my opinion good results.

They tried to make it mimic the way Japanese is full of really quick acknowledgement sounds and it seems to allow it to handle those pauses and interruptions really well.

https://en.nagoya-u.ac.jp/news/articles/say-hello-to-j-moshi... (english)

https://nu-dialogue.github.io/j-moshi/ (japanese and english)

I must admit it's a bit weird when LLMs laugh, I don't really know how I feel about that but it seems to laugh at the right times. Very tangential, but cockatoos have been known to mimic the right time to laugh presumably based on tonal cues that a joke was just made (I have experienced this first hand with rescue birds who li e amongst humans)

by ehnto

5/4/2026 at 9:24:59 PM

In voice conversations I tell it not to reply at all or only say “Understood” until I use some kind of code word. Not perfect, but less intrusive.

by saturdaysaint

5/4/2026 at 10:40:40 PM

Roger that, over.

by jdironman

5/5/2026 at 7:55:31 AM

Reducing the network latency helps with this exactly. OpenAI can make better timed decisions when to begin responding so it'll feel less like an interruption. I've also seen some research on full duplex voice models that handle interruption more like an organic conversation and low latency will help there as well

by futureshock

5/4/2026 at 9:05:29 PM

Hard problem. I find myself adding in filler to stop the thing from jabbering.

I also think it spends most of its iq on sounding good rather than thinking about the problem. “Yeah absolutely I can see why you’d like to…” etc. This is likely because it’s on a timer and maybe voice is more expensive to process? Text responses spend more time on the task.

by richardw

5/4/2026 at 11:10:26 PM

Their voice capable model is several generations behind the state of the art text-only one, as far as I know.

I don’t think it even has reasoning tokens, so it’s no surprise that it’s as most as smart as the “instant” models (i.e., not very).

by lxgr

5/4/2026 at 9:55:22 PM

Fwiw you can prompt it to respond differently to you.

by asdfman123

5/5/2026 at 1:10:05 AM

This is more of a VAD/turn detection issue. It's gotten a lot better over the last few years, but it's a hard problem. The extra ~100ms of latency makes a huge difference otherwise, especially when you have use cases that require tool calling that can easily add 500ms+ of latency.

by jameshush

5/5/2026 at 2:43:11 AM

It seems that tool calling shouldn't be 500ms of latency?

by angry_octet

5/5/2026 at 7:55:39 AM

If you have tool calling complex enough that it necessitates a higher reasoning level, and you would otherwise have reasoning set to "none", this can easily come out to 500ms.

by hobofan

5/5/2026 at 2:38:53 AM

People are migrating to the "End Of Thought" triggers. Deepgram does that wonderfully.

by miki_oomiri

5/5/2026 at 1:12:44 PM

Agreed. It’s stressful. I think they need to have an option to adopt a suffix, so they don’t start babbling until there is an “over” followed by a pause like in the old army walkie talkie days.

by christophilus

5/4/2026 at 11:17:10 PM

Strongly agree, some of us like to choose our words more carefully when interacting with an LLM.

I've tried to convey this to OpenAI through various available channels (dev forums, app feedback, etc.).

Grok solves this by having an optional push-to-talk mode, but this is not hands-free and thus more cumbersome than just having a user-configurable variable like seconds_delay_before_sending_voice_input.

by wnmurphy

5/4/2026 at 11:05:15 PM

yeh exactly, you cannot get a strong signal that a user is done speaking without some amount of “wait for 500ms of silence”. You could kick of processing and abandon if they continued talking, but that seems over optimized.

1-2s replies feel natural and like you pointed out pausing for 2-3s mid sentence is super normal.

by ericmcer

5/5/2026 at 1:08:42 AM

The AI should be able to model a probability for when is a natural moment to start talking.

by charcircuit

5/4/2026 at 9:13:11 PM

It’s possible to change the amount of time it waits if you’re using the API

by MagicMoonlight

5/4/2026 at 9:17:28 PM

[flagged]

by Barbing

5/5/2026 at 7:41:48 AM

[dead]

by NikolaosC

5/4/2026 at 9:46:50 PM

Wait a minute... I’m genuinely happy that they are sharing this, but keep in mind that realtime audio model from OpenAI are still stuck with the 4o family in terms of capabilities, sadly. I still find them so useful, such a pity that there’s no real competitor in this segment, having the experience a real conversation has helped me so much in expressing ideas and concepts.

Still, it’s worth to keep in mind that these are not frontier models, differently from when they were released.

(Please Sam, if you read this, release the new realtime audio models)

by Lucasoato

5/4/2026 at 10:23:42 PM

Yes the voice part of OpenAI realtime/voice mode is great but it’s pretty dumb compared to newer models and often gets stuck repeating itself.

Google’s Gemini flash live 3.1 is better, especially used via the API - it can do tool calling (including to other, even smarter LLMs if you set it up yourself), you can set the reasoning level (even high is still close enough to realtime) and it can ground answers in google search. I love bidirectional voice and right now it’s probably the best option. You can try it in AI studio

by dharma1

5/4/2026 at 10:26:06 PM

Thanks, I’ll try it, even if my experience wasn’t that great with Google models lately (503s)

by Lucasoato

5/4/2026 at 10:32:55 PM

Give it a shot, 3.1 live one in AI studio/API and max out reasoning - not the one in Gemini app it’s an older model.

Another option is to use pipecat with their VAD and separate STT and TTS and any (fast) LLM of your choice - but it’s more plumbing and not a true speech to speech model

by dharma1

5/4/2026 at 11:59:52 PM

Haha, wow, I never thought I'd see a voice model that was too quick, but 3.1 live felt like it responded unnaturally quickly! I'm kind of blown away, I'd want to insert a 100ms delay to make it sound more natural, wow. I never thought I'd see that.

by stavros

5/5/2026 at 9:49:15 AM

> Give it a shot, 3.1 live one in AI studio/API and max out reasoning - not the one in Gemini app it’s an older model.

Do you know why this is a thing? Despite the app technically being Gemini, I find it quite crap, while the AI Studio thing with thinking is my favorite LLM. Very jarring tbh.

by user_7832

5/4/2026 at 11:59:30 PM

Grok voice is surprisingly good, actually. It's still a dumber model than the thinking modes of frontier models, but it's less dumb than the voice modes of other providers.

by modeless

5/5/2026 at 1:59:57 AM

Grok voice model is also a thinking model. I agree that it’s far better than the other voice models

Just give me a option to have a slower response but better model…

by artdigital

5/4/2026 at 11:29:25 PM

This is what makes their voice mode unusable to me. I can’t stand the way 4o replies and it’s such a big jump in quality from text mode

by artdigital

5/5/2026 at 5:26:43 AM

Yeah I was quite surprised that the advanced chat gpt voice mode can’t itself go and message the frontier model underneath to retrieve data and then speak it. I basically tried asking it for that (something like “can you go and ask gpt5.5 to research this more in depth, and while we wait, tell me about XYZ”), but apparently that’s not a thing.

by radicality

5/5/2026 at 9:45:14 AM

Claude voice mode has come a long way! I'd say it's smarter than CGPT AVM last time i tried it.

But personally I've settled on just speaking to the slower models over a custom tts app, I find it being instant was not actually that important, and in the silence I find myself marinating in the discussion more anyway

by TomGarden

5/5/2026 at 6:43:34 AM

You can feel what is possible using Gemini speech to speech model, it can do tool calls and is very fast. It lacks somewhat in thinking capability but you can setup a tool call to a smarter model and it acts as a relay. I’ve been very impressed.

by sails

5/4/2026 at 11:08:33 PM

Yeah, the question in the title can be answered: "by using gpt-4o, a model 2 years behind the frontier, to serve audio responses"

by ddp26

5/4/2026 at 8:03:25 PM

> Voice AI only feels natural if conversation moves at the speed of speech […] At OpenAI’s scale, that translates into three concrete requirements: Global reach for more than 900 million weekly active users

Surely the number refers to the total users of ChatGPT overall, and the fraction of those who use voice features is considerably smaller, is it not?

That’s the kind of thing that influences business decisions like knowing how much hardware and software optimization to throw at a problem.

by thimabi

5/4/2026 at 8:21:03 PM

Yeah, that's why they've used "reach" - the total number of users who could be exposed to the feature regardless of engagement.

by stuartmemo

5/4/2026 at 9:32:35 PM

To defend them a little: voice is a little rough around the edges now, so there’s a chicken and egg problem of whether to prioritize improving voice if usage isn’t high partially because it’s clunky.

by janalsncm

5/5/2026 at 1:57:25 AM

id rather use the thinking models so the voice mode isnt' useful, i do use voice-to-text more and more just to speed things up though

by notfromhere

5/4/2026 at 8:08:25 PM

if anyone is looking to get into this. pipecat is a great open-source repo and community. https://github.com/pipecat-ai/pipecat

by Aeroi

5/4/2026 at 8:16:18 PM

I wish I had known about Pipecat a lot sooner. I found out about it a few weeks back, and since Gemma 4 launched, I've been building my own entirely local voice assistant using Gemma 4 + Kokoro TTS + Whisper from scratch - https://github.com/pncnmnp/strawberry.

Pipecat's smart turn model is really good for VAD - https://huggingface.co/pipecat-ai/smart-turn-v3

by pncnmnp

5/4/2026 at 9:54:07 PM

Yeah Gemma4 was and is great fun to do this with - I too am building pretty much the same as yourself in Go.

https://github.com/zarldev/zarl & https://www.zarl.dev/posts/hal-by-any-other-name

by zarldev

5/4/2026 at 11:06:33 PM

Looks like everyone is building one of these, I have my own little version that's using streaming STT, it can actually be too fast in some cases, and I have a little ring buffer grabbing audio from before the wake word detection fires (so it can hear "Hey Jarvis, turn on the lights" without deliberate pause) https://github.com/jaggederest/pronghorn/

by jaggederest

5/4/2026 at 8:27:36 PM

What do you have going on the hardware side? I want to plug this into hass but don’t know what hardware I need for reasonable latency

by AnthOlei

5/4/2026 at 9:25:02 PM

The whole setup works on my M2 MacBook Pro with 16 GB RAM. I use Gemma 4B via LiteRT-LM.

I've found that LiteRT-LM has a much lower DRAM footprint than Ollama. I've also made tons of optimizations in the code - for eg, you can do quite a bit with a 16k context window for a voice assistant while managing a good footprint, so I keep track of the token usage and then perform an auto-compaction after a while. I use sub-agents and only do deep-think calls with them, so the context window is separated out. In a multi-turn conversation, if Gemma 4 directly processes audio input, the KV cache fills up within a few turns, so I channel it all via Whisper.

Also, by far the biggest optimization is: 3-stage producer-consumer architecture. The LiteRT-LM streams tokens and I split them into sentences. A synthesizer thread then converts each sentence to audio via Kokoro TTS - the main thread then plays audio chunks sequentially. There's a parallel barge-in monitor thread. https://github.com/pncnmnp/strawberry/blob/main/main.py#L446

I did not want to use openWakeWord or Picovoice because they had limitations on which wake word you could choose. Alternative was to train a model of my own. So I created my own wake word detection pipeline using Whisper Tiny - works surprisingly well: https://github.com/pncnmnp/strawberry/blob/main/main.py#L143...

Also, I have VAD going with smart turn v3 (like I mentioned above) + I use browser/websocket for AEC + Barge-in (https://github.com/pncnmnp/strawberry/blob/main/audio_ws.py).

I'm using the MacBook's built-in microphones for this, though, and I haven't fully tested it with other microphones. I've been ironing out the rough edges on a daily basis. I should write a quick blog on this too.

by pncnmnp

5/4/2026 at 8:55:34 PM

Check out [0]. You can do 'Voice AI' on small/cheap hardware. It's the most fun you can have in the space ATM :) It's been a while, but posted a demo here [1]

[0] https://github.com/pipecat-ai/pipecat-esp32

[1] https://www.youtube.com/watch?v=6f0sUEUuruw

by Sean-Der

5/4/2026 at 9:16:30 PM

beautiful demo - is it running fully locally or talking to 3rd party API’s? That box was jaw dropping small

by AnthOlei

5/5/2026 at 1:12:58 AM

For the best experience, you'll still want it to communicate with 3rd party APIs to handle the speech to text, text to speech, and LLM.

by jameshush

5/4/2026 at 8:10:19 PM

I've been looking at this! Great project.

by BoxedEmpathy

5/4/2026 at 8:22:50 PM

what i learned from making a webrtc+kubernetes game streaming product:

- openai is wrong. almost of the issues they described are issues with libwebrtc, not with webrtc, kubernetes, network architecture, etc. the clue was when they said "the conventional one-port-per-session WebRTC model."

- there are no alternatives worth trying. everything else open source in the ecosystem, like pion, coturn, stunner, are too immature.

- libwebrtc is the only game in town.

- they haven't discovered libwebrtc feature flags or how it works with candidates, which directly fix a bunch of latency issues they are discovering. a correct feature flag can instantly reduce latency for free, compared to pay for twilio network traversal style solutions

- 99% of low latency voice END USERS will be in a network situation that can eliminate relays, transceivers, etc. it is totally first class on kubernetes. but you have to know something :)

this is the first time i'm experiencing gell mann amnesia with openai! look those guys are brilliant, but there is hardly anyone in the world who is doing this stuff correctly.

by doctorpangloss

5/4/2026 at 8:56:59 PM

Did you use libwebrtc on the backend? When you say `libwebrtc` is the only game in town are you talking about clients or servers?

Even for clients you have things like libpeer that libwebrtc can't hit.

by Sean-Der

5/4/2026 at 10:33:46 PM

yes - i used libwebrtc on the backend and, pre-LLM, patched it to work around a lot of the things i discovered that were directly related to low latency AV streaming. pion didn't exist then.

i think the challenge is that pion is an excellent product today. it would benefit me if its innovations were subsumed into libwebrtc, because eventually those innovations will show up in the iOS stack, which is one of the customers that matter to me. it is subjective if it is the MOST important customer, that is my belief and it is probably true of openai, at least until they get their own device out the door.

there can be many, many use cases though! not everything has to be, try to make the thing for 1b people that has to interact with all the most powerful and meanest businesses on the planet.

by doctorpangloss

5/4/2026 at 8:49:11 PM

Something I noticed is that companies that are vibe-coding their products miss out on the intelligence that (still) only humans can bring to bear. Just the knowledge cutoff alone puts AI at a serious disadvantage in any rapidly changing field.

by jiggawatts

5/4/2026 at 9:15:56 PM

GPT 5.5's knowledge cutoff is August 2025. Which aspect of WebRTC has meaningfully changed since then?

by fragmede

5/4/2026 at 11:13:45 PM

There's a difference between some piece of information being "officially published" and the AIs gaining a sufficient understanding of it.

Take any popular technology problem that has been around for a few years such as... wrangling Kubernetes with YAML config files. There's probably hundreds of thousands of discussions, source code samples from GitHub, official docs, blogs, bug reports, pull requests, etc... all discussing the nuances, pitfalls, pros/cons, etc. During pre-training the AIs internalise this and can utilise it later.

Now compare this with anything recent and (relatively) obscure, such as new .NET 10 features which were first officially publishing in November 2025, a month before GPT 5.5 cutoff.

As a human developer, these new language capabilities are on the same "level" for me in my day-to-day work as the features from .NET 9 or .NET 8. Similarly, my IDE has native refactoring and code cleanup support that can take C# code from the previous years and bring it up to the idiomatic style of $currentyear.

The AIs just can't do this, because one single Microsoft release note and one learn.microsoft.com page is nowhere near enough training data! The AI hasn't seen millions of lines of code written with .NET 10, taking advantage of .NET 10 improvements, and hasn't seen thousands of discussions about it. Not yet.

This is a fundamental issue with how LLMs are (currently) trained! Simply moving the cutoff date is not enough.

Human learning is second-order. If I see even the tiniest bit of updated information that invalidates a huge pile of older information, my memory marks everything old as outdated and from that second onwards I use only the new approach.

AI learning is first-order. It has to be given the discussions/blogs/posts that say "Stop using the legacy way, it's terrible! Start using the new hotness"! That, it can learn, but it'll be perpetually behind the rest of us by at least a few years.

Not to mention that thanks to AI forums like StackOverflow are dying, so... where is it going to get this kind of training data from in the future!?

AI training needs to switch to "second order", but AFAIK this is an unsolved problem at this time.

by jiggawatts

5/4/2026 at 10:52:05 PM

The problem is the sheer amount of knowledge out there. Particularly when using niche technologies (which webrtc and web audio still is, when measured by how many people develop using it), it is not surprising that AI doesn't have everything available in its responses, unless you specifically ask it about something you already know it should know.

by mschuster91

5/5/2026 at 12:18:54 AM

Dec 2025, actually: https://developers.openai.com/api/docs/models/gpt-5.5

(though knowledge cutoffs in practice can be bit fuzzy)

by tedsanders

5/5/2026 at 10:56:35 PM

That's what I get for trusting the output of the LLM instead of checking the docs!

by fragmede

5/4/2026 at 9:36:39 PM

When you have hard problems with unclear optimal solutions, taking this approach of a public show & tell will often (always?) solicit lots of interesting ideas the team may have not yet considered :)

by chevman

5/4/2026 at 8:46:47 PM

I wouldn't mind waiting longer for answers that would go through a better model with more thinking. As long as it has good support for interrupting and also it doesn't start answering as soon as I pause for 1 second and it's smart about knowing I'm done speaking.

by didibus

5/5/2026 at 5:29:18 AM

I find OpenAI's speech-to-text model the best of the lot. It can handle my & my 5-year old daughter's Indian accent pretty well.

I wonder if they run the STT model's output through the current model (that we're chatting with) as a final pass - since the text seem to be well aligned to the current conversation context.

For long prompts, I often speak to OAI web/app and copy-paste the text to Claude / Gemini :)

by amirathi

5/5/2026 at 10:09:31 AM

I don't like AI in general, and on youtube there are soooooo many horrible videos with voice AI. Having said that, I did notice AI has actually worked for some hobbyist-maintained games for the most part. Example: BG2EE (Baldur's Gate 2 Enhanced Edition). Yes, this is a forgotten game; and I actually have background music as audio rather than listen to the dialogue, save for testing it, but for the most part it worked here. So for poor-ressource hobbyists, AI is actually not totally useless. For youtube I find only horribly crap examples. I don't watch any AI-involved videos (if I can spot it; so much fake on youtube these days, Google does not realise how AI is killing many old users and visitors here).

by shevy-java

5/4/2026 at 8:59:12 PM

Am I reading this right that OpenAI is not using Livekit for WebRTC/audio anymore?

by qrush

5/12/2026 at 12:44:21 AM

they are not and haven’t been from what I hear since last Jan… I also have some friends work in listed companies on their website within real time divisions but haven’t used livekit, only signed up. It’s kinda shady tbh

by geekin23

5/4/2026 at 9:45:48 PM

It does appear that way. The LiveKit server is not what you would want for this architecture anyway (as they basically say with the SFU discussion), although it does have a lot of useful stuff in the client SDKs.

by fidotron

5/5/2026 at 12:34:04 AM

whats wrong with livekit ?

by zuzululu

5/4/2026 at 9:31:19 PM

IMO this probably isn't just about latency. keeping people in voice gives them training data text never will. is that why they were fine going transceiver over sfu and mostly ignoring multi-party?

by logickkk1

5/5/2026 at 7:13:33 AM

This is such a good write up, WebRTC is one of the coolest things ever! It's kinda genius to use the VIP approach, SFU is also pretty scalable but now they dont even have to do that

by vjay15

5/5/2026 at 5:44:07 AM

After all these, I still feel their voice AI interrupts quite a lot, especially when I pause just for 0.5 sec. Interestingly, when I tell it to interrupt less, it seems to be better.

by tracyhenry

5/5/2026 at 11:30:06 AM

>feels natural if conversation moves at the speed of speech

As someone use to podcast at 3x speech and sapi text to speech at much higher rate, listening to AI at human speech is a chore.

by maxglute

5/5/2026 at 12:16:21 PM

I never use the voice mode in the phone app, it's stuck with 4o for some reason. Same with Claude, that uses Haiku. Why can't they use a better model with thinking disabled?

by Saline9515

5/5/2026 at 10:05:19 AM

I have used voice mode on chatgpt, Gemini, Grok as I use it while driving. Best is from openAI. Natural conversation, smarter and meaningful replies.

by zerop

5/4/2026 at 8:48:43 PM

If a transceiver crashes during a stream, how is the active session recovered? Does the system automatically re-establish the context in a new WebRTC session?

by charisma123

5/4/2026 at 8:37:53 PM

> Global reach for more than 900 million weekly active users

lol, definitely didn't need to know there's 900M weekly users for this post. I mean yeah, there's a lot of users and they serve globally, that's relevant. But this is just pulling out your biggest stat because you can. How many voice users you have would actually be relevant and interesting but, to baselessly speculate on motivation here, might be a number that doesn't add as much fuel to an upcoming IPO as reminded people that you're almost at a billion users does.

by furyofantares

5/4/2026 at 11:08:28 PM

RFC 9297 support can't come quick enough in browsers. Would obviate having to deal with WebRTC in a client-server scenario.

by hnav

5/4/2026 at 8:00:52 PM

I hate the voice ai though, it's so much dumber

by anzerarkin

5/5/2026 at 3:57:45 AM

I used to use it all the time until about a year ago or so. Its responses are full of filler and the safeguards are really overbearing. It often will just give wrong answers in a way that GPT-5.x does not. I once asked it why a particular celebrity was canceled and it refused to tell me because it may harm me to know what they said!

by brett-jackson

5/4/2026 at 8:12:29 PM

Fwiw - I found the advanced AI voice feature to be actually detrimental. It's good if you just want a single sentence answer. I've turned it off though when I want a more detailed, structured, considered answer.

by NikolaNovak

5/4/2026 at 8:18:35 PM

Interestingly, that kind of parallels the real world too: if you want a quick and high level answer, talk to someone in person; if you want something detailed and info-dense, get them to write it down.

by drusepth

5/6/2026 at 6:38:16 PM

The AI attached to their voice chat is running a completely different model. Ask any question and you quickly realize it is completely, unapologetically lobotomized. If you want to talk to it about how you feel after your gf/bf broke up with you, it is fine. If you want to ask it something about tunneling machines and how tunneling through different types of rock impact engineering decisions, it is going to skim the first four sentences of some blog article and then defend whatever hill it has chosen until it dies, regardless of what the larger body of work on the topic says. OpenAI's voice chat being so bad and being totally divorced from their SOTA models is largely why I cancelled my subscription. I am tempted to wire up piper/whisper and the OAI api to get back what I actually want/need. But today you cannot have a conversation about engineering questions and get anything close to factually reliable answers out of it.

by hadlock

5/6/2026 at 2:30:50 AM

Turning advanced voice still leaves "regular" voice interaction which are actually (for me:) much much better - it's just the regular response, verbalized :). Voice quality isn't worse, it just doesn't try to summarize in one casual sentence.

(I still hate that the voice is getting more and more "natural" - the umms and ahhs and weird pauses)

by NikolaNovak

5/5/2026 at 5:52:03 AM

Interesting. What are the main latency bottlenecks in practice?

by hiroakiaizawa

5/4/2026 at 8:56:07 PM

It's bad enough having to speed-read the waffle of its written answers; even when told to be concise, the thought of having to listen to it waffle on in its smarmy, sycohpantic fashion makes me want to reach for the sick bag.

by CrzyLngPwd

5/5/2026 at 6:03:37 AM

why is the "How" included here? it is often removed

by whateveracct

5/4/2026 at 9:00:27 PM

OpenAI uses Go for the networking implementation for the relays and the services, which makes a ton of sense, instead of something immature as TypeScript / Node or whatever.

Yet another reason to not consider anything else like that for low-latency networking. Golang (or even Rust and C++) is unmatched for this use-case.

by rvz

5/4/2026 at 9:25:16 PM

"something immature as TypeScript / Node or whatever"

Node.js's initial release was May 27, 2009

Golang 's initial release was November 10, 2009

They're different, yes, but it's not like

by bananamogul

5/4/2026 at 9:30:01 PM

okay, sure, but one is by microsoft, the other by a 25 year old, and another by rob pike. The one by rob pike is going to be infinitely more mature and thought out than a hacky type system on JS because it isn't his first rodeo

by mghackerlady

5/4/2026 at 9:10:02 PM

Can golang do zero copy networking nowadays? In the past golang was terrible at this kind of thing due to allocations and copies of all relayed data.

by nvarsj

5/5/2026 at 3:07:46 PM

Even Java together with Netty supports zero copy networking... if Go misses that feature I wouldn't be very hard to implement it yourself

by troxhi

5/4/2026 at 9:16:37 PM

And the GC!

by fragmede

5/4/2026 at 9:37:21 PM

I think it's better to join some kind of club if you want to make friends?

by tom1IIIl1iIL

5/4/2026 at 8:00:22 PM

so is the answer

WebRTC + Kubernetes

by AIorNot

5/5/2026 at 1:52:17 AM

Inspired

by devopsengine

5/11/2026 at 5:18:39 PM

[dead]

by Daniel_Visovsky

5/6/2026 at 10:47:57 PM

[flagged]

by soleiman

5/6/2026 at 5:11:14 PM

[flagged]

by SandeepJawahar

5/5/2026 at 9:44:37 AM

[flagged]

by Ozzie-D

5/4/2026 at 9:14:22 PM

[dead]

by DumpoLumbo

5/4/2026 at 9:14:59 PM

[flagged]

by DumpoLumbo

5/5/2026 at 12:32:23 AM

[dead]

by mt_

5/5/2026 at 5:24:53 AM

[flagged]

by deferredgrant

5/5/2026 at 7:40:28 AM

[dead]

by NikolaosC

5/5/2026 at 1:04:53 AM

[dead]

by charcircuit

5/4/2026 at 8:14:31 PM

[flagged]

by Dorrell

5/5/2026 at 5:40:53 PM

[dead]

by ath3nd

5/4/2026 at 8:55:43 PM

[dead]

by testing_auth

5/4/2026 at 9:04:09 PM

[dead]

by testing_auth

5/4/2026 at 9:05:52 PM

Who cares? Their company is dying.

by jonahs197

5/4/2026 at 8:35:53 PM

Should I or shouldn't I be glad to see zero mention on Codex.

by flakiness

5/4/2026 at 8:45:59 PM

Shouldn’t, I think - advanced voice is a surprisingly slick feature, and if you’re someone who feels that they can think and speak more naturally than when they think and type, AI voice transcription is kind of huge.

by mock-possum

5/4/2026 at 9:01:06 PM

100% .. as a product designer/developer, i use it heavily for early feature ideation .. i’ll do a loose, exploratory back and forth on a long walk .. then pass the transcript to claude to validate and turn into a spec ..

by gyanchawdhary

5/4/2026 at 8:03:15 PM

It's missing the part where they explain how they obtained the training data for their voice AI.

by cdrnsf