This Github repo turns an ESP32-S3 into a realtime AI speech companion using the OpenAI Realtime API, Arduino WebSockets, Deno Edge Functions, and a full-stack web interface. You can talk to your own custom AI character, and it responds instantly.
I couldn't find a resource that helped set up a reliable, secure websocket (WSS) AI speech to speech service. While there are several useful Text-To-Speech (TTS) and Speech-To-Text (STT) repos out there, I believe none gets Speech-To-Speech right. OpenAI launched an embedded-repo late last year which sets up WebRTC with ESP-IDF. However, it's not beginner friendly and doesn't have a server side component for business logic.
This repo is an attempt at solving the above pains and creating a great speech to speech experience on Arduino with Secure Websockets using Edge Servers (with Deno/Supabase Edge Functions) for fast global connectivity and low latency.
----
If anyone is trying to build physical devices with Realtime API I would love to help. I work at OpenAI on Realtime API and worked on [0] (was upstreamed) and I really believe in this space. I want to see this all built with Open/Interoperable standards so we don't have vendor lock-in and developers can build the best thing possible :)
[0] https://github.com/openai/openai-realtime-embedded
[1] https://youtu.be/14leJ1fg4Pw?t=804
Offer is open for anyone. If you need help with WebRTC/Realtime API/Embedded I am here to help. I have an open meeting link on my website.
The OpenAI "Voice Mode" is closer, but when we can have near instantaneous and natural back and forth voice mode, that will be a big in terms of it feeling magical. Today, it is say something, awkwardly wait N seconds then listen to the reply and sometimes awkwardly interrupt it.
Even if the models were no smarter than they are today, if we could crack that "conversational" piece and performance piece, it would be a big difference in my opinion.
``` turn_detection: { type: "server_vad", threshold: 0.4, prefix_padding_ms: 400, silence_duration_ms: 1000, }, ```
What would be REALLY cool is if we had something that would interrupt you during conversation like talking with a real human.
I think it is closer, although still even it has a cold start problem. Once you are connected and in-session, it is a better experience.
There is still some "turn based" conversational aspect to it that can be awkward but it is much better. It also helps that you can "tap and hold" to override, which is a bit of a hack but works well in practice for that mobile use-case.
Both the supabase Api and OpenAI billing is per api call.
So the lovely talking toys can die if the company stops being profitable.
I would love to see a version with decent hardware that runs a local model, that could have a long lifespan and work offline.
This is a good point to me as a parent -- in a world where this becomes a precious toy, it would be a serious risk of emotional pain if the child experienced this scenario like the death of a pet or friend.
> version with decent hardware that runs a local model
I feel like something small and efficient enough to meet that (today) would be dumb as a post. Like Siri-level dumb.
Personally, I'd prefer a toy which was tethered to a home device. Without a cloud (and thus commercial) dependency, the toy wouldn't be 'smart' outside of Wi-fi range, but I'd design it so that it got 'sleepy' when away from Wi-fi, able to be "woken up" and, in that state, to respond to a few phrases with canned, Siri-like answers. Perhaps new content could be made up for it daily and downloaded to local storage while at home, so that it could still "tell me a story" offline etc.
We've already seen this exact scenario play out with "Moxie" a few months ago:
https://www.axios.com/2024/12/10/moxie-kids-robot-shuts-down
IMO this is only exacerbated by how little children (who are the presumably the target audience for stuffed animals that talk) often don't follow "normal" patterns of conversation or topics, so it feels like it'd be hard to accurately simulate/test ways in which unexpected & undesirable responses could come out.
Essentially, telling kids the truth before they're ready and without typical parental censorship? Or is there some other fear, like the AI will get compromised by a pedo and he'll talk your kid into who knows what? Or similar for "fill in state actor" using mind control on your kid (which, honestly, I feel like is normalized even for adults; eg. Fox News, etc., again US-centric)
https://youtu.be/0SfSx9ts46A
- why do you need nextjs frontend for what looks like a headless use case? - how much would be the OpenAI bill if there is 15 minutes of usage per day?
https://openai.com/index/introducing-the-realtime-api/
About the nextjs site, I was thinking maybe its difficult to have supabase hold long connections, or route the response? I'm curious too.
I noticed that it is dependent on openAIs realtime API, so it got me wondering what open alternatives there are as I would love a more realtime alexa-like device in my home that doesnt contact the cloud. I have only played with software, but the existing solutions have never felt realtime to me.
I could only find <https://github.com/fixie-ai/ultravox> that would seem to really work as realtime. It seems to be some model that wires up llama and whisper somehow, rather than treating them as separate steps which is common with other projects.
What other options are available for this kind of real-time behaviour?
The design of OpenAI + WebRTC was to lean on WebRTC as much as possible to make it easier for users.
[0] https://github.com/espressif/esp-webrtc-solution
[1] https://github.com/pipecat-ai/pipecat
I think Realtime API adoption would be higher if it is offered on Arduino rather than ESP-IDF as the latter is not very beginner friendly. That was one of the main reasons I built this repo using edge functions instead of a direct WebRTC connection.
[0] https://speaches.ai/ [1] https://huggingface.co/spaces/Xenova/kokoro-web
Pretty sure you'd need to host this on something more robust than an ESP32 though.
While I make no foolish claims that it's perfect, I've found Claude feels much less arrogant, and was genuinely appreciative when one of its replies started with an (accurate, of course I checked primary sources to verify that) analysis of the first half of my question, and then for the more obscure second half said "I'm not sure if I can answer that without hallucinating, but here's some stuff you could try researching."
Certainly Claude's tone and "attitude" (FSVO) works much better for me than any other LLM I've tried, though mileage will, of course, vary.
(I have zero connection to the company and am still on a free account, I'm just quietly impressed relative to the competition)
I believe there will be interest in extracting insights from speech-related fields, performing arts etc. Kind of how there was this transfer of design principles in the 90’s-00’s from traditional typographers, letterform revivals, print techniques.
It’ll be interesting to see an evolution of expectations and culture emerge around AI voices depending on role. Maybe we’ll see these positive voice vibes as silly and naive the same way we see MySpace aesthetics today?
Dead Comment
Deleted Comment