Show HN: openai-realtime-embedded-SDK Build AI assistants on microcontrollers

51 points · Sean-Der · 3 days ago

Hi HN! This is an SDK for ESP32s (microcontrollers) that runs against OpenAI's new WebRTC service [0] My hope is that people can easily add AI to lots of 'real' devices. Wearable devices, speakers around the house, toys etc... You don't have to write any code, just buy a device and set some env variables.

If you have any feedback/questions I would love to hear! I hope this kicks off a generation of new interesting devices. If you aren't familiar with WebRTC it can do some magical things. Check out WebRTC for the Curious[1] and would love to talk about all the cool things that does also.

[0] https://platform.openai.com/docs/guides/realtime-webrtc

[1] https://webrtcforthecurious.com

14 comments

kaycebasques · 19 hours ago

Took a bit of poking to figure out what the use case is. Doesn't seem to be mentioned in the README (usage section is empty) or the intro above. Looks like the main use case is speech-to-speech. Which makes sense since we're talking about embedded products, and text-to-speech (for example) wouldn't usually be relevant (because most embedded products don't have a keyboard interface). Congrats on the launch! Cool to see WebRTC applied to embedded space. Streaming speech-to-speech with WebRTC could make a lot of sense.

Show replies

jonathan-adly · 18 hours ago

Here is a nice use-case. Put this in a pharmacy - have people hit a button, and ask questions about over-the-counter medications.

Really - any physical place where people are easily overwhelmed, have something like that would be really nice.

With some work - you can probably even run RAG on the questions and answer esoteric things like where the food court in an airport or the ATM in a hotel.

Show replies

roland35 · 13 hours ago

Favorited and starred! I wonder if the real power of this could be in integrating large low cost sensor networks? I think with things like video and audio it might make more sense to bump up to a single board Linux board - but maybe the AI could help parse or create notifications based on sensor readings, and push back events to the real world (lights, solenoids, etc)

I think it would help to either have a freertos example, or if you want to go real crazy create a zephyr integration! It would be a lot of fun to work on AI and microcontroller combination - what a cool niche!

Show replies

johanam · 19 hours ago

Love this! Excited to give it a try.

Show replies

· 3 days ago

[deleted]