How about a button?

I'd prefer to physically press a button on an intercom box than having something churning away constantly processing sound.

If I have to go to a thing and push a button, I'd rather the button do the thing I wanted in the first place. Voice assistants are for when my hands are full or I don't want to get up. (I wrote more about my home automation philosophy in another comment[1]).

Also I have all my voice assistant devices mounted to the ceiling

1. https://news.ycombinator.com/item?id=47399909

What if you have two things? You'd then need two buttons.

The push button is a perfectly viable option, it just needs to be in a form factor that's works. Could be as simple as a tiny low-energy Bluetooth board with a coin battery that will last several months.

Luckily I carry around a device with infinite reconfigurable buttons!

Yeah, a touch-sensitive com-badge mounted on your chest would work.

Actually, I would think a small coin-sized button & transmitter that did nothing but emit a signal that your assistant (or phone) interprets as 'start listening' would be pretty useful. In your pocket, on a watch band, etc.

The pebble index seems like the optimal form for this.

https://repebble.com/index

Could be pressed even if your hands were busy.

If you want to relax some constraints, I made something similar for $10: https://www.stavros.io/posts/i-made-a-voice-note-taker/

Did you have any luck with the power issues on the new board?

The new board hasn't come yet, but a friend gave me a great idea, to power the mic from a GPIO, which powers it off completely when the ESP is off.

Hopefully the new boards will be here soon, but another issue is that I don't really have anything that can measure microamp consumption, so any testing takes days of waiting for the battery to run down :(

I do think these clones are the issue, though. They had a LED I couldn't turn off, so they'd literally shine forever. They don't seem engineered for low quiescent current, so fingers crossed with the new ones.

Makes a lot of sense :) thanks for the update.

I'll try to remember to creepy stalk you for updates as the device sounds great!

You can sign up to my mailing list to get emails if you want! It's at the end of each post.

Thank you for the blog/updates.

Since you're deep into this space, Do you use wall mount dashboards? There seems to be two camps now: Dashboards everywhere or none at all.

Is it worth removing the led from the board? Wont help with any other decisions by the designer that draw excess current, but maybe that's the only or largest one?

I did remove it :( It's still pretty bad. I ordered some Xiaos that do explicitly say 14 uA sleep current, but they seem to have gotten lost in the mail!

Most of what I (and in my experience many people) want a voice assistant for, is setting+ending timers... which for me happens mostly in the kitchen, while I'm simultaneously holding a hot pan or hand-tossing a salad or paper-towelling off some raw chicken. In none of those cases would I want a ring anywhere near my hands, let alone a smart ring. (And nor, in half of those cases, is it convenient/hygenic to use my oven timer.)

That being said, we could solve for fully 50% of in-home voice-assistant use-cases just by developing an extremely domain-specific voice assistant that has an extremely small (ideally burned-into-a-DSP) voice model that only knows how to recognize commands to manage kitchen timers. If such a device existed, and was cheap enough that you could assume anyone who wanted this functionality would just buy one, then this would make truly hands-free activation of a "real" voice-assistant much less necessary, as there'd be far fewer user-stories that would really "need" that. The rest of those user-stories really mostly could work with some kind of ring / belt buckle / shirt comm badge / etc.

I'm in if I can embed it into my forearm

In the mid 2000s I had a setup where some children's walkie talkie "spy watches" could be used to issue commands to a completely DIY, relay based smart home system.

I'm looking forward to whenever my Pebble ships so I can recreate that experience with this: https://github.com/skylord123/pebble-home-assistant-ws

apple watch gets you close.

Time for a real life Star Trek comm badge

We thought so too ;) https://www.linkedin.com/posts/y-combinator_button-computer-...

For real, if I had a fully self-hosted and private system, I'd love a real life Star Trek comm badge. Being able to say, "where is Jeff?" and have it tell me alone would be quite an awesome feature

Rules out a bunch of cases where your hands are busy handling ingredients in the kitchen, etc

Put it at foot level and kick it.

Like a light switch?

Or do you mean a button that activates chunked recording, passes it to a speech-to-text model, forwards to an LLM to infer intent, which triggers HA to issue a command, over a wireless network, to the computer with the light attached, to tell the light to turn on.

What's been surprising in my experience regarding the wake word is that it recognizes me (adult male) saying the wake word ~95% of the time. However, it only registers the rest of my family (women and children) ~30% of the time.

I have no firsthand knowledge, but I’d strongly bet that the home-assistant effort to donate training data is mostly get adult males, and nearly zero children.

This was 2021 (so pre-llm), but I used to work for a company that gathered data for training voice commands (Alexa, Toyota, Sonos, were some clients). Basically, we paid people to read digital assistant scripts at scale.

Your assumptions about training data do not match the demographics of data I collected. The majority of what our work revolved around was getting diversity into the training data. We specifically recruited kids, older folks, women, people with accented/dialected English and just about every variety of speech that we could get our hands on. The companies we worked with were insanely methodical about ensuring that different people were included.

You are reporting on a deliberately curated effort vs. what I understand is effectively voluntary data donation without incentives. It's not surprising to me that the later dataset ends up biased due to the differences in sourcing.

Your understanding of the datasets I helped create seems at odds with my experience actually creating the datasets. Do you have some insider experience or knowledge with dataset curation and creation for voice assistants that contradicts my own.

The guideline is that the newer your model, the more likely it is to have diverse voice recognition datasets since it solves the earlier problems caused by non representative data. The trend is moving towards better recognition for outliers. The training models are fed data that is very specific and not at all just whatever recordings they have collected in an S3 bucket. Given the amount of post recording work diarization, and QA we had to do on every single recording, I can’t imagine wanting to YOLO in bulk data.

You're missing the point. No one cares about the datasets you've created in a commercial context.

The effort being discussed is a volunteer effort among a community of tech enthusiasts, who are disproportionately privacy-oriented vs the average person. This will undoubtable skew towards middle-aged male audiences, and will be extra-selective against children. It's a best-effort collection, they're probably not turning anyone away, and it's only what they can get, they're (AFAIK) not paying anyone to collect underrepresented demographics.

Ah.

I thought you were talking about voice assistants in general. My mustake

Oh, I'm sure you're right. I've had people in my personal life (non-technical; "AI enthusiasts") laugh at me over concerns about training bias but this is likely a real world example of it.

I think you can train your own wake word with microWakeWord but I've never done it.

I remember when those systems first started collecting data they were worried kids wouldn't be handled - but they didn't know how to handle the privacy issuses with recording kids so discouraged it. Women being missed is not a surprise - but not anticipated.

I thought all people's voice had to be trained, and if you didn't go through it the match % was much smaller.

With Siri this is true. I'm not positive on the others.

I have a feeling beamforming microphone arrays might help here, something like this could improve the audio being processed substantially - https://www.minidsp.com/products/usb-audio-interface/uma-8-m....

That's a good call. I have a PS3(?) mic/camera that I was using when I was running the original Mycroft project on a Pi. I wonder if that would help with the inbuilt HA mic not waking for most of my family, most of the time. I will have to look at my VA Preview device and its specs later because I'm not sure if you can connect an external mic to it out-of-the-box.

Alexa devices have these (or used to at least), but Google Home's never did. So it shouldn't be necessary.

Yeah a small (ideally personalized) wakeword model would probably outperform just about any audio wizardry.

What about your wifi APs sensing which room you are in, with your choice of hilarious dance moves as the trigger ?

Funky chicken for Gemini

Penguin dance for OpenAI

Claude?

> Claude?

The Zoidberg Shuffle?

Why not use an easier to detect wake “word”, like two claps in quick succession? Or a couple of notes of a melody?

Can't clap if your hands are full and I would not subject my family to my attempts at delivering a melody.

I haven't tried training my own wake word though, I'm tempted to see if it improves things.

What about whistling?

Personally I'd pick "Cthulhu"

Ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn

On the plus side, mine misdetected a wake word during a funny conversation and said "Sorry, I can't find any area called _____[60 second repeat of funny conversation]___" and it made my family laugh harder than we've laughed in a really long time. I even went into the tts cache and saved the wav b/c it was sooo funny.

Ha, I had something similar happen as well that had us rolling. I think the hilarity was a result of the conversation snippet being taken completely out of context by the recording. Wish I'd saved the wav, I didn't even think of that :-(

Train a custom wakeword with this? https://github.com/TaterTotterson/microWakeWord-Trainer-Nvid...

or if you have an Apple Silicon Mac, this: https://github.com/TaterTotterson/microWakeWord-Trainer-Appl...

I used it personally, did a lot of research (including asking questions to the creator of microWakeWord), and submitted an upstream PR (I think it's already merged), which improved the resulting model slightly. I imagine the Nvidia version is similar, but I don't have experience with it. I also noticed that the model is so small (~25000 parameters), the actual training part doesn't even noticably improve with the GPU, only the TTS voice generation really only uses it.

if you are using this, I strongly recommend you create lots of personal samples with the recorder. I personally used 400, 200 from myself and 200 from my partner, with varying moods and in all the rooms we plan on using the assistant. I am considering re-training with more samples. it takes effort, but the resulting model seems to be well worth it.

Oh wow, 200 each? I just deployed it to the device, but I trained with 10. I'll need to redo it if it doesn't work well, but so far so good.

Wake word detection in low power DSP is a not-quite-COTS product but definitely exists. I believe PC manufacturers are looking at adding it to laptops soon, precisely to use with AI assistants.

How are you using Gemini in HA?

Why do you even need a wake word? Have a model look at full transcript and decide when to engage.