It's used because companies can cheap out on buying a license for other communication applications, it is fundamentally worse than anything else in any other metric. If voice lets me respond to a message without hunting for the hidden reply because Teams shoves it below the bottom of the screen then it could be a win. Considering UX is so low for Teams I doubt it will.
Natural language is a fundamentally wrong vehicle to convey information to a computer. It can be useful for some specific tasks, automated Q/A, simple interfaces to databases, stuff where I can't be properly f_ed to remember the syntax or the shortcut like IDE commands.
But the idea it can replace formal language is fundamentally and dangerously incorrect. I agree with Dijkstra's quip, we shouldn't regard formal language as a burden, but rather as a privilege.
What the hell? Is riding public transport or riding a bike either a burdain or a privilidge? Is Driving a car?
I am trying to control shit in my home, it should be neither.
1: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD06xx/E...
I don't see why a voice assistant for the masses couldn't "train it's own users", for example suggesting the language it does expect. But even then, most times people are talking in noisy environments or talk to fast or don't have an understand of how the machine might work. Regardless, who cares. They ruin the audio environment of a home. They're good for setting timers while you're cooking, that's about it.
So maybe it's just that the subfield of natural language understanding is still too early to be really useful. Speech recognition itself has gotten really good but then understanding the context, the intent, etc, all that is natural language understanding, and that is often the problem.
Citation needed, there's a lot of disagreements and misunderstandings (some have cost lives) that could've been avoided if we didn't have 10 different ways to say the same vague thing that can be interpreted in 20 ways. You think the military uses a phonetic alphabet and specifically structured communications for fun? Or the way planes talk to ATC for example. Where precision and unambiguity is crucial natural language always gets ditched for something more formal.
I only use voice assistants to set alarms. I cannot imagine voice as a primary input. Then again, many have opted out of owning desktops and laptops in favor of mobile phones. That also seems terribly inefficient.
A lot of people don't need computers in the general purpose sense. I admit my mind boggles a bit when co-workers tell me their kids don't want a computer to do their school papers because their phone is fine. But, then, I'm used to keyboards and what we think of an a "computer" and have been using one for decades--and grab one when I can for any remotely complex or input-heavy task.
https://www.theverge.com/2020/4/20/21227741/apple-ipad-pro-m...
Suggests a keyboard and large tablet is heavier than a laptop
I actually do wish there were good Mac or Chromebook choices for a travel 11" or so laptop but the market seems to have settled on a thin 13" as the floor and, admittedly, the weight/size difference isn't huge.
In response to a grandparent comment about weight for tablets: I prefer Apple’s folio old style of cases/keyboards because of weight. I have one for both my small and large iPad Pros. Whenever I travel, I usually just take one of my iPads if I don’t need a dev environment [1].
[1] but with GitHub Codespaces and Google Colab, development on an iPad is sort of OK.
Might as well go for the laptop at that point given that it can actually do far more imo, unless you ditch the phone and go for one of those half phone half tablets I guess.
For reading, I'm probably bringing my Kindle along if I don't bring my tablet.
Back to work? Sit on table, one cable and it's back to a desktop and charging up again.
Makes the whole thing make far more sense.
The killer-tech will be when we have a tablet that is as light as phone.
I grew up in the 1980s, when handwritten papers were still the norm. I do see the advantages of using a word-processor for writing papers, but don't see why it would be a necessity (at least, until University).
Around when Android appeared, and the first voice searches began, Google suddenly started to alias everything.
Search for 'Andy', 'Andrew' appears. Search for 'there', and 'they're' appears.
This has been taken further, now silly aliases such as debian .. ubuntu exist, and as google happily drops words in your search, to find a match, this makes precision impossible.
But, that's the only way to make voice search remotely work, so...
If you have terms you don't want interpreted broadly you can put them in quotes.
I preached the Gospel of Google when the competition was composed of web rings and Altavista, but Google in its infinite wisdom has abandoned the advanced user with changes of this nature.
https://blog.google/products/search/how-were-improving-searc...
Maybe we don't have clear intentions in the first place, maybe languages are not just ambiguous, but only meant to narrow realms of valid interpretations down to a desired precision, rather than intended to form a logically fully constrained statements. Maybe this is why intelligent entities are needed to "correctly" interpret natural language statements, because an act of interpretation itself is a decision making and an action.
Just my thoughts but I do think there are more to be said than "natural languages are ambiguous".
On the other hand, legalese exists and is the lingua franca of telling people what to do, and math exists.
And that's why all of aviation has moved to a tight phraseology, such that delegated commands are universally understood and their meaning is set in stone.
Natural language has cost many lives.
Using language to instruct humans goes wrong all the time. Just a short while ago on British Bakeoff I saw 2 of the contestants make white chocolate feathering on their biscuits by making actual feathers out of white chocolate and placing them on their biscuits. And I'm sure that will confuse quite a few people reading this too. It certainly confuses image searches. Language is a fuzzy interface. Compare to interface like clicking on a button that does the thing I want done.
Not always resulting in unambiguous instructions:
"Lord Raglan wishes the cavalry to advance rapidly to the front, follow the enemy, and try to prevent the enemy carrying away the guns." ~Lord Raglan, Balaclava
"I wish him to take Cemetery Hill if practicable." ~Robert E. Lee, Gettysburg
Every time we try to minimize errors, we formalize a language. I don't even think people use natural language to issue commands often. Commanding people is often considered rude.
I think this is really a characterization. Mostly human communication is full of errors and problems.
What is true is that when it is important enough, humans have come up with ways that minimize communication errors and frameworks to deal with ambiguity - mostly these involve training and effort though, it really doesn't come naturally.
- identify users by voice
- ask them clarifying questions
- remember the answers on a per-user basis
- understand "no, that was the wrong answer"
If you're going to provide a formal interface to the computer, you also have to provide teaching in that formal interface, which is far more of a burden to the user than the cost of the device. And we've completely moved away from that model (not necessarily a good thing, but that's what the market has chosen).
But I imagine there are significantly more who would appreciate clarifying requests by a teachable assistant capable of interacting with the entire digital world on their behalf, efficiently and intelligently.
I've said before, I would prefer a voice assistant that optimized for traversing its menu system, in response to unambiguous noises (could be high and low pitch hums or whatever) that lets me bypass the guessing game and use the menu it's hiding
That would be a fun universe.
[0] https://mw.lojban.org/index.php?title=Lojban&setlang=en-US
[1] https://en.wikipedia.org/wiki/Universal_Networking_Language
Hey Siri
Turn lights on 50 percent
For one hour
Dim over that time
Play music.
I can learn what I need to do; JUST LET ME KNOW THE MAGIC WORDS!
...I don't know how to do that
Alexa, turn lights on
...What do I turn the lights with?
Alexa, activate lights
...I don't know what you mean
...It is pitch black. You are likely to be eaten by a grue.
ALEXA TURN ON THE DAMN LIGHTS
...I don't know the word "lights"
...Oh no! You have walked into the slavering fangs of a grue!
** You have died **
Downstairs or upstairs bathroom?
Downstairs.
Sorry, I didn’t understand. Downstairs it upstairs bathroom?
Downstairs bathroom.
Sorry, I didn’t understand. Downstairs it upstairs bathroom?
Cancel.
Ok. Cancelling.
Siri turn on downstairs bathroom lights.
(Turns off all lights)
Because god help you if your device names are similar to your room names…
And I added scenes so I can say “Gondor calls for aid” and the beacons will light.
"hey siri?"
(no response, no icon),
"hey siri?"
(no response, no icon),
"hey siri?" (louder)
(no response, no icon),
"hey siri?" (louder and slower)
(no response, no icon),
reboot iphone 13 pro
"hey siri?"
works
I guess a list like language would be ideal and the pauses would be like parentheses
you great the prompt and add one or more actions to take.
Otherwise, it works great :-) We love the hands-off usage mode because we cook a lot, so adding things to shopping lists or looking stuff up doesn't require cleaning hands in the middle of prep. Also the speakers are pretty darn good for the size and work well for music.
Doing complicated things is right out though. But the simple stuff works fine.
It would be more error prone in a lot of ways, which is probably why nobody's done it yet, but it would also be a _lot_ more powerful, and fulfill the vision of conversational AI in a way the current rules-based assistants do not.
I think if powerful language models were easily accessible to normal people (in an inexpensive and completely unrestricted fashion, like with Stable Diffusion) we'd already see this happening in the open source world. Companies are going to be a lot more hesitant to try it though until they have a way to 100% prevent the models from making mistakes that could reflect poorly on the company, which is going to take _way_ longer to achieve.
In the spirit of the title of this post, someone else also has to say something.
If your argument is that this is a "non-visual command line" there's slim hope of the layperson learning a whole secret grammar without even a goddamn man page just to do their menial tasks.
Maybe it's possible to learn a working vocabulary and know how to command a voice assistant. I know my way around several command lines, but I have no idea what to say to Hey Google.
Also I know this is true in other domains as well, obviously there is a common "google-ese" that people learn to narrow down their searches.
This got me thinking. Voice recognition is basically a commodity now .. there are open source AI engines that can do it offline really well. So the recognition part is solved, you can just grab it from your distro's package manager. Now there's just the language part.
Thing is, I don't want to speak to my computer using English. Aside from the enormous practical problems in natural language processing you've outlined, I just find the idea creepy[1].
What I want is to unambiguously tell it to do arbitrary things. I.e. use it as an actual computer, not a toy that can do a few tricks. I.e. actually program it. In some kind of Turing complete shell language that is optimized for being spoken aloud. You would speak words into the open source voice recognizer, it writes those to stdout, then an interpreter reads from stdin and executes the instructions.
Is there any language like this? What should it look like?
And yeah that would take effort to learn to use it right, just like any other programming language; so be it. This would be a hobbyist thing.
[1] https://i.kym-cdn.com/photos/images/original/002/054/961/748...
If you're using an averaged American voice - maybe. But it's really not solved for everyone. Google assistant can't set the right timer for me 1/10 times. And that's before we get to heavy accent Scots and others.
There are quite a few hobbyists working on local on-prem privacy focused voice assistans with conversation support.
https://www.home-assistant.io/integrations/#voice https://www.home-assistant.io/integrations/conversation/
Have fun. It is a rabbit hole.
I personally don't consider this a fully-solved problem. The best transcription system I've used is OpenAI Whisper, and it doesn't work in realtime. Maybe it's fine on small amounts but it's still not perfect. You really need error to be driven down dramatically. Zoom auto-captions are a joke in terms of how badly they work for me, and Live Text (beta) on macOS is equally dreadful. YouTube auto-captions suck. All of these use industry-leading APIs. If I'm speaking a voice command and one single word is wrong, usually the whole thing fails.
There's an entirely separate issue about things that are Proper Nouns that don't exist. For example, "Todoist" is often misunderstood by Siri. Thus, people started saying "Two doist (where doist rhymes with joist)" to fool it into understanding "Todoist". Media like anime with strange titles from other languages often flat out trolls these transcription systems. ("Hey Siri, remind me to watch Kimetsu no Yaiba tomorrow".)
You knew that you were drawing something designed for a computer to recognise as unambiguously as possible, while being efficient to draw quickly and easy to learn for you. I feel like that's the kind of notion that voice interfaces should somehow expand upon.
This is potentially far from true, depending on how exactly you draw the line between "voice recognition" and "language". I've looked at quite a few transcription services, and they fail a lot of the time for most people - those who either have a non-native accent (even if very slight!) or those who do any amount of stammering or other vocal tics.
Manual transcription:
> So no: long story short, Slum is basically the way we can have an individual [, uhhh,] instance that carries all the licenses.
(Slum is a project name in this case)
Computer transcription (MS Teams):
> So no.
> A long story shorts. Love is basically the way we can have an individual.
> OHS instance that carries all the license.
But how? Even if those interfaces were actually working, it's still extremely inconvenient to talk when you can click. You have to be somewhere where talking out loud doesn't disturb the people around you. That excludes most situations: open space offices, restaurants, coffee shops, public transport, cars with passengers, and most places in the home except maybe the bathroom.
And even if you're all alone in a silent place, giving instructions out loud takes more time than configuring a screen, and will always be error prone, because the feedback will always be ambiguous and imprecise.
Except maybe if the feedback is on a screen, but then if there's already a screen, why not use it.
Then it would be called AirPods.
* Current time, and weather forecast for the day
* Upcoming meetings today
* My current commute time to work, including traffic
* NPR news podcast
So during my routine of letting the dogs out, starting the coffee, etc. in the morning, I get the daily "essential" info.
Working from home changes that. I can see many more opportunities for a multimodal input interface. Examples:
1. My fingertips now are closer to the "reply" button below this text area than they are even to the touchpad. Touching "reply" is half a second, moving one hand to the touchpad, aiming the pointer at the button and clicking takes longer. With a mouse: much longer. Anyway, my screen is not a touchscreen. I'll click.
2. Or, with an assistant, I could have said "Click reply", provided that the assistant knows where the focus is and that it can read the form I'm typing in.
I would separate out the two, actually. There's a "natural language control system for the entire OS" and then there's the actual voice part. Voice is often mostly useful for accessibility purposes -- hands full, running, driving, etc. However, the other side is that a text-based NL assistant would also be profoundly useful. On iOS, you can enable "Type-to-siri" and you can just type sentences and Siri will respond back in text.
If we make progress on NL-driven command-lines, we can actually make progress on voice-assistants, and vice versa. The catch is that the voice side still needs recognition work.
The would be trivial to use the interface on screen when appropriate, and a truly smart assistant should be able to follow the context and be aware of your preferences and mood.
This is not fundamentally impossible, we're simply not there yet.
Smart home light/etc while hands are occupied like with a baby. But usecases are quite limited
When they were a novelty I recall the excitement of trying new commands and layering in context, after many failures I've been conditioned to now only attempt and expect success with generic queries.
My biggest frustration with Alexa is getting it play the podcasts I want to listen to. Even popular podcasts with English names are hard to get just right for Alexa. The same goes for song titles and bands that are not popular, or they are in other languages.
Usually when I want to take a shower, I try to get the podcasts/music to play for 2 minutes, then sigh, give up and just say "Alexa play Britney Spears".
This is not power. This is just first-world problems.
Granted, this is for a specific user base and yes, not in coffee shops.
This kind of thing can't be built for modern mainstream operating systems because they generally prevent subjugation of the OS components and other programs, even if the user wants that, ostensibly for security reasons.
Unlike a human operator, an assistant "app" can only operate within the bounds of APIs defined by the OS vendors and third-party developers. Gone are the days of third-party software that extends the operating system in ways that the overlords couldn't (or wouldn't) dream of.
I think the problem with that is that even I, as a human, struggle to know for sure what you want.
You want to turn all the lights on in the house? Does that include the lamps in the bedroom? How about new lights that you add later? Or the ones in the garden? It's full of ambiguity. What device do you want to watch the news on? Or did you mean the radio? Do you want this to apply when you get back at 2am one night, meaning your family gets woken up when you turn on all the lights and start playing the news in their bedrooms?
I think that's probably why voice interfaces aren't likely to work well for anything beyond direct, specific, well-scoped requests: turn on the lights in the bedroom; turn off the heating at home; roll up the blinds; what's the weather like today; what's the remaining range on my car. They really struggle to deal with anything more complex – not so bad in theory, but really incredibly irritating when they make the wrong decision.
If you had some kind of 24-hour live-in assistant (a butler, maybe?), then they probably have the knowledge and intuition to make sensible decisions in response to fairly unstructured requests. But I think we're miles off getting a voice assistant to do it – not because they can't, necessarily, but because if they mess it up at all it's infuriating.
What about the shortcut for when I need to leave at 3am for some reason. then a different shortcut for when it isn't just me, but my whole family leaving at 3am. An still another for my son having to leave that early.
Jeeves can figure it out when I arrive at 2am so I don't need to program it.
Voice can do way more than we know, but we have no idea what it does or how to use it.
Standardizing the interface and providing tutorials would possibly change things dramatically.
And this goes for the back-end protocols as well.
The tech is way, way ahead of the UI and integration.
Imagine getting the power of 'git' with no tutorial and not really an understanding of what it does? Good luck with that.
90% of us would be using it in the car to do a lot of things if we really knew how to do it:
You: "Siri: Command. Open. Mail. Prompt. Recipients starting with S"
Siri: "Sarah, Sue, Sundar"
You: "Stop. Command. Message. To: Sunar. Thanks for the note. Stop. Send without Review"
Some of this already exists, but it's product specific etc. there needs to be some kind of natural universal interface - or we have to wait until the AI is really, really that good.
I don't really want them to be all that much more powerful, because natural language can be imprecise, and... there's just not much I that I want to automate in a home setting beyond some real simple timers for lights and stuff.
What if I had a bad day and didn't want to see depressing news? Or what if I came home and was talking on the phone when it turned the news on?
True automation as opposed to just telemetry and remote control can easily be annoying more than helpful.
I like the idea of automation... but I don't actually... automate anything aside from timers and reminders.
The problem is that you have many many billions of dollars have been sunk into making these devices about more than setting alarms and timers. There's actually been a lot of pretty amazing progress. But it's yet another one of those things that getting to 90% to anyone but techies who want to fiddle with their smarthome stuff or otherwise play with the technology.
Maybe they'll add an app that lets you browse possible commands so it's more discoverable.
But I'd observe that I'm going up to my brother's tomorrow and he has all manner of timers and other WiFi-connected stuff and none of it has any sort of centralized control and that's pretty normal even for people who have a lot of that sort of thing.
And, yeah, the only smart light thing I have at home is one thing that doesn't have a controlling light switch and I used X10 for it for years before I got an Alexa.
And even then, a voice assistant is essentially a user interface, not a product or service.
It could be a service if you could reliably say "Alexa, plan my trip to customer X the week of the 30th and send me my itinerary". But for now they are an alternative to a phone UI.
This seems a really high bar for voice assistants aspiring to do much more than set alarms or turn the odd light etc. on or off.
These days a large part of what people relied on secretaries for a computer can do faster, so only at the highest levels do you see them. There are still secretaries at the low levels, but not nearly as many, and they are not doing the same tasks.
And, yeah, assistants shared with a bunch of other people--as with travel agents in general--aren't really all that useful. If I'm mostly just giving fairly mechanical instructions to execute, it's probably easier for me to go online and figure out the options myself.
A secretary made a lot more sense when you dictated memos for inter-office mail and retrieving information often involved making multiple phone calls.
I know they know this about me not only because of my Gmail account but also because I use Google flights to find the flights before I book them.
Unfortunately they're not using this data to help me. Rather they're using it to target advertising to me. But they definitely have the data and the machinery to be more useful to me with more than just a few facts
It gets complicated in a hurry and for the cases where it is relatively simple (and when it gets into very complex international travel a voice interface is going to be completely useless), I can look up my options pretty quickly on a computer.
That's kind of my point. A voice assistant is just a fancy UI until they reach the level of AGI, and I don't see the point in spending billions of dollars on them to be a simple UI as Amazon seems to be doing.
Voice alone sucks, it's just too limited to be useful on a grand scale. Similarly, command lines suck too. The shell in general has the same problems that Voice assistants have, just that they have more value and had decades to mature into something actually useful. And toady we have unix-shells which reduce the problematic parts by many levels, and still receive constant improvements. This is missing for voice assistants, because unix-shells are growing and improving in an open space, where everyone can add their own things. This is not happening in big tech.
Edit: Maybe things like this happen because there are various nerds who lead these products and are good at talking the businesspeople into funding it. Maybe this was only possible at the big tech growth stage while business wasn't that good at telling the value proposition. So end result, lots more engineers get paid which is great in my book :-)
At the same time, siri seems to be getting slower and fatter every iteration so perhaps it is becoming more human ;)
“OK, I’ve created an infinite number of alarms, every five minutes, starting at 6 AM tomorrow!”
And you now have me wondering how open-ended calendar requests are actually implemented given that they can't literally have entries out to infinity. (I assume they go out some finite period and some background process periodically re-populates future entries.)
If these voice assistants were smarter about “alternative” names for every device it might be easier to use. But as it stands, it’s kind of a pain because the way you phrase each request is so unforgiving…
Oh yeah, and god help you if your device name is similar to your room name. If your room is “office” (or did I name it “the office”?) and your light is “office light” Alexa is gonna have a bad time figuring the two apart.
I have no clue how to fix this…
PS: this is why I question steering wheel free self driving cars. How will we tell these things exactly where to go when we cannot even reliably tell our voice assistants exactly what light to turn on?
I work at SoundHound where we've been worried about these issues. (I'm going to plug our recent work...) Our new approach is to do natural language understanding in real-time instead of at the utterance (turn) taking level. That way we can give the user constant feedback in real-time. In the case of a screen that means the user sees right away that they are understood, and if not, a better hint of what went wrong. For example a likely mistake is an ASR mistranscription for a word or two.
We still need to prove this is a better paradigm for VoiceAI in products that people can try for themselves, and are working towards that goal. I hope that voice interfaces that were clunky with turn-taking will finally be more naturally usable with real-time NLU.
A 'debug' area that lets you ask a command, see what was interpreted - and immediately edit or click "that's not what I wanted". But not an afterthought and not a cumbersome process like setting up an automation that is triggered by specific commands.
Imagine telling your voice assistant "You're wrong, as usual" and instead of it giving you the boiler plate "I'm sorry ", it actually offered a way to improve itself.
That sounds like a security nightmare. Someone walks by and starts changing your system settings? No thank you
In practical reality these interfaces feel, to me, as extremely inefficient. As someone who doesn't particularly like to speak, and prefers silent environments, these interfaces require more energy from me to use. Unless they are serving someone who has a physical impairment then I don't see what problems exist that these solve, but I can identify lots of problems that they introduce (not only noise but privacy / security vulnerabilities etc.)
Personal preference.
However Google's Assistant in comparison worked great, no memorization, and very useful. Sure time, weather, set timers, and alarms worked great with a very flexible set of natural language queries. Even more complex things like what will be the temperature tomorrow at 10pm, simple calculations and unit conversions. But also things like IMDB like queries about directors, actors, which movies someone was in, etc generally worked well. It seemed to really understand things, not just "A web search returned ...". Even more complex things like the wheelbase of a 2004 WRX would return an answer, not a search result.
With all that said I'm looking for a non-cloud/on site solution, even if it requires more work, most recently noticed https://github.com/rhasspy/rhasspy
There's a great little game series called Megaman Battle Network (Rockman.exe in Japan) which diverges from the mainline by showing an alternate universe where scientists focused on AI instead of robotics, resulting in a world where "Navis" are ubiquitous.
I wonder, what if our early software engineers focused on bringing natural voice control to CLIs, before perfecting GUIs first?
Instead, it's a guessing game about syntax and semantics, and frequently a source of frustration. There are many failure points: it can "hear" you wrong, it can miss the wake word, it can hear correctly but interpret wrong, miss context clues, or simply be unable to process whatever the request is. In my experience, most normal people either relegate voice commands to ultra-specific tasks, like timers, weather, and music, and that's that. Google and Alexa are relatively good at "trivia" questions, but Siri is a complete failure. All systems have edge cases that make them brittle.
I think there's potential here. Cortana was the most promising: an assistant that's integrated into the OS and can change any setting or perform anything on-screen would, again, be really awesome. We just don't have that. I think maybe OS-wide + GPT 4 (or later) might get closer to what we expect, but it's just not great right now. I really want to be able to say something as unstructured as "hey siri, create alarms every 5 minutes starting at 6am tomorrow" or "hey siri, when I get home every day, turn on all of the lights, change my focus to personal, and turn on the news". There /is/ power to-be-had, but nobody has really tapped it.