imho they're just used to give google/amazon a few more data points for their graphs
I sometimes feel that the trend to prefer NLP over formal query languages is comparable to the trend to prefer GUIs over consoles in the 80ies and 90ies.
I'd rather have an assistant type system with a fairly well defined query system that exposed its capabilities and limitations directly, rather than me having to guess at the corner cases and failure points.
Disclaimer: I work @ Google on display assistant devices, but I don't work on the actual assistant interaction pieces.
People like to reference Star Trek for stuff like NLP queries, but if you go back to TNG and pay close attention to the verbal queries to the computer, much of the time it isn't natural language. They seem to actually use some sort of formal query language that fits English a bit closer, but is still distinct from when the characters speak to each other.
"Computer, begin auto-destruct sequence, authorization Picard 4-7 Alpha Tango."
- Wake word. Command. Authorization stanza. (I bet the computer would prompt for authorization if missing.)
"Computer, Commander Beverly Crusher. Confirm auto-destruct sequence, authorization Crusher 2-2 Beta Charlie."
- Wake word (possibly superfluous). Identification stanza (probably superfluous for the usual crew, but I can see from an HCI perspective that you might want to make people provide it specifically for such a consequential protocol, and it may also be of merit if some random admiral usually halfway across the galaxy pops in to confirm). Command confirmation, authorization stanza.
"This is Captain Jean-Luc Picard. Destruct sequence Alpha-One. Fifteen minutes, silent countdown. Enable."
- Computer is very awake at this point, no wake word. Identification stanza. Sequence parameter. Time parameter. Verbosity parameter. Commit.
In any case I think this kind of speech is formulaic for the benefit of the audience, most of all, who are made aware through the formality that the speaker is addressing a machine. Additionally, we're watching navy men and women in space, so we expect them to speak to each other and to their computers in a formulaic manner ("Deck 5! Report!" etc, I can't think of good examples, brain's too tired).
Or perhaps the idea is that Trek AI is not really as advanced as to be able to understand natural language and that makes Data such a unique specimen.
Then again, there's the example of the Doctor in Voyager. I'm confused, I admit.
In any case, Trek isn't super internally consistent, anyhow.
In those aspects of UX, voice interfaces have the same drawbacks as console apps when compared to a good GUI.
Also, they have to work within the "bandwidth bottleneck" of audio - just imagine a phone system that tells you all the options you have, "Press 1 for something, Press 2 for another thing..." - they are so annoying because they are slow and inherently linear; a GUI can show the same options all at once, and you can read it much faster than listen to them.
So NLP as such is not dramatically more user-friendly unless it is at the "do what I mean" level which likely requries full human-level general artificial intelligence; before that it's just a voice equivalent of a console app, sharing all the problems of discoverability and needing to remember what the system can do and how should it be invoked.
They're even slower now because the brain trust decided adding voice control to the phone menu system was a great idea. So before, it said "For prescription refills, press 1." Now I have to wait for "For prescription refills, press 1, or say prescription refills." How on earth does that improve anything? I can just as easily press 1 as I can say a word, and when I press 1, there is a near 100% chance that the computer on the other end will understand my command.
Some phone menu voice automation are even worse. "Tell me what you want! <silence>" Then you say something, and it says "I didn't recognize that. Please tell me what you want!" Then it fails again and says "I didn't recognize that. For prescription refills, press 1, or say prescription refills..." Oh great so there was a menu? Why did you waste my time earlier?
Voice is just a terrible, low-fidelity, low-bandwidth way of commanding a computer. You might as well have handwriting input while you're at it: You write what you want on a piece of paper, and hold it up to the camera and the computer tries to figure out what you wrote. Just as silly.
So, I would rarely say something when I could push a button, but when using a smartphone on a call, it's not always easy or obvious how to push a button. Some people may have mobility issues making it hard to push a button, or be on speaker phone far away from the buttons. Or, maybe they haven't updated their telephone equipment in 50 years, and only have a rotary dial. Or, maybe on a terrible VoIP system that can't manage to get the tones through.
There's probably some way to clean up the script.
"(Please listen carefully, as our options have changed.) Please choose from the following options: Say prescription refills or press 1; say insurance denied or press 2; say referral to veterinary care or press 3"
I could get behind voice interfaces for more things if the commands words were documented, clear, and consistent, and the damn things worked. Until then, buttons seem good to me.
"Welcome to FedEx. [... blah blah blah ...] Tell me what I can help you with today."
"a package"
I mean, what the hell else can you help me with today anyway?
Ironically I used to love me some graffiti on PalmOS and Google Handwriting Input on my Droid 2, but I agree with the spirit of your comment.
foo --<tab>
GUIs are for discovery, CLIs are for power via composability, voice/NLP assistants are for convenience.
Was it? Even if you discard stuff like tmux as already a GUI, you can still send whatever is running at the moment to the background with CTRL-Z and typing "bg" on any modern Unix system. "jobs" will then list all your processes, and "fg <ID>" will bring it to the foreground. I am sure this functionality predates most modern GUIs.
You may be thinking of DOS, which yes had almost no multitasking ability available.
However there were multiple timesharing operating systems that existed before the PC and GUIs, Unix being the most famous and still around.
Multitasking is quite possible on a Linux console for example. It has 5 or more consoles, each handling different users, each being able to be split via screen/tmux. Each shell can run jobs in the background as well.
1) I get the correct response; Assistant first asks me if I want to use "air check", I say yes, and get the correct response.
2) I get appropriate responses when I ask "What is the closest gas station to Mt. Shasta." Because the approaches are mostly from Rt. 5, you'll have no problem getting a usable response. Another approach from Rt 89 exists, so I don't know how you expect Google to know which "mountain pass" you mean. However, should you ask Google "What is the closest gas station to Mt Shasta on Rt 89", you will get an appropriate response. Should you be approaching from the opposite side of the mountain, unlikely though that is, Rt 5 or Rt 89 would still be the closest
3) I don't know why you expect a computer to answer this question well at all. I'm not aware of an API that would track this sort of thing. Google doing this itself, at scale, for any popular destination would be a very large project in ML with a very high error rate: Instagram photos are unlikely to be representative of a whole, the data set for a given location may be sparse, and especially for an outside location it is entirely likely (as I did at a local pumpkin farm) that people at an outside tourist destination will move away from a crowd, remove their mask, and take a nice photo safely, while people in close proximation still adhere to appropriate social distancing. Effective use would also need to be real-time: from day-to-day a given location could attract people that wear masks, and the next day a large group does not, especially in the > 300 square mile area you specified. It is not reasonable to expect this question to be meaningfully answered by either a human or a computer.
Yeah, I don't. I just get "Sorry. I could not understand that." if I remember correctly.
> 3) I don't know why you expect a computer to answer this question well at all. [...] Instagram photos are unlikely to be representative of a whole
Sure, but I defined the query pretty clearly, and I just want to answer the "general" question of "Are people generally wearing masks in that area or not" and I already babysat that question into a data query that could use the Instagram API plus some object detectors.
I understand it's hard for one person to write code that could formulate and piece together that graph, but I feel like it should be a tractable engineering problem for Google.
If you expect more, than your problem does not lie with voice assistants, it lies with search technology itself. "Expecting" these questions to be answerable is unrealistic given current capabilities. Working backwards from the UX would produce nothing better because your expectations are thwarted not by poor design, but by the limits of state of the art technology.
Actually, in the Boston area, it's "Route 128." Despite it never being indicated as such at exits or on maps.
You have omitted the rather important qualifier “Southern” from your description, as that usage is a key differentiator between Northern and Southern California.
I moved to Northern California (Santa Clara County) in the late 80s and I still use the same monikers. If I take a trip to Big Sur, I tell folks I am going to take the 17 to PCH and drive down. Old habits die hard.
https://www.kcet.org/shows/lost-la/the-5-the-101-the-405-why...
— https://www.kcbx.org/post/how-you-refer-us-101-says-lot-abou...
“Playing <super specific punk rock song>”
“Alexa stop. Alexa play punk rock playlist”
“Cannot find punk rock playlist”
“Alexa play punk rock 00’s playlist”
“Can’t find”
“Alexa play early 2000’s punk rock”
“Playing punk rock 00’s playlist”
It’s like she’s trying to mock me.
Me: "Alright man, I'll tell you where is money ... ALEXA CALL THE POLICE!"
Alexa: "Shuffling songs by The Police"
* EVERY BREATH YOU TAKE plays as I get punched 24 times *
from https://twitter.com/ppathole/status/1092034892249079813
Drug addiction makes people do crazy things, but there's a reason why most burglars flee as soon as they are noticed
My fav is when she decides to interpret your words instead of playing the exact playlist you’re telling her to play.
“OK Google, what's the speed limit?” → “The speed limit is defined as the maximum rate at which a vehicle can legally travel on a given stretch of road.”
And not only that, but a specialized user interface is often preferable to voice even when the assistant passes the Turing test. There's a reason that people use apps to get food delivery rather than calling.
Your alarm example is frustrating. Of course, you could say "set alarm for tomorrow at 6:30 am" which would work, but then you're back in the realm not of natural language commands, but formal commands that exist in the uncanny valley, just similar enough to natural language to be irritating when they fail.
Edit: also, your "tomorrow at 6:30am" example is also open to interpretation if you're saying it two minutes after midnight. I'd really like it to recognize these sort of ambiguities and prompt me for clarification. "It's after midnight. By 'tomorrow at 6:30am', do you mean you want your alarm to go off about 6 hours from now or the next day?"
It doesn't even support 24 hour time - at least in English.
They use all kinds of languages with all kinds of accents and and ticks.
In my experience, and from the time I did a bit of NLP the situations is often along the line of. It works for mostly accent free simple English. Fails to get anywhere usable on most other languages and or accents. Sure that is to some degree because of missing training data. But for a consumer this doesn't change that for very many consumers this features work terrible bad.
Just out interest I tried out the youtube auto generated subtitle for a german video, but even at the parts where the text was super clear and well pronounced the result was hardly differentiable from randomly picking arbitrary words. It wasn't even that the algorithm choose similar sounding word. They where completely different words in many case. I think in a sentence of ~10 words in average 1 or 2 where correct. And that was at the parts where the text was unusually clear understandable. At other parts it wasn't even able to recognize that there where words...
Just imagine all the idioms they have to account for in these voice algorithms. I know in German they also handle "halb neun" correctly as 8:30.
Although it does have a problem with hours over 20 (“22 o’clock” becomes 20:02 for some reason, possibly because of my thick Italian accent).
Again, whether I would understand them or not, no one to my knowledge speaks like that in U. S. English. It is a great example to use to show the quirks of language. It is a bad example to use to show that Siri "doesn't even support 24 hour time - at least in English".
And in any case i said eighteen not eight a m.
Well, if I had some type of personal assistant who worked for me (as in, a real human), I would just call out "Hey, Sam, please order pizza for me" and continue with whatever I was doing.
The reason people don't make phone calls is they add a lot of additional friction. You have to dial the number, and wait for someone to pick up, and give them your address, and deal with frequently-questionable voice quality.
However, if you frequently order the same thing from a given restaurant, having a way to order "the usual" from that place via voice might be convenient. But I'm not sure it's much more convenient than being able to do the same thing with a button click or something.
For delivery, I agree. It is easier and faster to use a service, since my address and payment information are already saved.
For me, the ideal interface to engage (order something, create a note etc) when you know what you want is through speech. It just feels so effortless.
You can't get that with voice no matter how you do it. It's like if a restaurant had no menu and the wait staff just recited the menu in your face and asked you to pick. Even with real humans, voice isn't the best interface for presenting food choices.
That's exactly how food is presented to you at lots of high-end restaurants. Once you get above the fast-casual tier, menus never have pictures. If you know a lot about food and you trust the chef, voice is a perfect medium for explaining the choices. Pictures are just a discovery aid.
I understand it's a hard problem, but with current software capabilities and given Google's compute infrastructure I honestly think some of these things are well within the realm of what a team of several hundred Google engineers can do.
I'm not asking it to write Shakespeare, I'm asking it to crunch data with a sentence that could reasonably easily be parsed into a graph and be turned into a MapReduce query. I thought they were good at that stuff.
Context? I know it's hard, but I thought Google has been working on that. I'm very much adjusting my expectations to what I think Google should be able to accomplish in a decade. I expect some basic context capability now, at the very least these data-crunching type use cases.
There are two ways to look at this problem: (1) what is hard and what is easy for our current tech to do, and (2) what are the things that humans actually want to pawn off to assistants?
The problem is that those are two very different answers. I don't think I agree with the grandparent comment that some of those things should be easy, but I do think that comment contains good examples of the level of sophistication that would make an AI assistant more than just a curiosity for the couple times each day you prefer not to hit the button yourself (or if you listen to the radio while you shower).
The rest is easy, it could even be done completely on-device if you have a recent high end chipset. That's how powerful phones are nowadays.
Tbf, it is easier to just run your own server, and pre-program everything for yourself, a truly personalized experience.
Though Google could easily do it for the millions of people using Android. I really wish they allowed custom modules or something for their Assistant, the voice recognition is unmatched.
“Here are the results I found for the last week...”
“Show the same for the last 10 days”
Maybe the difficult part is whether or not you look for the 15 most recent photos containing people, or the 15 most recent photos of anything.
If I ask for recent wildfire news and I'm in a state that doesn't experience wildfires often, are you going to return 15 news articles about wildfires spread out over the 200 year history of the state? I almost certainly want 15 news articles about the current wildfires in some other parts of the country. Your algorithm doesn't really say what to do here.
If I ask for "recent relatively rare astronomical event" recent might mean hundreds of years or more. If I ask for "recent PC game releases" it might mean a month or the current year. If I ask for "recent public events in my town" it might mean over the last week.
In many cases, "there are no recent events" is a better answer than "here are the last 15 events."
The second query was hard for me to test but it seems like a reasonable one to make.
The third one is probably impossible due to Instagram's TOS (among other things.) Their TOS states: "You must not crawl, scrape, or otherwise cache any content from Instagram including but not limited to user profiles and photos." [0] After Cambridge Analytica I'm not surprised that this is the case. Even if Instagram allowed scraping of their data, this feels like a somewhat specialized (though currently very relevant) request. I tried reformulating this request as "What is the mask compliance rate in Shasta county?" which also didn't turn up results, but I'm not surprised by that either. I suspect that data doesn't exist anywhere, so it's hard to fault the assistant for not pulling it up.
[0] https://www.instagram.com/about/legal/terms/before-january-1....
I don't know how much of FAANG's budget goes towards improving voice assistants, but considering how much cash these companies have on hand and their operating budgets (the size of some smaller European countries), the progress in that area is just super disappointing.
The 3 most common use cases were refined a long time ago (Directions, Alarms and Play Music) and everything else ran into a hard wall.
- "Take me on the most scenic route to X." Can't you figure that out from social media tags? Simple first order solution: routes that have more photos with more likes = more scenic. Took me 1 minute to think of that. And 1000 engineers at Google couldn't implement that? These data crunching tasks are the kind of stuff computers are supposed to be good at.
- "Navigate to X but make sure you stay on paved roads." An actual problem if you are trying to use Google Maps in the back roads of California and don't have a 4WD vehicle. Google Maps loves taking you on 4WD dirt road detours. Don't you have satellite maps and street view? Can't you differentiate paved roads from unpaved ones? What the hell does your machine learning department do?
- "Stop at the last grocery store before the highway 120 junction." Yeah, it doesn't even begin to understand this type of query.
- "OK Google zoom out the map slightly." Nope. Sorry.
Paved/unpaved: that's one suck-ass assistant you've got there. I know the Garmin on the dash of my motorcycle will give that option. The Garmin RV-specific GPS has loads of other options, such as avoid any low overpasses that will rip the solar panels off my RV. Though it doesn't look like Apple is any better than Google, only because Apple Maps doesn't offer the option. Anyway, satellite and street view? Rand McNally had this information since dirt was first created, no need to get the satellites out.
The real frustration is when it fails on tasks I swear it has done before easily.
Like 'ok google' ends up googling ... the ok google command word for word sometimes. 'ok google set timer 10 minutes'
As soon as I have to babysit an assistant like that I'd rather just make sure the clock is on my home screen so I set a timer.
Raise your hand if you remember the early days of Xbox "recognition", as you listen to other people in your party:
"Xbox, record. XBOX, RECORD. X...BOX, RE...CORD."
It's the reason that "oh, we can't sell an Xbox without a Kinect" Kinect went into the garage after a few months.
It's really too bad that MS went with a windows only sdk for the 2.0, the 1.0 being external tech had a multiplatform sdk.
It could even be done on the phone itself, although battery life and heat would likely be a problem.
The fact that it doesn't work just shows that Google isn't that interested in their Assistant.
Which is rather strange, but maybe they're looking for better ways to monetize it?
Or maybe it's going the way of Reader once AR gains momentum or something... then again Google Lens is a really nice product and no one I know even heard of it.
I'm amazed at how well it can recognize any kind of writing, even my chicken scratch, and how it can look up any products/labels with decent results.
- you doing a manual search, and being tracked/targeted - them displaying ads as a result - them displaying contextual, higher paying ads
I've noticed that some things over the years, many things in fact, have been removed from Google Maps. I hypothesize that the entire reason is, these things reduce profit.
For example, you used to be able to pause Google Maps. You can't now, and therefore, if you have 'history' off, you have to stop, and manually re-start your destination by typing it in.
Well, history profits them.
And pausing is not the best, because then, you're not active 100% of the trip. Having you active lets them determine all sorts of things, like traffic patterns, where you go, who you're near, and so on.
There are lots of little things like this, which seem to be gone from earlier versions of Maps. Again, I presume, all to make more $.
Which is logical, and fine, but it gets a bit tiresome and sad at times.
While an artificial general intelligence would be capable of fluent speech, a perfectly good speech system does not necessarily need to be an AGI.
To give a more concrete example, here's a UPenn demo (https://cogcomp.seas.upenn.edu/page/demo_view/ShallowParse) for your instagram query:
> NP What percentage PP of NP people can NP you VP detect to be wearing NP masks PP on NP recent Instagram photos VP tagged PP at NP a location PP within NP a 10 mile radius PP of NP Mt. Shasta ?
Part of the reason we don't progress beyond that is that speech recognition like in the OP article is quite bad: 95 percent accuracy is considered "good." But it means we expect 1-2 words of your query to be misrecognized, so even if it did parse the query as you proposed, it would probably be answering the wrong question!
But, on the other hand, I have dictated these paragraphs, (pretty much just adding punctuation and minor edits at the end). I think the most useful feature I've found related to voice is dictation (speech to text). It is almost perfect, at least in English (and I am not a native speaker).
The thing we're discovering is how to marry NLP (which works pretty well) with structured databases and automated tools (which work really really well).
A pure AI play probably won't get this done. But if your AI starts knowing how to use information tools then I think we'll see a lot of near term progress.
https://arxiv.org/abs/2010.05243v1
more
https://scholar.google.com/scholar?as_ylo=2020&q=text-to-sql...
That's it.
People want "[wake word], [action] [modifiers]".
Companies want "[trademark product] [trademark product] [modifiers]" (eg the parents example "Hey Google (RTM), Live Transcribe (RTM) 'words to transcribe'").
I've only used Alexa (and only for fun) but the replacement of verbs with companies/products, and of nouns with proper nouns, is really annoying to me and gets in the way IMO.
"What's the weather in Yosemite" and "Yosemite weather" can get you two drastically different results. As of right now (10/26/2020 10:35 AM) the two results I get are 43F and 32F.
I made an "and finally ..." to add funny news to the end of the daily briefing and was super impressed how easy it was.
We're humans - we can deal with ambiguity. Systems should trust that we'll respect them more if they tell us they're unsure, rather than either jumping to the wrong conclusion or simply being unwilling to guess!
I have an admin. I can text, call, talk, email her and say “let’s get bob, mary and finance on the phone at 4” and she’ll move stuff around to make sure that happens. I can also delegate complex administrative tasks that need to be done in my name.
When I’m at the dentist, I usually do a “hey Siri” to book it as it saves time. At home, “hey Siri turn on the lights” is helpful. It’s magic, but not the same.
The tech companies try to frame this stuff “bigger”, but in doing so they create an unreasonable expectation. Google home, Siri, Alexa are amazing, but we bitch and moan about them as a result.
I don't have a human assistant, but if I did, the biggest things I would expect from them isn't about scheduling a meeting or scheduling a barber appointment. I can handle that stuff myself.
What I would REALLY expect of a human assistant:
"Can you call up my health insurance and fight this stupid bill of $400 for COVID testing that should have been $0. Please escalate if necessary. Thanks."
"Can you figure out how to fight this red light ticket that I got due to a malfunctioning red light camera? Thanks."
"Can you fight this parking ticket? I had a valid permit to park there, and here's documentation of that. Thanks."
"Can you register me to vote? Here's my ID. Thanks. Make sure they don't sign me up for spam."
"Can you call up Comcast and fight this bill increase? Threaten to switch to another provider if necessary, I heard that works."
"Can you call up this company that posted my personal information and ask them to remove it? If they refuse, threaten legal action."
"Can you dispute this electric bill for me? My heating shouldn't have been $400 a month. Something must be wrong with my meter or someone is leeching power from my line."
"Can you call up the 10 different grocery stores in the area and figure out which one has X in stock?"
In all honesty I do wish the Google automated assistant could do all of the above. Sorry Pichai, I don't care for scheduling haircuts automatically. I want your assistant to use its hundreds of thousands of hours of human conversations and use machine learning to craft and engineer responses to humans to know EXACTLY when to escalate, EXACTLY when to ask for a manager, HOW to threaten legal action, and basically fight tooth-and-nail with language to get me what I want against the companies and institutions I need to fight with on the phone. The job of an "assistant" should be to get sh*t done and get me what I want. Use machine learning and lots of data to master the art of negotiation with customer service reps.
All the cases you've indicated as "so simple" are only so simple as template matching.
When generalized, they are hard problems.
Sure, you could have humans sketch out a few thousand templates enabling "Ok Google" to support such simple things like "What is the air quality at <location> <Time>" which looks up from an API. But that doesn't generalize "for free" outside of your templates.
Google seems to want to avoid handcrafted templates and go straight for the generalized solution.
Ok, so you need the assistant to:
* Already have a trained dataset of people wearing masks
* Fetch ALL instagram pictures it can find
* Not only detect if there's a mask in the picture, but count them
* Fetch the location of Mount Shasta
* Calculate a 10 mile radius around it
* Apply that calculation to the people counted in the first step
* Calculate a percentage of people wearing masks which needs:
* Count people not wearing masks.
Those steps are sub-optimal. So you need not only understand that, but to run some sort of 'query planner' in order to get the results you are looking for.
You are thinking of Jarvis from Iron Man. Forget assistants. Allow someone to construct a query like this in a couple of minutes and you have a great product you can sell. Existing ones require quite a bit of domain knowledge and setup. Not even things like Wolfram Alpha would be able to parse this query.
This reminds me of: https://xkcd.com/1425/
Sure, but we're talking about Google here, not Wolfram. The masters at query optimization and MapReduce. I would have expected them to be able to parse this query, or at least fetch the AQI near Mt. Shasta but can't even do that.
"There is no 'Grocery' list. Would you like to make one?"
An example of how Apple's ecosystem breaks down if you want to use something outside it. Saying no should offer to add a reminder as requested, but it doesn't.
It's the only thing I use my Alexa for.
Siri play song “song-name” I’m sorry there was an error with Apple Misic Siri play song “exact-same-song-name” [song plays]
I’ve figured out what queries almost never seem to fail and use those almost exclusively. I don’t get creative.
Last weekend I had the following failed queries:
"OK Google, what is the air quality like at Mt. Shasta today?"
"OK Google, add a waypoint for the last gas station before the mountain pass"
"OK Google, what percentage of people can you detect to be wearing masks on recent Instagram photos tagged at a location within a 10 mile radius of Mt. Shasta?"
These are all things I would expect a computer assistant to do really well. They have access to so much data, and so many APIs, that they should be able to break down these sentences into a SQL-like query and give me results. The third, for example:
"recent Instagram photos" -> Instagram has an API
"tagged within a 10 mile radius" -> parse the cities within a 10 mile radius and look for tags in all of them
"people" -> use your wonderful person detection networks your friends at Waymo developed
"wearing masks" -> I'm sure your internal datasets have this label, so run an object detector
Then compile and reduce the data to give me the number I want.
That's what I want an assistant to do. But it couldn't even do the first, which just involves a single API query to fetch air quality index information. Bleh. And as for the second, it has no idea what "last gas station" or "mountain pass" means; it's a query a human would know to be extremely commonplace.
It turns out that the current generation of "assistants" are mostly just template-matchers which really doesn't help me much at all. I can set my own alarms, thank you.