> stop music and turn off all the lights
This is probably the easiest of the bunch because you are asking it to perform two distinct actions.
> turn off the bedroom light in 5 minutes
This is much more complex, because you are asking the application to setup some sort of workflow - after it understands what you want it to do, it then has to work out how to execute that, which will be utilising the device API's / services. This is a simple example, but there are lots of permutations of different actions here, for example you might want to say "turn off the sound system once this song finishes playing" which assumes that the assistant has the capability to then understand you want it to create a task specifically waiting for the trigger of a particular song finishing playing, and that it has the ability to setup that trigger.
> "keep the lights on” in a motion sensor system
Now this is where the orchestration gets tricky -
The assistant has to:
* Work out that the lights are being affected by a motion sensor system, which is likely outside it's own platform.
* Work out that your intent is that you want the assistant to override that.
* Understand how to connect to the platform in order to control it.
* Work out what parameter it is supposed to alter to achieve this task.
* Override the existing users settings, and presumably reinstate the settings after some portion of time.
I don't see conceptually the difference between the first and the second example. You're still executing two distinct actions, first being the waiting for x amount of time?
It could be issuing a command in the same way you do the first, but probably most lights just support switching them on/off right now, and won't take a delay argument.
So, if you can't just do the same you were doing, how do you do it? If there's some kind of local support, you can issue a delay command to some device in the network that will switch the lights on, but you still need to be able to control that action as the user might want to cancel or adjust the delay too.
If the execution need to happen remotely because there's no local support, then you don't even know that you'll be able to reach the device in 5 minutes (Internet down, router not letting random connections in). And keeping this request queued needs some infra in the server side too together with the necessary APIs to allow adjusting and cancelling requests.
The AI feeds the action queues, and another isolated component performs the actions, including delays and follow ups.
I'm pretty sure there is some form of local support, at least for Google. Whne I last used it you had to choose a speaker that automations executed from
Where is the hard part?
There is storage going on already
It's not a fixed amount of time. What if you rewind in the song a bit to catch something you missed in the lyrics?
I think this highlights that AI assistants are not really "I", they are good at doing fuzzy, hard to specify things like understanding speech, but there must still be an engine behind that interprets the text and transforms it into execution steps, and that must still be done by developers so it will be limited.
Is it just me or does it seem like these tasks would be not all that hard if you just, you know, programmed them rather than trying to be so fancy with ML?
That's how you do it at the moment. For example, in Google home you have "Routines"[1] which are a list of sequential actions triggered by a key phrase (or optionally something else).
i.e. you can program the use case "turn off the lights in 5 minutes" but that doesn't cover the use cases of "turn off the lights when i leave the house", "turn off the lights and turn on the tv" or "turn on the lights at 10pm tonight" - and there are so many potential scenarios here that it can quickly scale up.
It's really a shame the good language models are not deployed as voice assistants. It would probably be expensive to offer and they don't have the scale necessary. Just to load one of these models you need a $100K computer.
>99% of commands will be single intent, and they probably work 80% of the time at the moment, so getting those to 99% will have a much bigger short-term impact than focussing on solving the 1% (with the added benfit that once you have solved the first case of getting single-intent right all the time, solving the second more complex queries will be easier as you will have built a more robust base).
I can already do something similar with Siri. "Remind me to take out the trash in 5 minutes". Seems odd that "turn off the bedroom light" isn't trivial to support.
The bedroom light may not have such a structured concept already existing (or may not expose the full-featured API to Siri).
There is no "general ai" doing stuff in the background.
The reminder service needs "one line of code (tm)" to go "After five minutes, send <this> to the lights service".
- When I arrive at home, turn on the lights - After five minutes, play this song - Lock the front door at 10pm
And then the reminders service (which is a to-do app!) suddenly becomes 100x more complicated than it was to accoomodate all these strange use-cases that are outside of it's bounded context.
It's easy to implement a single use case, it's practically impossible to manually code every possible use case (which would also probably create a huge amount of tech debt).
I often get ready to leave the house and say "start the robovac in five minutes" and it just works. The Google home gadget confirms and states that it will start robovac "at $time-of-day". I've not got too many things connected but I assumed that it's generic among those things that can be turned on/off.
I'd go for one of those.
It's almost if that natural language thing is highly ambiguous which is why we needed to create a new grammar to give precise instructions for machine-driven processes.
Thread.sleep(5 * 60 * 1000); light.off();
Avoids reciting the steps you want and gives you reliability, but I suppose requires any 'variables' to be hardcoded (or for you to create multiple instances)
* Not recognize my voice/command
* Load infinitely when I give a command
* Do the wrong thing entirely
Me: already down the hall cursing my smart speaker.
This actually works already with Siri (and as mentioned in a sibling comment with google Home as well). I just tried that for fun a few days ago and was surprised that it actually worked.
“Turn off the lights in 5 minutes did but “turn off the floorstanding lamp in 5 minutes” did not
Honestly that’s more frustrating when it’s not uniform and now I’ve to remember this weird behavior.
In the Phillips* Hue app at least, they have the idea of lights independently, but you can also group them into rooms and stuff like that. So the multiple bulbs in my floor lamp are in a "lamp" grouping but also a "living room" grouping. It all seems quite flexible.
What about something like "all lights" "living room lights" etc etc?
* Or rather, whoever bought that brand from Phillips
This works for me, except when it mishears what I'm saying as "turn off the light for 5 minutes", which gets a bit annoying
Yes, maybe, gpu cluster server can understand you quickly, but taking whatever model you have and getting it to work quickly enough for people not to be upset is a giant problem.
You had one job, Amazon!
Google assistant works fine, for the most part anyways.
At what point can we say things like “turn off the bedroom light in 5 minutes” or “stop music and turn off all the lights”? Even something like “keep the lights on” in a motion sensor system is impossible it seems. Because to me they feel like low hanging fruit, and yet despite all the advances in machine learning and these systems being around for the better part of a decade, anything but the simplest single-task no-modifier commands result in a “sorry… I didn’t understand that” or completely unpredictable results. Is there something inherently difficult about these types of queries?