I am working on the same lines, creating an AI assistant which will pretty much do anything you want to do and you will do. I have been training the model on my UI actions on the computer and trying for my assistant to do the same using reinforcement learning. I haven't reached an appreciable working product yet it works great for me, it can screen cast, type passwords not only in browser but also on consoles. I am constantly adding features to it and I have had the same question as you for a long time.