I would assume that any RPA tool without a large multimodal (language) agent framework running it is basically already obsolete.

I hope we get more open source options like Llava or CogVLM.

The paper makes it seem like there may be some way to control the mouse and read screen captures, but doesn't give any details. There is a GitHub link though so maybe it's in there.