I actually built nearly the same tool under the same name: https://agent-desktop.dev And I've seen a couple other similar projects since then too! Seems like a lot of us are thinking in the same direction.

One wrinkle I found is that there wasn't a cross-platform library for accessibility APIs, and each platform is a bit different. I made an a11y library that supports Mac, Windows, and X11 and Wayland on Linux with consistent interface: https://xa11y.dev