_crowecawcaw

Show HN: Xa11y – cross-platform desktop automation via accessibility trees

Posted by _crowecawcaw on Monday June 08, 2026 @11:18AM
from the show-hn dept.

I built xa11y to simplify cross-platform desktop automation.Originally I was interested in using computer use agents to automate testing of desktop applications. The tools I tried gave agents the ability to click a mouse and fed them screenshots, but the agents struggled to use the mouse accurately. A couple projects like OmniParser use specialized screenshot reading models and element labeling...

( Read More... | no comments | on HN )

Re:Launch HN: Minicor (YC P26) – Windows desktop automations at scale on Wednesday May 27, 2026 @12:54AM (#48289826)

I also experimented with vision/screenshot based computer use tools for similar use cases but had inconsistent results. LLMs had trouble getting precise pixel coordinates from a screenshot to move a mouse. And the screenshots took extra tokens. I had a lot more success using accessibility APIs to replace screenshots + input simulation since accessibility data is easier for LLMs to process. The accessibility functionality is now released as a separate library for building automation tooling: https://xa11y.dev/

Re:I built the Playwright for desktop apps. 80% token savings on Saturday May 02, 2026 @12:49PM (#47988024)

I actually built nearly the same tool under the same name: https://agent-desktop.dev And I've seen a couple other similar projects since then too! Seems like a lot of us are thinking in the same direction.

One wrinkle I found is that there wasn't a cross-platform library for accessibility APIs, and each platform is a bit different. I made an a11y library that supports Mac, Windows, and X11 and Wayland on Linux with consistent interface: https://xa11y.dev