A lot of the UI scripting on macOS is accomplished via accessibility APIs. If you’re interested, check out Accessibility Inspector[0], which ships with Xcode. You can inspect macOS applications in a similar, albeit much more limited way to a web browser’s interactive inspect element function.

You can also accomplish UI scripting via C/ObjC/Swift using the Application Services framework [1].

[0] https://developer.apple.com/library/archive/documentation/Ac... [1] https://developer.apple.com/documentation/applicationservice...

Exactly.

We should decouple the core of the application from the UI. Have a standardized interface which exposes all available functionality to the screen reader and allows it to talk to the core directly, bypassing the UI.

Stop doing what everybody does today: stop having screen reader depend on the UI (ie. aria-labels, etc). This problem can't be solved by adding "decorations" to existing UI components. It's best solved by exposing an entirely separate interface which lets screen readers (and other hardware) interact with the core of the application directly.

This sounds like a spec, a set of language frameworks and a design pattern waiting to be written.

If it's done right and standardized it'd be worth it for everybody since even people who don't need accessibility features could then create their own interfaces to the programs they use customized for their preferences and workflow

These APIs have existed for like three or more decades for desktop apps. That's partly why most desktop UI toolkits are based on tree of widgets: Mac and Windows's native accessibility APIs expect a tree

FWIW, the oldest accessibility API was Microsoft Active Accessibility (MSAA), which was added to Windows 95 in a service pack in 1997. Classic MacOS never had an accessibility API; the Outspoken screen reader for that OS had to rely exclusively on ugly hacks (e.g. patching Toolbox functions). OS X introduced accessibility APIs for Carbon and Cocoa in an early version; not sure which one. Unix-based desktop environments got an accessibility API, AT-SPI (thank you Sun), in the early 2000s.

well TIL, I believed accessibility to already be there by the late 80s, in NeXT workstations and the first windows versions but it looks like that was not the case.