You'd lose all window managers, things like alfred and textexpander, screenshot tools, computer use agents, etc.
I want apps to be able to do that!
Is there an opinionated reason not to break out capabilities?
If you have a disability and need tools to use your computer the last thing you want to do is have those things not only off by default but complicated and involved to turn on.
accessibility.screenshot accessibility.paste
and whatever else there is. that completely removes the issues for apps like this.
The issue is with Apple's UX. Apple insists on asking permission for every little capability an app wants. So I would have to say "yes, allow this app to take screenshots" and "yes, allow this app to read the clipboard".
I wouldn't be surprised if, in the near future, Apple forced people to click "yes, allow this app to read the clipboard from app X" and then separately "yes, allow this app to read the clipboard for app Y" and so on for every single other app on my machine.
Apple does not allow you to say, "yes, I trust this #$@-ing app, please allow it to do whatever it needs."
If you don't have use of your hands you want that. The whole point of accessibility APIs is allowing arbitrary control of your computer via novel means. One of the big selling points of Dragon Natually Speaking is the ability to tell your computer to do things based on descriptions without a mouse. "open outlook", "click compose", "select subject", "type foo", etc.
And no the solution here is not computer vision with an LLM. Text and buttons rendered on my computer exist in memory somewhere as text and buttons. We should not need to convert them to pixels and back lossily to recover text and buttons. We should just expose things to the accessibility API and not guess.
Are we sure about this? At least on windows, NVDA works fine with chrome and any electron apps.
Also, even if you hypothetically wanted to use computer vision with an LLM… what API is that LLM going to use to take screenshots and click on stuff?
However, I would like to point out that Apple isn't totally wrong here because the accessibility API unfortunately is way too broadly scoped, and because of that you literally get access to everything on the computer like you you can screenshot listen and and move the cursor... This is completely ridiculous and the proper engineering solution would actually be to phase out the accessibility API and replace it with something that is narrowly scoped so you can grant specific permissions individually.
However, Apple, being Apple, is obviously not doing anything, and instead says no accessibility permission for anything that isn't demonstrable accessible. Now, there are obviously some exceptions because Apple is not particularly well known for applying its rule consistently and granting big exceptions for itself. However, they do have a valid point on privacy and data protection. And I say that as somebody who ended up distributing my MacOS app outside the App Store because I only got approval for iOS.
That said, I would definitely appreciate if Apple would gradually improve its developer program experience, because compared to its hardware lineup, the developer program is nothing short of abysmal.