What I liked a lot was that the framework relied upon accessibility. To automate the testing of an app, you firstly had to make sure that the app was really accessible and then you could built upon accessibility attributes to add automatic testing.
I found it very smart because developers were incentivized to make their app accessible. Accessibility was, in a way, a tool to describe semantically a UI, so a screen reader, a test framework, the OS could understand how the UI was structured and acted upon.
One of my (really) old post about UIAutomation: http://blog.manbolo.com/2012/04/08/ios-automated-tests-with-...
Nothing about HTML is low-level.
HTML (and most of web tech) suffers from being both not low-level enough and not being high-level enough.
You can't override the browser and provide your own rendering pipeline if you wanted to actually do your UI [1] You can't tell it to batch rendering/update instructions for a subset of elements on the page. You can't compose elements. You can't use existing building blocks to combine them and build a proper new element [2] You look at a web-page funny, and it does a layout re-calculation [3]. And that layout recalc? It's abysmally slow. That's why you can't even animate anything useful. And so on and so forth.
[1] It's now possible wit Canvas and WebGL, but this is not web tech, not really. It's desktop tech that has been around for ages, and is now being bolted on to browsers.
[2] No, not really. Try and make a full dropdown/combo box that is properly customizable, keyboard-accessible, etc. Many have tried, all are abominations.
[3] Even things like border widths and some times even border colors wil cause a layout recalc: https://csstriggers.com
The examples you gave of places where browsers are high-level in terms of what pixels get put where? I see those as features, not bugs. I'm actually happy that web browsers historically haven't given pages a lot of control over low-level rendering, and I am distressed by the ongoing push by large corporations to increase how much of that control they have access to.
I agree with your point about not being able to compose higher-order elements, though.
Google said a couple years ago that they would penalize sites with dark patterns (like forcing users to sign-in to view public pages). Doesn't look like Google enforced anything since all the major sites are using the same dark patterns now.
1. https://www.fastcompany.com/3063143/google-calls-out-one-of-...
HTML's building blocks are too low-level for our contemporary needs. We should gradually move to higher-order representations of the data we're exchanging online, and put pressure on companies/governments to expose their data using these representations. My browser (or the extensions I add to it) should be able to natively determine whether it's looking at a product listing, or a person's bio, or a social media post, because those blocks of information would adhere to a standard schema ("hooks", to use the article's term).
Naturally, companies that are in the business of putting things in front of your eyeballs don't want us to move in this direction, because it gives users much more control over what they see than they have today. If you don't want to see ads, tell your browser to skip rendering the <Advertisement /> element. (And if a company places ads outside of an <Advertisement />... I dunno, fine them?)
(Those of you who've been around long enough are probably wincing and rubbing your RDF scars, or your XML scars, or what have you. Yes, it's the same old fight. We have to just keep pushing until we make some headway.)