I think you need to think outside the Web in those one. The web already has many ways to access UI elements, like the DOM, CSS selectors, etc.
This is really more useful for things like desktop GUIs where you have no other option than to OCR then inject mouse clicks.
It's not clear exactly how you would implement it, though. Maybe by recursively dividing the problem into rectangles, directed by the model? E.g. start with full image, train the model to locate the first element of the html, and output an attention mask for that element and the corresponding html tag and maybe style. Then recursively run the model with the attention mask as an input and with the inverted attention mask as the input (two runs), and have it extract the next element of each.
Not sure if that would work, but it seems like it might.