Ask HN: Vision Models to Parse UI
Hello. I was curious whether there are any existing vision models out there for either object classification of individual UI elements, or object detection of UI elements within their broader context.
(
11 comments
| 2 points
| on HN
)
1. https://arxiv.org/abs/2204.08387
It's not clear exactly how you would implement it, though. Maybe by recursively dividing the problem into rectangles, directed by the model? E.g. start with full image, train the model to locate the first element of the html, and output an attention mask for that element and the corresponding html tag and maybe style. Then recursively run the model with the attention mask as an input and with the inverted attention mask as the input (two runs), and have it extract the next element of each.
Not sure if that would work, but it seems like it might.
This is really more useful for things like desktop GUIs where you have no other option than to OCR then inject mouse clicks.
But the hierarchy piece I think is a bit tricker.
I'm really curious to see what the comments come up with.
https://ai.googleblog.com/2021/01/improving-mobile-app-acces...
It's called Oxen and is super fast at versioning image data.
https://github.com/Oxen-AI/oxen-release