News for hackers. Stuff that matters.

Ask HN: Vision Models to Parse UI

Hello. I was curious whether there are any existing vision models out there for either object classification of individual UI elements, or object detection of UI elements within their broader context.
( 11 comments | 2 points | on HN )
Re:Ask HN: Vision Models to Parse UI by sharemywin on Tuesday January 24, 2023 @09:59AM (#34504356)
I'm interest too if any one knows of anything.
Re:Ask HN: Vision Models to Parse UI by carom on Tuesday January 24, 2023 @11:14AM (#34505521)
LayoutLM [1] is the closest that I have seen you what you are asking. It is applied to documents but essentially takes positional and visual information into account for text extraction. For example, extracting a total from the line that reads TOTAL - I think this would be the best place to start.

1. https://arxiv.org/abs/2204.08387

Re:Ask HN: Vision Models to Parse UI by smaddox on Tuesday January 24, 2023 @11:54AM (#34506270)
Not that I'm aware of, but that's a very interesting idea. If you had one that maps from ui to html with inline styling, you could automate turning image mock-ups into html with inline styling.

It's not clear exactly how you would implement it, though. Maybe by recursively dividing the problem into rectangles, directed by the model? E.g. start with full image, train the model to locate the first element of the html, and output an attention mask for that element and the corresponding html tag and maybe style. Then recursively run the model with the attention mask as an input and with the inverted attention mask as the input (two runs), and have it extract the next element of each.

Not sure if that would work, but it seems like it might.

Re:Ask HN: Vision Models to Parse UI by orev on Tuesday January 24, 2023 @12:27PM (#34506830)
I think you need to think outside the Web in those one. The web already has many ways to access UI elements, like the DOM, CSS selectors, etc.

This is really more useful for things like desktop GUIs where you have no other option than to OCR then inject mouse clicks.

Re:Ask HN: Vision Models to Parse UI by wruza on Wednesday January 25, 2023 @05:57PM (#34525338)
Is there an OCR that reports character coordinates? Couldn’t find a tesseract-ocr option for that, but maybe it’s in a block-detector-something source code?
Re:Ask HN: Vision Models to Parse UI by throwaway2016a on Tuesday January 24, 2023 @12:08PM (#34506531)
I too would like to see this! I'm pretty sure it can be accomplished by using screenshots as training data for an object recognition algorithm. As with a lot of machine learning, gathering and tagging the data would be tricky and overfitting to specific design systems could be a big problem.

But the hierarchy piece I think is a bit tricker.

I'm really curious to see what the comments come up with.

Re:Ask HN: Vision Models to Parse UI by felixr on Tuesday January 24, 2023 @12:33PM (#34506939)
* * *
Re:Ask HN: Vision Models to Parse UI by darepublic on Tuesday January 24, 2023 @12:57PM (#34507304)
Thanks will take a look at these
Re:Ask HN: Vision Models to Parse UI by lunixbochs on Tuesday January 24, 2023 @12:33PM (#34506947)
Android voice access blogged about how they use a model to detect and classify buttons:

https://ai.googleblog.com/2021/01/improving-mobile-app-acces...

Re:Ask HN: Vision Models to Parse UI by jetnew on Tuesday January 24, 2023 @10:51PM (#34514204)
I'm currently working on this! Currently using traditional computer vision methods (e.g. canny edge detection) which already works quite well for most websites or applications, but am working towards curating a dataset for deep learning. I'm keen to chat!
Re:Ask HN: Vision Models to Parse UI by gschoeni on Tuesday January 24, 2023 @11:56PM (#34514571)
Let me know if you want a good place to host and iterate on the dataset, I'm working on a version control system optimized for deep learning datasets. Free to host public datasets right now, just looking for feedback.

It's called Oxen and is super fast at versioning image data.

https://github.com/Oxen-AI/oxen-release