News for hackers. Stuff that matters.
Re:Accessibility Through Semantic HTML by yorwba on Saturday December 30, 2017 @08:55AM (#16035231)
Are you sure that extracting document structure from a screenshot is a solved task? I'm aware of "What You Get Is What You See" [1], but they are nowhere close to 100% accuracy even for reconstructing the original markup, which does not even require making inferences about the meaning of the content.

You seem to be overestimating how close to practicality neural-network approaches are for most problems. Very much of the potential application space is still firmly within the domain of research. That doesn't mean what you described is completely impossible, just that you could get a PhD for pulling it off.

[1] https://arxiv.org/abs/1609.04938v1 (incidentally, I noticed that the version of the paper that was accepted for ICML 2017 no longer has the HTML part)

Re:Accessibility Through Semantic HTML by nl on Saturday December 30, 2017 @07:20PM (#16038513)
Yeah reconstruction of HTML isn't solved.

The exciting thing is that in this case it doesn't need to be. Segmentation (which is solved) is sufficient to identify semantically separate parts of the page, and then OCR can be used to place the text from the html on the representation of the page.

Re:Accessibility Through Semantic HTML by yorwba on Saturday December 30, 2017 @11:16PM (#16039322)
Could you point me to a paper where they successfully segment random webpages in the wild semantically? Or other kinds of documents?

I expect that to be significantly different from the task of segmenting objects in a photo, because most of the boundaries are not revealed by texture differences, but implicit in the arrangement.

Re:Accessibility Through Semantic HTML by nl on Sunday December 31, 2017 @07:47AM (#16040382)
Most of the work is in document segmentation:

https://arxiv.org/abs/1704.01474

https://arxiv.org/abs/1711.07695

https://arxiv.org/abs/1710.07363

I'm pretty sure I've seen some webpage work too, but I don't have the reference.

Re:Accessibility Through Semantic HTML by yorwba on Sunday December 31, 2017 @08:25AM (#16040491)
Those are all about segmenting the text on a page from the background, and not semantic segmentation, which I thought you were talking about ("segment the page and decide what is the navigation").

Finding the text in a screenshot of a webpage should be easier than doing it for handwritten documents; it's the "decide what is the navigation" part that I'm doubtful about.