Re:Developers are your users too – a defence of 'View Source'
by JMTQp8lwXL on Saturday June 13, 2020 @06:51PM
(#23513523)
Even with the best of intentions, DOM structures can change without advanced notice. Instead of scraping via DOM structure, why not screenshot a webpage and use OCR? I think the solution to the scraping problem is being viewed narrowly.
Re:Developers are your users too – a defence of 'View Source'
by taeric on Saturday June 13, 2020 @08:09PM
(#23513974)
This suggestion is a hilarious admission that hypertext as conceived failed. And that makes me rather sad.
Re:Developers are your users too – a defence of 'View Source'
by JMTQp8lwXL on Sunday June 14, 2020 @08:15AM
(#23517125)
It didn't fail. People don't want others scraping to protect their businesses. And that's how we've ended up with highly obfuscated HTML. If it failed, it wouldn't work at all. HTML was designed to display documents, not for scraping purposes. You know what is designed to be scraped? An actual application programming interface.
Some user themes are for visual nicety, sure, but they might be essential aids to some. Making it easier to data-grab with BeautifulSoup etc means any data shown is more portable, too. Data scraping is all sorts of useful, and having classes that aren’t minified garble makes it a considerable amount easier to target.