Re:Facebook was used as a proxy by web scraping bots
by fiddlerwoaroof on Thursday November 05, 2020 @05:00PM
(#25002515)
It seems to me that what "previews" really want is an API for the page's content in a structured format: OpenGraph tags and other microformats are one representation, but it's annoying to have to load _all_ the HTML just to grab title and the OG tags.
Re:Facebook was used as a proxy by web scraping bots
by stickfigure on Thursday November 05, 2020 @04:18PM
(#25002075)
Accept: text/preview
Re:Facebook was used as a proxy by web scraping bots
by stingraycharles on Friday November 06, 2020 @12:41PM
(#25009451)
In what content type? Json? Xml? Html?
I guess what this is really about is, I hate to say it, but something in the direction of the semantic web, where web servers (and in this case, CloudFlare et al) actually gain a deeper understanding of the content they serve, and a web browser / crawler being able to query this content directly.