News for hackers. Stuff that matters.
Re:Facebook was used as a proxy by web scraping bots by fiddlerwoaroof on Thursday November 05, 2020 @04:03PM (#25001957)
I wonder if

    Accept: application/json
Would be a reasonable alternative? Wasn't this supposed to be the point of content negotiation?
Re:Facebook was used as a proxy by web scraping bots by stingraycharles on Thursday November 05, 2020 @04:12PM (#25002025)
Maybe, but not really; seems like this thread is more about intent (“I just want a preview”) while content type is more about representation (“I want the content as json”). I can imagine that there will be websites that are actively using the accept parameter to distinguish between “regular visitors” and have their APIs at the same paths (didn’t Reddit do this at some point?), and thus your approach would break in this case.

I guess what this is really about is, I hate to say it, but something in the direction of the semantic web, where web servers (and in this case, CloudFlare et al) actually gain a deeper understanding of the content they serve, and a web browser / crawler being able to query this content directly.

Re:Facebook was used as a proxy by web scraping bots by fiddlerwoaroof on Thursday November 05, 2020 @05:00PM (#25002515)
It seems to me that what "previews" really want is an API for the page's content in a structured format: OpenGraph tags and other microformats are one representation, but it's annoying to have to load _all_ the HTML just to grab title and the OG tags.
Re:Facebook was used as a proxy by web scraping bots by stickfigure on Thursday November 05, 2020 @04:18PM (#25002075)
Accept: text/preview
Re:Facebook was used as a proxy by web scraping bots by stingraycharles on Friday November 06, 2020 @12:41PM (#25009451)
In what content type? Json? Xml? Html?