News for hackers. Stuff that matters.

A comment on Web Scraping in 2016. Parent comment

Re:Web Scraping in 2016 by Doctor_Fegg on Tuesday August 23, 2016 @06:56PM (#12348436)
The protocol is also that you send a valid, non-faked User-Agent:

"The User-Agent request-header field contains information about the user agent originating the request. This is for [...] the tracing of protocol violations [...]. User agents SHOULD include this field with requests"

Many scrapers disregard this part of the protocol. Of course, whether a headless browser should send a different UA is an interesting question.

https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

[ Parent | on HN ]
Re:Web Scraping in 2016 by TheCoelacanth on Wednesday August 24, 2016 @02:06PM (#12353981)
User-Agent is a SHOULD, not a MUST. There are also practically no browsers that send a non-fake User-Agent, since they almost all claim to be Mozilla/5.0.
[ Parent | on HN ]