News for hackers. Stuff that matters.

A comment on Inventing Favicon.ico. Parent comment

Re:Inventing Favicon.ico by wareya on Sunday March 29, 2015 @10:27PM (#9287345)
Robots.txt is different. Without it, bots have no way of knowing whether to get any other data from the site. You would need "bots allowed" information in the HTTP handshake itself to prevent bots from accidentally hitting pages they shouldn't. This can already be Very Bad.
[ Parent | on HN ]
Re:Inventing Favicon.ico by TazeTSchnitzel on Sunday March 29, 2015 @10:38PM (#9287382)
> You would need "bots allowed" information in the HTTP handshake itself to prevent bots from accidentally hitting pages they shouldn't.

Humans could also hit such pages. If your GET requests change state, there's no helping you.

[ Parent | on HN ]
Re:Inventing Favicon.ico by wareya on Monday March 30, 2015 @08:20AM (#9288975)
The whole point of robots.txt is that there are pages which people may hit that bots can't. What are you on?
[ Parent | on HN ]