News for hackers. Stuff that matters.

A comment on Nike.com's robots.txt. Parent comment

Re:Nike.com's robots.txt by vmoore on Friday February 11, 2022 @11:10AM (#30303063)
> at least some parser will choke on them

Ah the never ending joy I get by putting a 1GB file as the robots.txt or even having a 1GB `favicon.ico` file.

[ Parent | on HN ]
Re:Nike.com's robots.txt by marginalia_nu on Friday February 11, 2022 @11:12AM (#30303092)
Dunno how other search engines do it, but I only ever read 512Kb from a connection :P
[ Parent | on HN ]
Re:Nike.com's robots.txt by vmoore on Friday February 11, 2022 @11:35AM (#30303443)
I once renamed a Ubuntu .ISO file to favicon.ico and placed it in the root. Some greedy bots choked on it, others had more sense and timed out.
[ Parent | on HN ]