News for hackers. Stuff that matters.
Re:Google's robots.txt by logotype on Saturday February 22, 2014 @11:07PM (#7284878)
Mine is more awesome: http://logotype.se/robots.txt
Re:Google's robots.txt by jonty on Sunday February 23, 2014 @08:03AM (#7285786)
Oh wow, I didn't realise my robots.txt joke had spread!

I added that to the last.fm/robots.txt many years ago (http://www.wired.com/business/2010/08/robot-laws/all/), and have just been made aware that it appears to have spread across the internet:

https://www.google.co.uk/search?safe=off&q="Disallow%3A+%2Fi...

Favourite sites I've found with it so far: php.net, princeton.edu, nest.com, songkick.com.

Re:Google's robots.txt by russellbeattie on Saturday February 22, 2014 @11:17PM (#7284903)
Heh, if only Yandex and Baidu respected robots.txt.
Re:Google's robots.txt by agwa on Saturday February 22, 2014 @11:22PM (#7284916)
I've found this GitHub project to be an invaluable resource for blocking bad bots: https://github.com/bluedragonz/bad-bot-blocker
Re:Google's robots.txt by anonymfus on Sunday February 23, 2014 @04:21AM (#7285378)
>Unless your website is written in Russian or Chinese, you probably don't get any traffic from them. They mostly just waste bandwidth and consume resources.

THIS is evil. You could use this argument for banning any new search engine.

Re:Google's robots.txt by agwa on Sunday February 23, 2014 @05:32PM (#7287946)
The problem is that both Yandex and Baidu are rather poorly behaved - they hit your website way too fast, downloading large bandwidth files in quick succession. That's actually what led me to the bad-bot-blocker project in the first place. Baidu has also been accused of not respecting robots.txt though I have not personally observed that.

This is the reason they're blocked, not because they're new or non-English.

Re:Google's robots.txt by ScottWhigham on Sunday February 23, 2014 @06:35AM (#7285615)
What "new search engine" has actually generated actual revenue for any webmaster in the past ten years? You could argue DDG but that's the only one I can think of.
Re:Google's robots.txt by joveian on Sunday February 23, 2014 @01:07PM (#7286756)
I think DDG primarily uses Yandex. They at least often put a "Powered by Yandex" logo along the side.
Re:Google's robots.txt by bmelton on Sunday February 23, 2014 @03:13AM (#7285305)
That's amazing. Know of anything like it for Nginx?
Re:Google's robots.txt by anon4 on Sunday February 23, 2014 @04:04AM (#7285358)
It's blocking by user agent and source ip. You should be able to port the list easily for Nginx. I'd even say you can write a simple awk script in a few minutes to convert from Apache's format to Nginx's.
Re:Google's robots.txt by coolj on Sunday February 23, 2014 @04:31AM (#7285393)
curl -s https://raw.github.com/bluedragonz/bad-bot-blocker/master/.h... | awk -F\" '/SetEnvIfNoCase/ {pattern = pattern $2 "|"} END {print "if ($http_user_agent ~ (" substr(pattern, 0, length(pattern)-1) ")) { return 403; }"}'

Not tested. :)

Re:Google's robots.txt by kercker on Sunday February 23, 2014 @01:09AM (#7285108)
I'm sure that Baidu respects robots.txt.

This site coolshell.cn is boycotting baidu, so it tells Baiduspider that it doesn't want to be indexed by baidu.

Baidu respects this and doesn't index anything from this website.

Re:Google's robots.txt by anonymfus on Sunday February 23, 2014 @04:15AM (#7285374)
Yandex's documentation about implemented robots.txt features:

http://help.yandex.com/webmaster/controlling-robot/robots-tx...

Yandex's online tool to check specific URL on allowness in robots.txt:

http://webmaster.yandex.com/robots.xml

Re:Google's robots.txt by wfn on Sunday February 23, 2014 @07:20AM (#7285709)
Along similar lines, personally I was fascinated with https://bridges.torproject.org/robots.txt :)
Re:Google's robots.txt by anonymfus on Sunday February 23, 2014 @03:48AM (#7285340)
Why you consider Yandex evil?
Re:Google's robots.txt by eksith on Sunday February 23, 2014 @05:05AM (#7285448)
I don't know if this is universal, but Yandex was quite bad in the past at stumbling around generated links (E.G. calendars that go to infinity) and it wasn't at all unusual to see 400+ links crawled in a day or so. Baidu was the same, but I think they're more well behaved these days.
Re:Google's robots.txt by Moru on Sunday February 23, 2014 @05:55PM (#7288064)
Lucky you with only 400+ per day. We had thousands per hour. Still the server survived but it wasn't fast...
Re:Google's robots.txt by Cthulhu_ on Sunday February 23, 2014 @08:59AM (#7285898)
I IP-banned Chinese search engine crawlers (yandex, baidu etc) because they did not respect the crawl-delay instruction in my robots.txt, causing my low-end crappy php server running vbulletin to bork out. Basically, their bots are too aggressive.
Re:Google's robots.txt by tbirdz on Sunday February 23, 2014 @01:14PM (#7286798)
Yandex is Russian, not Chinese.
Re:Google's robots.txt by drdeadringer on Saturday February 22, 2014 @11:27PM (#7284928)
Is it analogue nuclear?
Re:Google's robots.txt by vladtaltos on Sunday February 23, 2014 @03:42AM (#7285336)
Why evil yandex?