News for hackers. Stuff that matters.
Re:Bill Gates's new website: The Gates Notes by pierrefar on Wednesday January 20, 2010 @03:34PM (#1065708)
This is doing the rounds on the net too:

http://www.thegatesnotes.com/robots.txt

It doesn't mention Bing. Amusing, and useless.

Re:Bill Gates's new website: The Gates Notes by lolcraft on Thursday January 21, 2010 @06:09AM (#1067051)
Re:Bill Gates's new website: The Gates Notes by thwarted on Wednesday January 20, 2010 @06:47PM (#1066133)
It's a cut and paste from some sample file, so this doesn't indicate anything much.

google search for the text in the comments ( http://tinyurl.com/ygtuanh ) has 49 hits, there would be more if I had not joined the lines together, I'm sure.

Re:Bill Gates's new website: The Gates Notes by walkon on Wednesday January 20, 2010 @04:42PM (#1065867)
Why is it useless (I've never worked with robots.txt)? Are the /css and /js files not large enough to bother skipping?
Re:Bill Gates's new website: The Gates Notes by pierrefar on Wednesday January 20, 2010 @04:47PM (#1065886)
Useless in the sense it doesn't really need to mention Bing.

Blocking /css and /js makes sense because they don't add value in being crawled. Not necessary, but doesn't hurt.

Best practice: Always have a robots.txt file, even if it's empty.

Re:Bill Gates's new website: The Gates Notes by beza1e1 on Thursday January 21, 2010 @02:43AM (#1066836)
What is the reason for your Best Practice advice?
Re:Bill Gates's new website: The Gates Notes by mshafrir on Wednesday January 20, 2010 @04:49PM (#1065888)
Look at the comments...

# robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources.