Re:Bill Gates's new website: The Gates Notes
by walkon on Wednesday January 20, 2010 @04:42PM
(#1065867)
Why is it useless (I've never worked with robots.txt)? Are the /css and /js files not large enough to bother skipping?
[ Parent
| on HN ]
Re:Bill Gates's new website: The Gates Notes
by pierrefar on Wednesday January 20, 2010 @04:47PM
(#1065886)
Useless in the sense it doesn't really need to mention Bing.
Blocking /css and /js makes sense because they don't add value in being crawled. Not necessary, but doesn't hurt.
Best practice: Always have a robots.txt file, even if it's empty.
Re:Bill Gates's new website: The Gates Notes
by beza1e1 on Thursday January 21, 2010 @02:43AM
(#1066836)
What is the reason for your Best Practice advice?
Re:Bill Gates's new website: The Gates Notes
by mshafrir on Wednesday January 20, 2010 @04:49PM
(#1065888)
Look at the comments...
# robots.txt # # This file is to prevent the crawling and indexing of certain parts # of your site by web crawlers and spiders run by sites like Yahoo! # and Google. By telling these "robots" where not to go on your site, # you save bandwidth and server resources.