News for hackers. Stuff that matters.

A comment on The Onion cut 66% of their bandwidth by upstream caching 404s. Parent comment

Re:The Onion cut 66% of their bandwidth by upstream caching ... by joevandyk on Thursday March 25, 2010 @04:13PM (#1219369)
Is all that HTML? Spiders generally only care about HTML -- they're not going to download the js and css and images and flash and whatever.
[ Parent | on HN ]
Re:The Onion cut 66% of their bandwidth by upstream caching ... by bluesmoon on Thursday March 25, 2010 @11:58PM (#1220226)
8 years ago, maybe. Today all search engine spiders pull down and execute javascript & css as well just so that they can index a real user's experience.
[ Parent | on HN ]
Re:The Onion cut 66% of their bandwidth by upstream caching ... by code_duck on Friday March 26, 2010 @02:33AM (#1220408)
The better ones may, but perhaps not for all pages. Generally, when viewing logs I see search engine spiders make a single request, for only the html content and not js or css.
[ Parent | on HN ]
Re:The Onion cut 66% of their bandwidth by upstream caching ... by latch on Thursday March 25, 2010 @06:30PM (#1219711)
duh, I didn't think about that :) HTML is way smaller
[ Parent | on HN ]