News for hackers. Stuff that matters.
Re:Shoestring Budget? Starting to feel growth issues on you... by scumola on Tuesday September 09, 2008 @11:23PM (#300047)
Page size varies, but when I was doing the initial speed tests, pages were around 20k each (guestimate).

Yea, the crawler uses libcurl, which supports gzip compressed content.

My comcast link is (theoretically) 7Mbit down, 385Kbit up.

There are no comments on this comment yet.