News for hackers. Stuff that matters.

A comment on Shoestring Budget? Starting to feel growth issues on your back-end? Embrace unix and C.

Re:Shoestring Budget? Starting to feel growth issues on you... by natch on Tuesday September 09, 2008 @05:08PM (#299837)
What's your average page size?

Does your crawler support gzip compressed transfers?

What speed is your Comcast link?

[ Parent | on HN ]
Re:Shoestring Budget? Starting to feel growth issues on you... by scumola on Tuesday September 09, 2008 @11:23PM (#300047)
Page size varies, but when I was doing the initial speed tests, pages were around 20k each (guestimate).

Yea, the crawler uses libcurl, which supports gzip compressed content.

My comcast link is (theoretically) 7Mbit down, 385Kbit up.

[ Parent | on HN ]