News for hackers. Stuff that matters.

question222

question222 on Hacker News

Re:Pinterest Raises $150M at 2015 Share Price on Tuesday June 06, 2017 @07:12PM (#14502302)
Would you mind sharing your spider technology stack. How do you find new urls efficiently? What spider and what storage do you use? Do you append everything in one WARC file and split it if it gets too big?