News for hackers. Stuff that matters.
Re:The Death of Upcoming.org by creature on Saturday April 20, 2013 @10:00AM (#5581415)
I don't know for sure, but I expect it's a standard thing from any spidering. Let's say you start at one page and find 10 links; your tracker will then have 10 items. If each of those pages has 10 links, then you can then have up to 100 items to download. And so on.

Over time you're going to start seeing duplicates (ie. things you've already archived/spidered), so the number will go down. But while you're still in the 'discovery' stage it's going to go up and up.

Re:The Death of Upcoming.org by bambax on Saturday April 20, 2013 @10:17AM (#5581462)
Oh, ok, if you're spidering.

But here's what the OP said: "All of Upcoming's events and venues use autoincremented ids, making it dead simple to generate a list of URLs to scrape."

So if you go from 1 to {current_id} you get them all, and you have the list at the start.