I did a large-scale crawl of the web some years ago and we put together a billion pages.

The bulk of the content ended up being index pages - i.e. large list of links taking you to the content - pagination, other breadcrumbs etc.

You can exhaust a lot of resources without getting anything useful.

It is no longer a point and go kind of thing unfortunately, you need a good understanding of page structure, estimate what kinds of links are vital, etc. else there's a ton of crap you'll pick up.

Or maybe I was doing something wrong.