Framework isn't the issue in this case, it just looks like you're loading all 100 pages into memory at one time? Or keeping the previous pages in memory as you go "deeper" into the hierarchy.
I recommend you just keep a few pages in memory at any given time, keeping a simple list of the URLs scraped out of that HTML. If you're using PHP remember to unset() the variables storing the HTML and so on.
I haven't used PHP in a long time and of course I'm just speculating based on the error message, so slight disclaimer there.
I use PHP all the time, I am just a little dissatisfied with it in light of the alternatives that are out there.
Initially it started out as a 'what if I do this' kind of app. I would like to re-write the backend with more attention to detail...
A public service like that is a perfect candidate for their architecture, I think.
Looks like tomorrow is de-bugging day!
I wrote this app over the course of a day a couple of months ago and it's been languishing on my server since then. At the moment it's written in PHP and is not exactly what you'd call 'well tested'.
I'm considering migrating it to Rails (or Camping). Any suggestions for mini-frameworks (like Camping) suitable for a single-page app would be welcomed!