News for hackers. Stuff that matters.
Re:Archiving web sites by anarcat on Friday November 23, 2018 @10:22AM (#18516888)
author here. AMA.

since I wrote this, i started experimenting with grab-site:

https://github.com/ludios/grab-site

it's a wrapper around (and a fork of) wpull but the main advantage over it is it can do on-the-fly reconfiguration of delay, concurrency, ignore patterns and so on. it also provides a nice web interface. if you're only crawling one site every once in a while, wpull and crawl are fine, but for larger projects, grab-site is a must.

There are no comments on this comment yet.