Re:Internet Search Tips
by greglindahl on Thursday December 13, 2018 @02:16AM
(#18670441)
BTW "pip install warcio" is the latest hotness for processing warc files. Also, you can add a header to wget to download a byte range. Unfortunately archival tools are mostly targeted at crawling and then playback, not so much special collections like this one.
[ Parent
| on HN ]
How do you use wget? I know you can specify a start position, intended for resuming downloads, but you need to specify an end position as well (to extract just the single compressed file), and I don't see any way to specify an end in the wget manpage.
Re:Internet Search Tips
by greglindahl on Thursday December 13, 2018 @02:57PM
(#18675460)
Hm, come to think of it, I guess I was programming in python at the time. The internets say that wget doesn't support range requests and while curl's manpage says --range works, it doesn't work for me.