> I was really just thinking of grep and some other CLI utilities there. :)
Another technique I use is to spider a site (respectfully) and then use https://github.com/BurntSushi/ripgrep or load the spidered data into a local Elastic search.
Some of what you outline in searching other parts dovetail into handling poor access to the Internet. For folks with bad or transient access
pip install --upgrade youtube_dl
Re:Internet Search Tips
by greglindahl on Thursday December 13, 2018 @02:16AM
(#18670441)
BTW "pip install warcio" is the latest hotness for processing warc files. Also, you can add a header to wget to download a byte range. Unfortunately archival tools are mostly targeted at crawling and then playback, not so much special collections like this one.
How do you use wget? I know you can specify a start position, intended for resuming downloads, but you need to specify an end position as well (to extract just the single compressed file), and I don't see any way to specify an end in the wget manpage.
Re:Internet Search Tips
by greglindahl on Thursday December 13, 2018 @02:57PM
(#18675460)
Hm, come to think of it, I guess I was programming in python at the time. The internets say that wget doesn't support range requests and while curl's manpage says --range works, it doesn't work for me.
The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA.
> Nor digging into raw git and mercurial repositories.
I have never had to do that for any research task I've undertaken, so that would be both too obscure to mention and I wouldn't know anything about it.
> Universities often have blanket access to journals if you use their wifi.
A good point. My proxy superseded this for me but I used to do this, and simply forgot about it. I'll add that. (Another good trick I know is using the old Google Reader RSS exports hosted on IA to get fulltext of webpages. I'll have to add that too: https://www.gwern.net/Search#searching-the-google-reader-arc... )
> get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc)
I was really just thinking of grep and some other CLI utilities there. :)