I see no mention of the wayback machine.
Nor digging into raw git and mercurial repositories.
I have also had luck emailing folks that either inquired about document or last good email for the author.
Universities often have blanket access to journals if you use their wifi. SH has mostly supplanted the need for this, but not always. Universities are still great for interlibrary loans, if you are friendly with the library staff, they might let you IL a book and read it in the library if they don't checkout to citizens.
Last one on mining a local search database is like you said, to determine the boolean operators, get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc) and some lightweight un-obtrusive scripting (using in browser js console).
Like this snippet that extracts the pdfs and doi.org urls from a page. `copy` is a chrome specific helper. This snippet is great for extracting results that are lazily loaded in, something wget cannot do for you.
function gpdfs() {
aref = document.getElementsByTagName("a");
var result = [];
for(i = 0; i < aref.length; i++) {
var t = aref[i];
if (t.href.endsWith(".pdf") || t.href.includes("doi.org")) {
result.push(t.href);
}
}
return result;
}
function pc() {
copy(gpdfs());
}
pc();The IA is covered in multiple sections. The Wayback Machine is merely one area of the IA.
> Nor digging into raw git and mercurial repositories.
I have never had to do that for any research task I've undertaken, so that would be both too obscure to mention and I wouldn't know anything about it.
> Universities often have blanket access to journals if you use their wifi.
A good point. My proxy superseded this for me but I used to do this, and simply forgot about it. I'll add that. (Another good trick I know is using the old Google Reader RSS exports hosted on IA to get fulltext of webpages. I'll have to add that too: https://www.gwern.net/Search#searching-the-google-reader-arc... )
> get familiar with the FTS backends (SQLServer, Postgres, Elastic Search, etc)
I was really just thinking of grep and some other CLI utilities there. :)
Another technique I use is to spider a site (respectfully) and then use https://github.com/BurntSushi/ripgrep or load the spidered data into a local Elastic search.
Some of what you outline in searching other parts dovetail into handling poor access to the Internet. For folks with bad or transient access
pip install --upgrade youtube_dl