Is there any legal risk to users in North America that do this? Is this copyrighted material?

Probably; use a VPN.

Yes they're copyrighted - albeit not by the authors who actually wrote them, but by the publishers who require copyright assignment for the privilege of having your work hosted on their website.

So when academics casually share papers with their collaborators, are they technically exposing themselves to lawsuits? Or is being part of a research organization/university with subscriptions to these services enough to mitigate that risk?

No one talks about this, but for many (most?) of the papers they publish, there’s no copyright to assign. When I was in grad school, 100% of my research was federally funded (NSF, Navy, NASA) so they insisted that every product of that research be freely available with no restrictions. When we get an article ready for publication, the Journal would send us the standard “you assign your copyright to us” forms. We’d sign them but also send a copy of “the green form” stating that it was federally funded and we didn’t have any copyright to assign. They just ignore that.

It depends on the circumstances and the publisher. In many cases publishers permit authors to host the accepted version of the paper (but not the final version that includes revisions based on reviewer feedback) on their personal/institutional website and to email copies of the paper on an individual basis to people who request them.

For example, see https://www.elsevier.com/about/policies/copyright (under "Author Rights") for what Elsevier permits you to do with your own work.

On the other hand, publishers have sometimes filed lawsuits against sites where authors share their papers, e.g. ResearchGate: https://www.nature.com/articles/d41586-018-06945-6

Elsevier have also sent takedown notices to universities where academics have made the final version available on their institutional websites:

https://www.washingtonpost.com/news/the-switch/wp/2013/12/19...

https://osc.universityofcalifornia.edu/2013/12/elsevier-take...

https://news.harvard.edu/gazette/story/newsplus/elsevier-tak...

https://blogs.library.duke.edu/scholcomm/2014/01/28/setting-...

You are almost always allowed to self archive the final version, but you have to share the PDF you generated yourself, not the nicely formatted one from the publisher. And some publishers only allow self-archiving outside repositories like researchgate. But most importantly, Google Scholar and Semantic Scholar would pick up most links from blogs and Arxiv.

Use https://v2.sherpa.ac.uk/romeo/ to check. Also, EU projects in most cases now require open publishing and publishers make exceptions even when OA is forbidden (“self archiving is allowed if mandated by the funding agency”).

That is an interesting question without a good answer. As I understand it (I am far from being an expert), there are sometimes specific exemptions to allow that sharing. However, generally speaking, I think it is a gray area. The journals would surely be shooting themselves in the foot, though, if they tried to sue their contributors. Academics would bring down hell on any journal that tried that. Moreover, it is not clear to me that they would win as it seems to be a customary practice even if it is not an explicitly allowed one. Finally, sharing a single paper might get research or academic exemptions as you aren't copying the whole journal issue. I doubt any publisher wants to go down that road. They are probably better off with the law remaining vague.

They are usually shared under the title of “preprint” or “draft” often, but not always, before the time of actual publishing.

Always seemed like a grey area to me. We didn’t really distribute the copy of the paper with the journal/conference’s name + copyright - though a perhaps a line under the title: “To be published in…”

What happens in practice: researchers are usually free to share the draft they had before the publishing process. Which means you can read the text before every sentence was wordsmithed in huge pain to make the paper a quarter page shorter.

Sharing with collaborators may constitute fair use. I wouldn't put one of my own papers on my website, though, for example, for something that isn't published as open access.

> casually share papers with their collaborators, are they technically exposing themselves to lawsuits?

For the final, not open access published papers they certainly do.

From watching the Aaron Schwartz documentary, hopefully once scihub makes these publishers obsolete they will no longer have the power to press charges. But yeah they can and will get the Feds involved and will ruin your life like Schwartz

1. rent a seedbox for $10/mo hosted in a different country

2. seed from your seedbox, not your personal device

3. profit

I encourage you to do this in the most audacious, and visible way possible.

Let see how will they sue millions, upon millions of people.

Make them face a fait accompli. They already lost.

They never target millions. They target one, and ruin his life, preferably one with children so he also divorces.

Every dystopian regime does that.

Please look into the history of file sharing and how that went for individuals.

These companies will pick examples and ruin their lives. It's hard to say how big the risk is here but this is reckless advice.

Vote for copyright reform.

They only need to create some examples.

What we really need is an index of these torrents by DOI, and then ultimately by journal and issue. Are you aware of any work to make this happen?

The only one I'm aware of currently is https://github.com/sci-hub-p2p/sci-hub-p2p. Library genesis also hosts database dumps at https://libgen.rs/dbdumps/.

There's really a need though for more developers to get involved with building tools for more easily searching and working with the collection, ideally with a nice UI and integration with things like crossref. This is a massively valuable data set and it would be great to see what people can come up with. Lots of awesome potential for data mining too.

If it weren't for the legal issues (publishers using copyright law to restrict access to literature they got for free since they never pay authors for their work), there's no shortage of projects that could utilize this data and be enormously beneficial for the scientific community and humanity in general. Unfortunately such work can only be done in the shadows right now, which greatly limits the number of people/institutions likely to do so.

Isn't libgen already distributed via IPFS too?

Yes, LibGen is mirrored on IPFS:

https://news.ycombinator.com/item?id=25209246

I believe there is a database dump available with this information:

http://libgen.rs/dbdumps/scimag.sql.gz

This sounds very appealing. However, from a cursory search I can’t locate any NAS of that size anywhere close to that price point.

8-bay dock ~ 400$ (Icy Box 10-bay)

7x14TB HDD ~ 7*300$ (Toshiba MG07ACA14TE)

total 2500$

Is it compressible or already compressed?

It's all PDF files, which have their own compression, so it's unlikely there would be substantial gain from additional compression. Each torrent has 100 zip files, and each zip file has 1000 PDFs, but the files are stored uncompressed within the zips (i.e. using the STORE method).

> It's all PDF files, which have their own compression, so it's unlikely there would be substantial gain from additional compression.

You could write a custom compressor that decompiles journal PDFs to valid TeX, then compresses that.

Or at the simpler end of what's technologically possible, you could at least extract shared assets such as fonts that appear in multiple files. Keep files from the same journal together to find more overlaps.

I suspect there's quite a large gain to be had from further compression, at least theoretically. Even more if you could accept some level of non-semantic loss.

You could losslessly translate PDFs with compression to PDFs with no compression (bitmap images excepted), tar them up and compress the lot. This would get you a fair bit of gain for little pain.

However, I guess they use .zip STORE because it's fairly robust against minor corruption.

Is there some kind of searchable index included so that you can locate an article in a particular Zip? I'm assuming each article has some kind of ID numbers and the Zips are divided by ID range or something?

Yep! See https://github.com/sci-hub-p2p/artifacts/releases/tag/0

This project is in it's early stages and the documentation has quite some way to go, but the index that's part of the release contains all the necessary information. This tool also contains the code necessary to produce the index files if you have a local copy of the zips.

Each torrent contains 100,000 files, comprised of 100 zip files with 1,000 PDFs each. They are named by DOI. There's a database dump at (http://libgen.rs/dbdumps/) (scimag.sql.gz) which has the id -> DOI mapping and other information. The specific torrent and zip file can be determined based on the id; torrent = id/100000 and zip = id/1000.

Sci-Hub database/index is available here: http://libgen.rs/dbdumps/scimag.sql.gz

and database documentation is available here: https://gitlab.com/lucidhack/knowl/-/wikis/References/Libgen...

also see introduction to Sci-Hub for developers: https://www.reddit.com/r/scihub/comments/nh5dbu/a_brief_intr...

But each PDF is compressed individually. The textual content of the papers must have a lot of redundancy between them, maybe there is some gain to get there?

Illustrations easily outweigh the textual content, and those aren't shared. I mean, the text/formatting/latex code for an article compresses to something like 10kB, there's not much to save there.

Virtually all the works are published as PDFs. (There are some other formats, occasionally DJVU, etc.) There's integrated compression, though this can still vary tremendously by docuemnt.

Recent publications are virtually always based on direct PDF renders, and tend to be a few 100 kB per article.

Older publications are often scanned from paper-based copies, and can be about 10-20x larger, depending on the source. These may or may not have OCRed text, and OCR itself may be of variable quality. For documents with images or diagrams, those also add to both size and difficulty in vectorising copies.

It's possible to go through larger scans and regenerate them as rendered PDFs. That's intensive and error prone. There's also a range of viewpoints on archival as to whether it's preferable to retain the full expression of the original published version (and often accumulated marginalia and other marks of a specific instance), or to optimise for both storage and automated processing through reprocessed renders. The costs are high (typically you'll require a human or multiple humans to proof each work), though the storage and line-transmission savings are considerable.

I lean toward the latter myself. The attitude of other archivists (notably the Internet Archive) is to capturing as faithful a replication of originally-published formats as possible, at considerable cost in both storage and accessibility. (This applies to the Archives work in print, online / Web, and other document formats.)

Pressed, I'd strongly recommend a "capture what you can, reprocess according to need and demand as possible" approach.