News for hackers. Stuff that matters.

A comment on The Internet Archive is under a DDoS attack. Parent comment

Re:The Internet Archive is under a DDoS attack by 1vuio0pswjnm7 on Tuesday May 28, 2024 @12:31AM (#40497338)
Assume everyone is familiar with this project, dating back to 1996:

https://en.wikipedia.org/wiki/WWWOFFLE

https://ftp.netbsd.org/pub/pkgsrc/distfiles/wwwoffle-2.9j.tg...

The way the www is going, it seems like downloading a copy of libgen, i.e., nonfiction books, and scimag, i.e., academic journals, via torrent, would be more valuable than archiving websites, in general. These primary sources are part of the material used to train so-called "AI" anyway. The problem is that this so-called "AI" also includes all the garbage from the www.

Worst case is eventually these books and journals will again become publicly inaccessible but "AI" will be offered as a bogus substitute; a future where few people will do research using primary materials anymore, they will just submit questions to a remote "AI" server. Truth will be decimated.

[ Parent | on HN ]
Re:The Internet Archive is under a DDoS attack by dkz999 on Tuesday May 28, 2024 @02:06AM (#40497802)
We already kind of see this with search.
[ Parent | on HN ]
Re:The Internet Archive is under a DDoS attack by oredbored on Tuesday May 28, 2024 @11:08AM (#40501435)
Do we know for sure that they trained on data from libgen etc? It's such a powerful source of information you'd assume they must have, although they would never admit it. There must be a way to test if they have, via enquiring about some niche information only found in certain books.
[ Parent | on HN ]
Re:The Internet Archive is under a DDoS attack by Intralexical on Tuesday May 28, 2024 @04:31PM (#40505288)
It is apparently widely suspected that a certain "Books2" dataset mentioned by OpenAI is basically just LibGen:

https://blusharkmedia.medium.com/the-ongoing-battle-against-...

https://techhq.com/2023/09/can-libgen-shadow-library-survive...

https://www.twitter.com/theshawwn/status/1320282152689336320

https://qz.com/openai-books-piracy-microsoft-meta-google-cha...

https://qz.com/shadow-libraries-are-at-the-heart-of-the-moun...

https://goodereader.com/blog/e-book-news/authors-file-lawsui...

When asked about whether this was true, they refused to answer based on confidentiality concerns, then said they had deleted all copies of the dataset, stopped using it, and no longer employed the individuals that compiled it:

https://www.businessinsider.com/openai-destroyed-ai-training...

We do know for a fact that the (non-OpenAI-controlled) "Books3" dataset is just "all of bibliotik":

https://www.twitter.com/theshawwn/status/1320282149329784833

https://github.com/soskek/bookcorpus/issues/27

And we also apparently know for a fact that this was included in the datasets used to train LLAMA:

https://en.wikipedia.org/wiki/The_Pile_(dataset)

https://aicopyright.substack.com/p/the-books-used-to-train-l...

https://aicopyright.substack.com/p/has-your-book-been-used-t...

[ Parent | on HN ]
Re:The Internet Archive is under a DDoS attack by oredbored on Wednesday May 29, 2024 @06:24AM (#40510386)
Thanks a lot for all the links. Fascinating stuff.
[ Parent | on HN ]
Re:The Internet Archive is under a DDoS attack by 1vuio0pswjnm7 on Wednesday May 29, 2024 @12:59AM (#40508632)
This "AI" nonsense seems like a legitimate threat to literacy. Why would young people read a nonfiction book when they can just send questions to a so-called "tech" company that has used the book in training a LLM. These companies, needless intermediaries with zero experise on the subject matter of the book, exist only to collect data and use it for commercial purposes. Unlike the books' authors and publishers they have no responsibility for publishing information that is adequately researched and factually correct.
[ Parent | on HN ]
Re:The Internet Archive is under a DDoS attack by 1vuio0pswjnm7 on Wednesday May 29, 2024 @05:34PM (#40517294)
s/experise/expertise/
[ Parent | on HN ]