News for hackers. Stuff that matters.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by delusional on Sunday July 03, 2022 @05:28PM (#31971191)
Wasn't that what google books was supposed to be?
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by daniel_reetz on Sunday July 03, 2022 @08:08PM (#31972118)
Google Books, like so many Google projects, had a dual purpose. Making books accessible is noble and on-mission. But more importantly natural language models can be trained on the scanned corpus.

The same was true of the original GOOG 411, which provided a free service, but was really put in place to train up their voice recognition projects.

This is a long running strategy of Google, and it's a shrewd one. The main thing is not to mistake it for a public good. It is an act of privatization.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by visarga on Monday July 04, 2022 @12:33AM (#31973511)
That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection.

I think Google initially wanted to augment the web results with a large book collection to get "all the world information and make it searchable", same with Google News.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by vintermann on Monday July 04, 2022 @03:16AM (#31974343)
Before there were neural language models, there were n-gram models, skip n-gram models, latent dirichlet allocation models, a whole zoo of non-neural machine learning models. Google used some of them to power old Google Translate, which was still a very impressive piece of technology.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by philsnow on Monday July 04, 2022 @02:22AM (#31974035)
> That can't be true, Google Books was 15 years prior to the advent of large language models. Until 2020 nobody could train on such a large collection.

... pull the other one.

Okay, your statement could be true depending on what you mean by "large". But what makes you think that companies like Google haven't been working on language models without releasing them and/or without discussing them publicly? There's an advantage to be had by keeping corporate secrets.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by visarga on Monday July 04, 2022 @03:22AM (#31974390)
The gap between research and publication is real but its length is less than one year. I don't think even Google has the resources to do it secretly. Who would work on it and how could they have kept the secret so tight? AI researchers want to publish, especially the best ones, it's essential for their careers. Someone else could plant the flag on their discovery and claim the fame.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by philsnow on Tuesday July 05, 2022 @02:46AM (#31985134)
For a time while I was working at Google I nursed the idea of transferring to the natural language processing / language modeling areas, but I only have a bachelors in linguistics whereas it seems like they strongly prefer PhDs. Linguistics PhDs can pretty much teach linguistics or go into another field. It would not be hard for Google to find a couple hundred of them and entice them to join up.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by xavriley on Monday July 04, 2022 @03:53AM (#31974569)
At the time Google Books started, “big data” was in vogue and there were data-driven ML approaches to NLP e.g. word2vec, that would have benefited from a large corpus
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by voisin on Sunday July 03, 2022 @05:57PM (#31971374)
I assume Google abandoned this along with all their earlier mission statements in favour of building another chat app.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by bbarnett on Sunday July 03, 2022 @06:21PM (#31971501)
I can't imagine a life at google. So much promise, all turned to ash.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by UkrainianJew on Sunday July 03, 2022 @07:56PM (#31972047)
More like an ad company driven by dark patterns that needs to pretend being good and nice and innovative as otherwise it will lose goodwill just like Facebook did.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by klipt on Sunday July 03, 2022 @10:58PM (#31973056)
I think the Google founders were inherently nicer people than the Facebook founder, but their companies ended up converging on advertising because that's really the only way their products could be profitable.
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by discordance on Monday July 04, 2022 @02:01AM (#31973906)
Is that really the only way to be profitable though?

I believe going public inevitably leads to needing to follow growth, which then eventually corrupts virtuous missions.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by emanuensis on Sunday July 03, 2022 @09:01PM (#31972430)
Supposedly all at Library of Congress (LoC) anyone have experience using it there?
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by lupire on Sunday July 03, 2022 @06:23PM (#31971520)
Could you take a moment to check it Google Books search still exists?

I'll give you a hint: https://books.google.com/?hl=en

> Search the world's most comprehensive index of full-text books.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by voisin on Sunday July 03, 2022 @06:32PM (#31971570)
I know it exists, but it has appeared to languish for years. They rely on third parties now for inclusion of books, whereas in the early days they innovated on their own with specialized scanning technology. They seemed quite proud of it a decade ago. When was the last time Google has touted their books project? Have they even integrated searching books into their main search (which was supposed to catalogue And make searchable all the world’s information)?

It hasn’t been killed but it is clearly a zombie.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by londons_explore on Sunday July 03, 2022 @06:40PM (#31971609)
It was paralyzed by legal disputes with book publishers.

In the years the lawsuits were going on, nearly everyone left the project. And then the lawyers have put in so many red lines that it's nearly impossible to make any changes to it.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by voisin on Sunday July 03, 2022 @07:19PM (#31971847)
Yup, I read about that on Wikipedia, but I can't help but not care. If a company touts massive initiatives and then gets bogged down in lawsuits, it seems like they didn't do the basic due diligence to avoid that. (Uber, AirBNB, and others seem to also have these headwinds, though not to the extent that it led to permanent paralysis, so maybe Google made a bet they thought they'd win and then didn't, whereas these other companies did.). I can't help but wonder why, with Google's resources vs. Uber or AirBNB, they couldn't keep moving forward if they wanted to. Strike deals, pay people, whatever. If it matters (i.e. if it involved ads) they would have done it.

Given Google's behaviour since the end of their period of true innovative excellence, I don't cut them much slack.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by twicetwice on Sunday July 03, 2022 @07:44PM (#31971996)
This well-written article, "Torching the Modern-Day Library of Alexandria"[0], might change your mind on that. I found it to be a compelling and tragic story.

Edit: actually, I think this would make a good submission. Looks like it hasn't been posted since 2017.

[0]: https://www.theatlantic.com/technology/archive/2017/04/the-t...

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by yourad_io on Sunday July 03, 2022 @08:42PM (#31972326)
This was a great read and indeed quite tragic

> People have been trying to build a library like this for ages—to do so, they’ve said, would be to erect one of the great humanitarian artifacts of all time—and here we’ve done the work to make it real and we were about to give it to the world and now, instead, it’s 50 or 60 petabytes on disk, and the only people who can see it are half a dozen engineers on the project who happen to have access because they’re the ones responsible for locking it up.

Could have been great but it wasn't perfect for everyone so it was scraped because surely Congress would take up this noble quest...

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by thaumasiotes on Sunday July 03, 2022 @08:19PM (#31972186)
> I'll give you a hint: https://books.google.com/?hl=en

>> Search the world's most comprehensive index of full-text books.

I mean, the domain resolves, but that doesn't mean the product exists. You can run searches, but you're not allowed to see the results.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by IncRnd on Sunday July 03, 2022 @11:47PM (#31973308)
Why aren't you allowed to see the results?

I search, see the results, and then click on a result to view the applicable contents of a particular book. In the contents that get displayed, the search term I had entered is highlighted.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by iggldiggl on Monday July 04, 2022 @03:21PM (#31980760)
In some books it'll only show snippets, where

a) sometimes the snippet returned doesn't even contain the search term

b) you'll only get the snippets for the first three results, and that's that.

There might be even cases where the book is indexed, but the publisher has disabled even snippets, but I'm not sure about that. But a very limited snippets-only search instead of a full preview is certainly a thing.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by formerkrogemp on Sunday July 03, 2022 @07:14PM (#31971816)
> Could you take a moment to check it Google Books search still exists?

If only it still worked well enough to use. I use to use it every day, but how the mighty have fallen.

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by natrys on Sunday July 03, 2022 @07:05PM (#31971765)
Does anyone know of a good FOSS alternative for google books that could be self-hosted (for personal library)?
Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by MegaDeKay on Sunday July 03, 2022 @11:09PM (#31973119)
Does Calibre fit the bill? The program itself is great, but it supports a plugin system that really puts it over the top. One of them automagically strips off Adobe DRM for any book loaded in to it.

https://calibre-ebook.com/

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by natrys on Monday July 04, 2022 @07:09AM (#31975775)
Seems that with calibre-web[1] we can have nice web front-end. But it doesn't have full-text search inside the content of the books though, unlike google books.

[1] https://github.com/janeczku/calibre-web

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by MegaDeKay on Monday July 11, 2022 @11:12AM (#32057202)
Calibre 6 adds full text search, though not sure if that works from the web interface.

https://calibre-ebook.com/new-in/fifteen

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by mdp2021 on Monday July 04, 2022 @04:55PM (#31981669)
One product could be "Ambar: Document Search Engine"

https://github.com/RD17/ambar

> Ambar is an open-source document search engine with automated crawling, OCR, tagging and instant full-text search

> *Easily deploy Ambar with a single docker-compose file; *Perform Google-like search through your documents and contents of your images; *Tag your documents; *Use a simple REST API to integrate Ambar into your workflow

Re:Pirate Library Mirror: Preserving 7TB of books (that are ... by jrm4 on Sunday July 03, 2022 @08:38PM (#31972309)
But Google could never do it right. It strikes me as obvious that anything that's going try to be a genuinely modern library can't sidestep, or even "work with" the present capitalist+copyright regime. It will just have to be a fight.