Show HN: Goodreads Data Pipeline

Re:Show HN: Goodreads Data Pipeline by drusepth on Thursday February 27, 2020 @12:24PM (#22435406)

I've been working on a competitor for a while now, and the hardest part of replicating functionality is the data. OpenLibrary is probably the best source for book metadata online, but even their library dumps are riddled with mistakes that manifest in weird ways as you start building your own library. The Goodreads site sucks, but they've got surprising data quality that I don't think anyone else has; and they have a super restrictive data policy so you can't repurpose book data, reviews, shelf data, etc, even when users auth with a Goodreads account.

It's a small moat, but definitely penetrable with more than a little effort.

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by barkingcat on Thursday February 27, 2020 @12:45PM (#22435633)

The good data quality is actually an artifact of humans being involved in every step of the cataloging process. There's a large group in goodreads called the GoodReads Librarians, and that group has around a hundred thousand dedicated people who go through and flag anomalies, correct titles and indexes etc

Book publishers or people who've worked in book publishing will know that the book database is one area you don't want to mess with unless you know what you are doing. ISBN's are not the be all and end all of the story, and when you start taking into account special editions, covers, ebook editions, language translations, you'll start to realize that the Book Catalog system going back in history, including Dewey decimal system is a marvel of human achievement.

Of course establishing a good quality index is going to take work. People often forget that quality take human work and effort.

EDIT: I lied. I changed the number from my original estimate of a "few hundred" to "hundred thousand". The Goodreads Librarians group has 103718 members as of when I just peeked now - so it's actually a large number of humans submitting fixes to their catalog.

https://www.goodreads.com/group/show/220-goodreads-librarian...

If you take a look at the kind of discussions taking place, those are the kinds of things any competitor to Goodreads needs to know about.

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by johnmaguire2013 on Friday February 28, 2020 @12:23AM (#22440706)

I wonder if a partnership with Kobo, or even better Nook (Barnes & Noble) could help solve both the data problem and the issue with Kindle integration - while potentially bolstering the e-reader that integrates with it as well.

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by Kihashi on Friday February 28, 2020 @09:13AM (#22442868)

I for sure would take that sort of integration into account when looking at a new ereader. Although the primary thing I'd be looking for is library integration a la OverDrive.

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by spullara on Thursday February 27, 2020 @02:16PM (#22436722)

Facts are not copyrightable and scraping has been determined to be legal. IANAL, but I'm not sure law would protect them from the factual metadata about books being repurposed.

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by jjeaff on Thursday February 27, 2020 @02:37PM (#22436916)

Not being illegal wouldn't protect you from a crushing lawsuit though. Especially since the details likely vary (linked in data was publicly accessible, not sure if Goodreads requires a login).

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by hombre_fatal on Thursday February 27, 2020 @12:40PM (#22435592)

Demanding perfect data is a waste of time that will let you procrastinate your product indefinitely.

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by maw on Thursday February 27, 2020 @05:45PM (#22438799)

If you're aiming for "like X, but for people who are actually interested in the market X supposedly serves" then maybe this isn't true.

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by trollied on Thursday February 27, 2020 @12:35PM (#22435540)

You could nick the scraping code from Calibre...

[ Parent | on HN ]

Re:Show HN: Goodreads Data Pipeline by dewey on Thursday February 27, 2020 @02:44PM (#22437000)

The scraping part is probably not the complicated portion of the endeavor.

[ Parent | on HN ]