For an invite, please send me an email at [email protected] or go to: https://openlibrary.org/volunteer
# APIs & Data Dumps
- https://openlibrary.org/developers/api
- https://openlibrary.org/dev/docs/api/books
- https://openlibrary.org/developers/dumps monthly data dumps for if you need bulk access and the APIs are not enough.
# Spread the word
Also, if you want to help raise awareness of this resource, please help us get the word out on twitter!
1. https://twitter.com/openlibrary/status/1338185940469051392
2. https://twitter.com/openlibrary/status/1338186553915367425
# Issues
Thank you all for helping us discover some issues with our goodreads importer and search (recently migrated to Python3 + thanks @cdrini et al for these fast bug fixes! If you notice an problem, please help open an issue here: https://github.com/internetarchive/openlibrary/issues/new/ch...
# Learn More
- https://archive.org/details/openlibrary-tour-2020/openlibrar... if you want to learn more about Open Library, here's a short intro vid.
- https://github.com/internetarchive/openlibrary if you want to follow on github.
How do I go about claiming my author page? The current book listed there has been officially "retired" for over a decade now, and I have plenty of other books that I'd be happy to add.
As a project of the non-profit Internet Archive, having a trusted catalog is pretty paramount to what we're trying to accomplish and Aaron Swartz's original dream for an Open Library.
I've created an issue for this feature (as it's something we've discussed doing for a while). I'll also be adding this to our upcoming Tuesday 2021 roadmap discussion.
https://github.com/internetarchive/openlibrary/issues/4263
Thanks for raising this
Many of our librarian processes & FAQs are detailed here: https://openlibrary.org/librarians
Most data on Open Library is publicly editable by members. In this case, author merging is a capability that only folks who have been added to our Librarians usergroup can access because the process of reverting an accidental merge is quite time consuming (and so we have some training in place).
If you tweet @ our lead community librarian (http://twitter.com/seabelis) she can likely help you make this fix! Also, if you're interested in helping us make changes like this, we can invite you to our slack channel :)
Would anyone be interested in having an instant search experience for this books dataset, like the one I built for the 2M recipes database posted on HN earlier this week: https://news.ycombinator.com/item?id=25365397
ol_dump_latest.txt.gz 0%[ ] 28.73M 115KB/s eta 21h 54m[1] https://www.goodreads.com/review/import (export)
(granted, running on older versions of firefox, but still thought it might be a bug you would like to be aware of.)
If you use the https://openlibrary.org/volunteer page and click the Librarian link, this will send an email to Lisa, our head community librarian and we can help you access our slack channel and request access to our librarian features. We also have an optional weekly call for folks to raise issues and questions (e.g. "I want book series!")
More about librarianship @ Open Library: https://openlibrary.org/about/lib
If you hit any issues with this process, feel invited to send me an email to [email protected].
Ideally, it should probably bring you to the login page and (once you login) bring you back to the import page.
We'll see about creating an issue for this, thanks!
Although, I just tried importing my Goodreads export into Open Library and I get the following "Internal Error":
> Hmm... > Sorry. There seems to be a problem with what you were just looking at. > We've noted the error xxxx-xx-xx/yyyyyy and will look into it as soon as possible. Head for home?
Anyone else facing this issue?
I can confirm it works with at least my import.
Edit: Alas, some errors regarding "Book not in collection" and "No ISBN". But I think I can probably see if there is a way to contribute those books to the repository.
https://github.com/internetarchive/openlibrary/issues/new/ch...
The importer is a new project (as of August this year) and I think there's a lot of opportunity for improving it (including importing missing records on demand).
Other hat tips: I also want to call out the great work of https://inventaire.io, another great project in the space which uses wikidata, as they also have a goodreads import which we're learning from. Another project I'm excited about (which has goodreads import) is beta.thestorygraph.com. Not sure the status of its source code but I heard an interview from the founder and she and her work so far seems great.
Books pre ~1973 don't have ISBN. And goodreads covers far more material than just modern material.
The current Goodreads importer makes a big/unfortunate tradeoff of trying to have something which works for the majority of material on people's goodreads lists.
It doesn't preclude further efforts -- Open Library knows about millions of books that don't have ISBN. So if anyone wants to help us improve our importer (or at least register their interest by creating an issue which calls out what other identifiers should be considered -- e.g. LOC, etc) that would be very helpful.
# To open an issue:
https://github.com/internetarchive/openlibrary/issues/new/ch...
# Here's the code for the importer
https://github.com/internetarchive/openlibrary/blob/892adebb...
Having some free time on my hands I will take a look in the next few days. Looking for an alternative to GR and for something that tickles my interests.
Python, Search and literature. And open source.
e.g.
https://openlibrary.org/search.json?q=
https://openlibrary.org/books/OL28018730M.json
This is one of the best implementation decisions that Aaron Swartz, Anand Chitipothu, et al made when starting Open Library.
https://www.librarything.com/privacy
The site's also been around since 2005, so, that speaks well to both its longevity and its integrity. Of course, that's not to say they couldn't be acquired tomorrow and paywalled, but I think it unlikely.
...And they even have an API!
Made me realize how many booksellers(especially used) put proprietary barcode stickers on top of the 'actual' barcode stickers.
And yeah, searching needs some work! That's on my task list for this month. Just this Friday I spent most of my day working on updating our search engine, Solr, from 3.6 to 8.7 (wip!). But search is a _BIG_ pain point. We're a small team with a big long list of things to do, but we are making progress! This year we updated to Python 3, switched most of our production environments to docker-based for easier deploys and to give open source contributors more control of production infra, added reading history stats for users, added a new interface for exploring books, worked on a novel recommendation system, added text selection to the online BookReader for public domain books, added GoodReads importing, grew our community, added the ability to search by classification, and much, much more (you can see highlights from our year here: https://github.com/internetarchive/openlibrary/issues/3891 ).
There is still _definitely_ a lot to do, but I think the biggest reason worth using/contributing to Open Library is likely its open source community. Anyone can jump in and help make improvements to the system (as they very often do!). Personally, I think it's more likely that a system with a community will survive/flourish than one maintained by a single person (I also wondered whether I should just create my own before contributing to and now working on Open Library!). And there are also loads of different tasks associated with a site like OL, which would be impossible for me to do if I was going it alone.
If you would be interested, checkout the GitHub repo: https://github.com/internetarchive/openlibrary . It's very active, and you can get an idea of how we work :)
How do you even get an ISBN for a candle?
There is an explanation under "About the author" section :
This "author" was created to segregate those items which have ISBNs but are not actually books. For more information, see the manual and/or start a thread in the Librarians Group.
When an item which is not a book is imported via ISBN into Goodreads, it does no good to delete it: the item will only be re-imported as long as it remains on the feeder site. (Often these are book-related items which are assigned ISBNs by book publishers so that they can be tracked through their book systems.)
“Why not? What’s the big deal?” said someone who has never worked with garbage data.
"paying $2-10k+ to add an International Standard Book Accessory Number field to their software will never get approved for a bookmark that works as a compass, or a promotional Harry Potter bookend set."
Tough to disagree.
There's very little control. E.g I just published a novel,and Amazon did not in any way validate that the ISBN I have them actually belonged to me - I had not yet registered the book data, so what I told them would not have matched anything they might have looked up.
In this case, based on the reviews on Amazon, it looks like someone changed the description of an existing product, and that the ISBN probably actually belongs to a book.
ImportBot scrapes Amazon.com, matches product id with ISBN10 (which can be converted into ISBN13 without anything else IIRC), imports product as book.
Yes. There clearly only needs to be one way to identify things, so commonly larger systems just incorporate the smaller ones wholesale. The 13-digit system just incorporates the prior 10-digit system for International Standard Book Numbers with a 978 prefix called "bookland" (most prefixes in this system are geographic, but of course books aren't really from one single geographic region, so they're from "bookland") and adds a new set of possible codes. It also incorporates the entire 12-digit American "UPC" system.
Several other (less well known) systems were gobbled up the same way as "bookland", just allocating them imaginary geographic regions in the 13-digit system.
There's actually a fourteen digit system, but the lead digit tells you about how many of something are bundled e.g. so a distributor can distinguish a truck full of Pepsi cans from just one case or a single can in terms of things you can order. Lead digit 0 means "single" so if you know the 10 digit ISBN you can not only make a 13-digit EAN for that, you can make the 14-digit GTIN that means "just one of this book" which in most cases would be what you want.
Great points here
1. The banner happens last month of the year (Wikipedia being the perfect analog). Yes, there are mixed feelings and it's not the world's best experience :P
2. Our entire data set is available to download as in bulk https://openlibrary.org/developers/dumps because we'd love to see a decentralized p2p version
3. https://github.com/mouse-reeve/bookwyrm Mouse who used to work @ Internet Archive has a decentralized version of Open Library (Bookwyrm) and it's worth checking out.
4. For the last 5 or so years the Internet Archive has been cultivating a dweb/dapp community and integrating with IIIF, Dat, IPFS, gun, bittorent, webtorrents, and others and hosting regular summits and meetups https://blog.archive.org/2018/07/21/decentralized-web-faq/
5. The wayback machine is an interesting case study: it turns out, incentive structures (even things like FIL/filecoin) haven't been able to perfectly crack the nut on getting folks interested enough to preserve the whole wayback machine. There's petabytes of material and there's a powerlaw about what people care about today. Internet Archive realized what we care about today may not be the same as tomorrow, and so there's a cost eaten (the incentive comes from economies of scale generated by intrinsic desire rather than $). And in a way, this centralized solution (economies of scale) IS the solution a community came up with. It has flaws and advantages (tradeoffs), such as centralized points of failure, and I think the archive would be (and has been) ecstatic to explore improving these opportunities.
There's not an API directly, but https://www.howlongtobeatsteam.com looks promising. It's got an API that you can use to cache your (or all) games. Zapier might be able to do it as well w/ this scraper: https://www.npmjs.com/package/howlongtobeat (disclosure, I work at Zapier).
I used to have a CLI that would trigger a Zapier webhook that would use [this CLI app](https://github.com/xavdid/zapier-igdb) to populate data into Airtable, but I actually moved the whole process to [Airtable Apps](https://airtable.com/marketplace) since I could customize the UI a little more. I've been meaning to open source it, but it's pretty customized so I'm not sure if it'd be widely useful.
I'm curious to see where you end up with this though! It's something I've put a lot of thought into.
Time to beat is a very difficult metric. Does a Visual Novel take 10 hours to beat, or 2 minutes? I'd argue the latter, but I know plenty of people would disagree with me.
"Alright," says the webadmin, "I'll just implement a voting system and take the average". But now they'll be claiming that it takes 5 hours and 1 minute to beat, which everyone disagrees with.
The best system I've seen is SteamHunters' "Shortest/Median/Average", which gives you a pretty reliable indicator of what's possible, what's typical, and how much of a difference there is between optimal and casual play, respectively. It's by far the most comprehensive dataset out there, but unfortunately only mono-platform so probably not too useful for sites like igdb.
I haven't shared this yet -- it's more for the community, but I've tried to address various questions from the community and distill answers + resources for Open Library here:
https://blog.openlibrary.org/2020/12/13/importing-your-goodr...
Tried my best to include others players in the space (wikidata, inventaire, bookbrainz, worldcat, bookwyrm) who are doing great work and pay respects to readng, storygraph and other innovative services which are breaking onto the scene.
I get that Open Library doesn't have as much data as GoodReads, but I wish it would show me the data it couldn't import so I could add it manually to Open Library's data store.
Nevertheless, I love the idea and I'll be opening bug reports and maybe code contributions if something looks easy enough.
https://github.com/internetarchive/openlibrary/issues/new/ch...
If you tag @tabshaikh who helped implement the importer and me @mekarpeles we can make sure it gets triaged and tagged correctly this week :)
Dissatisfied with how slow and clunky Goodreads is I actually thought about making my own (albeit much simpler) version of Goodreads to keep track of my reading habits. I often dig through Goodreads to find books or authors I can't remember the names of -- and Goodreads isn't great for that.
Open Library actually provides the missing piece. The fact that they offer bulk downloads also makes it easier to be a good internet citizen and not send tons of API traffic their way.
Looks like I'll have to set up a monthly donation. I'd really like see openlibrary succeed.
https://openlibrary.org/developers/dumps
The project is also open source, and you can find the code (and contribute!) on GitHub: https://github.com/internetarchive/openlibrary
After some looking, there are Some private databases with millions of # but no official site. Eg
For books in the collections of large libraries (like the LoC) there will be a public catalogue entry with the ISBN attached, but they don't assign it.
There were also a lot of books published before ISBNs were created, and not every book has an ISBN attached even to this day.
https://en.wikipedia.org/wiki/International_Standard_Book_Nu...
> WorldCat is a union catalog that itemizes the collections of 17,900 libraries in 123 countries and territories[4] that participate in the OCLC global cooperative. It is operated by OCLC, Inc.[5] The subscribing member libraries collectively maintain WorldCat's database, the world's largest bibliographic database.[6]
My primary interest is in recording my thoughts on books and stories I've read, so a review site is what I'm looking for. This is more challenging for short stories as they don't have a ISBN, may appear online, or in a fiction magazine or in an anthology.
So, I may put down my thoughts on short story X that I read in book Y, but if I look up book Z that also features that short story, my thoughts on the story would also appear there.
In short, I'm looking for a site that also records short stories like the ISFDB [1], but allows users to add reviews.
So far, I haven't found one. I'm now putting down my notes on short stories in a Zotero database.
One web page for every book ever published. It's a lofty but achievable goal.
To build Open Library, we need hundreds of millions of book records, a wiki interface, and lots of people who are willing to contribute their time and effort to building the site.
To date, we have gathered over 20 million records from a variety of large catalogs as well as single contributions, with more on the way.
Open Library is an open project: the software is open, the data are open, the documentation is open, and we welcome your contribution. Whether you fix a typo, add a book, or write a widget--it's all welcome. We have a small team of fantastic programmers who have accomplished a lot, but we can't do it alone!
---
They also seem to provide an API[2].
[1] https://openlibrary.org/about
[2] https://openlibrary.org/developers/api