News for hackers. Stuff that matters.
Re:1.3B Worldcat scrape and data science mini-competition by hedora on Wednesday October 04, 2023 @12:08PM (#37767225)
I looked into using Dewey Decimal for a hobby project. OCLC has a de facto monopoly on it due to the Worldcat database. They're a non-profit, but they're supported by having libraries pay a subscription fee for Worldcat.

Back when OCLC was founded, the idea that people would want to have a copy of a card catalog for personal use was laughable, so I'm sympathetic to the people that set up their funding model. It's far cheaper for a library to subscribe to Worldcat than to hire a team to maintain such a database, so it created a win-win situation.

However, keeping the world's books' metadata a secret (and leaving control in the hands of a monopoly) is an anachronism.

It's well past the time when someone (such as an international coalition of Libraries of Congress) should figure out how to sustainably fund OCLC while also releasing their work into the public domain.

Re:1.3B Worldcat scrape and data science mini-competition by actuallyalys on Wednesday October 04, 2023 @01:10PM (#37768301)
I suppose one way to do it would be to allow patrons of subscriber libraries to access the database dumps and API.

The downside is that would still make it harder than necessary to access and leave some people out. The upside is that it's not that much of change from their existing model. I'm sure there would also be concerns about database dumps being shared publicly, although Anna's Archive has already released their entire database, and I suspect most people who would pay for formal access wouldn't use an authorized copy. Ultimately, I suspect OCLC would still be resistant to this change, as it would feel like a huge shift, even if I'm not sure it would change much from their perspective.

Re:1.3B Worldcat scrape and data science mini-competition by mgr86 on Wednesday October 04, 2023 @03:23PM (#37770286)
I have been told that my organization developed a system, HABS, that pre-dated OCLC [0]. That OCLC used this system as an inspiration. However, I cannot confirm this. Closest I can do is to find a footnote that Thanks Fred Kilgore, the founder of OCLC [1]. I should reach out to Koh, a friend of a friend, while she is still alive to confirm the story. Nevertheless, we have a collection of punch cards in a dusty room in an attic that was once the HABS system. I think it is a pretty fascinating legacy, and I wish it was better preserved.

[0] https://journals.sagepub.com/doi/pdf/10.1177/106939716900400... [1]https://journals.sagepub.com/doi/abs/10.1177/106939717300800...

Re:1.3B Worldcat scrape and data science mini-competition by gmcharlt on Wednesday October 04, 2023 @04:29PM (#37771264)
Neat! I hope you can learn more from Koh.

I know a bit about Henriette Avram and her work at LoC developing MARC, but it of course makes sense that other libraries were thinking along similar lines at the time.

Re:1.3B Worldcat scrape and data science mini-competition by empthought on Wednesday October 04, 2023 @04:58PM (#37771688)
OCLC is a nonprofit membership cooperative and would argue that it itself is that international coalition of national libraries and archives.
Re:1.3B Worldcat scrape and data science mini-competition by IKantRead on Wednesday October 04, 2023 @05:11PM (#37771856)
OCLC is a parasitic company masquerading as a "membership cooperative".

Libraries (often publicly funded) produce all the work, OCLC claims ownership of the results of that work, Libraries pay to get it back (but they do get a discount if they contribute).

The only reason OCLC continues to exist is because libraries don't have the support or resources to fight them. It's very similar to the Elsevier issue in academic publishing, but OCLC does a better job with PR.

Re:1.3B Worldcat scrape and data science mini-competition by empthought on Wednesday October 04, 2023 @05:34PM (#37772135)
I mean you’ve clearly read Aaron Swartz’s diatribes, but you also clearly have no clue about OCLC’s business model. The catalog data is intellectually interesting, but the value is in the holdings data and more importantly the interlibrary loan service it enables.

OCLC is exactly what happens when the libraries want to avoid another EBSCOhost or Proquest situation with ILL.

Re:1.3B Worldcat scrape and data science mini-competition by dredmorbius on Wednesday October 04, 2023 @08:45PM (#37773623)
What were the EBSOhost / ProQuest situations, if you don't mind?
Re:1.3B Worldcat scrape and data science mini-competition by partytax on Wednesday October 04, 2023 @08:13PM (#37773404)
The Melville Decimal System popularized by https://www.librarything.com may be of interest.

Here's an explanation from their footer: "Although Dewey invented his system in 1876, recent editions of his system are in copyright. LibraryThing's Melvil Decimal System is based on the classification work of libraries around the world, whose assignments are not copyrightable. The "schedules" (the words that describe the numbers) come from a pre-copyright edition of his system, John Mark Ockerbloom's Free Decimal System, and member contributions."

Re:1.3B Worldcat scrape and data science mini-competition by mannyv on Wednesday October 04, 2023 @10:19PM (#37774201)
One strange thing about the Dewey Decimal System is that it's copyrighted and libraries pay a fee to use it.

It came out when the Library Hotel in NYC used some of their notation as their room numbers and were sued. Everyone was WTF?