Re:1.3B Worldcat scrape and data science mini-competition
by throwawaykasc on Thursday October 05, 2023 @02:42AM
(#37775532)
Check out z39.50, you can use it to search and pull data from most national and university library catalogs. But also be prepared for some code archaeology along the way, as the protocol is around 50 years old.
https://en.wikipedia.org/wiki/Z39.50 https://z-brary.com/ https://github.com/asl2/PyZ3950/blob/master/PyZ3950/zoom.py
Re:1.3B Worldcat scrape and data science mini-competition
by gmcharlt on Thursday October 05, 2023 @01:34PM
(#37781456)
Depending on what you're looking for, a _lot_ more is being published by the Library of Congress as Linked Data nowadays, including LoC classification and subject headings. Check out https://id.loc.gov/.
Re:1.3B Worldcat scrape and data science mini-competition
by dredmorbius on Thursday October 05, 2023 @02:38PM
(#37782331)
[delayed]
Re:1.3B Worldcat scrape and data science mini-competition
by gmcharlt on Thursday October 05, 2023 @06:24PM
(#37785032)
https://id.loc.gov/download/ offers bulk downloads of many of the vocabularies, including the subject headings. In addition, individual records of various types can be search for.
Re:1.3B Worldcat scrape and data science mini-competition
by dredmorbius on Thursday October 05, 2023 @07:08PM
(#37785367)
Thanks.
That seems to have Subject Headings, but not the Classification, unless I'm blind (a fairly high likelihood).
I've spent quite some time poking around the LoC website over the years without finding anything more accessible than the PDFs of the Library of Congress Classification. Are you aware of that being generally available?
Re:1.3B Worldcat scrape and data science mini-competition
by gmcharlt on Thursday October 05, 2023 @09:31PM
(#37786275)
Near as I can tell, the best that's available is starting with the class(es) you want (e.g., https://id.loc.gov/authorities/classification/H.html for social sciences) and crawling recursively to grab whatever serialization of the RDF you want to consume. There's no SPARQL endpoint for id.loc.gov, alas, but it beats resorting to the PDFs.
Re:1.3B Worldcat scrape and data science mini-competition
by dredmorbius on Thursday October 05, 2023 @09:46PM
(#37786356)
Thanks.
(I've ... worked a bit with LoC classification and subject headings data, of which publicly-available data are only available in PDF or wordprocessing (MS Word or Wordperfect, if memory serves) formats. Which is ... somewhat unfortunate.)