The US Library of Congress has put 25M items free online

Re:The US Library of Congress has put 25M items free online by lgierth on Sunday May 28, 2017 @06:05PM (#14436291)

I added the raw .gz files to IPFS when Library of Congress announced this last week: https://github.com/ipfs/archives/issues/152

It's slightly more than 100GB and here it is: https://ipfs.io/ipfs/QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtH...

(Note that the filesizes in the directory listing are all wrong -- that's the original index.html from loc.gov/cds/downloads/MDSConnect/)

This makes it a lot easier to use this dataset at e.g. hackathons, where a lot of people would simultaneously pester that LoC server, which already seemed pretty bandwidth-limited on its own when I downloaded the files.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by lgierth on Sunday May 28, 2017 @06:12PM (#14436319)

You can pin it: `ipfs pin add QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

Or list it: `ipfs ls QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

Or copy it into your local filesystem `ipfs get QmWSzgkftVrkh2859bGT44ahzoqcGhFkjsrQUtHen9hVw9`

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by toomuchtodo on Sunday May 28, 2017 @07:13PM (#14436509)

Did you push them into the Internet Archive yet? If not, going to grab a beer and start iterating through your IPFS objects.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by lgierth on Sunday May 28, 2017 @07:20PM (#14436525)

Go for it! :) I've never pushed anything to IA so far to be honest.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by toomuchtodo on Sunday May 28, 2017 @07:24PM (#14436537)

For future reference then! https://github.com/jjjake/internetarchive

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by voltagex_ on Sunday May 28, 2017 @10:49PM (#14437120)

If someone wants to pay for / donate 100GB worth of bandwidth on a VPS somewhere, I'll do it.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by tty7 on Sunday May 28, 2017 @11:12PM (#14437173)

scaleway: €2.99/m 200Mbit/s Unmetered bandwidth

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by voltagex_ on Sunday May 28, 2017 @11:31PM (#14437240)

Okay, so who's going to spot me 3 EUR? ;)

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by toomuchtodo on Monday May 29, 2017 @12:04AM (#14437348)

Could TransferWise you the $3.35USD. That work?

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by voltagex_ on Monday May 29, 2017 @12:45AM (#14437448)

Having never done this before I've got to have a look at the internetarchive tool first, but yes, that would work (I'd hate to take your money then not be able to deliver).

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by voltagex_ on Monday May 29, 2017 @01:00AM (#14437481)

This thread is getting too long; how do I contact you?

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by tombrossman on Monday May 29, 2017 @08:19AM (#14438720)

This amount of usage falls well within most VPS companies' free trial / promo codes offerings and should cost you nothing. Use a throwaway email account and drop it after a month.

AWS will give you a whole year if you haven't tried them yet and the other popular VPS companies (DO, Linode, etc.) all will give you at least $10 startup credit. This is probably simpler and faster than figuring out how to receive <$4 from some random internet commenter.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by voltagex_ on Monday May 29, 2017 @08:59AM (#14438862)

Yes. This isn't the first time I've tried to contact toomuchtodo.

Anywho, the amount of data is actually ~19GB which is well within what I can upload with my home connection. Unfortunately the ia tool is failing for me: https://github.com/jjjake/internetarchive/issues/176

Also, it's not really about the $3, more that a tonne of "$3" projects really add up over a year or so.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by toomuchtodo on Monday May 29, 2017 @12:28PM (#14439889)

Email sent.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by voltagex_ on Tuesday May 30, 2017 @09:28PM (#14449878)

I have been advised that the data already exists on archive.org.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by toomuchtodo on Thursday June 01, 2017 @08:44PM (#14466492)

Email me back. Would still like to buy you a beer for your troubles.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by lgierth on Monday May 29, 2017 @07:20AM (#14438553)

Oops, it turns out I somehow ended up with decompressed files, that's why the filesizes are so far off. I'm creating a new fixed dump.

[ Parent | on HN ]

Re:The US Library of Congress has put 25M items free online by lgierth on Tuesday May 30, 2017 @10:34AM (#14445645)

The correct dataset (with gzip compression) is here: https://ipfs.io/ipfs/QmcvfB6pAqUfTnuAK8zFKVxbdhopnBPveJrDcy1...

[ Parent | on HN ]