Ask HN: Where can I host a large data-set (35GB) in public?

Posted by donohoe on Thursday August 19, 2010 @11:26AM
from the ask-hn dept.

I've been scraping various parts of a web site since 2008 and have accumulated an interesting data set (tracking changes every hour, and in some cases every 6 minutes) covering published RSS, markup changes etc.

The data is cool and I think people would have fun analyzing and making mashups. It consists of JSON and HTML fragments with a simple API of sorts to navigate:

  2010/index.js

The content of "index.js" is an array showing what months, within 2010, are available:

  [ "01" , "02" , "03" , "04" , "05" , "06" , "07" , "08" ]

Knowing the Months, you can then look up the days:

  2010/08/index.js
  [ "01" , "02" , "03" , "04" , "05" , and so on... , "17" , "18" , "19" ]

Within each day you can check the Hours:

  2010/08/19/index.js
  [ "00" , "01" , "02" , "03" , "04" , "and so on... , "09" , "10" ]

Within each hour, you can get the JSON file:

  2010/08/19/10/index.js

and so on and so on...

Freebase and Amazon Public Data Sets have been suggested but don't look like a good fit. Right now the file size of the 3 TAR files is about 35GB aprox.

Any further suggestions?

( 12 comments | 10 points | on HN )

Re:Ask HN: Where can I host a large data-set (35GB) in public? by paulgb on Thursday August 19, 2010 @12:32PM (#1617800)

How big is it if you gzip or bz2 it? I've found compression to be pretty effective for datasets like this.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by donohoe on Thursday August 19, 2010 @01:08PM (#1617908)

Good suggestion. gzip reduced one file from 2332528640 to 129440511

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by jerf on Thursday August 19, 2010 @01:44PM (#1617984)

Bzip may do yet better, though it will take longer. 7z may be worth a try too. You're getting into the range where you could just toss the whole thing up somewhere without much hassle. If you're not trying to make money, I'd just throw the whole data set out there on a torrent. Why take the time to write an access service when you don't even know what people are going to want to do with it? First thing a serious researcher is likely to do anyhow is just scrape the whole thing off your website, at much greater expense to you.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by oman121 on Thursday August 19, 2010 @12:54PM (#1617871)

Have you considered setting up a torrent of all the data thus far and then hosting only recent data? Then every month or so you can just create a new torrent with the data for that month and you will only be hosting a small amount of data at one time.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by there on Thursday August 19, 2010 @11:52AM (#1617690)

http://theinfo.org/

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by donohoe on Thursday August 19, 2010 @12:26PM (#1617786)

Doesn't look like they have resources on hosting the data but I'll dig around their discussion groups some more. Thanks.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by deno on Thursday August 19, 2010 @12:59PM (#1617885)

You can try using Amazon S3 (storage) + CoralCDN (distribution). Just use S3's permission system to restrict access other than through CoralCDN and you should have cheap and reliable storage that won't drive your bill crazy.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by ddispaltro on Thursday August 19, 2010 @12:49PM (#1617854)

Would infochimps.com be useful for this?

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by kellyjosephc on Sunday August 22, 2010 @09:24PM (#1625602)

Yes, we'd be love to host this dataset for free. Send an email to [email protected] and we can arrange the transfer.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by d_r on Sunday August 22, 2010 @11:07PM (#1625736)

Sorry! Accidentally clicked the "down" button on your comment and there's no way to undo it.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by aw3c2 on Thursday August 19, 2010 @01:01PM (#1617891)

If what you did is legal and you are legally able to share it, archive.org would be a good choice I guess.

[ Parent | on HN ]

Re:Ask HN: Where can I host a large data-set (35GB) in public? by hyuen on Thursday August 19, 2010 @12:53PM (#1617865)

make a torrent

[ Parent | on HN ]