I have 6.000.000 RSS articles in a DB. What can I do?
I created a website a couple of years ago (thatstoday.com) where members can add RSS feeds and read them in our "news reader". The years went by and we've indexed all feed articles in our db (and in Lucene).
Now I'm not sure what to do with all the data since the db it's getting large and difficult to handle.
Should I delete it or create a new service? What do you guys suggest?
Thanks, Rob
(
37 comments
| 30 points
| on HN
)
Did your store the raw content in the database? If so, you might consider writing files instead. These blobs, like pictures, are better stored as files. In your database, you should keep relational data: url, time, adder, etc, properly indexed (probably by adder, time and maybe keywords). Then a 6M rows table is quite a small thing for any RDBMS (if your SELECTs are filtering on indexed columns).
At least you'll have fun (YMMV)...
[1] http://www.icwsm.org/[2007-2011]
Hmm.. I know a few peeps at MS/Azure.. maybe they can sponsor..I'll get back on this one.
I hope this'll get a follow up, I'm sure someone can think of something awesome to do with it.
If you decide to open source it (copyrights?), use Bittorrent: it's the perfect tool for this job.
I know I would be interested in downloading it, and just poking it for interesting stats...