Re:How to write a crawler
by rb2k_ on Monday September 30, 2013 @08:29AM
(#6469155)
What would a filesystem do better than a DB in that case?
After all a filesystem is just a database who's primary keys are mostly filenames.
Most of them use the same data-structures (e.g. b+trees) and especially the newer ones turn into copy-on-write systems like e.g. CouchDB
I guess if you want to have backups of that table, you probably would like to keep it at a manageable size, but why not just save the html blobs in a separate table?
Most of them use the same data-structures (e.g. b+trees) and especially the newer ones turn into copy-on-write systems like e.g. CouchDB
I guess if you want to have backups of that table, you probably would like to keep it at a manageable size, but why not just save the html blobs in a separate table?