News for hackers. Stuff that matters.
Re:Why MongoDB is a bad choice for storing our scraped data by dkhenry on Tuesday May 14, 2013 @09:44AM (#5704740)
you lost me here

""" Ordered data

Some data (e.g. crawl logs) needs to be returned in the order it was written. Retrieving data in order requires sorting which is impractical when the number of records gets large. ""

it requires _indexing_ and is quite feasable as I do it every day with stock ticker logs ( also required to be retrieved incrementially )

There are a few other flags that make me wonder about the exact limitations you found, but I will be anticipating your follow up post to see what your fix was since some of those issues are very common.

Re:Why MongoDB is a bad choice for storing our scraped data by mayank on Tuesday May 14, 2013 @09:58AM (#5704831)
No kidding. Without details, it really sounds like the author is a bit clueless.

He mentions the lack of joins, but doesn't say a word about Mapreduce.

"MongoDB needs to walk the index from the beginning to the offset..." You don't "walk an index". It's an index.

"Too many databases" sounds a little suspicious. Why not add an indexed field to partition records?

Complaining about a lack of schema, transactions and triggers? Really? Did you read the docs at all before starting?

MongoDB is not without its problems, but friend, I think you wanted either Postgres or Hadoop.

Re:Why MongoDB is a bad choice for storing our scraped data by PavlovsCat on Tuesday May 14, 2013 @10:10AM (#5704915)
You don't "walk an index". It's an index

If you have an address book, you don't have to walk through the city to find an address, but you do have to look through your address book in some way or other. Of course, you can have an index of the index ("C starts at page 7"), but then you have to look through the index of the index.

Re:Why MongoDB is a bad choice for storing our scraped data by shane42 on Tuesday May 14, 2013 @10:20AM (#5704993)
The problems with pagination are explained better on this SO post: http://stackoverflow.com/questions/7228169/slow-pagination-o... Mongo docs used the "walk" terminology
Re:Why MongoDB is a bad choice for storing our scraped data by mayank on Tuesday May 14, 2013 @10:21AM (#5705000)
Not sure what you're getting at. "Walk" usually means a sequential scan. An index is sorted, so you can binary search.
Re:Why MongoDB is a bad choice for storing our scraped data by PavlovsCat on Tuesday May 14, 2013 @10:41AM (#5705138)
See other reply. If the docs say MongoDB "walks", it's hilarious to complain that someone obviously didn't read the docs for saying "walk", too.
Re:Why MongoDB is a bad choice for storing our scraped data by monstrado on Tuesday May 14, 2013 @10:27AM (#5705046)
It looks like they went with HBase for the replacement which means they can scan an index range, keys are in lexicographical order, so it makes it pretty easy to scan over a series of data with a single RPC call.
Re:Why MongoDB is a bad choice for storing our scraped data by monstrado on Tuesday May 14, 2013 @10:08AM (#5704903)
With a database like HBase, it's already ordered lexicographically which makes it easy to grab a range of data in the order it was written in. You could have a key design like <reverse_domain>-<epoch> which would allow quick scans over large amounts of data. ie..Scan from <object_id>-1368536860 to <object_id>-1368540450

HBase is multidimensional though, which allows you to keep N numbers of versions of a cell. By default you will get the latest version of the cell back, but you could also opt to receive N versions back, which is useful for time series use cases.

Re:Why MongoDB is a bad choice for storing our scraped data by swah on Tuesday May 14, 2013 @11:19AM (#5705455)
The last time I looked there weren't many resources explaining how to design NoSQL databases (how to compose your keys, when to avoid normalization, etc). Has this improved?
Re:Why MongoDB is a bad choice for storing our scraped data by monstrado on Tuesday May 14, 2013 @01:16PM (#5706357)
Well, NoSQL databases is a pretty broad term. Not all NoSQL databases are created alike, for example, MongoDB is a "document orientated database" where as HBase is a "column-oriented store" based on the Google BigTable whitepaper.

As far as I know, key design is not an important aspect with MongoDB but I could be mistaken. HBase has a pretty awesome book (http://www.hbasebook.com/), which has an entire chapter dedicated to key design. Lars (the author) also has a pretty in depth 1 hour video on key design (http://www.youtube.com/watch?v=_HLoH_PgrLk).

HBase is pretty widely used, I've seen 1200+ node clusters running production tables.

Re:Why MongoDB is a bad choice for storing our scraped data by shane42 on Tuesday May 14, 2013 @10:45AM (#5705169)
Take the example of crawl logs. Each log entry has a log level, timestamp and message. Typical use would be to view all ERROR (or higher) log levels, show all entries with a specific text in the message, or download the entire log. All of these should be in timestamp order. It's a shame that natural order is not insert order for non-capped collections.

It's a good point that some of this can be achieved with indexing, I should have given more details in the blog post.

Re:Why MongoDB is a bad choice for storing our scraped data by dkhenry on Tuesday May 14, 2013 @04:45PM (#5707979)
Right for that example ,not knowing the specifics, I would index on timestamp and then I could sort by timestamp which once indexed should be a relativly fast operation. I could even go one step better and make the _id a construct of { <timestamp>,<loglevel>,<fuzz> } Then Inserting would be done in order and I would get a magic index for free (depending on if I often query by loglevel I might leave it out) This gives locality of timestamps and helps keep "Hot" sections in memory.
Re:Why MongoDB is a bad choice for storing our scraped data by leothekim on Tuesday May 14, 2013 @10:09AM (#5704912)
Also, mongo has natural ordering which would do what the author wants without sorting.
Re:Why MongoDB is a bad choice for storing our scraped data by berito on Tuesday May 14, 2013 @01:02PM (#5706248)
No mongo natural order is just order on the disk. It is not always in reverse insertion order