The comment raised an interesting way of thinking about the problem that many developers would not consider, and while it might have had a fair bit of religious zeal, that is very common here and not worthy of comments like "unbelievable waste of a comment" and "your comment should be deleted".
The comment raised an interesting way of thinking about the
problem that many developers would not consider
Fantastic! Why can we not have an actual discussion about scaling MongoDB when the post is about scaling MongoDB! You want to have a conversation about different database engine solutions to the problem, go start a post about that. But this post was about MongoDB scaling. I may have been critical of the GP here, but you fail to appreciate the need and desire to keep HN to the level it should be.I think your comment only needed to be a polite "can we keep the conversation on the topic of MongoDB scaling" rather than a quite personal attack on the author.
Also, personally, I thought the main point raised about viewing the database choice from a business rather than development point of view was interesting and well worth discussion.
Because it might be more helpful to think about scaling your data needs than scaling with only a certain product.
Just to give you an idea where SFJulie1 is coming from:
http://www.reddit.com/r/programming/comments/2pwkcl/help_nee...
I think mikegioia was perfectly reasonable in calling out this comment, it is trash and he was right to recognize it as such.
It is hardly arguable that there is a segment of the developer population that has a religious zeal for their "stack" or against others (just pick any random topic on functional programming or PHP for examples of each). It is entirely appropriate to discuss and address that in topics like this on HN, in my opinion.
I suppose one might infer from it notion of a default assumption that developers are all zealots about their particular technology choices. I'm not sure such an assumption itself is terribly bad to have, honestly. Sometimes I think that if my only insight into the world of technology came from startup job postings and HN, I ought to be forgiven for thinking that the entire industry is a kind of elaborate and humorous play on role-playing game mechanics ("Ruby Dev" <==> "Developer/Ruby" class/skill combo), which are notorious for producing fanatical devotees of one combination or another.
Perhaps it's a deserved attitude.
Taking a look at the economics of a technology that's being used is a very sensible thing to do, especially for a business. If this analysis shows that one solution is inferior to another, it does not mean that the analyst has a "religious zeal" against the economically-inferior technology.
It would be doing this person a disservice to not mention alternative technologies that have solved very similar problems that have affected many other people/organizations.
It is total absurdity to avoid bringing up these solutions to the problem merely in order to maintain some vague purity of the discussion, or to otherwise keep the discussion unnecessarily and unhelpfully focused.
If I asked on a forum for help with the engine on my Dodge Dart, and you posted "Just buy a Prius", that would not be helpful. Do you see how that isn't helpful? That's what I'm trying to say, that I would like to see, for once a post stay on topic when it's about MongoDB (heck, even PHP for that matter!).
This submission is about a problem involving MongoDB. It is not about MongoDB itself. The submitter clearly wants a variety of ideas here. That's why the submission says, and I quote, "Any tips, stories, suggestions are welcome!"
"Any" means "any", including the suggestion that some other technology be used.
It is always a question of context. So I do think the question is lacking of business context.
The question asked is a if there is a silver bullet. The problem is optimization are always context related. You can even improve a software's sclalabity by changing your processes (who needs concurrencial access to a database if your process can ensure one and only one person is enough to do a certain provisioning that is Single Point of Performance loss?).
I mean, my life as a dev would be easier if managers where not asking for cost reductions and forcing technical solutions that increase costs and then convocating us because the company is having unexpected loss, and thus our wages have to be cut (or more often we have to do up to 50% extra hours coding for free) (and their wages are not). So yes, I am bitter, and unnice.
And even if you would be posting the "right question" here; then we would conclude that you are asking for a free audit, and also probably leaking business secrets (yep, btw, if you are on a competitive market, the concurrence and your customers now know you have a problem with scaling, and how you take costly decisions without caring about your costs... nice).
So with a brilliant reasoning ab absurdum, I can thus demonstrate that neither questions (the mongo oriented or the business oriented) are good questions to ask on HN.
It should be an heuristic on HN, and there maybe should a howto ask the right question like this one http://catb.org/~esr/faqs/smart-questions.html
Remember you are basically asking skilled people to do part of your job for free while aiming for a lot of money (I still wish you to be successful). It is quite logically itching poorly paid (or delusive (my case)) skilled coders.
And you are also leaking insider's information about your technology AND (lack of) business processes. And you may have future (and ex) cyberterrorists(joke)/concurrent/customers/partners/investors/employees here.
And you probably think security is about your 128 character gibberish SSO password OAUTH2 compliant chain you are using. Well most successful attacks are using social engineering, so now I know that I can hack you by proposing you a free software with a trojan aiming at solving your problem (using native active record so that it is not a first order backdoor). Or phone you as a contractor, and being paid to have a look at your documents because I know you are technologically a sitting duck.
I mean: seriously?
Advice #2: DON'T FEED THE TROLL (TL;DR) Yep, if you don't read until the end, you will not notice that I am serious, but choose deliberately a provocative tone because, I have fun, time, and I see this as the price to pay for my free consulting. I take the freedom to express my views in the formalism I choose. My intention is to see what will be the arbitration made by HN (and readers) between business (what I am saying is valid) and formalism (I say it in a clearly non consensual). So I intend to push my reasoning to the extreme just for measuring what is more important: business or being nice?
You completely ignore the question, make a comment about "religious adoptions of mongo" and then do an about-face by showing your OWN religious zeal AGAINST mongo.
Mongo is not killing his business. Your comment should be deleted.
To the OPs question: can you post any more specifics? 1M documents with an index shouldn't be an issue for the aggregation pipeline. What are the specs of the machine? What are your indexes? I think it would be tough to help without having that info, and sadly a question requiring that type of info may be best suited for something like Stack Overflow.