Re:LokiJS – In-memory JavaScript Datastore with Persistence
by andrewaylett on Tuesday July 05, 2016 @12:38PM
(#12037666)
That's true, but it's got to be a trade-off and a balance of risk, because "Performance over everything" makes no sense at all: the most performant thing to do is to store no data and return no data.
And even with that trade-off, I'd argue that there's a level of correctness that should be non-negotiable. Missing data might be OK, but corrupting data on the way in is bad, and failing to process everything might be OK (eventual consistency and all that) but you should meet your published guarantees and you shouldn't corrupt your results.
So I would argue for correctness over performance, with the caveat that "correct" has a technical meaning specific to the project.
What is important is being clear about what trade-offs they make.
As an example: Years ago I worked on a classifieds aggregation engine; we crawled hundreds of thousands of feeds, indexing tens of millions of listings.
All the queues in our processing pipeline were in-memory only: if we lost a few items (or a million) it didn't matter; worst case we'd pick them up no later than 24 hours later - in the case of more severe loss we'd simply trigger re-indexing of the feeds likely to have been "in flight" at the time. Failures would affect freshness and cause us to lose out on some updates, but as long as they were not severe enough or too frequent nobody would even notice, because the inventory was large and fleeting.
There are lots of cases like that where "performance over everything" can be a totally valid trade-off, because your data store is not the canonical source of information, doesn't need to be precise or comprehensive, and/or can be easily refreshed, for example.
This does require you to know and understand your requirements and what the datastore in question guarantees (or not), of course. If you don't, then you better stick with a "safe" option.