Re:How to make MongoDB not suck for analytics
by mitchell_h on Sunday July 01, 2018 @09:57PM
(#17438847)
This is such a common issue there's an entire architecture pattern developed to solve it. https://en.wikipedia.org/wiki/Lambda_architecture . No classic ETL, and any number of folks/systems can plug into the messaging system and get all the data.
Re:How to make MongoDB not suck for analytics
by jrochkind1 on Sunday July 01, 2018 @09:00PM
(#17438650)
Fair enough. Perhaps there are other uses of mongo going on in addition to exporting it to a different database/format? Otherwise I'd be curious what justifies the mongo choice.
Certainly sometimes you are in a position where you just gotta take what other units in the org give you and deal with it.
But _someone_ in the org is hopefully in the position to be able to articulate why they are using mongo in the first place...
Re:How to make MongoDB not suck for analytics
by dizzystar on Sunday July 01, 2018 @09:07PM
(#17438676)
Except that devs have to do ETL every day so analysts can do their query work.
Re:How to make MongoDB not suck for analytics
by marcinzm on Sunday July 01, 2018 @09:23PM
(#17438728)
Which you'll likely want with any DB since analytics workloads are very different than the usual production DB workloads.
Re:How to make MongoDB not suck for analytics
by twblalock on Sunday July 01, 2018 @09:23PM
(#17438725)
That can be automated.
Re:How to make MongoDB not suck for analytics
by dizzystar on Sunday July 01, 2018 @10:55PM
(#17439043)
In theory, yes; in practice, not really.
Re:How to make MongoDB not suck for analytics
by sdoering on Monday July 02, 2018 @12:49AM
(#17439399)
I tend to disagree. Having multiple automated ETL processes running for different projects/clients/colleagues I see that the code does not change as often, as I had anticipated.
Automation here (in my case) is a net win on time.
Re:How to make MongoDB not suck for analytics
by marcinzm on Monday July 02, 2018 @09:07AM
(#17441285)
Why? Technically speaking, simple ETL is easy to automate and not too much maintenance headache.
Re:How to make MongoDB not suck for analytics
by AznHisoka on Sunday July 01, 2018 @09:00PM
(#17438651)
"they can just copy the data into a different database that fits their needs."
That's easier said than done when your database is over 10 TB big.
Re:How to make MongoDB not suck for analytics
by twblalock on Sunday July 01, 2018 @09:14PM
(#17438694)
You only need to copy the stuff that changed since the last time you copied stuff.
If you are really generating 10TB of data more than a few times a day, you can look into putting it in something like Kafka for real-time consumption by the analytics team instead of batch copying.
You just need to read the oplog, so it only needs to track your saves.
In general, you probably should have at least something in your stack which reads all changes from your DB, at the very least for backup reasons.
If the devs want to use Mongo, it's their problem -- it shouldn't matter much to the analytics people, because they can just copy the data into a different database that fits their needs.