Re:How to make MongoDB not suck for analytics
by Ankhers on Sunday July 01, 2018 @11:38PM
(#17439194)
Mongo 3.6 introduced something called Change Streams[0] which is basically a safer way to tail the oplog. It is also supposed to work well in a sharded environment.
Re:How to make MongoDB not suck for analytics
by chiaolun on Sunday July 01, 2018 @11:26PM
(#17439152)
The oplog is awesome! It provides an immutable record which is really useful - we materialize the oplog directly in Athena to get a time-travelling database for debugging purposes.
Re:How to make MongoDB not suck for analytics
by kalendos on Tuesday July 03, 2018 @07:15AM
(#17449208)
Would you mind sharing the process? I'm curious which flow you use to materialize the oplog in Athena/S3.
Re:How to make MongoDB not suck for analytics
by sateesh on Sunday July 01, 2018 @11:03PM
(#17439068)
Yes you can tail oplog, but in a sharded setup with multiple clusters this becomes a pain.
Re:How to make MongoDB not suck for analytics
by sbr464 on Monday July 02, 2018 @01:01AM
(#17439442)
The new change streams api resolved a lot of this, since it allows your to utilize the aggregation framework to subscribe to highly filtered/specific queries. The upcoming 4.0 release is expanded further.
But I will say that many moons ago when I did actually write stuff for Mongo. The oplog was a god send. You can "tail" the oplog, and get every transaction in near real time. We used this for updating Elasticsearch indexes etc in what is basically realtime, without having to poll or modify existing code at all.