Re:Newsblur runs into MongoDB bug – can't replicate DB
by MichaelGG on Thursday March 21, 2013 @03:13PM
(#5418030)
100GB takes 6 hours? That's ~4.7MB/sec - is that right? That seems, sorta like slow?
Not to sound rude, but what is so difficult about managing 100GB of data? You can fit that in RAM without much work.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by conesus on Thursday March 21, 2013 @03:18PM
(#5418089)
The transfer speed is between data centers. West coast to east coast.
As for the 100GB, I have 12 task servers with 6-8 processes each reading 25-50 stories each every 4 seconds. I also have 18 app servers with 6-8 processes each reading 12 stories every second. Each story is on average 4KB. That's several MB a sec.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by ryangripp on Thursday March 21, 2013 @04:07PM
(#5418492)
I just signed up for newsblur and paid $24. Love it.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by base698 on Thursday March 21, 2013 @02:57PM
(#5417839)
Why didn't you just take a snapshot of the DB and start that up before adding it as a replica set member? I've had better luck doing that.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by conesus on Thursday March 21, 2013 @02:59PM
(#5417855)
Because that server isn't in the cloud. It's on leased hardware, so no ability to snapshot. Plus, I like it when I can replicate my data without worry. I don't want to have to rely on block-level replication.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by LukeShu on Thursday March 21, 2013 @03:47PM
(#5418348)
base698 isn't saying take a snapshot of the server, he's saying take a snapshot of the DB; find the directory it is stored in and `cp` that.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by mrkurt on Thursday March 21, 2013 @03:56PM
(#5418406)
That's not a good snapshot unless you're running LVM or can afford to suspend writes and fsync + lock.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by lwat on Thursday March 21, 2013 @05:45PM
(#5419309)
And again, this is yet another problem solved decades ago by RDBMSs.
Re:Newsblur runs into MongoDB bug – can't replicate DB
by mrkurt on Thursday March 21, 2013 @06:12PM
(#5419544)
And even earlier by Polaroid!
Re:Newsblur runs into MongoDB bug – can't replicate DB
by skorgu on Friday March 22, 2013 @12:07PM
(#5423813)
Armchair quarterbacking here of course but do you think it would be feasible/a good idea to try to switch to a less..troubled product at this point? I don't hear many horror stories about Cassandra for example and AIUI that's serving most of reddit.
I would love it if I could choose which server to sync from. This option used to exist, but they removed it. But that would only solve the performance penalty of replication.
To their credit, the MongoDB folks have been stellar. I had a hardware failure a year ago and the CTO ssh'ed into my machines to figure out what was wrong. This time I'm having a bit more difficulty getting the problem fixed, but it's understandable as I have 100GB of highly variable data.