News for hackers. Stuff that matters.

A comment on MongoDB has filed confidentially for IPO. Parent comment

Re:MongoDB has filed confidentially for IPO by dsmithatx on Tuesday August 15, 2017 @11:16PM (#15024690)
I worked with one of the largest ad publishing companies. They wanted to track data about every client served an ad. This generated over 1.2 Terabyte per hour of data when the MySQL master started to max out. We had the largest possible multiple core system. It was going to cost my client $30k to upgrade to SSD drives to get more out of MySQL. Also note we had to store this data on an expensive SAN in order to feed the data at a reasonable rate to MySQL or PostgreSQL.

I had just learned of MongoDB and went to school at 10gen for their sys admin class. I talked to the developer about storing the data in NoSQL using a small sharded cluster on a Friday. Monday morning he asked me to setup a MongoDB cluster. Tuesday we moved over from MySQL. They ended up using much smaller servers, got rid of the 3par and epsilon SAN's and saving tons of money.

My point is there are certain situations where NoSQL is still the answer unless you can cluster your SQL write server. I've moved on from working with Ad publishing clients but, I'm sure there are other places where SQL databases are not adequate.

NoSQL might be or, have been, a fad but, like any tool when used for the right job it works.

[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by qaq on Tuesday August 15, 2017 @11:36PM (#15024783)
A production large scale ad. system system that was doing 1.2 TB per hour in writes, was migrated over from MySQL to MongoDB in roughly 24 hour window cool story Bro.
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by DenisM on Wednesday August 16, 2017 @09:10AM (#15027136)
Good point. How is it possible to move that much data from SAN to mongo cluster in such a short time window?
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by qaq on Wednesday August 16, 2017 @09:12AM (#15027149)
and rewrite and validate all the code that was accessing MySQL to use MongoDB
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by mcintyre1994 on Wednesday August 16, 2017 @03:40AM (#15025652)
If more comments were like their story then this would be a way higher value comments section.
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by qaq on Wednesday August 16, 2017 @06:56AM (#15026489)
how can more made up stories result in higher value comments section?
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by tehlike on Wednesday August 16, 2017 @02:11AM (#15025296)
sounds like the wrong tool. what you are producing was probably logs data, which is immutable. There are far more efficient (storage, cpu) write-only stores.
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by skrebbel on Wednesday August 16, 2017 @02:45AM (#15025422)
Such as?
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by hvidgaard on Wednesday August 16, 2017 @03:00AM (#15025482)
Concider what a database does. It provides ACID properties and ability to query data. If all you need is writing data, the fastest you can do it write it directly to the disk, without the overhead a database comes with.

Using a loadbalancer in front of a farm of cheap logging machines, and aggreate the data you need for analysis to a suitable machine.

[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by LoSboccacc on Wednesday August 16, 2017 @04:30AM (#15025847)
they'd still need to query their stuff, I guess, so you'd need to trow in there somewhere something to aggregate logs and get the metrics they're tracking out of it - which can totally be done in streaming, without the need of going trough the logs every time, for most metrics.
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by hvidgaard on Wednesday August 16, 2017 @04:42AM (#15025904)
With that amount of data, streaming and only saving aggregated data is the only sane way. With 1.2TB/hour there is a limit to how much historical data that can be saved anyway, and we're talking about 30% utilization of a 10gbps network interface, so it's beyond using single machines for most usecases.
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by tehlike on Wednesday August 16, 2017 @06:16AM (#15026317)
Query? Most likely not. At least not in the traditional "lets on the fly create a dashboard" sense.
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by LoSboccacc on Wednesday August 16, 2017 @06:20AM (#15026337)
query as in 'how much I bill this guy for it's click' - doesn't have to be sql nor on the fly of course
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by foxh0und on Wednesday August 16, 2017 @01:18AM (#15025114)
Was $30k a lot of money for one of the largest add publishers?
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by roel_v on Wednesday August 16, 2017 @03:30AM (#15025607)
OTOH, if you're a consultant, and you can say 'hey guys I can save you 30k, my fee is only 15k', and you can do it in a week - I mean, there are weeks where I bill less than 15k...
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by quickthrower2 on Wednesday August 16, 2017 @06:44AM (#15026433)
3k a day consulting. I'm in!
[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by dx034 on Wednesday August 16, 2017 @08:49AM (#15027024)
How can MongoDB use that much less in such a situation? Especially prior to WireTiger?

An optimised schema in a relational database should be close to the minimum possible storage.

[ Parent | on HN ]
Re:MongoDB has filed confidentially for IPO by 0xbear on Wednesday August 16, 2017 @05:01AM (#15025983)
The largest ad system ever, Google AdSense, used MySQL until circa 2014 when it moved to a completely custom DB backend, F1. F1 is also a SQL database, however.

Google does use bigtable and such where appropriate, but for anything more complex than dumb key/value you can't do much better than regular DBs. Some people think they can, but 99% of the time they're mistaken.

[ Parent | on HN ]