News for hackers. Stuff that matters.
Re:Thoughts on Uber’s List of Postgres Limitations by Illniyar on Wednesday August 03, 2016 @05:59AM (#12216843)
I think its worth mentioning again that what Uber ended up using has no resemblence to an RDBMS (single table, manual indexes).

So regardless of whether their complaints are justified or not, it should not be taken as an endorsment of mySQL over postgres, but rather of an endorsement of NoSQL over RDBMS .

Which is really just what every company at these scales do (except for google and f5 if whitepapers are to be considered).

Re:Thoughts on Uber’s List of Postgres Limitations by mranney on Wednesday August 03, 2016 @07:28AM (#12217179)
Uber runs a lot of databases in production. We stopped using Postgres a while ago for new applications that talk SQL, and this article explains some of the reasoning.

We still have many applications that talk directly to MySQL, and we still have our original API monolith that talks directly to Postgres.

All new applications are being built using distributed databases like our in-house Schemaless system which happens to be backed by MySQL, and we also have Riak and Cassandra in production.

Re:Thoughts on Uber’s List of Postgres Limitations by im_down_w_otp on Wednesday August 03, 2016 @02:22PM (#12220209)
I would love to see a similarly detailed write-up on your guys' transition from Riak to Cassandra. Especially considering you would have been really, really deep into the details and guts of operating Riak at the red-line from your time at Voxer.
Re:Thoughts on Uber’s List of Postgres Limitations by andy_ppp on Wednesday August 03, 2016 @10:55AM (#12218345)
Will Schemaless ever be released open source? It sounds great...
Re:Thoughts on Uber’s List of Postgres Limitations by amaks on Wednesday August 03, 2016 @07:34PM (#12222326)
Does Uber still use Python for backend services?
Re:Thoughts on Uber’s List of Postgres Limitations by threeseed on Wednesday August 03, 2016 @07:51AM (#12217263)
> Which is really just what every company at these scales do

No they aren't. You need to understand that databases are being used in a whole range of different ways than you think.

Batch/streaming analytics are typically done with HDFS, Cassandra, HBase, MongoDB etc. Event aggregation often with a time series database like Druid, InfluxDB etc. Web API serving layers are still the domain of lightweight SQL databases like MySQL, PostgreSQL. Customer data warehouses are still dominated by Teradata, Oracle, SAP etc.

What you misconstrue as a single platform is often a multi headed beast with various architectural components. Each comprising their own technology stacks.

Re:Thoughts on Uber’s List of Postgres Limitations by ahachete on Wednesday August 03, 2016 @08:29AM (#12217414)
While I mostly agree with your comment, some NoSQL databases are not suitable for analytics. Indeed, they are terrible at them.

Unstructured information is, surprise, unstructured, which means it is harder to query / analyze, since the structure needs to be fully scanned and parsed to perform the computation.

I have seen many cases where MongoDB is used for this and suffers from really bad performance.

Re:Thoughts on Uber’s List of Postgres Limitations by brianwawok on Wednesday August 03, 2016 @11:02AM (#12218417)
I think the trap is simple POCS.

Do a simple POC where you ram 1 million rows in a MongoDB and then make a webapp to do some basic analytics. Look, it works and get responses within a second. Cool!

Then real data comes in at 12 billion rows and your analytics take 3 hours to run.

So you try to do the sharding thing, and realize that it works for a while, except now every analytics query needs to hit every shard....

Re:Thoughts on Uber’s List of Postgres Limitations by AndyNemmity on Wednesday August 03, 2016 @02:59PM (#12220521)
Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges.

Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame.

Any POC level show of a database would be about meaningless at that scale. You have to do a real test to view the challenges around it.

Re:Thoughts on Uber’s List of Postgres Limitations by pmontra on Wednesday August 03, 2016 @06:49PM (#12222115)
Which database did you use and what would you recommend?
Re:Thoughts on Uber’s List of Postgres Limitations by AndyNemmity on Sunday August 07, 2016 @12:50PM (#12242590)
We used SAP's HANA for a portion of the data, and then used an option on it called Dynamic Tiering which is essentially connecting Sybase IQ to HANA for a disk data store as well.

Keeping 1.3 trillion in memory is pretty expensive, so we were trying to cut some costs by using that to funnel data in and out.

Not a bad solution overall.

Re:Thoughts on Uber’s List of Postgres Limitations by askingqs on Friday August 05, 2016 @10:46AM (#12232705)
but do you need to access all 1.3 trillion rows to do predictive analytics? why not just use a very small sample of the data?
Re:Thoughts on Uber’s List of Postgres Limitations by AndyNemmity on Sunday August 07, 2016 @12:49PM (#12242583)
Yes, you need access to all 1.3 trillion rows (plus more) for the use case I worked on.

Ideally, you'd actually use more than that, however diminishing returns leaves you with that as what we were going with.

Re:Thoughts on Uber’s List of Postgres Limitations by kyrra on Wednesday August 03, 2016 @11:40AM (#12218729)
For scalable SQL products, I'd consider Google's Spanner to be in that boat. There is actually a class of products trying to do this under the name of NewSQL[0]. CockroachDB seems to be mimicking spanner's design principles and deliver a similar product[1].

[0] https://en.wikipedia.org/wiki/NewSQL

[1] https://github.com/cockroachdb/cockroach#design

Re:Thoughts on Uber’s List of Postgres Limitations by leftnode on Wednesday August 03, 2016 @06:38AM (#12217005)
Can you expand on what you mean by a single table with manual indexes? How does a frequently used table with indexes not resemble an RDBMS?
Re:Thoughts on Uber’s List of Postgres Limitations by Illniyar on Wednesday August 03, 2016 @07:08AM (#12217106)
the secondary indexes are saved in a different table and (I can only assume) the application layer (or one layer above mySql) is responsible for keeping that table up to date.

If you use only 1 table, then it's by no means "relational", so I don't see why you'll need a database system designed from start to finish to support a relational model.

Re:Thoughts on Uber’s List of Postgres Limitations by laszlokorte on Wednesday August 03, 2016 @10:07AM (#12217968)
> If you use only 1 table, then it's by no means "relational" [...]

FYI The term "relational" in "relational database" does not refer to the relation between tables but to the mathematical concept of a relation which is some set of tuples ie a single table.

Re:Thoughts on Uber’s List of Postgres Limitations by Illniyar on Wednesday August 03, 2016 @11:04AM (#12218433)
Well, the relational model does not describe or contain a notion of table really, so I guess you could model it in any way you want, even within a single table.

But I think that's a mute point, I doubt they kept any part of the relational model inside their schemaless database.

Re:Thoughts on Uber’s List of Postgres Limitations by slavik81 on Wednesday August 03, 2016 @11:20AM (#12218578)
Probably just an auto-correct mistake, but you mean "that's a moot point".
Re:Thoughts on Uber’s List of Postgres Limitations by hinkley on Wednesday August 03, 2016 @12:45PM (#12219330)
Autocorrect, don't take it for granite.
Re:Thoughts on Uber’s List of Postgres Limitations by pas on Wednesday August 03, 2016 @08:12AM (#12217334)
The "relationalness" is/was not really important here. It's all about MVCC and how storage engines handle it.

Postgres is lacking in these scenarios, whereas a particular fine tuned version (or fork) of InnoDB (or MyRocks or whatever they end up chosing) handles this better.

See Facebook's "mysql-5.6" branch, that has hundreds of patches piled on to support especially these taxing workloads.

Re:Thoughts on Uber’s List of Postgres Limitations by Illniyar on Wednesday August 03, 2016 @11:06AM (#12218452)
Well, for that matter, as I understood it, they also have no transactions nor atomicity and are basically eventually consistant , though these I'm simply inferring from their posts, it was not stated as far as I can remember.

So MVCC also has almost zero bearing on what they are doing.

Re:Thoughts on Uber’s List of Postgres Limitations by spotman on Wednesday August 03, 2016 @12:38PM (#12219276)
Just because you may not always use the database with the BEGIN/COMMIT semantics, that absolutely does not mean that "MVCC has almost no bearing on what they are doing".

MVCC and how it has been implemented in the two storage engines in discussion here absolutely has a lot to do with it.

Re:Thoughts on Uber’s List of Postgres Limitations by fauigerzigerk on Wednesday August 03, 2016 @06:57AM (#12217072)
If it was just about NoSQL vs RDBMS they could have used Postgres as a key-value store instead of mySQL. So yes they absolutely did endorse mySQL over Postgres.

Also, what is the scale of Uber that you think is unsuitable for RDBMS? Uber does a million rides per day but these rides are very predictably local. I don't know enough to make any firm claims here but at first sight this doesn't look like the sort of scale that is unachievable on an RDBMS in principle.

Re:Thoughts on Uber’s List of Postgres Limitations by duaneb on Wednesday August 03, 2016 @11:13AM (#12218515)
They endorsed MySQL over PostgreSQL for a type of structure and access very few people use. I don't see how the query language plays into the recommendation at all.
Re:Thoughts on Uber’s List of Postgres Limitations by marcosdumay on Wednesday August 03, 2016 @10:22AM (#12218087)
Well, Uber claims they are using MySQL because it's a better NoSQL database than Postgres.

From the context, I don't think they ever evaluated Postgres NoSQL features. But I don't think they would get a different conclusion if they did. MySQL trades some consistency guarantees for speed, and it looks like Uber doesn't need those extra guarantees anyway.

Re:Thoughts on Uber’s List of Postgres Limitations by Illniyar on Wednesday August 03, 2016 @07:38AM (#12217217)
"Also, what is the scale of Uber that you think is unsuitable for RDBMS"

Never said it's unsuitable, I have no idea. Just said the move from relational to NoSQL is what most billion dollar companies with mass-market apps or websites do.

Many of them also do sharding on normal RDBMS.