News for hackers. Stuff that matters.
Re:Thoughts on Uber’s List of Postgres Limitations by threeseed on Wednesday August 03, 2016 @07:51AM (#12217263)
> Which is really just what every company at these scales do

No they aren't. You need to understand that databases are being used in a whole range of different ways than you think.

Batch/streaming analytics are typically done with HDFS, Cassandra, HBase, MongoDB etc. Event aggregation often with a time series database like Druid, InfluxDB etc. Web API serving layers are still the domain of lightweight SQL databases like MySQL, PostgreSQL. Customer data warehouses are still dominated by Teradata, Oracle, SAP etc.

What you misconstrue as a single platform is often a multi headed beast with various architectural components. Each comprising their own technology stacks.

Re:Thoughts on Uber’s List of Postgres Limitations by ahachete on Wednesday August 03, 2016 @08:29AM (#12217414)
While I mostly agree with your comment, some NoSQL databases are not suitable for analytics. Indeed, they are terrible at them.

Unstructured information is, surprise, unstructured, which means it is harder to query / analyze, since the structure needs to be fully scanned and parsed to perform the computation.

I have seen many cases where MongoDB is used for this and suffers from really bad performance.

Re:Thoughts on Uber’s List of Postgres Limitations by brianwawok on Wednesday August 03, 2016 @11:02AM (#12218417)
I think the trap is simple POCS.

Do a simple POC where you ram 1 million rows in a MongoDB and then make a webapp to do some basic analytics. Look, it works and get responses within a second. Cool!

Then real data comes in at 12 billion rows and your analytics take 3 hours to run.

So you try to do the sharding thing, and realize that it works for a while, except now every analytics query needs to hit every shard....

Re:Thoughts on Uber’s List of Postgres Limitations by AndyNemmity on Wednesday August 03, 2016 @02:59PM (#12220521)
Absolutely true, and the scale keeps getting larger > 12 billion rows, and comes up with even harder challenges.

Worked on a 1.3 trillion row (for 1 table, others varying below this) database this year for predictive analytics, and it was mind boggling the hoops to jump through to try to get that thing to run anything in a manageable time frame.

Any POC level show of a database would be about meaningless at that scale. You have to do a real test to view the challenges around it.

Re:Thoughts on Uber’s List of Postgres Limitations by pmontra on Wednesday August 03, 2016 @06:49PM (#12222115)
Which database did you use and what would you recommend?
Re:Thoughts on Uber’s List of Postgres Limitations by AndyNemmity on Sunday August 07, 2016 @12:50PM (#12242590)
We used SAP's HANA for a portion of the data, and then used an option on it called Dynamic Tiering which is essentially connecting Sybase IQ to HANA for a disk data store as well.

Keeping 1.3 trillion in memory is pretty expensive, so we were trying to cut some costs by using that to funnel data in and out.

Not a bad solution overall.

Re:Thoughts on Uber’s List of Postgres Limitations by askingqs on Friday August 05, 2016 @10:46AM (#12232705)
but do you need to access all 1.3 trillion rows to do predictive analytics? why not just use a very small sample of the data?
Re:Thoughts on Uber’s List of Postgres Limitations by AndyNemmity on Sunday August 07, 2016 @12:49PM (#12242583)
Yes, you need access to all 1.3 trillion rows (plus more) for the use case I worked on.

Ideally, you'd actually use more than that, however diminishing returns leaves you with that as what we were going with.