madhusudhan000

I have taken the Cloud computing specialization in Coursera which was taught by Dr. Gupta. One of the best courses I have taken. Excited for this one. They should offer an Online masters for this, just like the data science one on Coursera

It hasn't changed much. But new OLAP features have come up such as parallel queries and for those, the existing cache architecture doesn't make sense since OLAP is full of sequential scans.

You can read the doc for the same - https://www.postgresql.org/docs/10/static/when-can-parallel-...

Author here.

> I was surprised to learn Postgres doesn't use direct read/write (O_DIRECT) to bypass kernel caches.

It is more related to the database being platform neutral. The community is very sensitive to putting in performance improvement/hacks which applies only to certain systems. There was a discussion back then - https://www.postgresql.org/message-id/50E4AAB1.9040902@optio.... There were several more, you can search around the mail archives, but the general tendency is not to put changes which is not universally applicable to all platforms.

With that said, a lot is changing, as their community grows we can expect optimizations like these.

> The article doesn't go into the details, but maybe someone can point me to some resources that explain the benefits of this double cache layer?

The article was already a bit long so I didn't go into much details. Perhaps ill write on it in another article.

So let me try to summarise this.

Poor replica MVCC support

They are actually pointing to a blog article written in 2010 -> http://blog.2ndquadrant.com/tradeoffs_in_hot_standby_deplo/

Do they realise that it is 2016 ?

Guess they did't bother to understand the hot standby feedback system.

> Postgres’s design resulted in inefficiencies and difficulties for our data at Uber.

What kind of inefficiency ? The explain what is purpose of WAL and replication which every database person knows about but didn't care to explain the actual problem at hand ?

Data corruption

> During a routine master database promotion to increase database capacity, we ran into a Postgres 9.2 bug

Why the heck didn't they upgrade to a newer version ? Did you report this bug to pg dev , did they take so much time to fix this, or were you just assuming that the bug could fix itself ?

> The bug we ran into only affected certain releases of Postgres 9.2 and has been fixed for a long time now. However, we still find it worrisome that this class of bug can happen at all.

Postgres 9.2 is pretty old and there has been 3 major releases after that. WTF ?

I can say countless instances where MySQL data corruption was a constant nuisance with version 5.5 and they have fixed it with newer releases.

Replication

> During peak traffic early on, our bandwidth to the storage web service simply wasn’t fast enough to keep up with the rate at which WALs were being written to it

So you have run into a hardware limitation and then blame postgres. What was limit that you hit ? I don't understand this point at all.

Concept of context switching

I am surprised that this is actually an issue, in a database the slowest part is always the disk and not the CPU. Confused on how did they hit this limitation first without actually touching others.

Time taken by a context switch : http://stackoverflow.com/questions/21887797/what-is-the-over...

Which is in microseconds.

InnoDB buffer pool

> By comparison, the InnoDB storage engine implements its own LRU in something it calls the InnoDB buffer pool

Postgres has something similar called shared_buffer. They are speaking as if postgres relies entirely on the operating system which is false.

> It makes it possible to implement a custom LRU design. For instance, it’s possible to detect pathological access patterns that would blow out the LRU and prevent them from doing too much damage

Not sure what kind of damage they are speaking. In a postgres sequential scan (full table scan), a ring buffer is used instead and does not result in the shared buffers being blown away.

If you need a custom LRU design, there is definitely something wrong in the way that you are using an OLTP database.

Connection Handling

This is complete BS. Nobody uses databases without connection pools. Agree that a thread is more lightweight than a process, but you would never hit this limit at all in real time which is in the order of microseconds again. In a production system, one would open connections immediately and then hold them in the connection pool. This overhead is almost not visible at all. If you are constantly opening and closing connections then there is something seriously wrong with your design.

> However, we have had occasional application bugs in our backend services that caused them to open more active connections (usually “idle in transaction” connections) than the services ought to be using, and these bugs have caused extended downtimes for us

So they are blaming the database for a bug in their design/system. Computers are no match for human stupidity.

> Accordingly, using pgbouncer to do connection pooling with Postgres has been generally successful for us.

Again what is the problem, the whole article smells more and more like a useless rant, just because you dont know how to use them ?

Conclusion

Another thing is that they have not given any kind of query/access pattern in which they use postgres/mysql. They put in a couple of low level things and then say that postgres is badly designed.

I can think of only two logical explanations

1) The article writer was already familiar with MySQL and they didn't bother to even dig into postgres deeper

2) They have been paid by oracle :P

Well, for one thing, if they say schemaless, then you are using the wrong tool for the job.

Sure postgres has JSONB (comparing to mongo db), Key value store such as HStore, but they do well if they fit on one machine. The moment you hit that scale you have to realize that there are tools specifically built for this.

There is the phoenix project https://phoenix.apache.org/ that salesforce is using for scaling. Definitely worth a try.

But again, the title of the article sounded nothing more than a rant to me.

>community of MySQL is much better and the tools available are more mature

Can you backup your statements with some facts. I have seen the postgres community to be much better particularly now with Oracle taking ownership.

Yeah, I sort of skipped over that detail, thanks for pointing it out, ill include it in the post.