News for hackers. Stuff that matters.
Re:New Features Coming in PostgreSQL 10 by api on Saturday April 08, 2017 @01:52PM (#14068066)
The feature I'd really love is master selection with Raft or similar and automatic query redirection to the master for all write queries (and maybe for reads with a query keyword).

That would make it very easy and robust to cluster pg without requiring a big complicated (a.k.a. high admin overhead and failure prone) stack with lots of secondary tools.

This kind of fire and forget cluster is really the killer feature of things like MongoDB and RethinkDB. Yes people with really huge deployments might want something more tunable, but that's only like 1% of the market.

Of course those NoSQL databases also offer eventual and other weaker but more scalable consistency modes, but like highly tuned manual deployment these too are features for the 1% of the market that actually needs that kind of scale.

A fire and forget cluster-able fully consistent SQL database would be nirvana for most of the market.

Re:New Features Coming in PostgreSQL 10 by mb4nck on Saturday April 08, 2017 @02:11PM (#14068159)
About redirection of write queries to the master, from 10 on, you will be able to specify all members of the cluster in the connection string and demand to connect to the master (like "postgresql://host1:5432,host2:5432/somedb?target_session_attrs=read-write"); libpq will do this automatically for you then, see the parameters "host" (now plural) and "target_session_attr" in section 33.1.2. here: https://www.postgresql.org/docs/devel/static/libpq-connect.h...

About raft-based leader-election, I believe the current recommendation is to look at patroni ( https://github.com/zalando/patroni), which has been built for docker and is now being integrated with Kubernetes; however, I don't think there is an inherent limitation that it couldn't be run on bare-metal.

Re:New Features Coming in PostgreSQL 10 by rachbelaid on Saturday April 08, 2017 @05:00PM (#14069008)
With 2ndquadrant working on Postgres-XL (http://www.postgres-xl.org/), I think that you can be confident that you will see a lot of the features being proposed to core postgres. It will just take some times to build the building block necessary like: global index, distributed sequence, repartition ...

I quite confident that the postgresql from 5y in future will be quite different in term of storage / server topology support. I won't be surprise pg_bouncer capacity to finally make its way to core when we have a coordinator.

Postgres has steady progression (even if not fast enough for some people) but they are moving without compromising robustness of their product for the users.

Re:New Features Coming in PostgreSQL 10 by api on Saturday April 08, 2017 @02:20PM (#14068204)
Yes on the first one, and a big nope on the second for now. I passionately loathe Rube Goldberg machine deployments and am the kind of engineer who constantly asks "do we really need that?". I love exterminating complexity. But maybe that will change when we get to millions of concurrent users and tens of millions of devices and actually need Kubernetes to scale.

Raft is not complex. I doubt leader elect would be terribly hard to implement.

Re:New Features Coming in PostgreSQL 10 by xyzzy_plugh on Saturday April 08, 2017 @06:09PM (#14069330)
Can't pgbouncer/pgpool2 solve query redirection? I don't understand the desire for all-in-one solutions.
Re:New Features Coming in PostgreSQL 10 by api on Sunday April 09, 2017 @12:12AM (#14070406)
That desire comes from three places:

1. Minimize cognitive load by minimizing the number of things you have to learn.

2. Minimize deployment complexity and dependencies.

3. Complexity is just evil I'm general. Linear increases in complexity result in exponential increases in bugs, vulnerabilities, and failure modes. It's just combinatorics.

Re:New Features Coming in PostgreSQL 10 by mb4nck on Monday April 10, 2017 @04:24AM (#14076861)
patroni builds upon etcd and (optionally) haproxy, two rather mature pieces of infrastructure (which can both be made HA on their own if SPOFs are to be avoided).

I understand where you are coming from, but having this kind of multi-server-who-is-master knowledge baked into Postgres itself will surely take another couple of releases, if it will be included at all.

Probably BDR (bi-directional replication, master-master like logical replication) will be there first, but whether the question is whether it will help a lot for local scale-out workloads (as opposed to glueing two datacenters together and allow transactions on both sides).

Re:New Features Coming in PostgreSQL 10 by xyzzy_plugh on Monday April 10, 2017 @06:08AM (#14077219)
Number 2 is the only point I see merit in. With respect to 3, isn't that an argument against increasing the complexity of postgresql?
Re:New Features Coming in PostgreSQL 10 by manigandham on Monday April 10, 2017 @01:28AM (#14076285)
> I don't understand the desire for all-in-one solutions

You really don't understand it? It's less moving pieces to think about, worry about, read about, deploy, maintain, fix and almost always leads to better performance and security.

Something as simple as connection pooling should've already been part of the database and query redirection is even more important to have included.

Re:New Features Coming in PostgreSQL 10 by xyzzy_plugh on Monday April 10, 2017 @06:07AM (#14077215)
Is it actually less moving pieces? The sum seems about the same. Your statement seems like FUD.

The unix philosophy of focusing on a single problem and solving it well seems to do just fine. I don't see why baking in featuress that the vast majority of users will never need is a good investment. If anything, it seems like it will only increase the complexity of postgresql. The protocol is well documented and open. Libpq provides all the functionality necessary. Why not bake the higher order logic into a discrete component?

Re:New Features Coming in PostgreSQL 10 by manigandham on Monday April 10, 2017 @03:27PM (#14081760)
That philosophy is just that - a philosophy that sounds nice and applies sometimes. It's not a rule.

The deployment and upkeep of components is a major hassle and yes, having it all bundled into the same running application (which is the only thing it can be used with anyway) decreases the amount of stuff the end-user has to worry about.

Scaling connections reasonably is something basic any database should handle natively - or better put hitting connection limits without yet another program to help is rather silly in 2017.

You can make the same argument about replication but half this thread is about logical replication being moved to core so we can get scalable distributed clusters in the future - something even fewers users would use but clearly has many benefits by being included in the main release.