News for hackers. Stuff that matters.
Re:Scalable PostgreSQL Connection Pooler by x4m on Friday November 12, 2021 @12:53PM (#29202351)
FWIW I maintain Odyssey and will be happy to answer any question regarding it. Actually there was just a CVE-triggered release... I really hope to release more often with more new functionality.

If you are interested in connection pooling maybe you will find interesing SPQR too https://github.com/pg-sharding/spqr Currently it's our experiment to build pooling-based PostgreSQL sharding.

Re:Scalable PostgreSQL Connection Pooler by 3np on Friday November 12, 2021 @10:35PM (#29207154)
Some questions on auth. We have one cluster with a large number of databases, all with automatically provisioned and rotating credentials for service roles using postgres password auth. Currently, clients connect directly to the db cluster (there's a tcp lb but no pgbouncer etc)

In this case, I'd like for services to continue authenticating as previously without changing authentication method, just switching address.

Should I use password_passthrough/auth_query? Does odyssey need pg credentials itself? Is storage_db referring to the db where odyssey stores internal data for itself...?

I guess I don't get why a connection pooler would need to handle auth, much like an HTTPS proxy doesn't need API tokens itself.

Re:Scalable PostgreSQL Connection Pooler by x4m on Saturday November 13, 2021 @12:26AM (#29207653)
Currently in Odyssey password_passthrough works only for cleartext auth, PAM and LDAP. Probably, we could make this for MD5 work too. And it's the whole purpose of SCRAM to defeat this feature :)

The other way to do so is auth_query - you provide a storage password to access auth data of the DB. This works for MD5 auth. When user wants to authenticate we just check credentials against what we see in the database.

Re:Scalable PostgreSQL Connection Pooler by 3np on Sunday November 14, 2021 @04:23AM (#29216217)
Thanks for the pointers! I did play around a bit but it seems to segfault and I'm not sure how to proceed.

https://github.com/yandex/odyssey/issues/361

Re:Scalable PostgreSQL Connection Pooler by x4m on Monday November 15, 2021 @06:49AM (#29226327)
It seems like there's issue when you have a user for accessing authentication table you login with the same user. It fails when due to mixing same pool for auth and for work. I'll look into differentiating these pools more closely. As a simple workaround you can setup different user for accessing password and not login into Odyssey with this user.
Re:Scalable PostgreSQL Connection Pooler by 3np on Monday November 15, 2021 @12:12PM (#29229701)
Thanks for following up. auth_query_user is not the same as the user I'm attempting to authenticate with, though. And I get the same behavior with other users, including non-existing ones.

Or are you referring to something else?

Re:Scalable PostgreSQL Connection Pooler by hn_throwaway_99 on Friday November 12, 2021 @12:58PM (#29202416)
Thanks very much for posting this. Odyssey is analogous to pgbouncer, correct? I haven't set up a connection pooler yet for our postgres instance but was planning to do it eventually with pgbouncer - are you able to comment how Odyssey compares?
Re:Scalable PostgreSQL Connection Pooler by x4m on Friday November 12, 2021 @01:20PM (#29202722)
Connection pooler allows you to get more transactions per second on the same hardware. All it does - pack a lot of connections into handful of concurrently running PG backends. In theory all it does - relay bytes between sockets. But there are workloads when it's CPU intensive, for example lots of concurrent TLS handshakes. And pgbouncer is single-threaded! Odyssey is more scalable.

PgBouncer is a great software, actually. We built Odyssey merely to create competitor for PgBouncer. But then started to add functionality that we needed for cloud installations. E.g. Odyssey computes transaction time quantiles, when PgBouncer computes average transaction time.

Re:Scalable PostgreSQL Connection Pooler by whitepoplar on Friday November 12, 2021 @01:38PM (#29202941)
How does Odyssey compare to other high-performance poolers like pgagroal?
Re:Scalable PostgreSQL Connection Pooler by x4m on Friday November 12, 2021 @01:49PM (#29203068)
Depends on how you measure performance :) pgagroal utilizes a lot of linux processes to track each connection state, we use coroutines packed into small number of processes. Actually if you have a bottleneck in multithreaded pooler - something really goes wrong. The database should consume most of resources. I'd be happy to help tuning Odyssey to someone who will bachmark both poolers (in fact there's only one number - number of worker processes..well, maybe pool_size too).

pgagroal claims performance superiority over all poolers [0]. I doubt that Odyssey was used in transaction pooling mode in those experiments.

[0] https://github.com/agroal/pgagroal/blob/master/doc/PERFORMAN...

Re:Scalable PostgreSQL Connection Pooler by whitepoplar on Friday November 12, 2021 @01:53PM (#29203121)
Thanks! For a Postgres n00b, can Odyssey work alongside a HA solution like pg_auto_failover running on the same nodes?
Re:Scalable PostgreSQL Connection Pooler by x4m on Friday November 12, 2021 @01:57PM (#29203160)
Sure. We use Odyssey as a component of HA cluster in Yandex.Cloud. Odyssey runs on each node of a Patroni-like system. One day we want to implement load balancing functionality in Odyssey. So that your queries go to other node if local is overloaded or lagging long behind primary.
Re:Scalable PostgreSQL Connection Pooler by tsenart on Friday November 12, 2021 @05:36PM (#29205334)
This, please! Native support for read-replicas would be awesome. Ideally it would now if a query is read-only or not without application changes.
Re:Scalable PostgreSQL Connection Pooler by jpgvm on Friday November 12, 2021 @08:36PM (#29206523)
For a variety of reasons this is incredibly difficult. Functions, etc make SELECT queries writes, not just UPDATE/DELETE, etc.

It's a lot easier for your application to know what a write is and just establish connections to 2 separate poolers (or hosts on the same poolers) and direct the reads/writes appropriately.

Re:Scalable PostgreSQL Connection Pooler by x4m on Saturday November 13, 2021 @12:28AM (#29207663)
There's already working part of libpq protocol for this - target_session_attrs. But the problem with target_session_attrs is that it just takes too long to discover new primary after failolver. We want to fix this within Odyssey.
Re:Scalable PostgreSQL Connection Pooler by kerblang on Friday November 12, 2021 @06:57PM (#29205951)
Maybe a dumb question but does it lock the postgres connection down to one pool client (no sharing) when the client has a transaction in progress? How does it compare to pgbouncer in this regard?
Re:Scalable PostgreSQL Connection Pooler by x4m on Monday November 15, 2021 @06:51AM (#29226337)
Transaction pooling rules are equivalent in Odyssey and PgBouncer.
Re:Scalable PostgreSQL Connection Pooler by jerrysievert on Friday November 12, 2021 @02:58PM (#29203865)
Any thoughts on an arm64 version?
Re:Scalable PostgreSQL Connection Pooler by x4m on Sunday November 14, 2021 @11:36AM (#29218250)
Odyssey could work fine on arm, but we've a small ASM function https://github.com/yandex/odyssey/blob/master/third_party/ma... . We need to reimplement it for ARM64 or at least fallback on some generic context swap.