News for hackers. Stuff that matters.
Re:Scaling Databases at Activision [pdf] by geenat on Friday April 21, 2023 @04:15PM (#35659433)
You'd certainly want to shard from the start in any case, but it's an order of magnitude less postgres boxes for the QPS.
Re:Scaling Databases at Activision [pdf] by leetrout on Friday April 21, 2023 @04:48PM (#35659873)
Wonder what the tradeoff on connection overhead would be? I've never been around a connection pooler that would be anywhere near this load.
Re:Scaling Databases at Activision [pdf] by aeyes on Friday April 21, 2023 @07:32PM (#35661628)
Not too much if connections are persistent. A problem is that pgbouncer is 1 CPU only so you need a couple of them which makes the pooling less efficient.
Re:Scaling Databases at Activision [pdf] by Mavvie on Friday April 21, 2023 @09:06PM (#35662282)
Does pgbouncer tend to run out of CPU? Or just at the large scale this article is about?
Re:Scaling Databases at Activision [pdf] by aeyes on Friday April 21, 2023 @10:03PM (#35662637)
I have a 100k qps workload and we run 10 pgBouncer instances. I ran into the CPU limit with 6. Connections to pgBouncer are persistent so it's not caused by connection churn. The amount of incoming connections is in the low hundreds per instance. I tried without SSL as well and it didn't make a measurable difference.

I'm not quite sure if this is a common problem or if it's our workload. It always made me wonder if moving pooling into Postgres is really a good idea.

Re:Scaling Databases at Activision [pdf] by geenat on Friday April 21, 2023 @09:00PM (#35662240)
Speaking only for postgres 15+, but in my testing, app drivers have been great about facilitating "connection pooling" without a separate server. Eliminates connection overhead at the app level for many projects.

Golang and ASGI Python (FastAPI, Starlette, Sanic) can do a small connection pool per worker, each worker can handle thousands of users at once.

PHP in 2023 has database connection persistence using FPM + PDO (effectively, one big connection pool shared between all workers on a server). Again, eliminates connection overhead- works well.

Just remember to raise your max_connections in postgres, like in this: https://gist.github.com/gnat/cfe3754c3dc817c7fb8b2225ef4db62...