Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a...
by mhink on Tuesday April 11, 2023 @02:25PM
(#35529149)
Just out of curiosity (as someone who hasn't done a lot of this kind of operational stuff) how does this approach to queueing with Postgres degrade as scale increases? Is it just that your job throughput starts to hit a ceiling?
[ Parent
| on HN ]
Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a...
by nightpool on Tuesday April 11, 2023 @06:21PM
(#35532429)
Throughput is less of an issue then queue size—Postgres can handle a truly incredible amount of throughput as long as the jobs table is small enough that it can safely remain in memory for every operation. We can handle 800k jobs/hr with postgres, but if you have more than 5k or 10k jobs in the table at any given time, you're in dangerous territory. It's a different way of thinking about queue design then some other systems, but it's definitely worth it if you're interested in the benefits Postgres can bring (atomicity, reliability, etc)
Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a...
by aetherson on Tuesday April 11, 2023 @06:44PM
(#35532673)
With Postgres, you also need to worry a lot about tombstoning and your ability to keep up with the vacuums necessary to deal with highly mutable data. This can depend a lot on what else is going on with the database and whether you have more than one index on the table.
Re:SQL Maxis: Why We Ditched RabbitMQ and Replaced It with a...
by Rapzid on Wednesday April 12, 2023 @02:37AM
(#35535808)
One strategy for mitigating vacuum costs would be to adopt an append-only strategy and partition the table. Then you can just drop partitions and avoid the vacuum costs.
Really depends on the needs but this can unlock some very impressive and sustainable throughputs.