People underestimate how bad databases are at implementing queuing systems. I’ve helped replace such systems multiple times.
But it doesn’t need to require 10x the time or code. That just sounds like overengineering. Database-as-queue vs. 10x ultimate solution is a false dichotomy.
Re:Just use Postgres for everything
by jbverschoor on Sunday December 11, 2022 @06:09AM
(#33942192)
People overestimate their requirements, because almost nobody had ANY idea what computers are capable of, and that big tech is HUUUUGE compared to the rest.
You are not Google and you are not Steve Jobs
Yeah they suck at it. There are so many good queuing systems, and if you’re on any cloud there’s gunna be a cheap managed queue that is super reliable and near zero effort. If you get beyond trivial scale just use a managed queue.
> how bad databases are at implementing queuing systems.
What difficulties did you meet?
I think there exists a midway, where you rely on the heavily-tested open source solution that implements what you want, and a sprinkle more. RabbitMQ for example is that layer that your 40 lines of code would do, but pays off dividends in good nights' sleep.
RabbitMQ is not free. You need to learn it, configure it, understand it, interface towards it, install it, upgrade it. Lots and lots of things.
So those 1000 lines of code (your prototype was 100 lines and this was 10x that) would still have been a week tops, right? That doesn't seem too bad a price to pay for scaling potential, even if in hindsight it wasn't needed. You can't always know in advance which features you are going to need and which ones are superfluous.
If what you need is a simple work queue, you really need to scale very far away (in the magnitude of several thousands of requests per second) to overgrow a solution built on top of PostgreSQL.
Re:Just use Postgres for everything
by jbverschoor on Sunday December 11, 2022 @06:07AM
(#33942181)
In other words: boss has no idea, boss doesn’t trust your technical nor business skills
Conclusion: abandon this company.. you’re probably just a cost center and nagging headache in his eyes. If you’re not valued or respected, there’s only way way up, and that’s out
My boss said he just wanted to install the best thing and be done with it forever. So we ended up spending 10x the time and wrote like 10x the code to integrate a third-party solution and we only ended up using the most basic features. Not to mention additional infrastructure.