News for hackers. Stuff that matters.
Re:System design hack: Postgres is a great pub/sub and job s... by nurettin on Friday November 08, 2019 @04:47PM (#21487186)
>> spin up a bunch of worker processes as EC2 spot instances that check for the next PENDING task

Does that mean workers poll the database?

Re:System design hack: Postgres is a great pub/sub and job s... by nostrademons on Friday November 08, 2019 @05:02PM (#21487328)
Yup, but they do so only once per shard (well, twice, once at the beginning and once at the end). If you've got a big job where each shard takes a few minutes to process and a hundred or so workers, the DB gets about 1 req/sec, which is well within the capabilities of Postgres.
Re:System design hack: Postgres is a great pub/sub and job s... by flavor8 on Friday November 08, 2019 @05:31PM (#21487619)
We had an "interesting" bug where we inadvertently ended up polling postgres advisory locks around 6 million times per minute from a cluster of 200 servers. It added about 20% load to the database, enough to be anomalous but not enough to outright trip any alarms. The overall database performance was fine.
Re:System design hack: Postgres is a great pub/sub and job s... by exikyut on Sunday November 10, 2019 @06:32AM (#21497564)
(That's 100,000 req/sec)

Was "the database" a single node, or multiple systems?

Re:System design hack: Postgres is a great pub/sub and job s... by wahern on Saturday November 09, 2019 @02:30AM (#21490180)
Postgres provides LISTEN/NOTIFY for precisely this sort of use case.

https://www.postgresql.org/docs/9.1/sql-notify.html

Re:System design hack: Postgres is a great pub/sub and job s... by seunosewa on Saturday November 09, 2019 @04:40AM (#21490557)
Yes, any idea about why isn't it used in the original article above?
Re:System design hack: Postgres is a great pub/sub and job s... by buckminster on Saturday November 09, 2019 @12:17PM (#21492730)
It is used in the article.
Re:System design hack: Postgres is a great pub/sub and job s... by nurettin on Saturday November 09, 2019 @12:22AM (#21489793)
I would lean towards maintaining a simple list on redis and letting the workers pop the next job from it whenever they are done.
Re:System design hack: Postgres is a great pub/sub and job s... by nostrademons on Saturday November 09, 2019 @01:07AM (#21489927)
That works fine too. The advantage of Postgres is that the master is a single point of failure, and if Postgres goes down all the state is persistent and you can just bring it back up again. Also, ad-hoc query tools are a little better for Postgres, which gives you a really simple interface to inspect the status of the job and the partial results.

This architecture originally evolved from an Amazon-SQS based queue system, where I'd stick the jobs in SQS and pop them off. Then I realized that I needed the DB anyway for my reducer-substitute, and if I'm writing SQL/ORM code anyway it was simpler to use the DB to manage the queue than to use SQS, and it gave me some other nifty abilities like being able to record timing & size stats with each shard, being able to query progress while the job is going, and being able to manually restart shards after correcting the errors that led to their failure. If you're running dozens-to-hundreds of workers, the cost of a t2 RDS instance (that's all you need for 1 QPS) is a rounding error, and a lot cheaper than EMR would be.

Re:System design hack: Postgres is a great pub/sub and job s... by nurettin on Saturday November 09, 2019 @02:45AM (#21490235)
To me, the message queue is simply a substitute for polling the DB, not a way of persisting message status. That is a job for the DB.
Re:System design hack: Postgres is a great pub/sub and job s... by blondin on Saturday November 09, 2019 @05:49AM (#21490752)
do you mean that there should be something like a controller that can spin up workers and also get rid of those that hanged (or retry) etc?