Re:Creating an advanced search engine with PostgreSQL
by xupybd on Wednesday July 12, 2023 @08:00PM
(#36703355)
You could put a Nats JetStream connection in line and you'd have a good buffer to catch the notifications. Not fool proof but then you'd need some kind of batch process that looks for any outstanding changes and syncs them.
We do something like this with our systems. External events get written to the event bus but all operations are idempotent on the event bus. So at night we send another round of the days events to clean up any inconsistencies.
Re:Creating an advanced search engine with PostgreSQL
by stoniejohnson on Wednesday July 12, 2023 @08:23PM
(#36703509)
Making jobs idempotent forces good design imo.
Re:Creating an advanced search engine with PostgreSQL
by freedomben on Thursday July 13, 2023 @11:29AM
(#36710769)
Yes agreed. It's (usually) more work, but the gains in robustness and fault tolerance are (usually) well worth it
That’s a significant risk for things that need to be in sync between two systems, so we stuck with listen/notify for info-level ops things and used polling/queue systems that offered better guarantees for more important tasks. Don’t want to be in a position where a quiet hiccup with a deploy or something results in fun bugs like 0.5% of rows being silently out of sync between ES and Postgres.