Delivering Billions of Messages Exactly Once

Re:Delivering Billions of Messages Exactly Once by deepsun on Thursday June 29, 2017 @04:47PM (#14666731)

Are you talking about distributed environment, where network partitions can occur? If yes, then there's Two Generals Problem and "FLP result", that just prove it impossible. So I guess you're talking about non-distributed environment.

In other words, to reliably agree on a system state (whether message id was delivered) you need the system to be Consistent. And per CAP theorem, it cannot be Available in presence of Partitions.

So other people you're referring to probably talk about distributed systems.

[ Parent | on HN ]

Re:Delivering Billions of Messages Exactly Once by rusanu on Thursday June 29, 2017 @04:56PM (#14666807)

Yes, I'm talking about distributed systems and I am aware of the CAP theorem. Hence my choice of the word 'practical'.

As I said, users had cases when the plumbing (messaging system) recovered and delivered messages after +40 days of network partitioning. Correctly written apps completed the business process associated with those messages as normal, no special case. Humans can identify and fix outages and databases can easily outlast network outages (everything is durable, transacted, with HA/DR). And many business processes make perfect sense to resume/continue after the outage, even if it lasted for days.

[ Parent | on HN ]

Re:Delivering Billions of Messages Exactly Once by alexbeloi on Thursday June 29, 2017 @06:41PM (#14667444)

I'm not really versed in this topic, but it seems like using a database for a socket makes the system entirely centralized around that database. Is there something I'm missing?

[ Parent | on HN ]

Re:Delivering Billions of Messages Exactly Once by ztorkelson on Thursday June 29, 2017 @11:31PM (#14668468)

ServiceBroker, at least, had the capability of (transactionally) sending messages between databases. So, if you drank the kool-aid (I did; it wasn't so bad), there needn't be "the centralized database". You can separate your databases and decompose your services, and indeed it's easier to do so correctly and with confidence because the technology eliminates a lot of hairy edge cases.

[ Parent | on HN ]

Re:Delivering Billions of Messages Exactly Once by rusanu on Thursday June 29, 2017 @05:05PM (#14666865)

Pat had some opinions about CAP and SOA and distributed systems, see [0]. I also remember a talk given by Pat and Eric Brewer together, that went deeper into the whole CAP ideas vis-a-vis the model Pat had been advocating (see Fiefdoms and Emissaries [1]), but I can't remember when it was or find a link for it.

[0] https://blogs.msdn.microsoft.com/pathelland/2007/05/20/soa-a...

[1] http://download.microsoft.com/documents/uk/msdn/architecture...

[ Parent | on HN ]

Re:Delivering Billions of Messages Exactly Once by aetherson on Thursday June 29, 2017 @05:32PM (#14667019)

Exactly once delivery is theoretically impossible.

Approaching exactly once delivery asymptotically is possible. Your parent poster's point is that this is one where you can get so close to exactly once in order that in practice you never violate for years and years.

[ Parent | on HN ]

Re:Delivering Billions of Messages Exactly Once by rusanu on Thursday June 29, 2017 @06:14PM (#14667281)

My point is that I've seen people making decisions to go with 'best effort delivery' and live with the (costly) consequences because they read here and there that EOIO is impossible, so why bother trying.

[ Parent | on HN ]

Re:Delivering Billions of Messages Exactly Once by inlined on Thursday June 29, 2017 @09:13PM (#14668056)

Because idempotentcy can be cheap at the application layer, so why try to solve something we know can never truly be solved?

[ Parent | on HN ]