News for hackers. Stuff that matters.
Re:From Kafka to ZeroMQ for real-time log aggregation by oldmanjay on Saturday September 17, 2016 @11:42AM (#12520947)
What if they crash without registering an exit?
Re:From Kafka to ZeroMQ for real-time log aggregation by kchoudhu on Saturday September 17, 2016 @01:18PM (#12521365)
Python context managers and heartbeats :)
Re:From Kafka to ZeroMQ for real-time log aggregation by vegabook on Saturday September 17, 2016 @01:42PM (#12521501)
They broadcast every 2 seconds. No heartbeat = dead.
Re:From Kafka to ZeroMQ for real-time log aggregation by oldmanjay on Monday September 19, 2016 @05:56PM (#12535105)
So there's more machinery waiting in the wings, not just the bog-simple query. Presumably you've barely described all of the mechanisms, and I won't bother socratically making the point that handwaving complexity away isn't the same thing as simplifying. I'll simply state it.
Re:From Kafka to ZeroMQ for real-time log aggregation by vegabook on Tuesday September 20, 2016 @03:55AM (#12537889)
The above code is wrapped in a thread and runs non stop in each node. It's really not very complicated. It's basically sticking a def around the top block of code and then thread.start() in the main. It's extremely cheap because it sends then sleeps for a few seconds. Any node than then listen to the broadcasts on a known port (but no IP address needed) and will know exactly what the state of the system is and all the IPs. Then in the main loop you just make sure that you've gotten a recent pulse otherwise you reconnect. If you're using zmq it's just another short poller entry amongst the others that you will already inevitably have. Literally a couple of lines. It allows you to bring nodes up and down at will for robustness and scalability.

I think it's much less complex than running an entire database node just for this, which btw will also require you constantly to poll, and will require you to bring in an (often heavyweight) client library into each node too, as opposed to standard-library sockets which if you're running multinode you're almost certainly already importing. If you're looking for "simple" distributed computing my sense is that that has yet to be invented.