it is trivially easy for any node to broadcast its IP address to the whole network periodically (in my case every 2 seconds) using a separate thread and UDP. Using this technique I have rock solid ZeroMQ topology that reconnects with max downtime about 2.5 seconds (because I broadcast every 2 sdconds) for any single node failure. I agree that this functionality could be better implemented in zmq but using this simple technique, the rest of zmq becomes amazing. In Python:

  import socket
  import time
  cs = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
  cs.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)
  cs.setsockopt(SOL_SOCKET, SO_BROADCAST, 1)
  while True:
      cs.sendto('Node ID', ('255.255.255.255', 54545))
      time.sleep(4)

Everybody listening on the same on port 54545 without knowing Node ID's IP address will get these messages which includes the broadcaster IP address.

  import socket
  s=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
  s.bind(('',54545))
  m=s.recvfrom(1024)
  print m[0]

This is a very useful technique when using ZeroMQ generally as you can broadcast services without knowning any IP address so they can come up and down on new addresses if / when necessary.

This would be great, but I don't think it works on AWS -- I don't think they support broadcast.

My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query.

Lots of ways to skin this cat...

indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.

For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works.

As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.

that's a very fair point. There are few applications where if the DB goes down, there's any point for the rest of the system to stay up. My use case happens to be distribution of financial tick info without persistence - ie: no cares on temporary data losses. Therefore it's very peer-oriented and doesn't care about lost messages or duplicated messages. All it cares about is throughput of latest data samples (note the world "samples". Not every single data point is needed though it's nice to have 99.999.. and we usually get that).

It's amazing how small use case differences can mean very big differences in the effectiveness of various strategies. The at least once / at most once / no guarantee either way but soft real time, use cases, can mean radically different toolchains. This is the real lesson of distributed streaming.

This is exactly how we solved our "discoverability" problem at the end of the day. It was pragmatic enough for us to have each node register itself in our production MongoDB in a collection with a TTL index. If the node went down, the DB itself removed its registration.

What if they crash without registering an exit?

Python context managers and heartbeats :)

They broadcast every 2 seconds. No heartbeat = dead.

So there's more machinery waiting in the wings, not just the bog-simple query. Presumably you've barely described all of the mechanisms, and I won't bother socratically making the point that handwaving complexity away isn't the same thing as simplifying. I'll simply state it.

The above code is wrapped in a thread and runs non stop in each node. It's really not very complicated. It's basically sticking a def around the top block of code and then thread.start() in the main. It's extremely cheap because it sends then sleeps for a few seconds. Any node than then listen to the broadcasts on a known port (but no IP address needed) and will know exactly what the state of the system is and all the IPs. Then in the main loop you just make sure that you've gotten a recent pulse otherwise you reconnect. If you're using zmq it's just another short poller entry amongst the others that you will already inevitably have. Literally a couple of lines. It allows you to bring nodes up and down at will for robustness and scalability.

I think it's much less complex than running an entire database node just for this, which btw will also require you constantly to poll, and will require you to bring in an (often heavyweight) client library into each node too, as opposed to standard-library sockets which if you're running multinode you're almost certainly already importing. If you're looking for "simple" distributed computing my sense is that that has yet to be invented.

That's very interesting and I did not know that. I happen to run my own modest (12-node) cluster, and had planned to move to cloud if I grew beyond 20 nodes. The fact that I would not own the transport layer in my case is quite severely problematic. Going to have to investigate if this is the case at Azure and the rest too. Can't I partition my instances into a VPN and UDP broadcast within that? If anybody has any insight, very happy to hear about it.

The ZeroMQ documentation is pretty up front about the need for you to build those pieces yourself. It would appear you chose the wrong tool for your requirements.

ZeroMQ wants to be a neutral wrapper in any language but in the end its a C++ library enforcing C++ concepts. You can't map in a straightforward way process concepts from C++ to other languages (also the OS and VM sits in between that). In the end its just mapping programming logic to state machines. There are much better ways to do this and end up with something much more powerful (with first class meta-programming).

I've found ZeroMQ to be an immensely useful and powerful library for certain kinds of distributed queueing applications. It's just a library, not an entire application or web framework to drop practically pre-dictated code into.

Also what "c++ concepts" does it enforce and how? And why is that a bad thing?

ZeroMQ has native stacks in Java, C#, Erlang, C. Also C++.

If you try to use ZeroMQ to replace TCP with the same semantics, then no, it won't work.

How should ZeroMQ replace TCP? Internet runs on TCP/IP (last time I checked). In private networks many problems of public networks don't appear. I'm aware that organisations use it for their internal systems. As soon as one bridges to the outside world, one is going to hit with a problem.

ZeroMQ is a messaging layer that runs over numerous network protocols (TCP, UDP, PGM, TIPC,...), as well as inter-process and inter-thread. It provides abstractions (like pub-sub) that work over all these protocols, and it shows the developer the abstraction using a socket-like API.

In practice you build services which are internally composed and scaled using threads and processes, and which talk to each other over plain (unencrypted) protocols, e.g. TCP between boxes, IPC between processes on the same box. And when you need to talk to the outside world you build bridges that can speak any protocol you like, such as HTTP, or ZeroMQ's encrypted TCP protocol (CurveZMQ). From the developer's point of view, it's mostly transparent. Obviously you need security infrastructure, e.g. key/certificate management.

You can build entire ZeroMQ applications using only encrypted TCP and they will run on public infrastructure. You can develop and test the same apps on a laptop. You might run the same code using IPC on a laptop, and CurveZMQ over the Internet.

So though ZeroMQ definitely uses TCP as one of its transport protocols, it's not a replacement for it.

Which means that doing things like asking for the IP address of a peer make no sense. What's the IP address of a thread, or another process on the same box? If you need identity information, you should not use the transport protocol for that.