I've worked with some pretty big systems that use a model that look an awful lot like 0mq and perform splendidly. I've also worked on several smaller systems (games) that needed to use UDP and all kinds of custom semantics for reliability to accomplish their task. I'd imagine that if such a thing happens when you scale down, what you're saying is possible when you scale up.
To anyone reading this thread: If you don't understand all the complexities of jandrewrogers' response, just assume YAGNI. In that case, if your problem lends itself to a stream oriented protocol, then ZeroMQ is a great choice. You can deal with the other complexities if the need arises.
Lots of things work very well with ZMQ, just not distributed systems. If your application scales efficiently on ZMQ then it is a giant pile of computers and not a "distributed system". I've seen systems that scale to thousands of nodes and also seen systems that won't scale past a few machines on ZMQ. System architects should be able to distinguish the two cases without standing up a cluster to see if it fails. All it requires is one part of your system to not fit the ZMQ case for things to start falling apart. That this is an expected result is not exotic computer science.
At this point, I'm invoking Poe's law.
What is interesting about the literature on designing systems with robustly stable and efficient network behaviors is how restrictive the known solutions to this problem are in terms of how a protocol must be designed. At the same time, it is enlightening to realize that every robust large-scale distributed system employs the mathematics pervasively. Email spam is a predictable failure mode for the SMTP protocol per the mathematics, once you start thinking about it that way.
The most important counter-intuitive idea is that robust distributed protocols are subtractive. You can't design a distributed system around creating good properties per se, you can only design a distributed system that suppresses bad properties. An optimal distributed protocol is one in which the various well-known pathologies are naturally and effectively suppressed by the behavior of the protocol without suppressing aggregate throughput.
Robust, high-throughput distributed systems are built around the concept of Nash equilibria. To ensure that, there is an optimal ordering to the set of possible message operations over those queues. It gives them a priority and you attempt to schedule operations in approximately optimal order. The ordering is not fixed; every message operation on a queue may have side effects that alters the total ordering of message operations for any process associated with that queue. Consequently, the schedule of operations over the entire set of message queues must be adaptively and dynamically imposed in order to approximate a good Nash behavior.
In real software, this is usually just a simple state machine that adaptively prioritizes operations based on the aggregate state of those messages and queues. Also, we usually approximate ideal scheduling for performance reasons (and good approximations are usually good enough). Priority modifying side effects propagate throughout the distributed system; local modifications in priority provide stabilizing negative feedback to the global behavior. (See also: greedy routing optimization problems etc. This is a pretty complex area of mathematics.)
Consequently, there are some popular design choices for network server systems that tend to work poorly for distributed systems, either because they prevent the creation of an effective operation scheduler or because they hinder the propagation of negative feedback that prevents pathological interactions.
Giving every socket its own OS thread or similar is right out. That lets the operating system decide when operations are scheduled and the operating system has no concept of prioritization in the sense that is important for distributed systems. Trying to use thread interlocks to impose an order leads to pathological context-switching storms. High-performance schedulers are cooperative. (You can design a schedule with multiple threads that executes with minimal interlocks, but it is not trivial.)
Unlimited buffering and hard-limited buffering are also poor abstractions because they obscure the state of a queue. Robust schedulers often prioritize queues based in part on how it reduces total buffering. For some good designs, you can prove the existence of a reasonable upper bound on the total buffers required for a node or queue with no hard limits on any particular buffer. It is an elegant side effect of scheduling interactions in a distributed system but requires a scheduler that is aware of many aspects of buffer state.
In principle, it should be possible to design a message queue abstraction designed for the requirements of distributed systems. It would not be as simple as ZMQ though. ZMQ is not designed to support that use case, so in cases where I am building a non-trivial distributed system protocol, we build on top of epoll directly. It is not a knock against ZMQ, it was designed for other purposes.