They solve slightly different (architecture) problems and they are on completely different ends of the message queue library spectrum of "usability vs. customizability".

RabbitMQ is a "batteries included" solution. ZeroMQ is a roll your own sort of library. If you just want a message queue use RabbitMQ. If you want to build your own message queue system (with complex or specific requirements) use ZeroMQ.