Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by Bogdanovich on Sunday October 18, 2015 @12:08AM
(#10407183)
You can have large queue sizes (larger than RAM size) and siberite would still consume small amount of resident memory. You basically don't need a separate server with decent amount of memory for it. You can also can get benefit from two-phase reliable fetch - if your client gets disconnected without confirming a message, the message will be served to another client (very convenient if you use amazon spot instances for your workers).
[ Parent
| on HN ]
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by eis on Sunday October 18, 2015 @01:15AM
(#10407299)
Note that this also means that messages can be delivered more than once and/or that the clients need to remember the messages that they processed. In some setups that can be a showstopper.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by Bogdanovich on Sunday October 18, 2015 @01:36AM
(#10407321)
Reliable fetch is a feature, not a protocol requirement.
You can use simple 'get work_queue' command to just get a message, or you can use 'get work_queue/open', 'get work_queue/close' - two phase fetch if you need a reliable fetch. You can also use 'get work_queue/close/open' command to acknowledge previous message and read a new one.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by eis on Sunday October 18, 2015 @01:54AM
(#10407344)
Ok so you can switch between at-most-once and at-least-once guarantees. While nice to have both options in a message queue, my point still stands.
Each of these have trade-offs and the way it is architectured here, in the at-least-once case you will have to either remember all the processed messages or be prepared to process a message multiple times, whatever that means in your specific use-case.