Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by Bogdanovich on Sunday October 18, 2015 @03:08AM
(#10407461)
It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.
[ Parent
| on HN ]
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by dave_ops on Sunday October 18, 2015 @10:32AM
(#10408317)
Also, you have to be paying one hell of a compaction penalty if this isn't a grow-only dataset. By ordering your keys you're at least minimizing the overhead of compaction on write by utilizing the happy-path for how LevelDB moves data out of the write buffer and into the SSTs.
But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by dave_ops on Sunday October 18, 2015 @10:19AM
(#10408283)
Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by Bogdanovich on Sunday October 18, 2015 @12:38PM
(#10408724)
There is no benefit in that except faster startup time. As a downside you'll get a lot head/tail db keys updates.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by dave_ops on Sunday October 18, 2015 @05:46PM
(#10409933)
Fast start times are a valuable thing for a service component.
Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens.
Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go
by Bogdanovich on Sunday October 25, 2015 @04:34PM
(#10448286)
I added some more benchmarks including packing with 200M small 64 byte messages (20Gb) and consumption of that queue. There is no slowdown because of mass delete. https://github.com/bogdanovich/siberite/blob/master/docs/ben...