News for hackers. Stuff that matters.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by krat0sprakhar on Saturday October 17, 2015 @11:30PM (#10407112)
This couldn't have come at a better time - I was actually looking for a durable message-queue written in Go. Is there any way to read more about the architecture of this system? I find systems like these to be quite fascinating but taking the time to go through the code can sometimes be very time-consuming. It would be awesome if more projects have a writeup as detailed as cockroachdb[0]!

Aside: There used to be a site sometime back which used to distribute compiled binaries of Go code for all platforms? Is it still up any chance?

[0] - https://github.com/cockroachdb/cockroach#architecture

Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by justinsaccount on Sunday October 18, 2015 @08:52AM (#10408087)
http://nsq.io/

http://nsq.io/overview/internals.html

The service you are thinking of might be https://github.com/ddollar/godist

You can see the download links on https://github.com/ddollar/forego use it.

Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by Bogdanovich on Sunday October 18, 2015 @03:08AM (#10407461)
It's really simple. Each queue is a separate leveldb database on disk. Messages are stored as key/value using incremental ids. Head and tail of the queue are kept in memory and get initialized on startup via db scan.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by dave_ops on Sunday October 18, 2015 @10:32AM (#10408317)
Also, you have to be paying one hell of a compaction penalty if this isn't a grow-only dataset. By ordering your keys you're at least minimizing the overhead of compaction on write by utilizing the happy-path for how LevelDB moves data out of the write buffer and into the SSTs.

But deletes are going to have a big impact still, and (working from my failing memory of LevelDB internals) I think might actually be the pathologically sad case.

Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by dave_ops on Sunday October 18, 2015 @10:19AM (#10408283)
Why don't you just store the head and tail as K/V entries? You have a durable K/V store at your disposal.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by Bogdanovich on Sunday October 18, 2015 @12:38PM (#10408724)
There is no benefit in that except faster startup time. As a downside you'll get a lot head/tail db keys updates.
Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by dave_ops on Sunday October 18, 2015 @05:46PM (#10409933)
Fast start times are a valuable thing for a service component.

Stick about 10GB of small entries in it (should be enough to create all the levels) and then see what happens.

Also, you could reserve the persisted [H|T] for controlled shutdown scenarios. Basically anything that isn't complete system failure if you're properly trapping signals.

Re:Siberite: A Simple LevelDB-Backed Message Queue in Go by Bogdanovich on Sunday October 25, 2015 @04:34PM (#10448286)
I added some more benchmarks including packing with 200M small 64 byte messages (20Gb) and consumption of that queue. There is no slowdown because of mass delete. https://github.com/bogdanovich/siberite/blob/master/docs/ben...