News for hackers. Stuff that matters.
Re:Sparkey – Key/value storage by Spotify by atdt on Monday September 02, 2013 @07:36AM (#6314637)
I was surprised to see LevelDB (https://code.google.com/p/leveldb/) was missing from the list of storage solutions you tried, because it seems optimal for your use-case. Were you aware of it?
Re:Sparkey – Key/value storage by Spotify by blippie on Monday September 02, 2013 @08:07AM (#6314749)
I'm not sure about the optimal use-case match. Sparkey is for "mostly static" datasets where on disk structures are generated by a batch process and pushed to servers providing read only access to the data to consumers.

leveldb on the other hand, supports concurrent writes and provides features to handle data consistency and cheap gradual reindexing.

Re:Sparkey – Key/value storage by Spotify by illumen on Monday September 02, 2013 @09:38AM (#6315074)
Also Sparkey seems to work well with bittorrent/rsync distribution. I recall spotify use bittorrent to distribute files to their servers.
Re:Sparkey – Key/value storage by Spotify by sorbits on Monday September 02, 2013 @08:19AM (#6314786)
According to GitHub, last commit (which is also initial checkin) is two years old.
Re:Sparkey – Key/value storage by Spotify by woogley on Monday September 02, 2013 @08:48AM (#6314887)
The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183...
Re:Sparkey – Key/value storage by Spotify by iampims on Monday September 02, 2013 @03:25PM (#6316661)
There is also the basho fork of LevelDB (https://github.com/basho/leveldb) which adds a few nice things.

If you're interested in the internals of LevelDB, I strongly recommend watching this talk http://www.youtube.com/watch?v=vo88IdglU_8 (slides here https://speakerdeck.com/basho/optimizing-leveldb-for-perform...)

Re:Sparkey – Key/value storage by Spotify by bouk on Monday September 02, 2013 @08:59AM (#6314923)
He means the last commit on sparkey
Re:Sparkey – Key/value storage by Spotify by krka on Monday September 02, 2013 @10:14AM (#6315233)
Right, this is actually the first commit - it's just that the history was squashed before publish - we had to remove sensitive Spotify specific things in it, and it seemed easiest to just do a big squash.
Re:Sparkey – Key/value storage by Spotify by tinco on Monday September 02, 2013 @08:02AM (#6314733)
Why do you think it seems optimal for their use case?

A quick glance over LevelDB's features gives me the impression that its bulk-write performance would not be sufficient.

Re:Sparkey – Key/value storage by Spotify by progx on Monday September 02, 2013 @08:00AM (#6314721)
The problem is, that so many projects for so many things exists. It is hard to find the matching project, thats why many people invent the wheele again.
Re:Sparkey – Key/value storage by Spotify by hyc_symas on Wednesday September 04, 2013 @09:48AM (#6327499)
LevelDB has very few good use cases and this certainly isn't one of them. http://symas.com/mdb/microbench/ http://symas.com/mdb/hyperdex/