Re:Sparkey – Key/value storage by Spotify
by atdt on Monday September 02, 2013 @07:36AM
(#6314637)
I was surprised to see LevelDB (https://code.google.com/p/leveldb/) was missing from the list of storage solutions you tried, because it seems optimal for your use-case. Were you aware of it?
[ Parent
| on HN ]
Re:Sparkey – Key/value storage by Spotify
by blippie on Monday September 02, 2013 @08:07AM
(#6314749)
I'm not sure about the optimal use-case match. Sparkey is for "mostly static" datasets where on disk structures are generated by a batch process and pushed to servers providing read only access to the data to consumers.
leveldb on the other hand, supports concurrent writes and provides features to handle data consistency and cheap gradual reindexing.
Re:Sparkey – Key/value storage by Spotify
by illumen on Monday September 02, 2013 @09:38AM
(#6315074)
Also Sparkey seems to work well with bittorrent/rsync distribution. I recall spotify use bittorrent to distribute files to their servers.
Re:Sparkey – Key/value storage by Spotify
by sorbits on Monday September 02, 2013 @08:19AM
(#6314786)
According to GitHub, last commit (which is also initial checkin) is two years old.
Re:Sparkey – Key/value storage by Spotify
by woogley on Monday September 02, 2013 @08:48AM
(#6314887)
The project is hosted at Google Code. Last commit was Aug 21: https://code.google.com/p/leveldb/source/detail?r=748539c183...
Re:Sparkey – Key/value storage by Spotify
by iampims on Monday September 02, 2013 @03:25PM
(#6316661)
There is also the basho fork of LevelDB (https://github.com/basho/leveldb) which adds a few nice things.
If you're interested in the internals of LevelDB, I strongly recommend watching this talk http://www.youtube.com/watch?v=vo88IdglU_8 (slides here https://speakerdeck.com/basho/optimizing-leveldb-for-perform...)
He means the last commit on sparkey
Right, this is actually the first commit - it's just that the history was squashed before publish - we had to remove sensitive Spotify specific things in it, and it seemed easiest to just do a big squash.
Why do you think it seems optimal for their use case?
A quick glance over LevelDB's features gives me the impression that its bulk-write performance would not be sufficient.
The problem is, that so many projects for so many things exists. It is hard to find the matching project, thats why many people invent the wheele again.
Re:Sparkey – Key/value storage by Spotify
by hyc_symas on Wednesday September 04, 2013 @09:48AM
(#6327499)
LevelDB has very few good use cases and this certainly isn't one of them. http://symas.com/mdb/microbench/ http://symas.com/mdb/hyperdex/