News for hackers. Stuff that matters.
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by ralusek on Friday March 09, 2018 @08:32AM (#16551567)
I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be.

What the world of data needs more of is continued development into novel indexing strategies/implementations. ElasticSearch, Postgres's GIN index on JSONB, MapReduce, graph databases.

I don't need another key value store...

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by zellyn on Friday March 09, 2018 @09:18AM (#16551874)
I don't mean this facetiously: did you read the article? This work presents novel ways of "solving" the problem that cause it to beat the other contenders fairly convincingly (in non-linearizable regimes).
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by SanFranManDan on Friday March 09, 2018 @10:19AM (#16552283)
I agree. However, I stumbled into the world of KV stores (like RocksDB, LMDB, LevelDB, etc..) last year, and what is most surprising is that they all stop in the same place. I understand that they should do one thing and one thing well, but it is still disappointing when you have to implement things like replication, sharding, and indexing yourself.

There really aren't even that many DBMS that are KV (like redis) out there to handle it either. They are normally much more complicated (like adding SQL layer on top of it).

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by hyc_symas on Wednesday March 21, 2018 @02:02PM (#16640742)
BerkeleyDB has a replication engine. IMO that's too much of a kitchen-sink approach. Judging by how many of those KV stores utterly fail to store data reliably, that's already a hard enough problem to solve. Focusing on the local storage is a clearly delineated realm of responsibility. Distribution obviously belongs to a higher logical layer.

Indexing requires knowledge of a higher level data model. (Again, BerkeleyDB has built in support for secondary indexing, but last time I checked it was a quite braindead and slow implementation. Faster to build your own indices instead, using the other facilities provided.)

With that said, while a KV store has no logical data model to apply to index generation, it can at least provide primitives for you to construct your own indices. BerkeleyDB and LMDB do this.

Distribution with transaction support may require help from the storage engine (offering something resembling multi-phase commit). BerkeleyDB provides this already; LMDB will probably provide this in 1.0.

An argument could be made that the storage engine should be able to handle replication/distribution even without understanding the higher level/application data model. BerkeleyDB does this with page-level replication. IME this results in gratuitously verbose replication traffic, as every high level operation plus all of its dependent index updates etc. are replicated as low level disk/page offset operations. IMO it makes more sense to leave this to a higher layer because you can just replicate logical operations, and save a huge amount of network overhead.

As for the possible higher layers - antoncohen's response below gives a few examples. There are plenty of higher level DBMSs implemented on top of LMDB, providing replication, sharding, etc.

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by digikata on Friday March 09, 2018 @01:10PM (#16553602)
I think we're just getting to the point where it may become more common to separate the simpler problem of a single node, non-distributed 'store' with some kv interface below, and then build more complex distributed algorithms in a layer or two above. You can see some of the larger monolithic codebases that had to start by having their own code all the way up and down the stack, but now are starting to experiment with backend store interfaces so you can trade-off some of the strengths and weaknesses of various local store performance areas.

Along the same lines, a few newer codebases for distributed stores seem to be building with those delineations in mind. Another comment brought up Tidb/Tikv for example. Tikv iirc uses RocksDB as its local store.

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by hyc_symas on Wednesday March 21, 2018 @02:06PM (#16640776)
"Just getting to the point" ? OpenLDAP has been architected this way for ~20 years. I think the same could be said for MySQL, as well as SQLServer (built on top of ESENT/JET). Large monolithic data stores are an obvious anti-pattern, reflects short-sighted design process.
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by digikata on Wednesday March 21, 2018 @08:45PM (#16644274)
Ha, you have a good point. I think my comment was directed more to "distributed cloud store" applications, much more so than databases. That begs the question - then what capabilities have those apps been focusing on that they didn't get from existing databases (or thought that they didn't get from existing databases)...
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by antoncohen on Friday March 09, 2018 @01:14PM (#16553615)
RocksDB, LMDB, and LevelDB are basically low level disk representations used by the databases that do things like provide network access, sharding, and replication. OpenLDAP (LMDB), MySQL (MyRocks), Bigtable (LevelDB-like), Riak (LevelDB), etc.

Many are or can be used as key-value stores. MySQL actually has a memcached compatible KV store, using InnoDB for storage. Postgres has HStore. A lot of the distributed databases roughly fall into the category of KV stores: HBase, Riak, Cassandra, DynamoDB, etc.

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by naasking on Friday March 09, 2018 @08:08PM (#16556496)
> I feel like between Redis, S3, Cloud Storage, RocksDB, Cassandra, etc...this area strikes me as one that has been solved as well as we could reasonably expect it to be

This paper literally shows this statement is false. In what other circumstances could you provide one and two orders of magnitude performance improvement, and people would just shrug and say, "meh, I really don't think we need improved performance".

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by tracker1 on Friday March 09, 2018 @04:36PM (#16555271)
All the options you mention have varying features that give it advantages and disadvantages in various scenarios. What works well in on, won't in others, etc.

That said, I was mostly interested on how this compares to etcd, since the use case seems pretty similar. How does Anna scale out (differing data centers) compared to it being a bad idea with etcd for example.

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by doorbumper on Sunday March 11, 2018 @01:45AM (#16561954)
I've just finished reading the paper. One difference is that Anna offers a weaker form of consistency than etcd. Etcd offers sequential consistency, and Anna offers casual consistency. I believe that this could eliminate many etcd use cases, since ordering for non-casual events many be different depending on the node you access.

I'm not sure I fully understand etcd's durability guarantees. When an operation is completed, does that mean that the data is durable on a single machine, and then becomes durable on other machines at a later point? If so, it seems like Anna could offer the same durability guarantees.

I think Anna's architecture could be more of a competitor to Cassandra, or DynamoDB as long as you only need casual consistency. The performance implications do seem pretty interesting.

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by jaricftw on Sunday March 18, 2018 @10:53PM (#16615936)
etcd does not provide key partitioning/sharding, it achieves linearizable consistency using replica state machine via Raft. Anna provides weaker consistency guarantee (causal or read committed), but has partitioning/sharding.
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by xstartup on Friday March 09, 2018 @09:41AM (#16552010)
Cassandra is JVM GC issues

Redis is in-memory only

Cloud Storage - Not sure, how we can use it outside of cloud vendors

RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it? I am sure, they use one internally, why is it not being open-sourced?

There is also badger but most of these only offer low-level operation.

Sorry, most of my developers are unable to consume them just like Redis.

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by majidazimi on Friday March 09, 2018 @10:36AM (#16552421)
> where is the service which adds replication, clustering and network interface on top of it?

Actually I would prefer such kind of database (just the engine).

1. Expose it to network via what ever framework you like. Thrift, Rest, grpc, ... You don't have to include different kind of libraries for each network service. I would love to connect to every network service (Redis, Elasticsearch, Cassandra, MySQL, ...) via a single framework (say grpc).

2. In most large scale scenarios, there is already some kind of log service (DistributedLog, NATS, Kafka, ...). Why not take benefit of that for replication? Isn't it great to separate the engine layer from replication layer? Currently we are doing double replication actually. Replicate data from master DB to slave DB. Then replicate the same data, from any DB to cache, search, ... components. The data is already there on log. Let everyone (slave DB as well as cache/search module) consume it. This is basically state machine replication idiom. PNUTS[0], Twitter K/V database, LinkedIn Espresso [1] (as well as Ambry[2] which is their internal object store), ... use this approach for replication.

3. I would agree with that, they only support basic low level operations.

[0] http://www.vldb.org/pvldb/1/1454167.pdf

[1] http://www.csce.uark.edu/~xintaowu/BDAM/p1135-qiao.pdf

[2] http://dprg.cs.uiuc.edu/docs/SIGMOD2016-a/ambry.pdf

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by kodablah on Friday March 09, 2018 @10:04AM (#16552180)
Agree w/ all these points (forgot "S3 has provider lock-in" though others replicate the API). I do use Cassandra for most use cases and I don't hit GC things, but I understand the concerns that come with the JVM (no, haven't subbed Scylla in yet). One that I haven't tried but want to hear the downsides of is https://github.com/pingcap/tikv (not the DB built on top, but just that one for KV). A nice published list of all cons of all database systems would be ideal.
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by SanFranManDan on Friday March 09, 2018 @10:13AM (#16552248)
From what I could tell, its tikv is virtually useless without the rest of the tidb (the MySQL layer that sits on top written in go) to be useful.

TiKV has no replication / sharding built in, that is actually handled by a Placement Driver (PD)

From the docs: > TiKV is a component in the TiDB project, you must build and run it with TiDB and PD together.

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by bpicolo on Friday March 09, 2018 @12:30PM (#16553285)
You can disk-back redis at various consistency levels if you so choose

https://redis.io/topics/persistence

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by nilkn on Friday March 09, 2018 @01:29PM (#16553747)
Redis supports at least two forms of persistence: (1) periodic checkpoints; (2) writing an append-only log which can be periodically compacted to save space.
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by electrum on Friday March 09, 2018 @02:27PM (#16554226)
> RocksDB - Facebook just outsourced the engine to the community, where is the service which adds replication, clustering and network interface on top of it?

https://myrocks.io/

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by petre on Saturday March 10, 2018 @01:27AM (#16557395)
They also released MyRocks, a RocksDB engine for MySQL which will is being intergrated in MariaDB 10.3 and Percona Server 5.7. I don't think it supports replication yet, but it's a start.
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by zilchers on Saturday March 10, 2018 @01:01PM (#16559435)
Cassandra has a host of other trade offs besides java (tombstones and data garbage collection come to mind).
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by mspoof on Monday March 12, 2018 @06:39AM (#16567211)
Rocks Storage engine in Cassandra which claims to reduce the GC stalls

https://engineering.instagram.com/open-sourcing-a-10x-reduct...

Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by mlevental on Friday March 09, 2018 @08:45AM (#16551663)
there's a new one of those posted like every day - and every single one is "crazy fast" and "super scalable"
Re:Anna: A Fast, Scalable, Flexibly Consistent Key-Value Store by michaelcampbell on Friday March 09, 2018 @09:40AM (#16552003)
Almost as frequent as the new hotness JS framework.