I have questions, I'd be glad if you could answer them:
1. DGraph v0.2 isn't production ready?
2. DGraph API doc is missing?
3a. Does DGraph support bulk loading of RDFs only? No support for graphSON?
3b. Does DGraph support incremental loading of a Graph? Or just bulk loads?
4. Is 'distribution' achieved by maintaining a copy of entire Graph data across all instances? Or is data distributed too?
5. What's with the UID generation? Is to establish a partioning scheme?
3a. Yes, with the 2 phase loader. Only RDFs are supported right now, nothing else. https://github.com/dgraph-io/dgraph#distributed-bulk-data-lo...
3b. Yes, with mutations. https://github.com/dgraph-io/dgraph#queries-and-mutations
4. It's truly distributed. The data is actually sharded, with each shard containing part of the data and served by a separate instance. The bulk loader instructions generate 3 instances.
5. To keep the queries, data storage and data transfer efficient, we assign a uint64 ID to all entities. UID assignment is that operation.
A few Qs abt the storage layer:
DGraph supports replication too, in case a node fails...?
Given your description, I take that you've implemented a custom data distribution protocol on top of rocksdb? Do you have plans to extract this 'distributed rocksdb' out to its own implementation? How would something like this compare to actordb.com and/or rqlite?
Thx again.
DGraph would support high availability, which means all our shards would be replicated 3x across servers, so in case one server fails, the shards would still be available for querying and mutations. In addition, shard movements to other servers would happen so the replication factor remains the same. We aim to achieve this using (Etcd's) RAFT protocol, by version 0.4.
RocksDB is just a medium for us to have something between the database and disk. All the data arrangement, handling, movement etc. happens above RocksDB. So, no there's no "distributed rocksdb" here.
DGraph, OTOH, is a native graph database. We do use RocksDB, but the data distribution and maintenance is done by us. It's optimized for decreasing the number of network calls, to keep them linear to the complexity of the query, not the number of results. This is of incredible value when you're running real time queries, serving results directly to end user. The query latency hence, isn't too affected by high fan out of intermediate results and should remain low; while providing high throughput.
In fact, the entire HN traffic is being served by one GCE n1-standard-4 instance right now, using all 4 cores really well :-).