News for hackers. Stuff that matters.
Re:Show HN: Nebula – a distributed graph database written in... by bane on Tuesday January 14, 2020 @11:48PM (#22051830)
This is very interesting. Anybody use it? Alternatives?
Re:Show HN: Nebula – a distributed graph database written in... by lmeyerov on Wednesday January 15, 2020 @04:00AM (#22052726)
We help bring gpu visual analytics & investigation automation to users of all sorts of graph DBs (think tableau & servicenow for graph), so based on our enterprise/big tech/gov/startup interactions:

1. Shortlist (and in no order): Neo4j, AWS Neptune, Datastax Graph, TigerGraph, Azure CosmosDB, and JanusGraph (Titan fork) are the ones we see the most in practice, and not in production but rumor-mill, Dgraph, RedisGraph, & ArangoDB. The three-and-four-letter types seem to roll their own, for better or worse. There are also some super cool ones that don't get visibility outside of the HPC+DoD world, like Stinger & Gunrock. Interestingly, the reality is a ton of our graph users aren't even on graph DBs (think Splunk/ELK/SQL), and for data scientists, just do ephemeral Pandas/Spark. As someone from the early days of the end-to-end GPU computing movement, we're incorporating cuGraph (part of nvidia rapids.ai) into our middle tier, so you get to transparently benefit from it while looking at data in any of the above.

2. I now slice graph DB's more in terms of OLTP (neo4j, janus, neptune, maybe tiger) vs OLAP (spark graphx, cugraph) vs batch (janus, tiger) vs friendly BI/data science (neo4j) vs friendly app dev / multi-modal add-on (CosmosDB, Neo4j, Arango, Redis). Curious to see how this goes -- given the number of contributors, I'm guessing it's doing well in at least one of these. +1 to hearing reports from others!

Re:Show HN: Nebula – a distributed graph database written in... by bane on Wednesday January 15, 2020 @08:55AM (#22054110)
Thanks, I really appreciate the comprehensive write up of what your team is seeing. Any chance of a longer blog post that expands on this, especially pro-cons and performance?
Re:Show HN: Nebula – a distributed graph database written in... by lmeyerov on Wednesday January 15, 2020 @02:04PM (#22057303)
Yes, that is a great idea!
Re:Show HN: Nebula – a distributed graph database written in... by derefr on Wednesday January 15, 2020 @11:11AM (#22055434)
For someone who just wants to run some (intensive) OLAP graph queries on the “graph formulation” of a relational or hierarchical dataset every once in a while (maybe batch, maybe user-initiated, but either way <1QPS), but doesn’t yet have a graph DB and doesn’t really want to maintain their data in a canonical graph formulation, which type of graph DB would you recommend as the simplest-to-maintain, simplest-to-scale “adjunct” to their existing infra?

I.e. what’s the graph DB that best fits the use-case equivalent to “having your data in an RDBMS and then running an indexer agent to feed ElasticSearch for searching”?

Re:Show HN: Nebula – a distributed graph database written in... by lmeyerov on Wednesday January 15, 2020 @01:57PM (#22057229)
My default nowadays is minimize work via "no graph db": csv/parquet extract -> jupyter notebook of pandas/cugraph/graphistry, and if that isn't enough, then dockerized (=throwaway) neo4j , or if the env has it, spark+graphistry. The answers to some questions can easily switch the answer to say "kafka -> tigergraph/janusgraph/neptune", or some push button neo4j/cosmosdb stuff:

* Primary DB: type / scale, and how fresh do the extracts need to be (daily, last minute?)

* Are queries more search-centric ("entities 4 hops out") or analytics ("personalized pagerank")?

* Graph size: 10M relations, or 10B? Document heavy, or mostly ints & short strings?

* Is the client consuming the graph via a graph UI, or API-only?

* Licensing and $ cost restrictions?

* Push-button or inhouse-developer-managed?

The result of (valid) engineering trade-offs by graph db dev teams means that, currently, adding a graph db as a second system can be tricky. The above represent potential mismatches between source db / graph stack / workload and team burden. Feels like this needs a flow chart!

Happy to answer based on the above, and you can see why I'm curious which areas Nebula will help straddle :)

Re:Show HN: Nebula – a distributed graph database written in... by jamie-vesoft on Thursday January 16, 2020 @05:24AM (#22063236)
Very insightful answer! Thanks for sharing your opinions here. Nebula Graph is good at OLTP use cases where high QPS and low latency are required.
Re:Show HN: Nebula – a distributed graph database written in... by sfgweilr4f on Wednesday January 15, 2020 @03:18AM (#22052535)
I'd say dgraph (https://dgraph.io/) is closest competitor but Neo4j (https://neo4j.com) as well which has longer heritage.

I'd also include redis because of the graph module (https://oss.redislabs.com/redisgraph/).

I've likely missed a bunch of others. Add them as I'm interested in graph db and have only scratched the surface myself.

Re:Show HN: Nebula – a distributed graph database written in... by rapnie on Wednesday January 15, 2020 @03:59AM (#22052719)
There is also decentralized Gun DB: https://gun.eco/

OrientDB is an alternative to Neo: https://orientdb.org/

There are a bunch of DB's compatible with Tinkerpop and e.g. query-able with Gremlin: http://tinkerpop.apache.org/

Re:Show HN: Nebula – a distributed graph database written in... by rapnie on Wednesday January 15, 2020 @08:06AM (#22053827)
There is also Weaviate, still in development, which has a flavor of GraphQL for querying: https://github.com/semi-technologies/weaviate

And this awesome page has some good entries: https://github.com/jbmusso/awesome-graph/blob/master/README....

Re:Show HN: Nebula – a distributed graph database written in... by jhoechtl on Wednesday January 15, 2020 @12:31AM (#22051966)
Alternative:Dgraph

https://dgraph.io/

In architecture and goals it actually closely resembles Dgraph, would love to see an (opinionated) comparison by Manish, the CEO of Dgraph

Re:Show HN: Nebula – a distributed graph database written in... by mrjn on Thursday January 16, 2020 @02:12PM (#22067803)
(Manish here) Don't know much about Nebula. Feels quite inspired by Dgraph.
Re:Show HN: Nebula – a distributed graph database written in... by shermanye on Thursday January 16, 2020 @09:31PM (#22071334)
(Sherman here. I'm the founder of Nebula) Nice to meet you here, Manish. Nebula is actually inspired by the Facebook internal project Dragon (https://engineering.fb.com/data-infrastructure/dragon-a-dist...). Fortunately I was one of the founding members of the project. The project was started in 2012. We never heard of dgraph at that time. So I'm not sure who was inspired :-)

The goal of Nebula is to be a general graph database, not just a knowledge graph database. There are some fundamental differences between the two.

We welcome any positive feedback and technical discussion. We would love to learn to the community and to provide a product which truly satisfies customers' needs.

Re:Show HN: Nebula – a distributed graph database written in... by jhoechtl on Thursday January 23, 2020 @10:41AM (#22128381)
> The goal of Nebula is to be a general graph database, not just a knowledge graph database. There are some fundamental differences between the two.

I am by no means a Graph expert but what are some of the mentioned fundamental differences?

Re:Show HN: Nebula – a distributed graph database written in... by w3clan on Wednesday January 15, 2020 @02:39AM (#22052385)
I had the very same feeling, dgraph is older and has a larger community plus additional features like:

- geospatial features

- good speed as it is based on badgerdb key value database and ristello cache library.

- http library and other features

One of the advantage I saw in nebula graph is security role based access which is not available in dgraph until today.

I am very curious about benchmark between nebula graph and dgraph.

Also what is storage system used in nebula graph.

Re:Show HN: Nebula – a distributed graph database written in... by ayorosmage on Wednesday January 15, 2020 @04:15AM (#22052812)
ACL is an enterprise dgraph feature: https://dgraph.io/support
Re:Show HN: Nebula – a distributed graph database written in... by jamie-vesoft on Wednesday January 15, 2020 @07:23AM (#22053591)
geospatial is also available in Nebula Graph.:)

As to the storage system, Nebula Graph is based on multi-group raft and RocksDB.

Re:Show HN: Nebula – a distributed graph database written in... by w3clan on Thursday January 16, 2020 @08:43AM (#22064398)
Can you add support for module based storage system like, if someone wants to use badgerdb or leveldb or any other storage system instead of rocksdb
Re:Show HN: Nebula – a distributed graph database written in... by jamie-vesoft on Friday January 17, 2020 @12:07AM (#22072052)
Yes, Nebula Graph supports multiple backend storages by design. So theoretically you are able to use whatever storage you want for whichever graph space in Nebula Graph.

You may take a look at this article about the design of our storage engine: https://github.com/vesoft-inc/nebula/blob/master/docs/manual...

In 2020 we will be working on more plugins. You may stay tuned if that interests you. :)

Re:Show HN: Nebula – a distributed graph database written in... by jhoechtl on Wednesday January 15, 2020 @04:49AM (#22052970)
Geospatial is on the TODO list
Re:Show HN: Nebula – a distributed graph database written in... by jamie-vesoft on Wednesday January 15, 2020 @05:05AM (#22053046)
Geospatial support has already been merged to the code base. :)
Re:Show HN: Nebula – a distributed graph database written in... by w3clan on Thursday January 16, 2020 @09:07AM (#22064593)
Dgraph has another disadvantage of data redundancy, where data associated with multiple index are stored multiple times for speed.

Does nebula also store data multiple time for multiple index?

Re:Show HN: Nebula – a distributed graph database written in... by jamie-vesoft on Friday January 17, 2020 @03:51AM (#22073007)
Thanks for asking! Sorry I missed this question earlier.

Nebula doesn't store data multiple times for index.

And here's how the indexing works in Nebula Graph:

You are allowed to create multiple indexes for one tag or edge type in Nebula Graph. For example, if a vertex has 5 properties attached to it, then you can create one index for each if it's necessary for you. Both indexes and the raw data are stored in the same partition with their own data structure for quick query statement scanning. Whenever there are "where" clause/syntax in the queries, the index optimizer decides which index file should be traversed.

Re:Show HN: Nebula – a distributed graph database written in... by campoy on Thursday January 16, 2020 @01:56PM (#22067649)
Hi there, out of curiosity what do you mean the data is stored multiple times for speed?

We (I work at Dgraph) have data redundancy when you have multiple replicas for a given group - but that's an optional feature.

Thanks!

Re:Show HN: Nebula – a distributed graph database written in... by emmanueloga_ on Wednesday January 15, 2020 @03:06AM (#22052484)
Check the list of implementations of SPARQL [1].

One of the most interesting picks: RDF4j (java based). It can connect to a lot of different SPARQL servers, but the rdf4j Native Store should be good enough for data sets in the order of the "100 million triples", according to the docs.

I don't know much about it, but not long ago they announced integrated support for "federated queries", which means that if you data set can't fit in a single node, they have a solution to query different servers in the same query [2].

I'm slowly learning through the forest of related technologies, one of the most useful is SHACL [3], which is a language to validate and extract pieces of the graph that match a pattern (very loosely, think a "schema" for graphs).

1: https://en.wikipedia.org/wiki/List_of_SPARQL_implementations

2: https://rdf4j.org/news/2019/10/15/fedx-joins-rdf4j/

3: https://rdf4j.org/documentation/programming/shacl/

Re:Show HN: Nebula – a distributed graph database written in... by rapnie on Wednesday January 15, 2020 @04:10AM (#22052786)
Before using RDF for graphs one should inform themselves on the differences between labeled property graphs and triple stores, and choose the model that best fit their use case.

Take your time and beware of objectivity of article. Vendors try to lure you in. Following has some good info (but a Neo4j bias): https://dzone.com/articles/rdf-triple-stores-vs-labeled-prop...

Re:Show HN: Nebula – a distributed graph database written in... by emmanueloga_ on Wednesday January 15, 2020 @04:45AM (#22052950)
Good point. Funny you mention that article: I remember encountering both that article and another one that provides some counterpoints! [1]

Also, both those articles are a bit old: RDF* ([2],[3]) is a new extension for RDF that makes it easier to accomplish the same kind of things you can do with property graphs. RDF4j has support for RDF* in the roadmap! [4].

To me, the fact that RDF is 1) a simpler and more general model and 2) an open standard with multiple free and commercial implementations; makes RDF a more a attractive option than locking into a single proprietary implementation like Neo4j.

--

1: http://www.snee.com/bobdc.blog/2018/04/reification-is-a-red-...

2: http://olafhartig.de/slides/RDFStarInvitedTalkWSP2018.pdf

3: http://blog.liu.se/olafhartig/2019/01/10/position-statement-...

4: https://github.com/eclipse/rdf4j/issues/1484

Re:Show HN: Nebula – a distributed graph database written in... by zozbot234 on Wednesday January 15, 2020 @07:34AM (#22053644)
RDF is an interoperability mechanism, it has nothing to do with the architecture you use internally for your database. You can have a PostgreSQL database and offer an endpoint for querying it via RDF.
Re:Show HN: Nebula – a distributed graph database written in... by jamie-vesoft on Wednesday January 15, 2020 @05:16AM (#22053085)
Currently the project has been deployed in multiple leading internet companies in China, including Tencent, MeiTuan (Chinese Yelp), Red (Chinese Pinterest), Vivo, and so on.
Re:Show HN: Nebula – a distributed graph database written in... by bane on Wednesday January 15, 2020 @08:57AM (#22054119)
That's pretty impressive. I'd love to see some details blog posts about setting it up, or using it in production (things to watch out for, good practices for provisioning hardware, etc.).
Re:Show HN: Nebula – a distributed graph database written in... by jamie-vesoft on Wednesday January 15, 2020 @09:50AM (#22054586)
There is a Getting Started series in the GitHub wiki page: https://github.com/vesoft-inc/nebula/blob/master/docs/manual...

Check out the Getting started YouTube video here if you prefer video tutorials: https://www.youtube.com/channel/UC73V8q795eSEMxDX4Pvdwmw

Also some FAQs: https://github.com/vesoft-inc/nebula/blob/master/docs/manual...

If you are interested in the architectural design of the project, here are some articles for your reference:

Overview: https://github.com/vesoft-inc/nebula/blob/master/docs/manual...

Storage engine: https://github.com/vesoft-inc/nebula/blob/master/docs/manual...

Query engine: https://github.com/vesoft-inc/nebula/blob/master/docs/manual...

Feel free to contact us if anything is missing. :)

Re:Show HN: Nebula – a distributed graph database written in... by gibsonf1 on Wednesday January 15, 2020 @12:15PM (#22056055)
We're using Blazegraph (The db that Amazon Neptune uses) with great success. We only use the Sparql api with quads so we get nested graphs.