News for hackers. Stuff that matters.
Re:Why You Should Never Use MongoDB by hkarthik on Monday November 11, 2013 @01:19PM (#6713015)
>> Some folks say graph databases are more natural, but I’m not going to cover those here, since graph databases are too niche to be put into production.

Is this really true? It sounds like both relational DBs and document DBs are a poor choice for the social network problem posed. I've actually dealt with this exact problem at my last job when we started on Mongo, went to Postgres, and ultimately realized we traded one set of problems for another.

I'd love to see a response blog post from a Graph DB expert that can break down the problem so that laymen like myself can understand the implementation.

Re:Why You Should Never Use MongoDB by mazelife on Monday November 11, 2013 @04:11PM (#6714069)
At my current employer, we're working on a product that relies heavily on a graph DB (Titan, in this case). Performance characteristics vary dramatically based on the type of query you're trying to run so you have to be careful about how you use it. There are certain types of things you might do in a relational DB with no worries but that would perform horribly in Titan. The converse is also true, of course. For example, a query along the lines of "give me a list of friends of friends of person X" is very fast indeed on a graph database, whereas a query like "give me a random person" tend to perform horribly. But we've been able to get impressively fast, real-time performance on graphs with millions of vertices and tens of millions of edges. They're still niche products compared to NoSQL systems like Mongo, Redis, etc. But I don't see any reason to think think that Titan or Neo4J aren't production ready.

Here's a good intro to Titan and how it works: http://www.slideshare.net/knowfrominfo/titan-big-graph-data-...

Re:Why You Should Never Use MongoDB by fat0wl on Monday November 11, 2013 @01:50PM (#6713181)
I would look at Neo4j. I originally came across it when vetting Grails (it has a Grails plug-in) and it seems to be one of the heavy contenders in terms of a production-ready graph DB. People (this article's included) seem to say that production-ready graph DBs don't exist. Maybe these projects are still trying to gain traction? I expect some stable builds will be out there soon if they aren't already...

http://www.neo4j.org/

Re:Why You Should Never Use MongoDB by drone on Monday November 11, 2013 @02:12PM (#6713305)
My experience with Neo4j (this year) was abysmal. The take-away I had was: it's only good for very small graphs.

Generally, I'd spend some time writing a script to load data into it, start loading data, respond to it crashing a few hours later, increase the memory available to the process, start up again, and respond to it crashing a few hours later. I was never able to get any reasonably-sized graph[0] working reliably well without using an egregious amount of memory, and knowing that I would continue to face memory issues, I gave up on Neo4j and found another way to solve my problem.

It may be that I simply was not competent at setting it up properly, but no other data store I've worked with has been as hard to get stable over a moderately sized data set. I spoke with some other people who had worked with Neo4j at the time, and they expressed the same issues - they couldn't make it work for any reasonably-sized dataset and had to find another solution.

[0] Not big, mind you, just reasonably-sized. E.g. 4 million nodes, with each node having an average of 5 edges and 2-4 properties.

Re:Why You Should Never Use MongoDB by mhluongo on Monday November 11, 2013 @02:19PM (#6713344)
Hm, I assume you reached out to the mailing list and what not? I know a number of installations with numbers well above that. Were you using the batch insertion API?
Re:Why You Should Never Use MongoDB by drone on Monday November 11, 2013 @02:34PM (#6713446)
No, I'm sure there are some great running instances out there - but I was put off by the difficulty of getting it reliably running without being an expert in its configuration. Additionally, the fact that I'd have to spend at least $12k/year to have only 3 nodes in a cluster, knowing we'd need a lot more than that as time went on sealed the deal.

We found that we could do everything we needed with secondary processing against our document store at runtime for so much less without adding another layer of complexity to the architecture.

Edit: forgot to mention - no we weren't use batch-insertion in all cases, IIRC, we had issues with duplication and had to do check-if-exists -> create-if-not as we were reading from raw data sources that were heavy with duplicates.

Re:Why You Should Never Use MongoDB by jexp on Monday November 11, 2013 @07:42PM (#6715131)
Many heavy duty production customers of Neo4j run with just a 3 node cluster, no need to scale out as with other NoSQL datastores. And actually they replaced larger clusters with a small Neo4j one.

I would love to learn about your Neo4j setup, and the issues in detail, I want to make it easier for people in your circumstances in the future to get quickly up and running with Neo4j in a reliable manner. If you're willing to help out, please drop me an email at michael at neotechnology dot com.

Re:Why You Should Never Use MongoDB by fat0wl on Monday November 11, 2013 @01:53PM (#6713209)
And I remember in flipping through a book on graph DB engines that some can be mounted as an extra layer on top of relational stores, so there is always that backdoor back into it.
Re:Why You Should Never Use MongoDB by mhluongo on Monday November 11, 2013 @02:21PM (#6713364)
Yeah, FlockDB (https://github.com/twitter/flockdb) comes to mind. I think Titan (https://github.com/thinkaurelius/titan) should / will be able to handle this too.
Re:Why You Should Never Use MongoDB by mhluongo on Monday November 11, 2013 @02:15PM (#6713322)
I can attest that Neo4j is production-ready- I know they're being used at companies like Adobe and Cisco, and we were happy with it at Scholrly.
Re:Why You Should Never Use MongoDB by prathle on Monday November 11, 2013 @06:10PM (#6714746)
More than 30 of the Global 2000 now use Neo4j in production for a wide range of use cases, many of them surprising, even to us! (And we invented the property graph.)

A partial list of customers can be found below:

www.neotechnology.com/customers

The "too niche" comment might have been true a few years ago. I won't speak for all graph databases, since many are clearly very new and haven't had much time to mature yet. But Neo4j has been in 24x7 production for 10 years, and while the product has of course evolved significantly since then, it'd built on a very solid foundation.

Most the companies moving to graph databases--speaking for Neo4j, which is what I know about-- are doing so because either a) their RDBMSs weren't able to handle the scope & scale of their connected query requirements, and/or b) the immense convenience and speed that comes from modeling domains that are a graph (social, network & data center management, fraud, portfolios, identity, etc.) as a graph, not as tables.

For kicks, you can find a number of customer talks here, from the four (soon five) GraphConnect conferences that were held this year in major cities around the world:

http://watch.neo4j.org/

If you're in London, the last one will be held next week:

www.graphconnect.com

You'll find a summary below of some of the technology behind, with some customer examples.

www.neotechnology.com/neo4j-scales-for-the-enterprise/

One of the world's largest postal delivery services does all of their real-time package routing with Neo4j. Several customers have more than half of the Facebook social graph running 24x7 on a web application with millions of members, running on a Neo4j cluster. Railroads are building routing systems on Neo4j. Some of the world's largest customers are using them for HR and data governance, alternate-path routing, etc. etc.

The best way to really understand why graph databases are amazing is to try. Check out the latest v2.0 M06 beta milestone of Neo4j (www.neo4j.org) which includes a brand-new query environment. I've seen connected queries ("shortest path", "find all dependencies", etc.) that are four lines in the Cypher query language and 50-100 lines in SQL. I've seen queries go from minutes to milliseconds. It's convenient and fast. Glad to see you exploring graphs!

Re:Why You Should Never Use MongoDB by yid on Monday November 11, 2013 @01:24PM (#6713041)
> Is this really true?

Facebook's TAO is a giant graph database built on top of MySQL [1]. I'd say it's pretty production-ready, because Facebook's social graph probably has at least hundreds of vertices.

[1] https://www.facebook.com/notes/facebook-engineering/tao-the-...

Re:Why You Should Never Use MongoDB by ris on Monday November 11, 2013 @02:58PM (#6713592)
There's a difference between ready-for-production and ready-for-production-if-you-have-the-entire-team-of-developers-that-wrote-it-on-hand-all-the-time.
Re:Why You Should Never Use MongoDB by petern on Wednesday November 13, 2013 @09:07AM (#6725190)
See http://blog.neo4j.org/2013/11/why-graph-databases-are-best-t... on how to use Neo4j for the mentioned Diaspora cases (Neo4j was actually proposed back in 2010 to the team). Comments are very welcome.
Re:Why You Should Never Use MongoDB by msane on Monday November 11, 2013 @09:56PM (#6715573)
Exactly what I thought. Mongo has its purpose. But it's a tree. If your data is a graph with many nodes, it's going to take some elbow grease. Don't use mongo in that case, use something that is built for that, like Neo4j ...
Re:Why You Should Never Use MongoDB by the1 on Monday November 11, 2013 @01:30PM (#6713080)
what's wrong with symlinks on transactional filesystem?
Re:Why You Should Never Use MongoDB by jahewson on Monday November 11, 2013 @02:05PM (#6713261)
Millions of small files is the worst workload for pretty much every filesystem. Data locality and fragmentation can end up becoming real problems too.