News for hackers. Stuff that matters.
Re:Dragon: A distributed graph query engine by mark_l_watson on Tuesday March 22, 2016 @09:34AM (#11336410)
The query examples looked like Clojure code; is that correct?

The technology of Facebook and Google (where I worked as a contractor) impresses! That said, I like a simpler Internet with smaller scale services like Gnu Social, and simply using email to stay in touch with family and friends. I have this preference both for privacy reasons and also prefer more one on one communication.

Re:Dragon: A distributed graph query engine by hellofunk on Tuesday March 22, 2016 @10:09AM (#11336683)
It looks a lot like Clojure but I'm not sure it is. For example, this expression:

(filter (> age 20))

That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.

Re:Dragon: A distributed graph query engine by adsharma on Tuesday March 22, 2016 @11:48AM (#11337404)
Right, it's not Clojure, but we tried to borrow some syntactical elements for readability. More discussion here:

https://groups.google.com/forum/#!topic/clojure/P3eW6Vi2QcU

Re:Dragon: A distributed graph query engine by okram on Tuesday March 22, 2016 @01:08PM (#11338017)
It has a very similar look and feel to Gremlin (http://tinkerpop.apache.org).

   (->> ($alice) (assoc $friends) (assoc $friends) (filter (> age 20)) (count))
...in Gremlin is:

   g.V(alice).out("friends").out("friends").has("age",gt(20)).count()
Re:Dragon: A distributed graph query engine by j_m_b on Tuesday March 22, 2016 @11:16AM (#11337177)
>That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.

I'm pretty sure this is Clojure or something very close to it. ->> is the thread-last macro. What it does is take the first expression, ($alice) and inserts it as the last expr in the next form so that the second step (assoc $friends) would eval to (assoc $friends $alice). It continues this by inserting this expr into the last part of the next form. So the example would reduce to: (filter (> age 20) (assoc $friends (assoc $friends $alice)))

Re:Dragon: A distributed graph query engine by hellofunk on Tuesday March 22, 2016 @12:15PM (#11337603)
But as I mentioned, the second argument to Clojure's filter is a function, and (> age 20) is not a function; therefore, this is not normal Clojure (or Clojure or at all). You'd have to replace the standard library's filter function to get the syntax you see in this article, which I doubt they are doing. therefore: not Clojure.
Re:Dragon: A distributed graph query engine by emidln on Tuesday March 22, 2016 @02:59PM (#11339024)
You could also walk the ast and do something special with unknown symbols (and functions that contain them).

i.e. while walking, we notice a function (> age 20) and age isn't visible in scope. We thus rewrite that into something like (fn [age] (> age 20)) or (fn [x] (> (:age x) 20)).

I've taken this approach before with DSLs. It makes for extremely fast to write business logic where business users don't have to care as much about where their data comes from (they just need field names).

Re:Dragon: A distributed graph query engine by hellofunk on Tuesday March 22, 2016 @03:45PM (#11339423)
In general code, that would be very hard to read, not knowing if a form represented an actual value or a function. Clojure being dynamically typed makes things hard enough; changing the language syntax as you describe sounds pretty risky to me.
Re:Dragon: A distributed graph query engine by emidln on Tuesday March 22, 2016 @04:11PM (#11339618)
The purpose of such a DSL isn't to let you write generic clojure faster. The DSL isn't even targetted at traditional programmers. The purpose is to enable training people who have no idea what Clojure or Lisp to get things done that are complicated (or expensive) to design graphical UX for. These people might have some excel experience or maybe a programming class in college, but are mostly reliant on examples and documentation to get things done. Why teach them things like how to create a function when they don't need to know or care? There is a huge usability difference for these people between (fn [age] (> age 18)) and (> age 18).
Re:Dragon: A distributed graph query engine by nemoniac on Friday March 25, 2016 @07:16PM (#11363472)
It's Lisp code, more particularly Lisp syntax, and I mean that in the most general sense in which Clojure is Lisp syntax. So are Common Lisp, Scheme, Racket and Emacs Lisp.

S-expressions: fully parenthesized prefix notation.

Re:Dragon: A distributed graph query engine by eranation on Tuesday March 22, 2016 @08:58AM (#11336177)
Nice! What are the benefits over TitanDB, Apache Spark/GraphX, GraphLab, Giraph, Neo4j, openCypher Tinkerpop etc?
Re:Dragon: A distributed graph query engine by jimbokun on Tuesday March 22, 2016 @09:57AM (#11336594)
Seems like the big improvement is heuristically keeping related data on the same hosts as much as possible, to improve query performance.
Re:Dragon: A distributed graph query engine by grandalf on Tuesday March 22, 2016 @09:26AM (#11336361)
This one creates optimized indexes based on the social graph's intrinsic semantics.
Re:Dragon: A distributed graph query engine by jimbokun on Tuesday March 22, 2016 @09:57AM (#11336591)
This sounds really cool.

For distributed relational or graph databases, seems like the key trick for making queries efficient is to get related data on the same host, whenever possible.

So would be cool to dig into the specifics of the algorithms they are using, to see exactly how they are optimizing where to store the data. With a graph database, it's impossible to guarantee having all related data together (eventually, friend of a friend of a friend...will be on a different host). So needs to be heuristics based.

For tree shaped data, on the other hand, it is possible to have the root of each tree and all of its related data on the same host (assuming each tree is "reasonable" size). Google's F1 project took this approach.

Re:Dragon: A distributed graph query engine by adsharma on Tuesday March 22, 2016 @10:43AM (#11336924)
Hello, I wrote the blog post. There is a link to a video in the blog and we published something at NSDI last week:

https://www.usenix.org/conference/nsdi16/technical-sessions/...

Re:Dragon: A distributed graph query engine by jimbokun on Wednesday March 23, 2016 @06:29PM (#11348773)
Thanks for the link! Looks like an interesting read.
Re:Dragon: A distributed graph query engine by deforciant on Tuesday March 22, 2016 @03:54AM (#11334994)
looks interesting, where can I download Dragon? :D
Re:Dragon: A distributed graph query engine by lynxaegon on Tuesday March 22, 2016 @04:30AM (#11335109)
I don't think it's open source yet. Maybe in the near future
Re:Dragon: A distributed graph query engine by rubyfan on Tuesday March 22, 2016 @07:16AM (#11335662)
Slight tangent here but is it me or has the experience in social networks taken a hit? Feels like it was better with low complexity hack technology than with technology solutions focused on scaling the experience. I sometimes wonder if we are changing experience too much for technology optimization.

- LinkedIn has been terrible for about 2-3 years since things stopped updating real time and the timeline started getting random

- Facebook randomly suggesting things from your graph based on what it thinks you like

- Twitter I heard soon will be no longer in chronological order?!?

I think there is something to be said for low complexity MySQL and memcached.

Re:Dragon: A distributed graph query engine by bananaoomarang on Tuesday March 22, 2016 @07:28AM (#11335703)
I don't think that's to do with stack complexity, more like:

"Crap how do we make money"

"Crap how do we retain users/keep them here longer"

The stack is to do with providing a fast performance to vast numbers + making it easier to work on for engineers than it might be at that complexity.

Re:Dragon: A distributed graph query engine by rubyfan on Tuesday March 22, 2016 @07:43AM (#11335761)
yeah I hear that and to be clear I'm not blaming all their experience issues with the stack.

But there are definitely stack complexity trade offs when you get to the scale of these guys.

Also could be more correlation not causation that when you get to a certain scale and more stack complexity you also start hiring specialized staff that end up diluting the culture and spirit of a startup.

Re:Dragon: A distributed graph query engine by chrisseaton on Tuesday March 22, 2016 @07:28AM (#11335701)
I don't think any of those things are caused by optimisations or needing to scale.

Saying that Facebook is recommending things simultaneously at random, based on a reccomendation algorithm, and also as an optimisation which wouldn't be needed if they used MySQL, is obviously nonsense.

Re:Dragon: A distributed graph query engine by rubyfan on Tuesday March 22, 2016 @07:56AM (#11335809)
@chrisseaton - big fan of your work, eagerly awaiting Truffle JRuby production ready release.

I'm not saying MySQL > some specialized solution for scale... I'm guess I'm drawing the correlation that when companies have to make investments like this they are also trading off experience, or maybe to your point screwing up the experience using recommenders and algorithms instead of letting my feed flow.

Re:Dragon: A distributed graph query engine by chrisseaton on Tuesday March 22, 2016 @08:16AM (#11335893)
Yeah I see your clarification in your other comment where you say that when they start to scale it opens the door for tons of other complexity. That makes more sense.