Re:Dragon: A distributed graph query engine
by mark_l_watson on Tuesday March 22, 2016 @09:34AM
(#11336410)
The query examples looked like Clojure code; is that correct?
The technology of Facebook and Google (where I worked as a contractor) impresses! That said, I like a simpler Internet with smaller scale services like Gnu Social, and simply using email to stay in touch with family and friends. I have this preference both for privacy reasons and also prefer more one on one communication.
Re:Dragon: A distributed graph query engine
by j_m_b on Tuesday March 22, 2016 @11:16AM
(#11337177)
>That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.
I'm pretty sure this is Clojure or something very close to it. ->> is the thread-last macro. What it does is take the first expression, ($alice) and inserts it as the last expr in the next form so that the second step (assoc $friends) would eval to (assoc $friends $alice). It continues this by inserting this expr into the last part of the next form. So the example would reduce to:
(filter (> age 20) (assoc $friends (assoc $friends $alice)))
Re:Dragon: A distributed graph query engine
by hellofunk on Tuesday March 22, 2016 @12:15PM
(#11337603)
But as I mentioned, the second argument to Clojure's filter is a function, and (> age 20) is not a function; therefore, this is not normal Clojure (or Clojure or at all). You'd have to replace the standard library's filter function to get the syntax you see in this article, which I doubt they are doing. therefore: not Clojure.
Re:Dragon: A distributed graph query engine
by emidln on Tuesday March 22, 2016 @02:59PM
(#11339024)
You could also walk the ast and do something special with unknown symbols (and functions that contain them).
i.e. while walking, we notice a function (> age 20) and age isn't visible in scope. We thus rewrite that into something like (fn [age] (> age 20)) or (fn [x] (> (:age x) 20)).
I've taken this approach before with DSLs. It makes for extremely fast to write business logic where business users don't have to care as much about where their data comes from (they just need field names).
Re:Dragon: A distributed graph query engine
by hellofunk on Tuesday March 22, 2016 @03:45PM
(#11339423)
In general code, that would be very hard to read, not knowing if a form represented an actual value or a function. Clojure being dynamically typed makes things hard enough; changing the language syntax as you describe sounds pretty risky to me.
Re:Dragon: A distributed graph query engine
by emidln on Tuesday March 22, 2016 @04:11PM
(#11339618)
The purpose of such a DSL isn't to let you write generic clojure faster. The DSL isn't even targetted at traditional programmers. The purpose is to enable training people who have no idea what Clojure or Lisp to get things done that are complicated (or expensive) to design graphical UX for. These people might have some excel experience or maybe a programming class in college, but are mostly reliant on examples and documentation to get things done. Why teach them things like how to create a function when they don't need to know or care? There is a huge usability difference for these people between (fn [age] (> age 18)) and (> age 18).
Re:Dragon: A distributed graph query engine
by nemoniac on Friday March 25, 2016 @07:16PM
(#11363472)
It's Lisp code, more particularly Lisp syntax, and I mean that in the most general sense in which Clojure is Lisp syntax. So are Common Lisp, Scheme, Racket and Emacs Lisp.
Re:Dragon: A distributed graph query engine
by jimbokun on Tuesday March 22, 2016 @09:57AM
(#11336591)
This sounds really cool.
For distributed relational or graph databases, seems like the key trick for making queries efficient is to get related data on the same host, whenever possible.
So would be cool to dig into the specifics of the algorithms they are using, to see exactly how they are optimizing where to store the data. With a graph database, it's impossible to guarantee having all related data together (eventually, friend of a friend of a friend...will be on a different host). So needs to be heuristics based.
For tree shaped data, on the other hand, it is possible to have the root of each tree and all of its related data on the same host (assuming each tree is "reasonable" size). Google's F1 project took this approach.
Re:Dragon: A distributed graph query engine
by rubyfan on Tuesday March 22, 2016 @07:16AM
(#11335662)
Slight tangent here but is it me or has the experience in social networks taken a hit? Feels like it was better with low complexity hack technology than with technology solutions focused on scaling the experience. I sometimes wonder if we are changing experience too much for technology optimization.
- LinkedIn has been terrible for about 2-3 years since things stopped updating real time and the timeline started getting random
- Facebook randomly suggesting things from your graph based on what it thinks you like
- Twitter I heard soon will be no longer in chronological order?!?
I think there is something to be said for low complexity MySQL and memcached.
Re:Dragon: A distributed graph query engine
by rubyfan on Tuesday March 22, 2016 @07:43AM
(#11335761)
yeah I hear that and to be clear I'm not blaming all their experience issues with the stack.
But there are definitely stack complexity trade offs when you get to the scale of these guys.
Also could be more correlation not causation that when you get to a certain scale and more stack complexity you also start hiring specialized staff that end up diluting the culture and spirit of a startup.
Re:Dragon: A distributed graph query engine
by chrisseaton on Tuesday March 22, 2016 @07:28AM
(#11335701)
I don't think any of those things are caused by optimisations or needing to scale.
Saying that Facebook is recommending things simultaneously at random, based on a reccomendation algorithm, and also as an optimisation which wouldn't be needed if they used MySQL, is obviously nonsense.
Re:Dragon: A distributed graph query engine
by rubyfan on Tuesday March 22, 2016 @07:56AM
(#11335809)
@chrisseaton - big fan of your work, eagerly awaiting Truffle JRuby production ready release.
I'm not saying MySQL > some specialized solution for scale... I'm guess I'm drawing the correlation that when companies have to make investments like this they are also trading off experience, or maybe to your point screwing up the experience using recommenders and algorithms instead of letting my feed flow.
Re:Dragon: A distributed graph query engine
by chrisseaton on Tuesday March 22, 2016 @08:16AM
(#11335893)
Yeah I see your clarification in your other comment where you say that when they start to scale it opens the door for tons of other complexity. That makes more sense.
The technology of Facebook and Google (where I worked as a contractor) impresses! That said, I like a simpler Internet with smaller scale services like Gnu Social, and simply using email to stay in touch with family and friends. I have this preference both for privacy reasons and also prefer more one on one communication.
(filter (> age 20))
That second form to filter should be a function, but in this case it is an expression that returns a boolean and is not itself a function.
https://groups.google.com/forum/#!topic/clojure/P3eW6Vi2QcU
I'm pretty sure this is Clojure or something very close to it. ->> is the thread-last macro. What it does is take the first expression, ($alice) and inserts it as the last expr in the next form so that the second step (assoc $friends) would eval to (assoc $friends $alice). It continues this by inserting this expr into the last part of the next form. So the example would reduce to: (filter (> age 20) (assoc $friends (assoc $friends $alice)))
i.e. while walking, we notice a function (> age 20) and age isn't visible in scope. We thus rewrite that into something like (fn [age] (> age 20)) or (fn [x] (> (:age x) 20)).
I've taken this approach before with DSLs. It makes for extremely fast to write business logic where business users don't have to care as much about where their data comes from (they just need field names).
S-expressions: fully parenthesized prefix notation.
For distributed relational or graph databases, seems like the key trick for making queries efficient is to get related data on the same host, whenever possible.
So would be cool to dig into the specifics of the algorithms they are using, to see exactly how they are optimizing where to store the data. With a graph database, it's impossible to guarantee having all related data together (eventually, friend of a friend of a friend...will be on a different host). So needs to be heuristics based.
For tree shaped data, on the other hand, it is possible to have the root of each tree and all of its related data on the same host (assuming each tree is "reasonable" size). Google's F1 project took this approach.
https://www.usenix.org/conference/nsdi16/technical-sessions/...
- LinkedIn has been terrible for about 2-3 years since things stopped updating real time and the timeline started getting random
- Facebook randomly suggesting things from your graph based on what it thinks you like
- Twitter I heard soon will be no longer in chronological order?!?
I think there is something to be said for low complexity MySQL and memcached.
"Crap how do we make money"
"Crap how do we retain users/keep them here longer"
The stack is to do with providing a fast performance to vast numbers + making it easier to work on for engineers than it might be at that complexity.
But there are definitely stack complexity trade offs when you get to the scale of these guys.
Also could be more correlation not causation that when you get to a certain scale and more stack complexity you also start hiring specialized staff that end up diluting the culture and spirit of a startup.
Saying that Facebook is recommending things simultaneously at random, based on a reccomendation algorithm, and also as an optimisation which wouldn't be needed if they used MySQL, is obviously nonsense.
I'm not saying MySQL > some specialized solution for scale... I'm guess I'm drawing the correlation that when companies have to make investments like this they are also trading off experience, or maybe to your point screwing up the experience using recommenders and algorithms instead of letting my feed flow.