Re:FastSpark: A New Fast Native Implementation of Spark from...
by truth_seeker on Thursday October 31, 2019 @02:54PM
(#21412306)
Please give me an example. I can't think of any transform which cannot be done by using SQL or inbuilt Functions or new UDF.
[ Parent
| on HN ]
Re:FastSpark: A New Fast Native Implementation of Spark from...
by brokensegue on Thursday October 31, 2019 @04:39PM
(#21413483)
train a set of sklearn models one each per a random partition of the data (computed distributed). then combine all those models using averaging and evaluate them all against an even larger dataset. how do you do that in SQL
Re:FastSpark: A New Fast Native Implementation of Spark from...
by truth_seeker on Friday November 01, 2019 @01:19AM
(#21416464)
Sharding the table can help scale the problem across many machines and as I mentioned earlier you can use PL/R or PL/Python language extension to lift all sorts of ML functions to SQL functions.
Re:FastSpark: A New Fast Native Implementation of Spark from...
by brokensegue on Friday November 01, 2019 @11:18AM
(#21419722)
I'm unfamiliar with PL/Python. Can you have a Python object be the returned value of a sql query? Because that's a requirement of my example.
Re:FastSpark: A New Fast Native Implementation of Spark from...
by nxpnsv on Friday November 01, 2019 @09:32AM
(#21418551)
It’s also possible to do a lot in excel, it is just not always the best tool for the job.
Re:FastSpark: A New Fast Native Implementation of Spark from...
by ai_ja_nai on Thursday October 31, 2019 @04:29PM
(#21413394)
Spark != SQL
It's also graph analysis and ML models.
Re:FastSpark: A New Fast Native Implementation of Spark from...
by truth_seeker on Friday November 01, 2019 @01:24AM
(#21416476)
Graph analysis -> Recursive common table expression (https://www.postgresql.org/docs/current/queries-with.html)
ML models - I already mentioned how to uplift R and Python functions to SQL function. even if you are not using PostgreSQL many other databases help you with uplifting and interfacing with existing ML libraries through FFI