Re:FastSpark: A New Fast Native Implementation of Spark from...
by brokensegue on Thursday October 31, 2019 @04:39PM
(#21413483)
train a set of sklearn models one each per a random partition of the data (computed distributed). then combine all those models using averaging and evaluate them all against an even larger dataset. how do you do that in SQL
[ Parent
| on HN ]
Re:FastSpark: A New Fast Native Implementation of Spark from...
by truth_seeker on Friday November 01, 2019 @01:19AM
(#21416464)
Sharding the table can help scale the problem across many machines and as I mentioned earlier you can use PL/R or PL/Python language extension to lift all sorts of ML functions to SQL functions.
Re:FastSpark: A New Fast Native Implementation of Spark from...
by brokensegue on Friday November 01, 2019 @11:18AM
(#21419722)
I'm unfamiliar with PL/Python. Can you have a Python object be the returned value of a sql query? Because that's a requirement of my example.