News for hackers. Stuff that matters.
Re:FastSpark: A New Fast Native Implementation of Spark from... by brokensegue on Thursday October 31, 2019 @04:39PM (#21413483)
train a set of sklearn models one each per a random partition of the data (computed distributed). then combine all those models using averaging and evaluate them all against an even larger dataset. how do you do that in SQL
Re:FastSpark: A New Fast Native Implementation of Spark from... by truth_seeker on Friday November 01, 2019 @01:19AM (#21416464)
Sharding the table can help scale the problem across many machines and as I mentioned earlier you can use PL/R or PL/Python language extension to lift all sorts of ML functions to SQL functions.
Re:FastSpark: A New Fast Native Implementation of Spark from... by brokensegue on Friday November 01, 2019 @11:18AM (#21419722)
I'm unfamiliar with PL/Python. Can you have a Python object be the returned value of a sql query? Because that's a requirement of my example.