Stochastic gradient descent written in SQL

Re:Stochastic gradient descent written in SQL by ISL on Tuesday March 07, 2023 @09:55AM (#35056012)

The challenge I, with limited knowledge, see with developing detailed algorithms in SQL is a lack of good testing, abstraction, and review tooling. Similarly for a lot of the user-defined-functions for the larger data warehouses (redshift, bigQuery, etc.)

dbt solves a lot of it, but I'd love to learn more about good resources for building reliable and readily-readable SQL algorithms for complex logic.

[ Parent | on HN ]

Re:Stochastic gradient descent written in SQL by AndrewKemendo on Tuesday March 07, 2023 @10:55AM (#35056766)

I mean this is an opportunity right? Build those things

[ Parent | on HN ]

Re:Stochastic gradient descent written in SQL by Lemaxoxo on Tuesday March 07, 2023 @11:13AM (#35057039)

I agree. Databases are going to be here for a long time, and we're barely scratching the surface of making people productive with them. dbt is just the beginning.

[ Parent | on HN ]

Re:Stochastic gradient descent written in SQL by adeelk93 on Tuesday March 07, 2023 @12:25PM (#35058159)

I've supported 3 different models over the years with inference implemented in SQL. First one I inherited, loved it so much that I implemented it twice again. Amazingly fast for TBs of data and no waiting on other tech teams.

That tooling you're describing is definitely not there. Bigquery has BQML but it's very much in its infancy. I tend to do all the modeling in Python/R on sampled data and then deploy to SQL.

[ Parent | on HN ]

Re:Stochastic gradient descent written in SQL by fithisux on Tuesday March 07, 2023 @03:09PM (#35060486)

BQML should become standard.

[ Parent | on HN ]