News for hackers. Stuff that matters.
Re:There was PayPal mafia. Meet the Berkeley mafia by solidangle on Friday January 20, 2023 @06:35PM (#34461724)
But you don't have to write map-reduce jobs at all? You can just write SQL queries or Pandas programs, and they automatically get parallelized by Databricks. Databricks is a data warehouse (just like Snowflake).

https://www.databricks.com/product/databricks-sql

Re:There was PayPal mafia. Meet the Berkeley mafia by legerdemain on Friday January 20, 2023 @07:41PM (#34462436)
In a twist, pandas programs don't get parallelized on Spark. Someone had to go and write a parallel layer that duplicated the pandas API, because otherwise you ended up with the entire pandas program executing on a single executor.
Re:There was PayPal mafia. Meet the Berkeley mafia by alexott on Sunday January 29, 2023 @09:11AM (#34568244)
there is Pandas on Spark, included into Spark itself (originally Koalas) - the switch to it is very easy, and you get parallelization.