Re:There was PayPal mafia. Meet the Berkeley mafia
by legerdemain on Friday January 20, 2023 @07:41PM
(#34462436)
In a twist, pandas programs don't get parallelized on Spark. Someone had to go and write a parallel layer that duplicated the pandas API, because otherwise you ended up with the entire pandas program executing on a single executor.
https://www.databricks.com/product/databricks-sql