Re:Databricks and Neon
by disgruntledphd2 on Wednesday May 14, 2025 @09:14AM
(#43984155)
The nice thing about spark is the scala/python/R APIs. That helps to avoid lots of the irritating things about SQL (the same transformation applied to multiple columns is a big one).
Re:Databricks and Neon
by lelandbatey on Thursday May 15, 2025 @01:08AM
(#43992020)
I really can't speak highly enough of Trino (though I used it as AWS Athena, and this was back when Trino was called Presto). It's impressive how well it took "ever growing pile of CSV/JSON/Excel/Parquet/whatever" and let you query it via SQL as-is without transforming it and putting it into some other system.
What an impressive feat of engineering.
But these days just use trino or whatever. There are lots of new ways to work on data that are all bigger steps up - ergonomically, performance and price - over spark as spark was over hadoop.