Re:SQL should be your default choice for data engineering pi...
by ramraj07 on Monday January 30, 2023 @09:54AM
(#34580298)
Either you work with data that’s not really big data or you have some super awesome engineers who know spark very well, because spark is pretty much not declarative as much as it says so. You have to constantly battle crap like GC errors, data skew, etc. Even on supposedly managed solutions like databricks. As opposed to solutions like snowflake which are a lot more declarative (nothing is perfect of course).
[ Parent
| on HN ]
Re:SQL should be your default choice for data engineering pi...
by hobs on Monday January 30, 2023 @10:13AM
(#34580620)
And the vendors are happy to tell you to throw more cores at it instead of grapple with the core issues.
One of the biggest problems with spark imo is that people think that the magic cloud is going to make it go fast, when it actually means that the REPL is just muuuuch slower to iterate on.