Re:Ask HN: Learning PySpark and Related Tools
by almosthere on Friday February 07, 2025 @01:46PM
(#42976052)
Having used spark for the past 8 years or so, it's definitely a solid basic for data engineering. I use it for generating reports the most, but sometimes we have large projects to get data into different staging databases. I use it a lot with ElasticSearch or a parquet. Basically it helps you write large joins and flatten the result to a database that can more quickly perform aggregations on that flattened result (like Elasticsearch) or a columnar database.
[ Parent
| on HN ]
There are no comments on this comment yet.