Launch HN: ParaQuery (YC X25) – GPU Accelerated Spark/SQL

Re:Launch HN: ParaQuery (YC X25) – GPU Accelerated Spark/SQL by dogman123 on Monday May 12, 2025 @02:58PM (#43966388)

This could be incredibly useful for me. Currently struggling to complete jobs with massive amounts of shuffle with Spark on EMR (large joins yielding 150+ billion rows). We use Glue currently, but it has become cost prohibitive.

[ Parent | on HN ]

Re:Launch HN: ParaQuery (YC X25) – GPU Accelerated Spark/SQL by winwang on Monday May 12, 2025 @03:05PM (#43966473)

Is the shuffle the biggest issue? Not too sure about joins but one of the datasets we're currently dealing with has a couple trillion rows. Would love to chat about this!

[ Parent | on HN ]

Re:Launch HN: ParaQuery (YC X25) – GPU Accelerated Spark/SQL by threeseed on Monday May 12, 2025 @05:01PM (#43967420)

You should try using an S3 based shuffle plugin: https://github.com/IBM/spark-s3-shuffle

Then mount FSX for Lustre on all of your EMR nodes and have it write shuffle data there. It will massively improve performance and shuffle issues will disappear.

Is expensive though. But you can offset the cost now because you can run entirely Spot instances for your workers as if you lose a node there's no recomputation of the shuffle data.

[ Parent | on HN ]