News for hackers. Stuff that matters.

ericxiao251

Re:Show HN: A Hands-On Guide on PySpark Coding and Best Practices on Wednesday February 20, 2019 @04:35PM (#19212232)
Oh awesome thanks for the resources! I will definitely see how i can incorporate it into my guide :).
Re:Show HN: A Hands-On Guide on PySpark Coding and Best Practices on Wednesday February 20, 2019 @02:41PM (#19211264)
I haven't looked into/keep up with Arrow much, but if I see fit, I can add more stuff about it :)!
Re:Show HN: A Hands-On Guide on PySpark Coding and Best Practices on Wednesday February 20, 2019 @02:40PM (#19211254)
Yes I agree, that was the whole premise of the repo 2 years ago!

I'm glad you like it :)!

Re:Show HN: A Hands-On Guide on PySpark Coding and Best Practices on Wednesday February 20, 2019 @10:39AM (#19208677)
Hey quadrature, thanks for the feedback! Would you be able to go into more details about what skew you see :)?

In chapter 7 I go into some methods of fixing skewed data when performing joins. This solved a majority of our skew problems, but we still see skew on aggregates I believe. I am working on how to debug/find skews in a spark application in Chapter 6, wanted to initially release this as I've been procrastinating over 2 years to do so lol.

We have done more spark parameter optimizations but that helps after the data skew have been resolved.