News for hackers. Stuff that matters.

A comment on FireDucks: Pandas but 100x Faster. Parent comment

Re:FireDucks: Pandas but 100x Faster by OutOfHere on Wednesday November 20, 2024 @11:16AM (#42195381)
Using `lambda` without care is dangerous because it risks being not vectorized at all. It risks being super slow, operating one row at a time. Is `d` a single row or the entire series or the entire dataframe?
[ Parent | on HN ]
Re:FireDucks: Pandas but 100x Faster by rogue7 on Wednesday November 20, 2024 @11:20AM (#42195423)
In this case `d` is the entire dataframe. It's just a way of "piping" the object without having to rename it.

You are probably thinking about `df.apply(lambda row: ..., axis=1)` which operates on each row at a time and is indeed very slow since it's not vectorized. Here this is different and vectorized.

[ Parent | on HN ]
Re:FireDucks: Pandas but 100x Faster by OutOfHere on Wednesday November 20, 2024 @11:48AM (#42195757)
That's excellent.
[ Parent | on HN ]
Re:FireDucks: Pandas but 100x Faster by almostkorean on Wednesday November 20, 2024 @01:41PM (#42196869)
Appreciate the explanation, this is something I should know by now but don't
[ Parent | on HN ]