News for hackers. Stuff that matters.
Re:Analyzing Data 180,000x Faster with Rust by georgeecollins on Saturday October 21, 2023 @01:36AM (#37964357)
I like Python too but let's face it, it is not performant. Why do we have to pretend otherwise?
Re:Analyzing Data 180,000x Faster with Rust by wenc on Saturday October 21, 2023 @01:50AM (#37964408)
I think everybody agrees and is not pretending otherwise.

But my thing is, I think the problem could be solved in a different way in Python where we can use performant libraries to get an answer in a reasonable time.

The author says the naive Python implementation with a for-loop takes 36 milliseconds per iteration, and the problem requires 2.5 billion iterations (= 2.9 years, which is unreasonable) while optimized Rust takes 8 mins (corrected).

I believe we can solve the problem in Python in a reasonable amount of time (not 2.9 years) by expressing it differently. And I believe we can do it in Python without trying to optimize Python operations like the author is doing with Rust.

Imagine your boss came up to you and said I need the answer by this week and that you could only use Python, you would need to come up with a way to solve it. I wouldn't start by trying to optimizing Python's for-loop -- I would break out of the loop paradigm altogether and use arrays, database indices, optimized dataframe libraries (probably written in C++ or Rust) to get there. Because Python is not fast -- everyone knows this -- Python programmers will often think of other ways (generally reaching for libraries) to solve the problem.

Re:Analyzing Data 180,000x Faster with Rust by bibmbop on Saturday October 21, 2023 @02:23AM (#37964552)
8 minutes.
Re:Analyzing Data 180,000x Faster with Rust by tetsuhamu on Saturday October 21, 2023 @01:50AM (#37964409)
The industry agreed to standardize on Python for the task of describing compute-graphs that get executed by compute engines implemented in something other than Python. Python is not meant to be used for the computation itself.
Re:Analyzing Data 180,000x Faster with Rust by FridgeSeal on Saturday October 21, 2023 @04:13AM (#37965020)
And then other people came along, using other languages, and realised we could do the same compute, with a fraction of the hardware, in about the same time, in the languages they’re already using, which really diminishes the appeal of Python, which boils down to:

“Setup some infra, and get someone who knows how to operate it, and then write new code in python, and babysit it as it ossifies into a core piece of infrastructure”

_or_ optimise it in a language you’re already using, or is semi-common in your stack already, and then move on with things.