Why I Use Nim instead of Python for Data Processing

Re:Why I Use Nim instead of Python for Data Processing by m_mueller on Saturday September 25, 2021 @03:15AM (#28651165)

I agree with your entire post but, and I‘m saying this as a fulltime python dev, there‘s often a point where it starts being bothersome, and that usually comes only later in the lifecycle of an application after it had some organic growth. Some day e.g. a sales manager comes down to your lair and asks you if you couldn‘t just also parse this little 200MB Excel spreadsheet after it came over the network such that your ETL process could save it into a new table. And boom you‘re into CPU bound lands now. Often it‘s fine, you can wait those 1-2min for a daily occuring process. But what if for example you put this whole component behind a REST API that is behind a load balancer that is set with a certain timeout? There are even strict upper limits if for example you chose AWS Lambda for your stuff.

And suddenly you need to introduce quite a bit more technical complexity into this story that‘s gonna be hard to explain to management - all they see is that you now can insert a couple of millions of DB rows and their Big Data consultants[TM] told them that this is nowadays not even worth thinking about.

Point being: If your performance ceiling is low, you‘re gonna hit it sooner.

[ Parent | on HN ]

Re:Why I Use Nim instead of Python for Data Processing by rsyring on Saturday September 25, 2021 @10:44AM (#28653573)

> there‘s often a point where it starts being bothersome

My team and I have been using Python for web, scripting, and ETL development since 2007. I don't recall the last time Python wasn't "fast enough" for anything I needed to do. I'm sure it's legitimately too slow for plenty of use cases and classes of programming domains. But for a general purpose language that makes our developers incredibly productive (which is what we optimize for organizationally), I've not "often" found the point where it becomes bothersome. On the contrary, I'd say I've rarely found it. And in those cases, the workaround to make it fast enough is there.

That's not to dispute your experience. I just want to provide a counter example.

[ Parent | on HN ]

Re:Why I Use Nim instead of Python for Data Processing by akho on Saturday September 25, 2021 @02:42PM (#28655220)

Does Python give you an actual productivity edge? I do not see how.

Python developers are easier to hire (certainly than Nim), have a monolanguage experience (meaning less onboarding), but are not magically more productive. Developer availability is still a great reason to choose Python.

[ Parent | on HN ]

Re:Why I Use Nim instead of Python for Data Processing by m_mueller on Saturday September 25, 2021 @05:16PM (#28656241)

YMMV. My background is in financial applications and there it’s a quite common problem popping up, especially (but not exclusively) in analytics.

[ Parent | on HN ]

Re:Why I Use Nim instead of Python for Data Processing by DeathArrow on Saturday September 25, 2021 @04:02AM (#28651390)

>Point being: If your performance ceiling is low, you‘re gonna hit it sooner.

And that isn't the only culprit. Large code bases will be hard to structure, maintain and organize. And you will spend more time writing tests than writing productive code. Because you don't have many defenses against errors and because once a bug will hit production it will be very difficult to debug.

[ Parent | on HN ]

Re:Why I Use Nim instead of Python for Data Processing by m_mueller on Saturday September 25, 2021 @06:48AM (#28652073)

I somewhat disagree there. IMO you can structure a large application just as well in python as in Java, but it requires more architecture guidance as your degree of freedom is higher. Regarding testing, mypy and later python versions have been a big help there, I don't think it's much of a difference anymore and I'd take it over Node.js for that reason because the core language IMO is more consistent and strict and requires way way less outside dependencies. In fact the standard lib for me is the main reason to choose python today.

So for me it's really performance being the main thing to think about with python. If one anticipates tighter latency or throughput requirements I'd go for something else.

[ Parent | on HN ]

Re:Why I Use Nim instead of Python for Data Processing by someguy101010 on Saturday September 25, 2021 @01:33PM (#28654805)

Is there a problem with binding c or c++ to your python project in these situations?

[ Parent | on HN ]

Re:Why I Use Nim instead of Python for Data Processing by m_mueller on Saturday September 25, 2021 @05:08PM (#28656199)

The whole build gets quite complex. Numba is interesting but also still a bit limited in what it supports. There are no silver bullets and sometimes you can find a good fit, but IMO the JVM or C#/.net overall have a better story when it comes to write performance critical applications (but don’t need to get the last 20% such as HFT or HPC sims), as you can also find people on the market who understand how to do and maintain that more easily. There are tons of python devs but in my experience at least on European market their understanding of underlying systems tends to be limited.

[ Parent | on HN ]