News for hackers. Stuff that matters.
Re:Test for lists in Cython by mhh__ on Monday April 19, 2021 @07:30AM (#26861509)
If you are genuinely hitting a wall like that surely it's time to move on from python? Or at least think about it
Re:Test for lists in Cython by ZuLuuuuuu on Monday April 19, 2021 @07:42AM (#26861615)
The thing is that the current ecosystem (numpy + scipy + pandas + PyCharm) fits our company structure perfectly (very few software engineers, mostly test engineers who are not very proficient coders). And we already have tens of thousands lines of code. So changing the whole ecosystem just because 5% of the problems are slow is too big of a jump for us. For now, it is easier for us to write a bit of C code for places where we absolutely need speed. But it would have been so convenient if we could just have native threads, so that we don't even need that 5% C code.
Re:Test for lists in Cython by mhh__ on Monday April 19, 2021 @07:50AM (#26861692)
Do you know what you are actually stalled on? i.e. Memory or Instructions, if it's the latter you can probably eek out some more performance just by fiddling with compiler flags (you'd be surprised simultaneously how clever and how utterly braindead some compilers are if you play with them on Compiler Explorer for a bit)

Also, if you are running on Intel you might have some luck with Intel's profiling tools as, although I've never used them for python they are by far and away the best things in this area, they do support Python.

I don't know what details they give you in from Python code directly, but if you've used perf to profile code at a low level before, vTune is like a Tom & Jerry shotgun in comparison.

Re:Test for lists in Cython by meepmorp on Monday April 19, 2021 @09:51AM (#26862744)
> Compiler Explorer

Never seen this before. Thanks!

Re:Test for lists in Cython by mhh__ on Monday April 19, 2021 @10:01AM (#26862843)
Re:Test for lists in Cython by pantsforbirds on Monday April 19, 2021 @03:28PM (#26866953)
You don't have to replace the core python parts. We moved a decent amount of our data pre-processing into Scala, Golang, and Data Base queries and then just store the processed data for the python models to use to train/run (which ends up using the super efficient C/Fortran/Cuda code).