When you have so many large companies with a vested interest in optimization I believe that Python can become faster by doing realistic and targeted optimizations . The other strategies to optimize didn’t work at all or just served internal problems at large companies .
What exactly do you do with Python that slows you down ?
I still use Python for cases when dev time is more important than execution time (which is rarer now that I'm working adjacent to "big data") or when I'm doing things like writing python to close gaps in the various arrays of web apps provided for navigating the corporate work flow, and if I went as fast as a 64 core box let me, we'd have some outages in corp github or artifactory or the like, so I just do it one slow thing at a time on 1 core and wait for the results. Maybe multiprocessing with 10 process worker pool once I'm somewhat confident in the back end system I am talking to.
(edit: removed my normal email signoff)
Don't take this the wrong way but I think you could be more specific. Are you saying that similar to Go it should be just faster in general?
So many successful projects/technologies started out that way. The Web, JavaScript, e-mail. DNS started out as a HOSTS.TXT file that people copied around. Linus Torvalds announced Linux as "just a hobby, won't be big and professional like gnu". Minecraft rendered huge worlds with unoptimized Java and fixed-function OpenGL.
I took a graduate level data structures class and the professor used Python among other things "because it's about 80 times slower than C, so you have to think hard about your algorithms". At scale, that matters.
It prevents you from taking advantage of multiple cores. Doesn't really impact straight-line execution speed.
A data structures course is primarily not going to be concerned with multithreading.
Most computers running Python today will have multiple cores. If your program can only use a fraction of its available computing power, it affects the program's execution speed. Python is too widely used in compute-intensive applications (e.g. machine learning) for GIL-related issues to be ignored in the long term.
Ideally, Python should have automatic load scaling and thread-safe data structures in the standard library to take advantage of all available CPU cores.
Java has had concurrent data structures in its standard library for years and is adding support for "easy" multi-threaded execution with Project Loom.
Python needs to add its own Pythonic versions of thread-safe data structures and compute load scaling to take advantage of the multi-core CPUs that it runs on.
>> A data structures course is primarily not going to be concerned with multithreading.
A graduate-level data structures course might include concurrent and parallel data structures.
Being fast isn't contradictory with this goal. If anything, this is a lesson that so many developers forget. Things should be fast by default.
I think some people can do this and some can't. For some, writing slow code is much easier, and their contributions are still valuable. Once the bottlenecks are problems, someone with more performance-oriented skills can help speed up the critical path, and slow code outside of the critical path is just fine to leave as-is.
If you somehow limited contributions only to those who write fast code, I think you'd be leaving way too much on the table.
There are a few much better options like mergesort or quicksort, but they have their tricks.
But to sort real data really fast, you should use something like timsort, that detects if the data is just the union of two (or a few) sorted parts, so it's faster in many cases where the usual sorting methods don't detect the sorted initial parts. https://en.wikipedia.org/wiki/Timsort
Are you sorting integers? Strings? Ascii-only strings? Perhaps the code should detect some of them and run an specialized version.
It absolutely is contradictory. If you look at the development of programming languages interpreters/VMs, after a certain point, improvements in speed become a matter of more complex algorithms and data structures.
Check out garbage collectors - it's true that Golang keeps a simple one, but other languages progressively increase its sophistication - think about Java or Ruby.
Or JITs, for example, which are the latest and greatest in terms of programming languages optimization; they are complicated beasts.
Go’s GC was intentionally slow at first. They wanted to get it right THEN make it fast.
No offense but you’re not making a strong case. You’re sounding like an inexperienced coder that hasn’t yet learned that premature optimization is bad.
Of course one can't release optimal code from version one, that would be absurd.
Also your last sentence is extremely condescending.
In over 90% of my work in the SW industry, being fast(er) was of no benefit to anyone.
So no, it should not be fast by default.
Maybe better to elaborate on what it should be, if not fast? Surely you aren’t advocating things should be intentionally slow by default, or carelessly inefficient?
There’s a valid tradeoff between perf and developer time, and it’s fair to want to prioritize developer time. There’s a valid reason to not care about fast if the process is fast enough that a human doesn’t notice.
That said, depends on what your work is, but defaulting to writing faster, more efficient code might benefit a lot of people indirectly. Lower power is valuable for server code and for electricity bills and at some level for air quality in places where power isn’t renewable. Faster benefits parallel processing, it leaves more room for other processes than yours. Faster means companies and users can buy cheaper hardware.
It should satisfy the needs of the customer and it should be reasonably secure.
Everything else is a luxury.
My point was that in most of my time in the industry being faster would not have benefited my customers in any manner worth measuring.
I'm not anti-performance. I fantasize about how to make my software faster to maintain my geek credentials. But neither my bosses nor my customers pay for it.
If a customer says they want it faster we'll oblige.
Languages that tried to be all things to all people really havent done so well.
-- Donald Knuth
https://softwareengineering.stackexchange.com/a/80092
> Yet we should not pass up our opportunities in that critical 3%. A good programmer will not be lulled into complacency by such reasoning, he will be wise to look carefully at the critical code; but only after that code has been identified.
I'm confused. What do you think the context changes? At least as I read it, both the short form and full context convey the same idea.
Instead, what he meant is that you should profile what part of the code is slow and focus on it first. Knuth didn't say you should be fine with 10-1000x slower code overall.
Yes, if you quote Knuth here (whether the short quote or a longer version) you are probably responding to someone whom you believe is engaged in premature optimization.
It remains that the person quoting Knuth isn't claiming that there isn't such a thing as justified optimization. As such, pointing to the context doesn't really add to the conversation. (Nor does a thoughtless quote of Knuth either)
But either way I think the original statement is so uncontroversial and common-sense I actually think it doesn't help any argument unless you're talking to an absolutely clueless dunce or unless you're dealing with someone who somehow believes every optimization is premature.
As was said, profile, find user-impacting bottlenecks, and then optimize.
But, they aren't.
I mean, profile, identify critical components, and rewrite in C (or some other low-level language) for performance is a normal thing for most scripting languages.
> You've got to decide upfront whether the trade off in a rapid prototyping language is good or not, not wait until you've written the project and then profile it.
No, you absolutely don't.
Or to look at it another way: Python with C extensions is effectively another language. You have to consider it as an option along with Pure Python, Rust, Go, C++, Java, FORTRAN, or what have you. Each language has different trade-offs in development time vs performance.
a) gives a rule-of-thumb estimate of how much optimisation to do (maybe 3% of all opportunities);
b) explains that non-premature opimisation is not just not the root of all evil but actually a good thing to do; and
c) gives some information about how to do non-premature optimisation, by carefully identifying performance bottlenecks after the unoptimised code has been written.
I agree with GP that unless we know what Knuth meant by "premature" it is tempting to use this quote to justify too little optimisation.
* Some of the many applications relying on these pieces, could almost certainly use the speedup and for those it wouldn't be premature
* The return of investment is massive due to the scale
* There are tremendous productivity gains by increasing the performance baseline because that reduces the time people have to spend optimizing applications
And it's got commercial interests willing to throw money at performance now because of that.
Seems like the Python community, whether as top-down strategy or emergent aggregate of grassroots decisions made the right choices here.
Languages don't succeed on their technical merit. They succeed by being good enough to gain traction, after which it is more about market forces. People choose Python for it's great ecosystem and the availability of developers, and they accept the price they pay in performance. But that doesn't imply that performance wasn't an issue in the past, or that Python couldn't have been even more successful if it had been more performant.
And to be clear, I use Python every day, and I deeply appreciate the work that's been put into 3.10 and 3.11, as well as the decades prior. I'm not interested in prosecuting the decisions about priorities that were made in the past. But I do think there are lessons to be learned there.
In my experience it's far more common for "optimizations" to be technical debt than the absence of them.
> only ever think about performance in retrospect
From the extra context it pretty much does mean that. "but only after that code has been identified" - 99.999% of programmers who think they can identify performance bottlenecks other than in retrospect are wrong, IME.
It is not possible to predict bottlenecks in computation, no. But the implications of putting global state behind a mutex in a concurrent application should be clear to the programmer, and they should think seriously before making a choice like that. If you think of a different way to do that while the code is still being written, you'll avoid trapping yourself in an irreversible decision.