Because I have a 1 million unique users video streaming service still running python 2.7, using a few servers. The thing has a mobile version serving a different media on the fly, encodes user uploaded videos, features comments, tagging, and even has machine learning detection of content now.

It is still maintained by one single person, and he is not a professional dev.

And I'd say it's already very rare to reach that size in any project, in any company. Hell my last 3 paid projects as a freelancer have less than 20 concurrent users. And that's professional. That's people's real life.

2 out of 3 IT projects fail, no code is even shipped. I would worry about the scaling later. Once you do have the problem, pay the price for more hardware or a rewrite, you made it!

And if you do know for sure you will have the problem (you already work for a GAFAM), then not choosing python is ok, it's not a trap, it's a trade off.

But for the vast majority of us out there, having an easy, solid, battle tested and clean language with a huge ecosystem is worth a 10000 times more than some scaling potential in 5 years down the road.

I have seen so many big C++ or Java project fail I don't associate any language with big and failure anyway. I've seen devs arguing about the purity of this in Haskell, or that in Ocaml, and nothing gets to the end user because it's never perfect, or it's a toy, and doesn't handle IRL.

But what powers most companies ? Rail apps that you won't upgrade ever, but are still running. Spaghetti PHP code and wordpress that just won't dies. Horrible SAP scripts, VBA macros and excel sheets that actually deal with your real data.

Because they shipped. They did the job.

But solve the problem first, and solve it fast. It's why so many startups from the 2010's on used Ruby; they were productive in it, solving a real problem, rolling out features fast.

I've worked in a few projects where they dove onto the cargo cult, of wishing they had the kind of requests and load that e.g. a microservices architecture might help with. Massively overpriced projects too, because they hired lots of consultants and self-employed people.

The one time there was a guy who rocked up in a Maserati (used, lol); he took two weeks to build a page that was just a centered bit of text and a button, and it didn't even work.

When rewriting you already know how the project is now, so you can design your app around that, which makes things simpler and faster.

It doesn't require any extra work either. Just requires knowing what you're doing.

Congratulations.

•How many servers 2, 5, 10?

•How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale?

•How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed library are still using Python2.7.

> It is still maintained by one single person, and he is not a professional dev.

So he cannot upgrade the project to Python3.x along with all matching dependencies either, So the project stays with all bugs and vulnerabilities.

Not everyone would be happy with such a setup and aforementioned drawbacks, that's why I think the parent comment says Python is a trap at scale and I concur. I've faced issues at at scale (few hundred thousand) and there's no way a python project can meet that scale without relying upon Cython dependencies.

> A trap for your dream world were you suddenly get google size ?

So only Google size is a benchmark for scale? If I were to start a web project which has just 10 concurrent users I would choose Go because I know I can develop the entire project with little to no dependencies, it can scale to N users without touching the code and with the confidence that the number of servers required will be < than that of python.

I still use Python every day, for where it's good i.e. small scripts for personal use, prototyping algorithms. For professional products I use Python only as an Machine Learning endpoint(With all the bandages) because I have no other choice.

7

But the question is rather, whats the cost / ratio ? The 7 servers costs 7 less times money that they bring. The solo dev makes $150k/years with it, and doesn't need to work full time, have flexible hours and its own business.

> How many pure python libraries did you have to replace with a Cython extension after you realized that it cannot support your scale?

Many. Being extensible with compiled extension is a feature of Python. Numpy and co are part of the ecosystem. That's an awesome thing about it: I can code in Python and get c speed anyway.

> •How many external dependencies? How many of them did you replace after they were abandoned? Surely not all actively developed library are still using Python2.7.

Many. And the project still runs, with so little resource, bring the money in and serve its purpose, after 10 years of service.

> So he cannot upgrade the project to Python3.x along with all matching dependencies either, So the project stays with all bugs and vulnerabilities.

Sure he can, it's just an expense he won't do. Bugs are workaround, vulnerabilities are shielded by other parts of the system. Essential things like injections are dealt with. Some trade off are made, like always.

> So only Google size is a benchmark for scale? If I were to start a web project which has just 10 concurrent users I would choose Go because I know I can develop the entire project with little to no dependencies,

No you can't. Not if you want the first version to come out in a few months. Because you need a scrapper, and a bdd, and registration, and auth, and permissions, and screenshotting, and upload, and videos encoding, with background tasks, and load balancing all the videos and pics, tagging, i18n, xss protection, and search, etc. So either you pay to use cloud services to do all those things for you, or you leverage a rich ecosystem. Or you take a year to do it. Even if Go were the best language in the world, it can't beat 10 years of django plugins, 25 years of python modules and the iterative speed of a dynamic syntax. Well, it can, with AWS :)

In fact, Go is a very niche language. If you take it out of the I/O and concurrency specialty, it's quite average. Rust is better at raw perfs. At same perf, java has a better ecosystem. For simplicity and easy compilation, you have nim and zig that are more expressive. Resilience ? Erlang. And for speed of dev, python is better.

So for our use case it is not were it would shine, because:

> it can scale to N users without touching the code and with the confidence that the number of servers required will be < than that of python.

1 server only run the python code, the others are for the video hosting and the db. Python is not the bottleneck in this site. It almost never is in web dev. nginx serves the content, postgres and redis deals with the data.

There is also the stdlib. JS doesn't have simple things built in like left trim a string, csv loading, uuid generation, etc.

Then there is the API. Sorting in JS requires this weird -1/1 function. Getting the last element or truthiness to manipulate length.

Suddenly not only you have to install tons of things, but libs and frameworks can't rely on a common ground, nothing seems to integrate well. You glue a lot of things manually.

The popularity of JS is because it has a monopoly on the browser, and so a lot of people must use it, and a lot of users know it by default, and look for it elsewhere.

But when you have Ruby, Python, or even modern PHP frameworks in front of it, it's not a great deal.

Of course, not all programming tasks are about the Web. And this is where JS collapses completely. Sysadmin, data analysis, pen testing or automation using JS is a pain compared to the fluent versatility of alternatives. You know, things like connect to this ftp, download the excel files every hour, put the average of every column in the mysql db then export the thing as a PDF and send that to the boss.

The best testament to this is the reality that popular web services were able to limp along on a slow language for a long time before they had to rewrite their code.

But if one is doing something that requires raw power or precise control over resources, or precise timing then not choosing Python is not merely a tradeoff to be considered, but an obvious architecture decision.

If you're a millionaire by that time, or have a strong established business, then it doesn't matter, it's a nice problem to have. Have your engineers have a go at it.

See: Dropbox, Disqus, AirBnB, and others...

And if you're not, then going from 0->100 in 2 years instead of 0->1 in a few months with Python wasn't worth it anyway (and you also don't have enough customers/users to make use of those 100x improvement).

I wrote a few things in what I’m pretty sure was correct, idiomatic, best-practices Python 3 and almost immediately I could see some familiar problems from my Perl days looming on the horizon, especially around testing, packaging and maintainability.

I enjoyed writing Python code and for the right kind of startup or hobby project I might use it in the future, but there is so much black-boxed weirdness in the ecosystem, so much inconsistency in paradigms, and so much temptation to “cheat” that I’d definitely think twice before committing a team to it. And I didn’t even get as far as the performance issues.

On the other hand, of course, you can opportunistically “move fast and break things” within a larger org too. But I worry that if you get too committed to Python, you will be unreasonably dependent on your lead engineers sticking around.

Python might not be the most performant, but once you need the scale, it's a good problem to have because at that point, your product has already made it.

[0] https://books.google.co.uk/books?id=eulODwAAQBAJ&lpg=PA136&d...

Making good engineering choices from the beginning matters a lot.

If the choice is between a sub-optimally performant but working system, or a boondoggle that never gets off the ground, the “better engineering choice” is probably the former.

At what point do you embrace the trap and happily live with it?

For example Dropbox runs many millions of lines of Python, has massive traffic and their server costs aren't completely out of line for the service they offer. Their code base is also over 10 years old.

It seems to be working very well for them.

I talked to one of their engineers a few months ago. He gave me a complete run down of how they build and deploy Dropbox at https://runninginproduction.com/podcast/82-dropbox-gives-you....

Like many companies, they had initial scaling successes with Python,hit performance bottlenecks and looked for a way out with faster languages.

In a ton of cases you'll be completely fine serving a few million monthly page views of your SAAS app on a single $20-40 a month server using Flask, Django or whatever Python web framework you prefer. Performance will be really good too. Just talking about most apps where it's mainly reading and writing data to a DB.

I write all my backend code in Go (used Python before Go existed) because I can deploy Go executable on the cheapest render.com server and it runs using fraction of the available resources (~50 MB out of 512 MB available and literally 0.01% CPU use).

If I used Python (or Ruby, node or even Java) I would likely have to go higher (and pay more) because those languages are not only slow, they are also memory hogs.

I'm as productive in Go as I used to be in Python, I see no reason not to use Go.

As a frame of reference, ASP.NET services in C# can exceed the throughput and performance of an equivalent Django app, but Django (and Python) may be the best choice for some people. What's not considered here is the experience of the developer and whether they can avail of the platform capabilities fast. Do they need to rewrite libraries? How much work is needed to be as proficient as you are with Go? The language and platform choice is rarely a differentiator here.

Over the long-term you can see some patterns. Python companies are able to scale up their teams more quickly and cheaply in some markets than Go ones. You choose to optimize hosting cost while they optimize other costs. They may need to avail of libs like Pandas and these are readily available. I don't know what the situation is with Go today, but similar libs weren't available originally and the Go community didn't have a big foothold in data science.

If you were NOT able to do that, that would have actually been a reason to not build your app in Python.

Even 5-10 years ago, that's not a "huge success story"; that's table stakes for any serious language.

I did not mean "huge win" as in it is something unique to Python. I meant it in the sense that it greatly adds to the value proposition of the language.

it's always the same

prototype in an easy language

tighten the bolts closer to the metal after some time

... but I think you're forgetting the real motivations that drive people to use Python: it's easy to learn, it's very readable, there's a large (and not so much toxic) community behind, it's versatile. No one is saying that it will replace every other language, but it still can find its space even among large projects.

Python already does so much for you, and provides so much tooling to do even more, it's hard to write bad code if your daily job is to be a coder.

The iterator protocol means you rarely get off by one errors, context managers and GC deals with resource handling, the GIL, for all its faults, helps a lot of concurrency errors...

Now, add a good IDE, and syntax errors, names errors and attribute errors go away. Play a little in the REPL, and use cases become more obvious, behaviors more clear.

Want more security? You can add type hints and unit tests (which are incredibly easy to write, thanks to pytest).

Even without all that, the size of the ecosystem means you won't write most of the code anyway, and less code, means less potential for mistakes. Adds to that the language bare bone syntax, and you really has something that actively helps with writing good code.

Not to say you can't write bad code in Python, but unless you are not a professional dev (in that case, you can write bad code in any lang), you have a lot to help you there.

Writing bad code is easy in most statically typed languages too.

I’ve used Python for more scripting type tasks and creating basic APIs to get specific jobs done. I’m very weary to go “all in” on Python for an app based on all the feedback about performance.

All that being said I’ve never heard someone say we used X language and it scaled remarkably without issues. I’ve noticed lots of companies eventually go to Java once they get big but nobody really likes to brag about Java apps.

JavaScript (V8 at least) is extremely performant, near native code performance. Considering how dynamic it is, it's not an easy feat but Google, Apple and Mozilla work a lot on it.

Here's a benchmark where JS is 50x faster than Python: https://github.com/kostya/benchmarks

Note that PyPy does much better.

For example, here are filters for both JS and Python: https://www.techempower.com/benchmarks/#section=data-r20&hw=...

Do note that for most of the "realistic" stacks out there, you'd probably want to filter out all of the micro or no framework approaches, to compare something like Express and Koa against Django and Flask, like in the following link: https://www.techempower.com/benchmarks/#section=data-r20&hw=...

Here's the summary page with composite scores across all tests (though you might also want to filter by data storage solution, for example, MySQL): https://www.techempower.com/benchmarks/#section=data-r20&hw=...

To summarize, the performance from the best to worst currently is:

  Composite scores: Koa (JS) > Express (JS) > Flask (Python) > Django (Python)

> "more real world scenario" "realistic"

I'm of the opinion that "Real programs may not be representative either".

http://www.larcenists.org/Twobit/bmcrock.temp.html

We have to show that they are or discover they are not.

Of course, if you can, take benchmarks with a grain of salt and ideally do some prototyping and load testing.

Is "exactly" really the situation, or is it that the benchmark does something kind-of like one of the use cases?

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

For example, look at a full platform Java Spring Boot example here: https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...

The controller code, the model code and even the repository code are all very reflective of the primitives you'd find in production code.

The only differences would be when you add additional complexity, such as validations and other code into the mix, such as additional service calls etc., but if even the simpler stuff in Python (as an example) would be noticeably slower, then it stands to reason that it wouldn't get magically faster on the basis of less overhead alone.

The exceptions to this which i could imagine would be calls to native code, since the aforementioned Python has an amazing number crunching or machine learning support, but that is hardly relevant to web development in most cases.

Otherwise we risk running into the "No true Scotsman" fallacy: https://en.wikipedia.org/wiki/No_true_Scotsman

In such a case, we'd adamantly claim that these benchmarks are simply not good enough even for educated guesses and we'd have to build out our full system to benchmark it, which no one actually has the resources for - in practice you'd develop a system and then would have to do a big rewrite, as was the case with PHP and Facebook, as well as many other companies.

On the other hand, premature optimization and dogmatic beliefs are the opposite end of the spectrum: you don't need to write everything in Rust if using Ruby would get your small application out the door in a more reasonable amount of time.

There are no easy answers, everything depends on the context (e.g. small web dev shop vs an enterprise platform to be used by millions) and the concerns (e.g. resource usage and server budget vs being the first to market, as well as developer knowledge of tech stacks, for example market conditions of PHP vs Rust).

Regardless, in my eyes, there's definitely a lot of value in these attempts to compare idiomatic interpretations of common logic for web development primitives (serving network requests in a known set of formats).

> … which no one actually has the resources for…

That webpage references "JSMeter: Characterizing Real-World Behavior of JavaScript Programs".

iirc they instrumented a web browser to collect "real-world" data and demontrated that wasn't like the behavior of benchmark code.

That webpage references "A comparison of three programming languages for a full-fledged next-generation sequencing tool" — "reimplemented … in all three languages and benchmarked their runtime performance and memory use."

That's in-practice.

> … you don't need to write everything in Rust if using Ruby…

Performance doesn't matter until it matters.

As-in the home page quote and reference — "It's important to be realistic: most people don't care about program performance most of the time."

https://benchmarksgame-team.pages.debian.net/benchmarksgame/

Python basically using C types and GMP to write C in Python...

It’s not really what you’d expect for a site called computer language benchmarks game. Each example just calls out to a very fast c library (pcre2) to perform the heavy lifting regardless of which language is being “benchmarked”.

Seems a pretty pointless site.

The other examples have similar nonsense.

> "... a pretty solid study on the boredom of performance-oriented software engineers grouped by programming language."

I think that's probably the best way to describe it.

Edit to answer yours: There may be less Python programmers that are bored, or less performance-oriented Python programmers.

No.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Your python 3 one isnt the fastest python 3 solution, it’s this one which uses pcre2

The slower program is an example from that website of a regex program which does not " just calls out to a very fast c library (pcre2)…".

  from re import sub, findall

https://docs.python.org/3/library/re.html

pcre2 is not —

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

Here's some of a comment from _operator.c

"This module exports a set of functions implemented in C corresponding\n\ to the intrinsic operators of Python. For example, operator.add(x, y)\n\ is equivalent to the expression x+y."

Why wouldn't you expect that CPython regex would "ultimately" be written in C?

Did you? :-)

That Node pidigits program also uses GMP.

Besides those 2, did you notice other JS programs not 50x faster than the corresponding Python programs ?

But everyone knows it is significantly faster than Python. Probably at least a few times for pure JS vs. pure Python programs. Not that it matters, Python is basically glue to run mostly C and C++ programs.

(Java probably is also fine; I've less personal experience there so I can't say).

A lot of them have huge amounts of code that runs in browsers, so they essentially have to invest in JS whether they like it or not.

> All that being said I’ve never heard someone say we used X language and it scaled remarkably without issues.

Sometimes you have to go by what people don't say. Humans tend to talk most about problems or surprises. The former are actionable and the latter are more interesting to talk about because of their rarity. That means when things work well as expected, you get silence.

I remember when people constantly talked about how Java was too slow. Then there was a period of time where people talked a lot about how Java was fast enough because that was an interesting change. Now people don't talk about Java performance much at all, which is a good sign (but easy to overlook), that Java performance is now consistently reliably good for most users.

I think the rewriting part is only necessary if there’s some characteristic in your use case that you need that Python doesn’t provide — like raw computational speed, low latencies etc. If you’re a senior engineer familiar with Python, these trade offs are always front of mind.

For the majority of software I’ve developed, these characteristics were not essential. Python was a good choice for 80-90% of all of the important software products I’ve ever written.

For many data science projects, maintaining large code bases in Python is often the optimal decision (rather than reinventing the wheel and writing your own data frame and machine learning libraries, or using immature poorly maintained ones in other languages — for data manipulation and scientific algorithms, these libraries are highly optimized in Python anyway since the underlying code is in C or Fortran). Python is also a good choice for data engineering pipelines.

Where I might hesitate to recommend Python is when you have to write web services or desktop apps or any kind of application which requires speed or scale that cannot be handed down to a lower level library in Python.

Also for very large code bases, static typing truly helps to keep things sane, especially when you have different teams working on different parts of the codebase — with statically typed languages, no type checks in unit tests are needed, and refactoring is much more solid and error free. Python’s type annotations are an attempt to move in this direction, but static languages truly excel at type integrity (which are sometimes the cause of subtle errors in Python).

Otherwise my experience has often been that Python is a good first or second choice, depending on what you’re doing. Making that choice correctly is what sets senior/principal engineers apart from junior folks.

I wouldn't call it an attempt. I mean, sure it has no chance against for example Rust, but Python's type system is actually quite decent and I think it is more powerful than the one from Go. Also you have freedom of using both nominal and structural typing if you chose.

The only problem I have with it when I have to use a package that don't have types, but fortunately that is happening less and less.

Some package authors refuse to add types, but others provide stubs to solve that problem. For example boto3-stubs provides types for boto3.

I really like that thanks to types I can also easily refactor code without worrying about breaking something in the process. I suspect if types existed and were popular with Python 2 then the whole Python 2 -> Python 3 migration would be a non issue.

Or you could use R, instead of the half-baked immature clones in Python :) I'm only slightly joking here, even if Python is much much better for string processing, it's much less useful for DS tasks (unless it's NLP or DL, to be fair).

That being said, Python is the second best DS language, as well as second best for everything else, so it can be a good choice for these projects.

For our use case of deep learning, REST APIs, and image processing (often a mix of these), Python seems like the best choice we have. The GIL isn't a problem at all because all our computations are done with NumPy or PyTorch, which release the GIL during most of their operations.

Personally I've got a lot of mileage out of Hypothesis for property-based testing. It's good at exercising edge-cases, and works particularly well when we sprinkle assertions through a codebase (where the "property" we're testing is simply "calling Foo doesn't throw an exception").

One of the benefits of typing (that is rarely discussed) is that it deters people from writing code whose type would otherwise be crazy (e.g., "if someone passes the string 'foo' in for a parameter, then the return type is a string, otherwise it's a bytestring"). In other words, it deters a lot of crappy code. When you annotate after the fact, the annotation becomes really difficult and people get crabby that they're having to make this really complicated annotation and they blame typing (rather than their own poor coding). To your point, typing after-the-fact is still better than nothing, but you're missing out on a lot by waiting. Moreover, Python's typing story is still really immature with respect to syntax and tooling.

The technical solution is a very useful way of preventing mistakes by well-intentioned people though.

The claim is very narrow: typing can inhibit certain kinds of bad code and guide people toward better solutions. Whether someone is writing bad code because they're lazy or inexperienced, typing keeps some of the bad code from entering the code base.

Of course it's not a replacement for good management, hiring practices, etc.

I argue that somebody who would try something like your example above is either lazy or trying to wreak havoc. In my experience, people like that won't be deterred by strict typing, they'll keep typing nonsense and copying dangerous lines from StackOverflow until the thing outputs what they want. Having them write code that's complicated or important enough to require typing checks will result in disaster anyway, the only solution is to keep them out of the codebase, or review (and often rewrite) all their work.

I agree that typing is a very good guardrail for well-intentioned people with a minimum of competence, but in that case there's nothing to lose by making typing optional: they'll follow it anyway, and the added flexibility is often useful when well-used.

An inexperienced dev who wants to learn also won't need to be forced to follow typing guidelines, we just have to explain it to them...

Any engineer who would be qualified to write these kinds of guidelines would tell you not to waste time drafting and enforcing guidelines because type checkers exist. And if they're a really good engineer, they'll politely rebuke you for attributing poor code quality to the author's moral character, as this is a self-limiting outlook and generally toxic behavior.

- The typing argument is impossible to argue against, but isn't it the same problem you'd have with any other dynamic language? How is, say, Javascript any better? I don't see this as something that's specifically Python's fault, rather, yet another argument as to why starting a large project in 2021 without seriously considering static types is bordering professional malpractice.

- As for the GIL thing, I'm with you that it's never going to be lifted in any meaningful way, but wouldn't the subinterpreters idea along the lines of Ruby's ractors fix most of the pain in most cases?

Also, why are you skeptical about "the 5x plan"? I don't really have an opinion about it, but I'm interested in hearing any input.

Type inference & hinting at the IDE level is just not as developed either

I used Python and I noticed over the years that things like type interference drastically improved.

Similarly type checker in Rust extension for IntelliJ was lacking even though Rust has superior type system.

I assume you're saying this out of experience, so can you give a practical example of what you call 'notable scale'? And are you talking about desktop or web or mobile, or just all of them? Just so that we know what you are talking about in a concrete way. Not the 'large software with large number of developers is always a problem' way.

edit I also see others here using 'scale' in a 'performance / number of request per second' etc meaning - I thought we were talking codebase size though. I mean if you know on beforehand this type of performace is a requirement it seems weird to turn to Python, of all things?

The codebase is being slowly migrated to static typing. On one hand, as the parent says, the typing module is still immature and there are still some Python constructs (not too weird ones, see [1] for an example) that you can't type-check correctly. On the other hand, I like the fact that you can include typing slowly and not all at once, it makes the effort much easier to tackle. And, if typing works, it works well.

Regarding performance, well. Parallelism is pretty hard to do well, and the language itself is not the fastest thing. Some parts are migrated to a C extension but that's costly in terms of development and debugging time.

Despite all of that, I do think that Python was the best choice for our situation, and still is. Maybe from this point onwards another language would make things easier, but without Python's library support, ease and speed of development and expressiveness the cost of just getting to market would have been far higher, and probably we wouldn't have reached this point with other languages. And migrating the codebase to another language is just not worth it at all, as there are still a lot of areas we can improve quite a lot with far less effort than a full rewrite.

1: https://github.com/python/mypy/issues/2756#issuecomment-8772...

Would be cool if some experienced dev could share some estimates when the Python scaling issues start.

When you build a backend using Python + Django/FastAPI, I assume in most cases the DB and not Python is the limiting factor. Moreover, you could always spin up more workers to mitigate scaling issues.

When you train ML models, your Python code just calls C++ functions. Python is not a limiting factor here either.

For us, the scaling problems started when we had two independent teams working in the same codebase. The extreme dynamism of the language meant that classes and data structures were being mutated willy-nilly in ridiculous ways across the execution flow. The lack of static typing made onboarding new developers difficult as they had to parse generations of excessively clever code and magic left behind by departed developers. This problem has only gotten worse in Python 3, which keeps piling on more ways to accomplish the same task.

The deployment story was also awful, but I don't think that's a surprise to anyone who has deployed Python at scale.

In terms of raw performance, at one point we estimated that our Python stack was adding 3-400ms of request latency compared to a comparable system written in Go. With the Python 3 deadline coming, we convinced management to invest in rewriting performance-critical parts of the service in Go, instead of the migration to Python 3. I left before the project was completed but we were already seeing massive improvements.

Edit: code example https://gist.github.com/asemic-horizon/2830ed3637cfd278e7937...

Otherwise you’re right — whenever you have “for” loops you’re always going to end up ahead if you rewrite those in C, Rust, Julia or Fortran. And that’s exactly what authors of high performance numerical codes do.

I did my dissertation on symplectic exponential Runge-Kutta schemes with stable numerics for the big matrix exponentials needed (which have this specific structure that allow for theorems to be proven). But I didn't have time to write any code at all. I wonder if open source ODE solvers are getting good high-order symplectic methods by now...

Up to 10th order https://diffeq.sciml.ai/stable/solvers/dynamical_solve/#Symp... . Also Magnus methods https://diffeq.sciml.ai/stable/solvers/nonautonomous_linear_... and exponential integrators https://diffeq.sciml.ai/stable/solvers/split_ode_solve/#Ordi... . And the expmv implementations are highly optimized as well: https://github.com/SciML/ExponentialUtilities.jl . A lot of this specialized matrix exponential stuff is rather fun, for example see https://github.com/SciML/ExponentialUtilities.jl/pull/64

That said, matrix notation can sometimes be the more readable way of expressing a calculation.

Anyways, my point was more in response to this from the person I responded to: "my instinct is to look for opportunities to vectorize by rewriting loops into matrix notation on paper and then expressing them as array calculations". In some languages (Julia in particular) that is slower than the equivalent loop based code.

1 - https://nim-lang.org

This lets you gradually transition hot path Python modules to Nim, get compiled performance generally on par with C and Rust, whilst enjoying strong, static typing with great type inference.

In my experience (6-7 years 4 of which are full time) Nim strikes the perfect balance of the productivity you get with Python with high performance at the same time.

Also the metaprogramming features are incredible and, importantly, don't use a language subset but use the base Nim language itself.

There's also a QML wrapper here: https://github.com/filcuc/nimqml

But sometimes the productivity benefit is all that matters and you accept you may have to throw it all out later (or invest insanely in making it work)... it's all about making an informed decision.

To an outsider who knows a bit of python because of Airflow and Spark, it seems Python has become a popular metaprogramming language that various disparate ecosystems have all adapated.

Pyspark is python but kind of its own language and much of the processing is happening outside the python runtime. I think you could say the same for people doing Numpy or Pandas.

Tensorflow probably even more so where I believe Python is the most popular interface, but you're really just programming a program to run somewhere else. Again this is the same with Airflow conceptually, though I believe the runtime is python.

If my hypothesis is overall correct, and that a majority (or large share) of python programmers aren't sharing the same ecosystem, libraries and packages, then shifting to a different runtime or language that just encapsulates a subset of the language is much less challenging.

This is the opposite problem of the JVM, where no one really likes Java the langauge, so everyone tries to create enjoyable (and productive) languages for the jvm to keep using the libraries and the runtime optimizations.

But now in hindsight, I'm shocked that Python 3.0 wasn't seized as an opportunity to lose more of its baggage - the under-optimized interpreter, the poor parallelization story, the excessive surface area of the interpreter exposed to Python itself, making it impractical to re-implement in other interpreters, better typing, etc.

They broke backwards compatibility and all we got for it was Unicode?

Personally I refuse to touch our Python services with a ten-foot pole. My local app hits the testing environment and that's that.

For most python projects, that can be as simple as `pip install tox & tox`

That's certainly my impression. I'm not a professional Python programmer, though I have used it in a few projects that went to prod. I really want to like Python, and it is really nice for some quick one-offs. However, in my project, where the codebase was 30-40% Python, I estimate that 80-90% of bugs and time spent were in Python. The other side of things was more or less write once, never revisit.

This is exactly as it should be: the nature of the interpreter is that it has to do a lot of the work a compiler does when it compiles whenever a new command is run (and in Python's case, that is before you cover the whole "everything is an object" part). Yes, some of it can be mitigated, but the expectation that an interpreter keep pace with compiled code is somewhat of a pipe dream. What I'd love is a language that can be compiled (and be highly optimized) and interpreted without changing it's behavior. The developer experience with an interpreter is fantastic, the performance of compiled code is... better.

You mean something like a byte code like language with a JIT system? Like Java and C#? C# in particular now supports AOT compilation, but can still be run as “interpreted” CIL.

The VM abstraction is not a barrier to fully optimizing for your architecture, but rather how much time you can spend converting IR/bytecode into assembly, whether you do it at runtime, and whether runtime information lets you optimize even further.

This has been a great feature of many lisps for 4+ decades now. Not unique to them of course.

By the way, there are some (high-level) compiled languages with REPLs like Lisp, Haskell and Elixir.

Does elixirc compile to "native code" ?

Does the Haskell REPL use an interpreter not the GHC compiler?

http://downloads.haskell.org/~ghc/latest/docs/html/users_gui...

Anything done at significant enough scale /must/ be multi-threaded to take full advantage of modern hardware which makes languages which don't provide a reasonable threading model effectively non-starters or at best a toy language for internal prototyping. I shifted from Python to Ruby, simply because Ruby provided a realistic way to interface with real threads, and then to Go, and haven't looked back. Simple applications are massively more performant in Go than in Python, and it's not just due to typing and compilation.

I love how simple Python is to learn and how it brings more people into the fold in approaching solving complex problems, but at the end of the day you should treat it like runnable pseudocode to get shared understanding so you can implement in a real language.

PyPy is 4x faster (on benchmarks), no xp with Jython.

But when Cython fits, it's great, and a good tool to have in your toolbox if you're working with Python.

* specifically, this happened long after the tragically many-ways-to-do-it of packaging systems and virtual environments, which survive to this day.

For example, mypyc is a python to C compiler that is part of mypy. It uses types for both correctness and optimization:

https://mypyc.readthedocs.io/en/latest/using_type_annotation...

Micropython seems to make embedded code more readable and bug free than Arduino C-based programs, judging from a few projects I looked into.

Yes, it would probably be a mistake to write a large distributed, high performance database (say, for a login system handling billions of users) in python, but many, if not most, projects never hit the wall you are referring to.

In that sense, I feel that writing something in the most scalable language (if such a thing existed) would just be premature optimization at the highest level.

So I still look suspiciously to all mid to large projects what are written in Python. It takes some serious effort and good engineering to maintain good quality, and still the tooling is not there for error detection, like in other better typed languages would be. I wouldn't feel confident relying on such project for a critical infrastructure piece. The idea should have been validated quickly with Python, then implemented in a more appropriate language.

EDIT: Obviously that's just my possibly wrong opinion. There are huge projects out there proving otherwise, but the issues discussed here are _very_ real.

To use a metaphor: Python is the duplo of programming languages.

Shrug.

Typed python is safer and more maintainable than Go. Slow as a dog, but it's really a hard choice when choosing performance and a bad type system with a half baked language vs a good language with a good type system that is a bit of a drag on performance.

And you really don't have to go all in typing. I have worked on gradually typing code bases and it pays off from day one almost.