I am seeing what the GP comment says in a codebase I manage, around 47k LOC. Not massive at all, but enough where the problems mentioned above start to pop up. This is an application that runs in servers to analyze traffic data, so it has both analysis code and also a lot of code for the analysis framework. It gets hard to manage. I have unit tests and also integration tests that cover some, but not all of the code paths (it's very hard in this case to have everything covered). Most of the times, when they fail is due to something that would have been caught by static typing.

The codebase is being slowly migrated to static typing. On one hand, as the parent says, the typing module is still immature and there are still some Python constructs (not too weird ones, see [1] for an example) that you can't type-check correctly. On the other hand, I like the fact that you can include typing slowly and not all at once, it makes the effort much easier to tackle. And, if typing works, it works well.

Regarding performance, well. Parallelism is pretty hard to do well, and the language itself is not the fastest thing. Some parts are migrated to a C extension but that's costly in terms of development and debugging time.

Despite all of that, I do think that Python was the best choice for our situation, and still is. Maybe from this point onwards another language would make things easier, but without Python's library support, ease and speed of development and expressiveness the cost of just getting to market would have been far higher, and probably we wouldn't have reached this point with other languages. And migrating the codebase to another language is just not worth it at all, as there are still a lot of areas we can improve quite a lot with far less effort than a full rewrite.

1: https://github.com/python/mypy/issues/2756#issuecomment-8772...

+1

Would be cool if some experienced dev could share some estimates when the Python scaling issues start.

When you build a backend using Python + Django/FastAPI, I assume in most cases the DB and not Python is the limiting factor. Moreover, you could always spin up more workers to mitigate scaling issues.

When you train ML models, your Python code just calls C++ functions. Python is not a limiting factor here either.

I worked for a startup that built their services using Python and Twisted. The codebase was a monolith broken up into several Twisted services, by the time I left it was probably 200k LOC of Python.

For us, the scaling problems started when we had two independent teams working in the same codebase. The extreme dynamism of the language meant that classes and data structures were being mutated willy-nilly in ridiculous ways across the execution flow. The lack of static typing made onboarding new developers difficult as they had to parse generations of excessively clever code and magic left behind by departed developers. This problem has only gotten worse in Python 3, which keeps piling on more ways to accomplish the same task.

The deployment story was also awful, but I don't think that's a surprise to anyone who has deployed Python at scale.

In terms of raw performance, at one point we estimated that our Python stack was adding 3-400ms of request latency compared to a comparable system written in Go. With the Python 3 deadline coming, we convinced management to invest in rewriting performance-critical parts of the service in Go, instead of the migration to Python 3. I left before the project was completed but we were already seeing massive improvements.