I mean, what kind of performance issues were you running into?

You're not going to get maximum performance out of a python program, generally. But unless your python program is using 100% cpu that's not really an issue, and there are very few cases where I'm using python and it's CPU that's slowing down the program. Mostly it ends up being some kind of IO.

If you're looking to do complicated mathematical operations, you can call out to an external library like `numpy` of tensorflow to take better advantage of cpu/gpu compute.

You can also use a faster python interpreter, like `pypy`, although then you start getting slower startup times and more memory usage. The performance is closer to something like java but drawbacks are generally enough that they don't get used, since it's very rare that you need that extra performance.

There's also micropython, which lets you run python on a microcontroller and makes it pretty easy to optimize things for your hardware, although you lose some useful debugging and introspection features.

But for the most part, I haven't run into anything I'd call "performance issues", just the knowledge that python is a scripting language and it's never going to be the fastest thing around. Python's performance just isn't an issue very often.

In my experience most people creating CRUD websites with Django will hit a CPU bottleneck.

I've got more than a decade of on-and-off web-dev experience, a lot of integrating data-science projects with a web-interface, and I've never seen anything like that.

You have to work pretty hard to hit a CPU bottleneck, and I can't imagine how you'd do that building a simple CRUD website.

Can you explain a bit more about how the people you know are hitting that bottleneck? I mean I've literally built CRUD apps on an esp32 microcontroller using python, and they work performantly. If real python with a real web-framework can't do the same then something is going horrible wrong.

If you do a bunch of filtering/joining etc that should be done in SQL by grabbing whole tables you can hit a CPU bottleneck...

But then maybe your problems are deeper that choice of language.

That's trend that I see often. The other day I saw someone providing a response to argument that when using NoSQL database you need to plan in advance how the data is accessed (so you can pick the right key). The response was "don't you need to do that in every database?".

So many people immediately dismiss relational databases, then re-implement all that functionality in their apps with various bugs and performance bottlenecks.

ORM is another bad technique. It supposedly promises you that you don't need to know SQL to use, and that is true for the simplest examples, but you absolutely have to know it for anything less trivial, then you have to figure out how to write query in ORM to get a desired SQL statement (it makes it very difficult to use advanced SQL functionality), and that's not the end. ORM constantly will make unnecessary SQL queries and by default request all fields, even if you don't use them, adding additional performance bottleneck.

In my current workplace thanks to ORM we make on average 10 queries per request and at peak generate 1Gbps throughput from/to database because of those inefficiencies.

I think the way to go is SQL support in IDE, I recently saw PyCharm with DataGrip where if you configure it to connect to a database it will start recognizing SQL in string statements and start treating it like code (so you now have autocomplete, refactoring etc). I think this is probably the proper way to do it and I wish that other IDEs would have similar features.

> It supposedly promises you that you don't need to know SQL to use

please support this assertion with an ORM whose documentation promises this.

> ORM constantly will make unnecessary SQL queries and by default request all fields,

as does "SELECT * FROM table" if you don't write out the fields and use a buffering database adapter (which is the case for nearly all Python database adapters), so, when using an ORM, you need to give it instructions over what columns you need to fetch. This is not unusual nor even anything a library could possibly guess for you if you do not give it this intent.

> please support this assertion with an ORM whose documentation promises this.

I don't think any ORM would officially claim that, but many people decide to go that way to avoid SQL. For many simple examples it looks simpler than SQL.

I also experimented myself and wrote my code using SQLAlchemy's ORM. Then edited my code to use psycopg2 natively. I realized that SQLAlchemy didn't save any code for me, in fact it was more verbose. It also heavily encouraged me to get the raw data from database and do processing of it in the application. It also did unnecessary extra queries.

So with it now I need to:

1. still understand SQL

2. trying to figure how to write the SQL statement using ORM

3. figure out how to make ORM not to do extra queries

> as does "SELECT * FROM table" if you don't write out the fields and use a buffering database adapter (which is the case for nearly all Python database adapters), so, when using an ORM, you need to give it instructions over what columns you need to fetch. This is not unusual nor even anything a library could possibly guess for you if you do not give it this intent.

My problem is that in addition to knowing SQL, I also need to know how to do in that specific ORM that's for the specific language I'm using.

The problem is that once you start fetching specific columns in your queries, the resulting objects aren't entities, just records - i.e. it's not really object-oriented, which is the main allure of ORM.

ORMs that are more honest about this, such as SQLAlchemy, are generally better than those that try to pretend that you really are dealing with entities.

I use .values() and .values_list() to fetch raw records instead of objects. It still allows me to construct the query using the ORM which is nice.

I think that "ORM" is a bit of a misnomer for that. I'd call it "language-integrated query", if not for the potential confusion with LINQ (which stands for exactly that, not coincidentally).

There's a clear distinction between query builder and ORM though you can especially see this with SQLAlchemy.

But anyway even query builder is not that great. Recently used PyCharm's integration with DataGrip. Basically the way it works is that if you configured a database in your project and let PyCharm fetch its schema, suddenly the IDE started recognizing the SQL statements providing autocomplete not only the statements but also table names etc, it also offered refactoring which created migration scripts.

After using that I think that's the proper way of solving the impedance mismatch and at that point you no longer need ORM or query builders. I hope other IDEs will start doing the same thing.

> I think that "ORM" is a bit of a misnomer for that.

Good point: I'm no longer doing object mapping in that case. :) Though I think the most complicated part of the process that the Django "ORM" does for you is build the query. Maybe it should be called Django query generator.

As someone that knows SQL I find that the ORM enables me to have decent queries rather then getting in the way.

I don't know what kind of operations you're doing, but I started having issues for anything that's not trivial. Perhaps those are achievable, but then in addition to knowing SQL I also need to know ORM (which is also different per language)

Some things:

- to make my application scalable I want two endpoints one that is read only, that goes to replicas and one that goes to master for modification. That way in my application I have full control which queries go to replica and which need master

- use datatypes that are recommended by the database

- do upsert

- do select and only return fields that I need

- perform insert and return only fields that I want (my ORM was performing another select)

- ORM also does bunch of queries that I don't need, I don't want it to add extra load to database if I never use the results of it

Those are very simple things, but I'm also curious how you can do some advanced operations like group by having, window functions, CTEs, aggregate functions that return JSONB structure (so you don't have to send duplicate data and avoid N+1 queries). How do you use PostGIS?

> Those are very simple things, but I'm also curious how you can do some advanced operations like group by having, window functions, CTEs, aggregate functions that return JSONB structure (so you don't have to send duplicate data and avoid N+1 queries). How do you use PostGIS?

You don't! That is not what an ORM is for. Use SQL. Any decent ORM with allow you to integrate SQL.

Check out the Fluent (query builder) and Eloquent (ORM) from Laravel. They allow all the features you want.

Your experience is very much unlike my own. Every project I’ve seen hit issues with the database (e.g. unindexed queries, changing data access patterns) first, followed by I/O (especially other services like S3, search, etc.), and RAM usage before CPU became a major factor.

It’s possible, of course, but I’ve usually seen it as a symptom of not having a good culture around monitoring and troubleshooting — e.g. I remember someone porting an entire site to Jinja2 alleging performance wins, and it’s true that template rendering got (IIRC) 10-15% faster but they’d missed that 99.999% of the total runtime was being taken up by unoptimized database access generating many thousands of queries.

Of course, if you are incompetent enough to write queries that use no indexes and you don't detect that before you push to prod, you will have problems there.

What I mean is: when you start optimising, no matter how well you do, you will eventually hit a CPU performance wall. Then you will realise that only a rewrite will get you out of that hole, and by then it will be late.

In my experience, many sites never hit that point of having CPU-bound Python code which they can’t afford; the ones which do hit that wall are usually large enough that they can carve out room for a C/Rust library, microservice, etc. rather than rewriting the entire thing.

I concur, I’ve worked on at least a dozen “web scale” Django sites and I can’t think of a single one where we were CPU bound.

In my experience it doesn't sound like you have a lot of experience anywhere near CRUD websites with Django or otherwise.

just going to post my usual answer to these types of comments: if it's good enough for Instagram, it's good enough for us.

IME database always bottlenecks first.

Would Django going async ease that a bit or not really?

async doesn't decrease the amount of CPU cycles needed to do work, in any case it only increases it.

Yes and no. The Python interpreter/bytecode-VM is not as fast as it could be, because only a few people work on it. PyPy is great, but not drop in. Lack of funding is the proximate cause, of course, but that's just masking the ultimate cause of lack of interest. Because people use Python as a glue, just like PHP, or Node. To do the heavy lifting you have many options. (Even in Python, you can just use Cython, use FFI to call out to anything with A C interface, like Rust, or C++, or C. Many Python packages do exactly that, wrap a battle tested old native lib.)

That said the Python VM is a well optimized beast, it's very dynamic, you can monkey patch anything and everything at runtime on-the-fly. This is of course what makes it hard to speed up, too many things to "check for overloadedness" when doing anything. A JIT could help, but that's a lot of work. (Hence why PyPy took a decade, and the whole Python C API compatibility problem is still there - PyPy can only JIT the Py parts.)

Anyway. With that much dynamic stuff and with the VM and the language already well tuned for concurrent memory safety its 'async' story is great and performant. (Like NodeJS'.)

So Python is great for orchestrating whatever you need to handle requests. And if you want to scale up, just spin up more instances. (Just like with Java, but you don't have to think much about the overhead garbage collection, because CPython uses simple reference counting.)

Finally, my concern with Python was the lack of static typing, which made any large project hard to manage. (And resulted in a lot of boilerplate tests and excessive defensive coding.) But mypy is great, and with that it's a joy to work in Python. (Just like TypeScript made the JS world exponentially much more saner.)

> PyPy is great, but not drop in.

Nowadays it works correctly with almost all C modules. Mostly you just need to use `pypy3 -m pip install foo` instead of `python3 -m pip install foo`.

Last year I had to fight with it quite a bit to get it to work with uwsgi.

It seems that any new/greenfield Python project should target PyPy though, just in case, and the performance never hurts.

What about the type hints they introduced? I enjoyed how you can go back and add them after the fact in separate files.

It's always prioritized delivery speed and quality over raw performance. If you want an ultra optimized web framework it will always be the wrong choice - it was never designed for that.

The are some new-ish features like async and there some new parallelization options in 3.8 that help in some very specific cases, but generally you choose Python because you value development speed more than execution speed.

You can use Python (or Ruby, etc) to build fast websites that scale well if you are smart about how you use it and how you design your application, but it still isn't a particularly fast language compared to others.

Yes it does, and almost everything else will have performance issues except assembly. With web services, chances are that you'll hit other bottlenecks before you hit it. In python you have multiple ways to optimize performance, for example use PyPy, or various libraries like Cython that compile your performance sensitive code.

That's how asyncpg can outperform[1] some libraries that you normally would think was not possible.

[1] https://github.com/MagicStack/asyncpg

nothing significant changed.

Yes. If you can afford the CPU power, by all means go with Python. Otherwise just don't.

Instagram can afford it so it works fine for them. Reddit can't, so their site is always slow or down.

I think reddits issues are more related to database IO than cpu time.

You don't know what you're talking about.

And you know it is becouse of python reddit is often slow or down? Or are you just blaming pythin without any actual data on the underlying reasons?