Your comment states the problem space addressed by SQLAlchemy 2.0, which is what the above document is about, really well! So for your comment to have value, what did you think of SQLAlchemy 2.0's direction in how it seeks to vastly improve the very issues you speak of, "where the transactions are", "how committing works", etc.?
For example: "accidental cross joins" - SQLAlchemy 1.4 now warns/ raises for these: https://docs.sqlalchemy.org/en/14/changelog/migration_14.htm... cool huh? The new docs are worth a read before commenting.
It sounds like you were relying on flask-sqlalchemy in any case which unfortunately makes some poor decisions in these areas, but in SQLAlchemy 2.0 these things are brought forward unconditionally in all cases so you really can't talk to the database without having your transaction and its scope front and center.
The problem I'm describing isn't really the ORM libraries' fault as its outside the scope (hah). Your library is great. Most of the trouble I see is where the ORM meets the web framework. I think that gets tricky for lower experience developers because it's fundamentally tricky in Python. What the scheduler (e.g. gunicorn) does and how it forks and how sessions are handled at the wsgi layer are where you have to be very careful. Django has brought that in scope and you haven't -- which is why thats a better wholistic webdev experience and SQLAlchemy is the more powerful ORM.
With that said I stand by my comment having read the full changelog (commendably thorough btw). Unless you can afford to figure those things out as a team, it would behoove most web devs to reach for Django first.
2.0 definitely looks like a step in the right direction. Can't wait to use it with psycopg3 and the latest Python. Thanks. ;-)
That's one of my pet peeve in the Python world: people thinks Flask is for beginners and Django for advanced users. The Flask API is simple, the hello world is 5 lines. By the time you render your first Django ViewModel, you have read and tried out the full Flask doc.
So it's easy to say, "I'll start with Flask, and check out this Django thing later".
But in reality, it's exactly the opposite. Flask requires you to know a lot of things and make a lot of decisions, not to mention select/write additional libraries for features Django packs.
So if you use Flask for anything non trivial, you should show your trade or you are going to mess up, big time.
If you are a beginner, you should always start with Django. Yes, it's more work up front, but it will give you an idea of how important concepts work, plug in to each others, and can be organized in a code base.
Django is not perfect, and its ORM is really far from SQLAlachemy, but it's good enough for a loads of thing. And it will be mostly secure and extensible by default.
The upfront cost of Django makes people thing it's much more complicated than a micro-framework like Flask.
I like Flask, but not because it's a simple easy to use framework, especially once you chuck SQLAlchemy into the mix. Shit, even now if I don't touch it for a while I end up trying to remember how things work
Like an adjacent commenter said, ORM is ok for simple blog websites. But I can’t imagine using an ORM for complex data tables and relationships with normalized data.
I created a video streaming website, with 600k visitor days, that features tagging, categories, related vidéos, user upload, encoding queue, search with auto-completion and so on. It relies heavily on the ORM.
Django migration system and db model generation also make it a very nice tool for data export/migration/exploration.
The ORM provides a nice API, but above all, a standardized one, which lead the Django ecosystem to become so reach because of the assumptions the libraries could do, making you very productive.
Now, one has to remember:
- you can, and should by pass the ORM if you need it, writing raw SQL. Django let you do that.
- plenty of things can end up be in a cache, using varnish, redis, etc. Plus a lot of things should be done in task queues anyway, not in the django views. So the cost of the ORM is rarely an issue.
- complex data tables are sometimes better presented using SQL views, making the ORM model simple.
As usual with coding, it's a matter of gain and cost, and the right answer is "it depends". But you can go very far, and very comfortably with this tools, as long as you don't fight against its nature.
So yes, definitely use Django with the ORM. I am not advocating to use Django + SQLA, but rather contrast Django with Flask + SQLA.
Whenever I read a statement like this from someone, I always feel that they actually haven't thoroughly read the docs for Django's ORM. Can you give me one concrete complex example that you can't imagine doing with the ORM? Because 95% of the time, these "complex" models are trivially easy to pull off with the ORM (from my experience).
Not only it makes creating an API from ORM models quick and easy (although the documentation doesn't let you believe so), but it also let you do custom views seaminglessly, including goodies such as standardized permissions, authentication and pagination with several levels of granularity. Having that being able to be integrated with all the rest of the Django ecosystem transparently (your website, a wagtail CRM, etc.) is really sweet.
But the biggest advantage for a beginner is again that it will prevent them from having to make decisions they are likely to get wrong. Especially on the serialization, format, identifier policy and homogeneity of the API: people crafting that by hand often make a big mess and don't even know about it.
I caution people against using Django because the ORM makes so many weak, simplified assumptions about how a database works that don't bear out in practice. It's fine for a blog. It falls apart when you're working with Real Data.
Whereas I've used Sqlalchemy and getting it to do a basic join, turns out there's 3 different ways of doing it all with an insane amount of crap to find the right way of doing things. I actually left my old job in part due to Sqlalchemy being too painful to work with.
The doc is very formal, and if you want to make a website, a quick script or a data migration, you are left on your own to find the proper setup.
Being more hands on with sqlalchemy is probably the right approach.
I always found Flask-SQLAlchemy a strange approach because it ties opposite and somewhat orthogonal parts of the request lifecycle together. When requests-to-data-transactions is a 1-1 relationship it's a convenience, but once you step out of that I think it's better to do things by hand.
My main app has a simple layer structure of "request > business logic > data layer". All of the business logic functions start a SQLAlchemy transaction using a Python decorator that is ~10 lines of SQLAlchemy code.
https://github.com/Pylons/pyramid-cookiecutter-starter/blob/...
I think same approach should work well for flask - just tie the session to the request object on first access.
Side note: I think we met on IRC few years ago ;-)
Just curious, but how many employees? How many specifically are responsible for writing code that interacts with SQLAlchemy and/or the objects it creates. While I don't doubt it is _possible_ to make SQLAlchemy work great, I think it's mandatory to overstaff with engineers/programmers in order to do so.
I don't think this blame is as unfair as you imply, even if if were the primary cause of ORM induced performance disasters. In my experience the main selling point of ORMs has always been the promise for people who fancy themselves programmers to leverage DB technology without having to sully themselves or their cherished OO model by learning the relational model, databases or SQL. The other, equally false promise has been that the ORM would somehow allow you to "abstract" over your concrete DB so you could easily swap out one for another. Both these ideas now seem utterly misguided to me (although many years ago I probably fell for the same nonsense). For all of SQL's warts, I find the relational model conceptually much superior to OO. Moreover since the way data moves in and out of the DB (and the consistency/transactional and performance constraints around this) is often more architecturally central than the python code that orchestrates it, what ORMs are trying to do, shoe-horning the well thought out relational model into the badly thought out OO model seems utterly backwards to me -- spanning the cart before the horse so to speak. Now I will readily admit that SQLAlechemy is about the least bad ORM I have encountered, but I still absolutely fail to see what the point of it's ORM part is (I can see a bit of the appeal of being able to dynamically construct or compose queries with core, although I'd try hard to avoid the necessity).
Clearly you know what you are doing with hundreds of millions of users and you are not the sole large user of SQLAlchemy either, so I am evidently missing something. Can you maybe give one or two examples of things that are much easier/robust/... because of SQLAlchemy, compared to just writing SQL, in separate files, and executing that from you python code?
To be clear, there are various things that I think are broken about how databases work, and I'd really like to see them fixed but I don't see how SQLAlchemy helps much with most of the below (with the notable exception of query parameterization).
- no high level support for migrations: it should be possible to get a readable aggregate schema defintion out for example and it isn't
- parameterization and composition of queries is awkward and limited
- (for most DBs) lack of in-process support
- bad testing framework support
I am, and I'm warning them, whats wrong with that?
That said, I really like SQLAlchemy.
But I need SQLAlchemy to really work with ... SQL! And it is solid at that. The glue libraries between bigger ones like SQLAlchemy and Flask usually are less maintained and have less eyeballs on them. These are weaker points IMHO, not SQLAlchemy.
People cargo cult the flask intro tutorial and have no clue how session binding works, where the transactions are, how committing works, etc because it’s all tucked away in magical middleware and singletons. As the code base grows so does the mess of blocking txns, accidental cross joins, pool exhaustion, and so on.
It’s a great tool from a technical expressiveness perspective but terribly full of operational foot guns. Beware and use Django until you’re sure you AND your team know what you’re doing.