News for hackers. Stuff that matters.
Re:Python: Just Write SQL by zzzeek on Monday August 14, 2023 @10:19AM (#37121322)
ORMs do much more than "write SQL". This is about 40% of the value they add.

As this argument comes up over, and over, and over, and over again, writers of the "bah ORM" club continuously thinking, well I'm not sure, that ORMs are just going to go "poof" one day? I wrote some years back the "SQL is Just As Easy as an ORM Challenge" which demonstrates maybe a few little things that ORMs do for you besides "write SQL", like persisting and loading data between classes and tables that are joined in various very common ways to represent associations between classes:

https://gist.github.com/zzzeek/5f58d007698c4a0c372edd95ab8e0...

this is why whenever someone writes one of these "just write SQL" comments, or wow here a whole blog post! wow. I just shake my head. Because this is not at all what the ORM is really getting you. Plenty of ORMs let you write raw SQL or something very close to it. The SQL is not really the point. It's about the rows and objects, moving the data from the objects to the INSERT statement, moving the data from the rows you SELECTed back to the objects. Not to mention abstraction over all the other messy things the database drivers do like dealing with datatypes and stuff like that.

It looks like in this blog post, they actually implemented their own nano-ORM that stores one row and queries one table. Well great, now scale that approach up and see how much fun it is to write the same boilerplate XYZRepository / XYZPostgresqlRepository code with the same INSERT / SELECT statement over, and over again. I'd sure want to automate all that tedium. I'd want a one-to-many collection too maybe.

You can use SQLAlchemy (which I wrote) and write all the SQL 100% yourself as literal strings, and still use the ORM, and still be using an enormous amount of automation to deal with the database drivers and moving data between your objects and rows. But why would anyone really want to, writing SQL for CRUD is really repetitive and tedious. Computer can do that for you.

Re:Python: Just Write SQL by gmassman on Monday August 14, 2023 @10:44AM (#37121615)
Thanks for you perspective, Mike. Completely agree that the interface between data and code should be handled by a single tool. That tool must meet some minimum complexity, because it’s solving a very hard problem! Also just want to say that my team has benefited greatly from your work on SQLAlchemy, and we appreciate you immensely!
Re:Python: Just Write SQL by dagmx on Monday August 14, 2023 @10:48AM (#37121656)
I completely agree with you. Every single time someone says: “Just write your own abstraction over an sql generator ”, it eventually devolves into a full blown ORM.

I swear the majority of these opinion posts against ORMs are from people who must have worked in a badly implemented project that left them with a bad experience and they blamed the pattern rather than the technology.

One of the best bits about an ORM is making it consistent for non-db users on the team to simply grok and work with, without creating monstrous and hard to debug joins everywhere. But when badly set up it can lead to a lot of debugging spaghetti. Which is the same as can happen with SQL but I suppose people think that at least there’s one less layer to debug while ignoring the problem is actually how they got where they are and not the technology

I switched a project from manually written sql to sql alchemy on a project that’s used by multiple Oscar nominated films daily for reviews. The SQL version was gross and impossible for the team to manage because it had bloated over the years, with no nice way to detangle the statement generations and joins. SQL Alchemy made it so any one of the technical directors on the team could step in and add new functionality, without serious performance footguns. Instead of me having to clean up bad sql every year (projects would fork per film and merge at the end) to keep performance up, I could trust the ORM to do that for me.

At its worst, it was way too easy for TDs to get the raw SQL to be tens of minutes per review session by structuring their logic incorrectly, but it was so difficult to see. Switching to an ORM meant I could get the performance down to seconds per session and they couldn’t destroy the performance in subtle ways.

Re:Python: Just Write SQL by joaodlf on Monday August 14, 2023 @11:22AM (#37122079)
> I swear the majority of these opinion posts against ORMs are from people who must have worked in a badly implemented project that left them with a bad experience and they blamed the pattern rather than the technology.

Fair. But you then proceed to detail a personal experience on the other side of the spectrum: Badly written code, without an ORM, and how it was fixed by introducing an ORM.

I think we can all agree that you can write bad code, with or without an ORM :). Not that this is entirely relevant to my post, I am simply advocating for writing more SQL, not how to write a good object mapper. That's a different beast, and I purposely kept that simple just to illustrate that it is possible to get started without too much pain.

Re:Python: Just Write SQL by dagmx on Monday August 14, 2023 @11:44AM (#37122366)
My point with the personal anecdote is precisely that you can write bad or good code with both. however that an ORM can allow for more consistent experience across many more people.

My point is different than your takeaway. In my point I’m not blaming the technology, I’m blaming the people involved with using it in production.

I specifically point out that a well versed engineer can write a SQL based system well. An ORM just means I can diffuse that responsibility over multiple people more reliably.

Re:Python: Just Write SQL by lelanthran on Monday August 14, 2023 @02:30PM (#37124724)
> My point with the personal anecdote is precisely that you can write bad or good code with both. however that an ORM can allow for more consistent experience across many more people.

I find it interesting that you say that; my takes is that it's the other way around!

SQL join statements look the same no matter what programming language the reader is used to, but each ORM differs in the way the join looks to the reader. The EF method of filtering your results set in C# looks very different indeed to how the ORM for Python would do it.

Every ORM looks different, which results in a very inconsistent experience for people, especially when you bring in a DB expert to figure out why there's a slowdown somewhere, and he cannot just visually inspect the EF/SQLAlchemy code and say "Well, here's what's wrong".

DB experts can usually very easily do that just by looking at the SQL, no matter what programming language was used.

Re:Python: Just Write SQL by dagmx on Monday August 14, 2023 @03:14PM (#37125477)
DB experts tend to be the minority though. It’s about indexing (pun intended) on the expert experience vs the generalist experience.

Often those may be very competing goals , where nativeness to the primary language used by the team might be more important than the nativeness of the expert.

Re:Python: Just Write SQL by aidos on Monday August 14, 2023 @10:23PM (#37129421)
To be fair, in sqlalchemy you could just do “print(query)” and you’d see your sql.
Re:Python: Just Write SQL by lelanthran on Tuesday August 15, 2023 @12:33AM (#37130235)
> To be fair, in sqlalchemy you could just do “print(query)” and you’d see your sql.

I wasn't saying that you couldn't see the raw SQL, I'm saying that the ORM syntax does not result in a consistent experience for everyone.

Re:Python: Just Write SQL by aidos on Tuesday August 15, 2023 @07:25AM (#37132569)
Sure, but if you’re working in a sqlalchemy codebase, you should really learn enough sqlalchemy to get by. I don’t jump between languages and frameworks and expect a consistent experience but ideally I’d have an experience streamlined for the context.

If you’re a db expert that’s doing drive-by optimisation of queries in sqla then you can print them to see what’s going on - is that not broadly what you’re concerned with in your original comment?

Re:Python: Just Write SQL by aforwardslash on Tuesday August 15, 2023 @05:03AM (#37131708)
> Badly written code, without an ORM, and how it was fixed by introducing an ORM.

Funny how those anecdotal bits never get published, eh? Its always "I shaved my db requirements in half by actually understanding what was going on because my orm is magic". Odd, innit?

Re:Python: Just Write SQL by waffletower on Monday August 14, 2023 @12:01PM (#37122599)
Enough people have had bad experiences with ORMs who have decided to evolve and find alternatives. ORMs are bulky and do not provide value to many modern functional, data-first development paradigms.
Re:Python: Just Write SQL by dagmx on Monday August 14, 2023 @12:29PM (#37122958)
Why speak in absolutes? Swift data for example is a data first ORM https://developer.apple.com/documentation/SwiftData

Nobody is saying only use an ORM, but the anti-ORM crowd seem to think that there’s only one true way to do things.

I don’t know why so many programmers think purely in binary “good or bad”. Different projects may have different requirements.

Re:Python: Just Write SQL by waffletower on Monday August 14, 2023 @01:14PM (#37123653)
Why hear in absolutes? I said essentially what you did: "Different projects may have different requirements"

You find ORMs useful, I do not, they aren't mutually exclusive.

Also, An ORM is inappropriate for certain forms of inter-team collaboration. Data scientists often prepare SQL that we need to integrate into our services. Asking all at a company to learn a language specific ORM isn't practical. SQL is, for better and worse, an important common tool.

Re:Python: Just Write SQL by hnfong on Monday August 14, 2023 @10:15PM (#37129364)
> I swear the majority of these opinion posts against ORMs are from people who must have ...

"microservices" - https://news.ycombinator.com/item?id=37125636

That cleared things up, at least for me. I totally agree that if your job is to essentially write a custom, RESTful, fault tolerant, 99.999% uptime accessor for a table or two, basically you're hired to be the ORM, so you don't want to just suggest people to use a ORM framework and effectively make yourself redundant.

Re:Python: Just Write SQL by joaodlf on Monday August 14, 2023 @10:56AM (#37121766)
First of all, thank you for SQLAlchemy! If I ever had to make a final choice in how I would interact with a database for a very large project that involves a considerable dev team, I would always bet on SQLAlchemy. Not that I would necessarily like all aspects of it, but when it comes to Python and SQL - “Nobody ever got fired for picking SQLAlchemy.”.

With that out of the way, despite ORMs doing much more than "just writing SQL", it is exactly on that point that I flinch: Most devs should be exposed to SQL. And if your project allows you to build around simple enough abstractions so that you aren't reinventing the wheel, you should definitely be writing SQL. Especially if you don't know SQL yet - which is the growing case of new devs coming into the job market.

You can achieve a lot with SQlAlchemy Core, a tool that I absolutely recommend, but my post is just a simple alternative to get developers to think about their approach. If that results in some devs reconsidering using "full fat" SQLAlchemy and to try SQLAlchemy Core, that's a win for me!

Your gist tries to highlight the difficulty of doing certain things without an ORM. Migrations (as just 1 example) doesn't need to be hard, simple tools like flyway, or migrate (https://github.com/golang-migrate/migrate) achieve a similar result (while also keeping you on the path of writing SQL!). Deep and complex relationships between objects also don't need to be hard - typically people approach this subject with a requirement to be very flexible in the way they want to build queries and objects, but that to me in a sign that maybe they should reconsider their business logic AND reconsider that, just maybe, their project doesn't require all that flexibility, it is fairly straightforward to extend objects and introduce some more complex representations as and when it is needed - will all of this make me write code faster? Absolutely not. That is why you have spent so much time perfecting SQLAlchemy, but then again, I am not advocating for devs to go and replace their usage of ORMs, just presenting an alternative that may or may not fit their needs for a new project + give devs the chance to learn something that the ORM might have taken away.

Re:Python: Just Write SQL by jpc0 on Monday August 14, 2023 @11:05AM (#37121868)
> and still be using an enormous amount of automation to deal with the database drivers and moving data between your objects and rows

I've found I would generally only need the handling of database drivers and query building since I'm already writing a validation layer and to add data marshalling to that is pretty trivial.

Likewise practicing YAGNI, what is the chances I need multiple database drivers for different databases, it's extremely unlikely that I'm going to be chopping and changing between different databases so I'm really only writing that code once.

I would argue to start with writing the basic SQL queries and adding an ORM later when you know you actually need it.

It's much easier to onboard someone into pure python code + a db driver vs having to onboard someone to SQLAlchemy, since it is quite a complex piece of software, necessarily complex for what it is trying to achieve but if you don't need it it's not a good fit.

Re:Python: Just Write SQL by jacurtis on Monday August 14, 2023 @11:32AM (#37122233)
> I would argue to start with writing the basic SQL queries and adding an ORM later when you know you actually need it.

I think you missed the point of the parent comment, which is that ORM's are not about writing SQL queries (although they do that). But ORMs are about moving data around, transforming it from rows and columns into meaningful objects in the project's language and data model.

As the parent comment suggested, if you are dying to write your own SQL (which does often happen as queries get more complex and don't fit into ORM language model) then you can write raw SQL but still let the ORM do the heavy lifting so you can take advantage of those features, which is the majority justification for the ORM in the first place.

You are basically suggesting what the original post author is suggesting. The comment above provided a rebuttal to their argument and you replied by suggesting the same thing they originally rebutted to. Hence a circular argument.

Re:Python: Just Write SQL by jpc0 on Monday August 14, 2023 @04:47PM (#37126613)
> But ORMs are about moving data around, transforming it from rows and columns into meaningful objects in the project's language and data model.

I made my point about this but will repeat, I am generally writing validation for this, adding the code that converts this into known types isn't significantly more work, if I was never writing the validation this point would make sense. Also the ORM the parent comment is an author of doesn't do this, it generates a class with a ton of ancillary ORM specific functions that I don't care for and many ORMs do that.

I don't even agree with what the article states, the article is effectively creating a custom ORM, I'm saying write a function that converts the data from your database driver to the type of thing you want, write a function that validates that data, those two can be 1 function for simple data but likely will be 1 function which is a composition of other functions.

When you get to the point where that is tedious you now likely have a good idea of what you want in an ORM and are significantly better equipped to make an informed decision for your specific project, you might find you end up only wanting a query builder.

Regarding abstraction over database drivers, well how often do you need an abstraction over database drivers, are you really using two or more different databases in a single project?

My ideal workflow is: 1. Query exactly what I want 2. Make it valid data in the form I want

That valid data shouldn't have a ton of extra stuff attached to it, I want plain old data that is validated and the correct type.

Why would I add a dependency that is complex before I can make an informed decision that I would even need it?

Re:Python: Just Write SQL by jddj on Monday August 14, 2023 @05:10PM (#37126835)
> My ideal workflow is: 1. Query exactly what I want 2. Make it valid data in the form I want

> That valid data shouldn't have a ton of extra stuff attached to it, I want plain old data that is validated and the correct type.

I'm in this camp too, and I really don't mind it. The database call is an API boundary, so right there is exactly where that occasionally tedious mapping + validating code is supposed to live. If it's not there then where is it?

Copilot took some of the tedium away, too.

Re:Python: Just Write SQL by aforwardslash on Tuesday August 15, 2023 @05:12AM (#37131772)
> Regarding abstraction over database drivers, well how often do you need an abstraction over database drivers, are you really using two or more different databases in a single project?

Extending on this, you actually shouldn't even use different databases. There are bugs and quirks between different databases on things like string matching :) Having a multiple database project is often a nightmare, as you need to track all of these small differences and nuances that may affect the way your system works.

Re:Python: Just Write SQL by ris on Monday August 14, 2023 @02:19PM (#37124551)
> Likewise practicing YAGNI

This is why I have come to hate YAGNI. Nowadays when it's said what I hear is "I don't understand why I need it (yet)".

Re:Python: Just Write SQL by jpc0 on Monday August 14, 2023 @04:51PM (#37126653)
An app with 3 endpoints that makes 12 SQL queries won't need it.

An app with 30 endpoints that makes 100s of queries might need it.

In this case, it's a matter of not knowing whether you need it or even what iteration of it you need. Different ORMs have different tradeoffs, you don't even know which tradeoffs you want to make.

Re:Python: Just Write SQL by tru1ock on Monday August 14, 2023 @11:06AM (#37121874)
I picked up some SQL knowledge through osmosis by using ActiveRecord and I do wish there was a better connect between what the ORM did and the end result. There were some tools to see what code generated what queries but it was not that intuitive and in your face like for example how you would have to deep dive frequently in your generated front end javascript and css code.

In other word I think there are some tooling left on the table that can assist in increasing SQL literacy and comprehension.

Re:Python: Just Write SQL by taeric on Monday August 14, 2023 @11:26AM (#37122155)
My only caveat to what you are saying, is that I have yet to see anything that successfully scales up to many tables. I can also count on one hand the number of products I have seen that successfully migrated between big databases in a meaningful way without a ton of ancillary rewriting in the process.

That said, I fully agree that the ORM isn't necessarily the problem. I point the blame at over eager data modelling that ignores the OLTP/OLAP divide. Or that ignores that row creation has a different contract than row editing, such that using the same object for both leads to confusion with missing fields. Heck, even different edits have different contracts, such that mapping a single "object" to a database row is often a mistake.

Re:Python: Just Write SQL by siva7 on Monday August 14, 2023 @11:33AM (#37122243)
badass mic drop. i will refer to your comment whenever i see this discussion coming up again and it will come up again as long as people are still learning the art of software development.
Re:Python: Just Write SQL by ansc on Monday August 14, 2023 @11:40AM (#37122329)
I’m using SQLAlchemy in my job, and have worked with Python for many years. Never have I seen a good case of someone using SQLAlchemy to hydrate objects from raw SQL queries. I’ll definitely admit — I have not gone out of my way to search for it. It seems that it is a common want to do this kind of 60% benefit ORM you speak of, but it’s definitely unclear to me how to pick those parts together with the daunting (and fantastic) piece SQLAlchemy is.
Re:Python: Just Write SQL by marcosdumay on Monday August 14, 2023 @11:55AM (#37122544)
The one distinguishing feature of an ORM is that it drops SQL and the relational paradigm and places the developer completely within the OOP world. If you drop this, you have merely a database connector library, with much more freedom of behavior than an ORM.

Yes, since they are an notoriously bad abstraction, every ORM will give you an escape hatch for the minority of tasks that it can't abstract at all. That escape hatch is not in any way a defining feature of the system.

Now, about that extra freedom that you get from dropping the requirement that your connector is an ORM... well, neither your data at rest, the presentation to the user, the abstraction for a service client, nor your validation rules benefit from OOP. Proof of that is that OOP interfaces from all of those things precede the current popular ones, and all of them were abandoned due to fundamental dissonance between the model and the data.

The rationale for an ORM is that, even though none of the interfaces you actually want for your data is best done in OOP, somehow OOP is still the best way to integrate them so that you can reuse data and metadata. This thought is not completely without merit, but there is very little empirical motivation for it, and the tiny amount that exists is more than completely explained by the OOP hype that only started to die around a decade ago.

EDIT: Oh I saw you wrote SQL Alchemy! First, thank you for that great piece of software.

Now, SQL Alchemy does provide a lot of useful ways to postpone the object representation or even to map your values into dumb data. My comment up there is on theoretical limitations, but on practice, I do think it's the best option available on Python. (And maybe about the best option afforded, since the language is intrinsically biased into OOP.)

Re:Python: Just Write SQL by BeefWellington on Monday August 14, 2023 @12:18PM (#37122830)
While I somewhat agree that a lot of these articles are people who just don't actually try/use the full feature set of ORMs, I don't agree with the overall premise you're presenting that they really do more than write SQL for you. The other things they provide are largely just abstractions around how the queried data is returned and some additional metadata tracking of the relationships. Your estimation of those parts being 60% of the value added is probably generally wrong for most users of SQLAlchemy. Not having to consider or use another language syntax while writing your code is probably closer to 70-80% of the value for most people.

Your example gist is essentially just "ORMs excel at this one thing, anything else is worthless" and also effectively hides code ("the benefit of the library!!!") to make it a very much disingenuous comparison.

> But why would anyone really want to, writing SQL for CRUD is really repetitive and tedious. Computer can do that for you.

Performance.

I wanted to love SQLAlchemy but even in some relatively simple things it generates really asinine queries that take insane amounts of time. It's just usually not noticeably insane amounts of time at first. So you query for some data and wind up with a query that returns in 20ms when there's 20 rows in the database but falls apart when there's 200,000.

It's also bad at projecting analytics type queries onto a transactionally-normalized database, but then most ORMs are not great at that.

I had both of these issues enough times that I instead just opted to start handcrafting queries in those cases and using a single "ResultSet" type class that projects as both a SimpleNamespace and a dict.

Re:Python: Just Write SQL by TX81Z on Monday August 14, 2023 @06:11PM (#37127432)
Agree, my datasets have multiple billions of rows and if I don’t know the details of the query, or have the ability to tune it, it’s utterly insufficient for my needs.

I still fail to see how anybody who actually knows sql and works with “Big Kid” datasets would use an ORM.

Re:Python: Just Write SQL by ubercore on Monday August 14, 2023 @07:15PM (#37128062)
When I've worked with datasets that size, the common operations tend towards the analytic or bulk operations, IME. That's not the sweet spot for an ORM, and SQL will often make a lot of sense. But, and again this is my experience, you often want to work with some more traditionally sized slice of that data, doing CRUDish things, and ORM can exist alongside the nicely tuned bulk or analytic operations. If you have the pure data operations side of a DB that size, I agree ORM isn't it.
Re:Python: Just Write SQL by BeefWellington on Monday August 14, 2023 @08:54PM (#37128798)
Yeah I agree here. If you're doing mostly transactional work then it can work for the 99%. As soon as you step towards analytics queries you start to see things fall apart and it's better to hand-tune your queries.

In my recent projects that's essentially what I do - I use SQLAlchemy for very basic CRUD and then anything analytical gets handcrafted.

Re:Python: Just Write SQL by TX81Z on Tuesday August 15, 2023 @12:25PM (#37135899)
Doesn’t that increase complexity? I just have a database wrapper class and keep all my db code segregated there. In my case also using an ORM would blow up that segregated approach.
Re:Python: Just Write SQL by ubercore on Wednesday August 16, 2023 @06:33AM (#37145081)
It's very very easy to keep the sql alongside models and other business logic. SqlAlchemy lets you drop down to fairly complex sql while remaining in Python, and will even take in a direct sql query as well.

The added complexity has always been a non-issue for me. Even a simple `sql.py` next to `models.py` works really well, for a pretty straightforward solution.

Re:Python: Just Write SQL by zzzeek on Monday August 14, 2023 @08:01PM (#37128411)
Would need to know what versions of SQLAlchemy you spent time with and what exactly were these "asinine" queries you refer towards. SQLAlchemy generates very good queries these days for the information its given. Years ago there was heavy use of large complicated subqueries for many cases and those days are long gone.

Also my 40% / 60% breakdown is based on the SQLAlchemy source code itself regarding what parts of the code deal with generating SQL and what parts deal with all the rest. 40% is likely a large overestimate.

Re:Python: Just Write SQL by BeefWellington on Monday August 14, 2023 @08:50PM (#37128769)
40% LOC != 40% of the value, which is what your comment stated.

My most recent experience of needing to hand-craft queries because SQLAlchemy was doing something truly hilarious was on 1.4.40-something from 2022 (I think possibly .41 or .42?).

I don't have the specifics in front of me as it was for a prior job but rough outline of what I observed:

A simple select including several related tables with filters on each table resulted in an individual subquery for each *filter* being applied. The resulting SQL was incredibly large, selected data in a poor way, and when replaced with a simplified single query that let the DB do its thing sped up about 100x.

Those days may be gone now with 2.0 but I don't consider barely a year ago "long gone".

I think there's a great place for Create/Update/Delete, and generally when doing trivial Reads. But the lesson I've learned with basically every ORM is that once a third table becomes involved you probably want to write your own SQL because they'll break in some weird way.

This is essentially how I use SQLAlchemy in current projects and it's been great in that regard.

Re:Python: Just Write SQL by zzzeek on Monday August 14, 2023 @09:47PM (#37129176)
this sounds like the select() object was being used incorrectly, using a pattern that currently emits very prominent deprecation warnings which is:

   stmt = select(...)
   stmt = stmt.where(stmt.c.foo == 'bar')
   stmt = stmt.where(stmt.c.bar == 'xyz')
That above pattern is one I've seen people do even recently, using the "select().c" attribute which from very early versions of SQLAlchemy is defined as "the columns from a subquery of the SELECT" ; this usage began raising deprecation warnings in 1.4 and should be fully removed by 2.1 as it was a remnant of a much earlier version of SQLAlchemy. it will do exactly as you say, "make a subquery for each filter condition" and will produce disastrous queries. people have started tripping over it recently as a result of 1.4 / 2.0's move away from the ORM `Query` object back towards `select()`.

the moment you see SQLAlchemy doing something you see that seems "asinine", send an example to https://github.com/sqlalchemy/sqlalchemy/discussions and I will clarify what's going on, correct the usage so that the query you have is what you expect, and quite often we will add new warnings or documentation when we see people doing things we didn't anticipate (such as in this case, I thought we had removed .c by 2.0 but apparently it's still just deprecated. it will be gone in 2.1).

Re:Python: Just Write SQL by xp84 on Monday August 14, 2023 @12:32PM (#37122997)
> writing SQL for CRUD is really repetitive and tedious

Agreed, and this is the primary reason that ORMs are a necessary tool. And possibly that it's easier to train a junior developer to use one than it is to get them to a basic level of proficiency (and security awareness!) in SQL.

That said, I think this is one of those areas where It's Complicated, because inefficient database calls due to ORM usage is one of the primary ways I see applications completely break down. For most types of apps I've seen, which are very read-heavy and which aren't doing intensive writes, roughly all your writes should probably be using the ORM (including such niceties as validations, default scopes, all that nice stuff), and if it's simple enough, your "show" actions (fetch and display one entity) may be fine as well, but every "dashboard" and "index" action (show many entities, basically things with joins) likely need to be written in SQL.

In my experience (Rails), the object instantiation cost is insane, much greater than the actual time talking to the DB, so not only do you need to write SQL, but you need to handle the data that comes back without instantiating ActiveRecord models.

This is much harder work (more specifically, it needs much greater skill and experience, and is easier to make a mess of) versus using the ORM and models, but only on apps with tiny amounts of data per request, or very low request volume, can the ORM be a serious exclusive option for this task. Unless you want to end up like an app I once was asked to help fix, where they were on the $9,000 per month Heroku postgres instance, and since that's the biggest one, they could "scale" no bigger. (Okay, this wasn't their only problem, their main one was not understanding that you don't sprinkle analytics DB writes all over the place because now the simple high-volume "read" pageviews can't be generated using just a read replica).

Re:Python: Just Write SQL by SPBS on Monday August 14, 2023 @12:48PM (#37123248)
> The SQL is not really the point. It's about the rows and objects, moving the data from the objects to the INSERT statement, moving the data from the rows you SELECTed back to the objects.

That's easy. It's a database mapping library. Write the query, give it an object and it fills it in or reads from it. You can do this with annotations, or struct tags in the Go world. There's no need to introduce abstractions over SQL joins which I find very off-putting, because the abstractions are never perfect and suddenly you have to learn invented concepts just because you didn't want to write a JOIN in an SQL query but rather have some clever framework introspect some classes and automagically write the JOINs for you.

Re:Python: Just Write SQL by hot_gril on Monday August 14, 2023 @01:01PM (#37123449)
> writing SQL for CRUD is really repetitive and tedious

If you think of relational DBs as just CRUD machines, an ORM makes total sense, but that's the original mistake.

Re:Python: Just Write SQL by audunw on Monday August 14, 2023 @02:59PM (#37125231)
Is it a mistake if that's all you need?

I'm kind of on the side of avoiding ORM unless you have a clear need for it. But I've seen projects where they are very valuable. If you mainly want to register, update and delete a bunch of structured data, it's not a bad idea to put it in a dabatase. And if you do those operations a lot in some part of the application, it's not a bad idea to use an ORM there. For analysis and reports and such I would probably suggest just writing SQL directly though.

Re:Python: Just Write SQL by hot_gril on Monday August 14, 2023 @03:39PM (#37125803)
It's hard to be sure that that's all you need. Maybe you'll be fine, maybe you'll get stuck between keeping a slow DB or rewriting everything. Doesn't cost much to go proper relational from the start. And if you're really sure you want objects, there are NoSQL solutions that are designed for that.
Re:Python: Just Write SQL by TX81Z on Monday August 14, 2023 @06:13PM (#37127451)
I think that’s part of why I’m having trouble with this framing - it’s treating CRUD as the entire universe of why you’d need to connect to a database.

Some of us do very intense compute in very large datasets, and ORM are not capable in those tasks. At all.

Re:Python: Just Write SQL by hot_gril on Monday August 14, 2023 @06:44PM (#37127699)
Doesn't even have to get very intense. A simple timeseries schema gets slow/annoying quickly with an ORM.
Re:Python: Just Write SQL by TX81Z on Tuesday August 15, 2023 @12:28PM (#37135941)
I think for a lot of things an ORM is “good” at, you could likely just have a bunch of static JSON files and retrieve them from disk with zero practical difference.

I just can’t wrap my head around why you even *need* a database in those cases.

Re:Python: Just Write SQL by hot_gril on Tuesday August 15, 2023 @02:06PM (#37137259)
You need a DB if you need to insert/update those objects transactionally, plus it's more convenient than mapping DB entries to files on disk. But you don't need an ORM for that anymore because Postgres and MySQL have json/jsonb types.
Re:Python: Just Write SQL by lelanthran on Monday August 14, 2023 @02:14PM (#37124480)
Okay, you're the expert here, and I'll happily concede that I am not (and apologise in advance if I seem to be disrespectful), but ...

> The SQL is not really the point. It's about the rows and objects, moving the data from the objects to the INSERT statement, moving the data from the rows you SELECTed back to the objects.

I think that that is the problem: mapping a relational dataset to a hierarchical dataset is the digital equivalent of pounding a square peg into a round hole.

I know you've read Ted Neward's "The Vietnam of Computer Science." (it's a short read, so search for it), and he articulated the same thoughts I had each time I had to work with an ORM in a mainstream language.

There is an impedance mismatch of sorts between relational data and hierarchical data. In cases where there is no hierarchy, you can do away with the ORM completely and turn out much easier to read code, for example with `sqlc` and `Go`.[1]

The problem with the ORM is the programming language - one in which hierarchical objects are the idiomatic way to code (all of the mainstream OO languages) is always going to require some wrangling.

> still be using an enormous amount of automation to deal with the database drivers and moving data between your objects and rows.

Unless you're using `sqlc` and Go, where the automation is provided

[1] Now you may argue that sqlc is technically an ORM, but then where do we draw the line for calling something an ORM? Is it the case that any method for "generating boilerplate for mapping relational datasets to in-program structures" is going to be called an ORM? Because to my mind, the result is so different for in-program representation that they aren't the same thing.

Re:Python: Just Write SQL by zzzeek on Monday August 14, 2023 @02:28PM (#37124699)
yah I read Neward's thing, and it was one of the main reasons I wrote SQLAlchemy in the first place, because he was just so wrong. It read like he tried to write some object relational thing and it didn't work out, so he goes off and rant rant ORMs are wrong. Kind of proving that post wrong was one of the primary goals of SQLAlchemy, really, where I sought to change the question of "impedance mismatch" and "leaky abstraction" and all that and redefined the ORM / SQL abstraction layer as *automation*, not any attempt to "hide" anything.

Yeah if your app has one table and two queries, write the SQL and marshall the data yourself. I think everyone should do that approach anyway for awhile so they know what's actually going on. But if you are writing for 1200 tables, it's just not practical. if the Go thing is also automating that and generating SQL / data marshalling boilerplate code for 1200 classes, yes that's kind of ORMish. that's a totally valid way ORMs are written and if I had multiple lives I'd probably write a stored procedure ORM that does something like that too.

Re:Python: Just Write SQL by joaodlf on Monday August 14, 2023 @03:25PM (#37125636)
I don't think it is farfetched to say that, for the good and bad, modern software development is moving away from a single project having to handle "1200 tables". As we see the growth of "services" (gasp, microservices!), the scope for codebases is reduced, hence why the pattern in my post (OP here) is so common to see in Go. Are Go developers masochists? No... But when you're working on (micro)services and your immediate work only touches 5 tables and the relationship between them, it's really not inconceivable to just reach for a database adapter and simple abstractions.
Re:Python: Just Write SQL by zzzeek on Monday August 14, 2023 @04:18PM (#37126266)
I'm going to propose, really just out of my butt so to speak, that most Go apps that use databases nonetheless are themselves middleware kinds of services that are involved with software infrastructure, as opposed to what we might call "business cases". That is, American Express might have a bunch of Go services that are deriving data from small configurationally-oriented databases all around the organization, but "the database with everyone's account information and credit card statements" is absolutely not a five table DB with a single Go application on top of it. 1200-table databases at the center of business cases will continue to exist, it's just Go applications are not themselves centralized business applications, Go is currently an infrastructure language (I googled around for this conjecture and it's obviously debated, but is still a pretty prevalent assertion I can find being made a lot). The boring business stuff is still in places like Java, Python, C#, etc.
Re:Python: Just Write SQL by hnfong on Monday August 14, 2023 @10:08PM (#37129325)
> I mean, that was really an important point in time when there really werent ORMs that were easy to work with, there was Hibernate in a very early stage and there were overly simplistic things for Perl, so I thought it was important that this "better way" I had in mind could be put out there, before the idea that "yeah let's all avoid ORMs unconditionally" could take hold.

I totally remember this time. It was around 2002~2005.

I started off with writing a discussion forum and a calendar booking webapp for my high school in PHP + MySQL. After working on it for a year or so, I realized I was basically re-typing a lot of SQL and boilerplate code, which repeated no matter whether I was working on a Post object, a User object, or a Event object. I almost wrote a half-assed ORM in PHP.

I then got an internship job, and they had a simple Java webapp project. Hibernate. In retrospect it was a huge pain, and we could probably have completed the project in 30% of the time if we used something more effective (but then, web frameworks weren't a thing in 2004), but still Hibernate was probably better than writing raw SQL for our cases.

Then later, at the same company, they had another webapp project in Perl. I don't remember the name of the ORM but it was kind of "magic" - as you said, overly simplistic, everything was implicit but seems to worked.

Shortly after I encountered Django, which seems to have solved all my problems with ORMs and I been happy with the way it works for the almost 20 years of using it.

To me the utility of ORMs is just obvious once you've been through writing CRUD without ORMs on a deadline, and tried anything that's not called Hibernate.

I'm not sure why there are so many people who apparently develops or maintains a codebase that deals with only a table or two ranting about ORMs. To me this isn't an issue with ORMs at all, but rather the curious phenomenon in the industry there is extreme specialization to the point that businesses can afford to put a person (or even a team), full time, working on a small part of the business only involving one table or two. (Can't say I'm not in this situation in my professional work today either!)

To those who say don't use ORMs -- try implementing HN in a week, without an ORM. It can probably be done either way, but with an ORM you end up with 50% less code and probably less sore fingers.

Re:Python: Just Write SQL by aforwardslash on Tuesday August 15, 2023 @05:09AM (#37131751)
> try implementing HN in a week

Not all of us work with prototypes someone will rewrite later on. Design a high traffic, low budget site with Django, and you will quickly understand the trade offs. It's not all about less coding effort; in most dev positions that matter, your work is cheap compared to the runtime cost - eg. optimizing something to take 10% less has the potential to shave multiple times your salary off the global cost pool.

Re:Python: Just Write SQL by aforwardslash on Tuesday August 15, 2023 @05:01AM (#37131696)
> But if you are writing for 1200 tables

Have you ever managed 1200 "live" (in-development) tables on a SqlAlchemy app? That is impressive. Not only because of the amount of work that entices, but also because you can just read the schema and provide a dynamic object representation of it (think dataclasses, but based on a table schema). Also, if you're querying 1200 tables, python might not be such a good idea. Impedance mismatch at a language level.

> boilerplate code for 1200 classes, yes that's kind of ORMish

Yes, please do explain how a Python ORM can save you from having 1200 classes you need to maintain, plus the need for specialized queries.

Re:Python: Just Write SQL by zzzeek on Tuesday August 15, 2023 @01:17PM (#37136616)
Not personally but we absolutely have users dealing with 1000+ table databases . SQLAlchemy includes mass reflection and dynamic mapping features and there are also extensions like SQLACodeGen that generate mappings as python source code. Additionally, its not unreasonable that applications within an organization are composed such that they refer only to a particular segment of such a database, but across the organization there is source code that refers to the database as a whole in the aggregate.
Re:Python: Just Write SQL by hintymad on Monday August 14, 2023 @02:53PM (#37125122)
Do we know what Hibernate did wrong? It used to be very popular among Java engineers but later seemed have become an obscure technology.
Re:Python: Just Write SQL by akkad33 on Monday August 14, 2023 @03:10PM (#37125427)
Do you know why? In our company we still use hibernate heavily in our java web app
Re:Python: Just Write SQL by IKantRead on Monday August 14, 2023 @03:10PM (#37125418)
To add to what you're saying: It's not like the Object-relational impedance mismatch[0] is some great unknown property of ORMs. Since people have been putting ORMs into production designers of these systems have been well aware that you must always chose a trade-off between a full functional object system and a fully relational one.

And the mismatch has two sides. If one's answer is "just use SQL!" then you're going to have new problems dealing with the mismatch coming from the SQL side of things.

The pre-ORM solution to this was not simply to shove a bunch of SQL in your application logic (though this was done), but to have a much, much more complex set of data layer logic. I'm guessing most people today writing about eschewing ORMs in favor of pure SQL have never used: stored procedures, triggers, cascades etc. I personally do miss some of the features from that era of software, but there's a lot more complexity to the "just write SQL" approach than most people realize.

0. https://en.wikipedia.org/wiki/Object%E2%80%93relational_impe...

Re:Python: Just Write SQL by ckmar on Monday August 14, 2023 @03:38PM (#37125793)
> The SQL is not really the point. It's about the rows and objects, moving the data from the objects to the INSERT statement, moving the data from the rows you SELECTed back to the objects.

If anyone is looking for this "pure" ORM (no query builder API) in Node there is https://github.com/craigmichaelmartin/pure-orm

Re:Python: Just Write SQL by habitue on Monday August 14, 2023 @04:45PM (#37126599)
Honestly, sqlalchemy is such a different breed of ORM, whenever people slander ORMs, I'm imagining they're thinking of like Django ORM or ActiveRecord which are like the duplos to SQLalchemy's Legos.
Re:Python: Just Write SQL by TX81Z on Monday August 14, 2023 @06:04PM (#37127371)
I’m sorry but 90% of the time I encounter somebody who swears by an ORM I’ll figure out the reason they use it is because they didn’t know SQL to start with and didn’t commit to learning a new language.

The number of people who know SQL well and still choose an ORM seems to be very, very low in my experience.

Re:Python: Just Write SQL by aidos on Monday August 14, 2023 @10:34PM (#37129509)
Low, maybe, but we’re out there. Sqla just saves me so much time in general. Sometimes it’s easier to drop down a level, but for 99% of my db interactions the code will be significantly shorter using sqla, so I choose to use it.
Re:Python: Just Write SQL by TX81Z on Tuesday August 15, 2023 @12:23PM (#37135875)
Fair, I always figured there were some people in that bucket, just not that many!

I think I’m also just cranky over the general trend of junior devs who know a “framework” but none of the underlying concepts or code and can’t cope when their “framework” doesn’t…work.

Re:Python: Just Write SQL by epgui on Monday August 14, 2023 @07:01PM (#37127899)
> It's about the rows and objects, moving the data from the objects to the INSERT statement, moving the data from the rows you SELECTed back to the objects. Not to mention abstraction

That's actually the problem.

I'm sure you're aware that ORMs have this fundamental problem called the ORM impedance mismatch problem. AFAIK it remains unsolved to this day.

> You can use SQLAlchemy (which I wrote) and write all the SQL 100% yourself as literal strings

This is true, but it's like saying you can disassemble a Boeing 737 to build chopsticks. It's true but completely unnecessary.

Re:Python: Just Write SQL by aforwardslash on Tuesday August 15, 2023 @04:55AM (#37131664)
> I'd sure want to automate all that tedium. I'd want a one-to-many collection too maybe.

You don't actually need an ORM for that. And some of us (I know I'm a minority) actually want to have pure data objects. Some of us (talking about myself again) actually design schema-first - as in, you use DDL and your code just consumes the definition. Some of us don't care about code knowing about relationships which are an intrinsic part of the data model.

> You can use SQLAlchemy (which I wrote) and write all the SQL 100% yourself as literal strings, and still use the ORM

SqlAlchemy is an amazing piece of software, and it is relevant becaise it is actually part of the problem. You don't really have an ecosystem of database abstraction libraries in python, as you have in other languages. The options are SqlAlchemy or Django ORM. Golang? You have 50 different libraries that implement 100 ways of doing the same stuff. C#? You have probably a dozen libraries besides Entity framework that provide different level of abstractions? Java? You got low level stuff up to high level ORMS. PHP? The same.

Python? You have SqlAlchemy, Django or roll your own. Both SqlAlchemy and Django can do raw queries with little effort - but if you're doing this, arent't you discarding most of the functionality that comes with it - namely, code maintenance? If you have a big project with a bunch handwritten queries (because you can do it, and quite easily, I may add), aren't you just ignoring any of the advantages of the orm? If, when you edit a model, you need to also check handwritten sql queries, you shouldn't be using an orm in the first place.

And with SqlAlchemy and Django, there is no middle ground. You either are all in, code-first definitions, or you are in a world of pain.

That is why specialized applications tend to be rewritten in other languages. Because optimizing the data layer alone often boasts 10-100x performance increase, when compared to this kind of abstraction.

> writing SQL for CRUD is really repetitive and tedious

It is. But between that and SqlAlchemy - in most languages, there are options. Not in Python. Between orm and no orm, I'll take no orm any day of the week - at least I can understand the queries.