As I see it, being declarative and limiting depth of function calls go hand-in-hand. Also very, very closely related is locality.

It's really hard to give a good example (because it's not an easy problem), but I really want to see a high-level procedure that describes (declares) every important thing that is going on (from a business standpoint). Then ideally, I only have to dive into each function once to get enough context on the main procedure. Any further dives are library/util-type calls (again, I'm talking super ideally here).

And even though I used the term "procedure" twice there, I'm talking about either FP or OO styles.

And of course (oh boy) the context can change everything.

I agree so much with this. I prefer that all lines in a function are at the same level of abstraction w.r.t your problem domain. One line can't be about customer accounts and the very next one about the file system!

I also feel that Local Reasoning is a very powerful capability in code. In fact, Martin Odersky the creator of Scala pointed to an article that headlined Local Reasoning as one of the clearest explanation of Why FP matters (https://twitter.com/odersky/status/1271182333467602945?lang=...).

Any abstraction (here I include complex types) is a trade-off against local reasoning. Anything that makes the reader of the code drift far away from what they're currently reading on the screen is a hit against local reasoning. So they've to be introduced judiciously.

I really don't know about this, I'm writing audio & media effects in a fairly declarative style with https://github.com/celtera/avendish and I'm so much more productive that it's not even funny - I can rewrite entire effects from scratch in the time that it used to take me to find a bug somewhere

> I’ve spent way too many hours stepping through over-abstracted declarative code, getting frustrated at the opacity of it all.

Then it’s not declarative

True from the reader's perspective. The writer begged to differ!

Arguably, the writer confused "abstract" for "generic".

> The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise. (Djikstra)

That sounds akin to the OO tendency to refactor everything to the point of one executable line of code per class.

Maybe that is the juice that makes Java the 800lb enterprise gorilla that Java is, but that juice seems hardly worth the squeeze of stack traces that go on for multiple screens.

If your abstraction leaks it’s a nightmare, but if not it’s a dream. When was the last time you had to debug a problem in the Django model code itself?

All abstractions eventually leak the higher you go. It's just about whether you want to ignore it.

These leaks are usually performance related.

As someone who works on performance sensitive code with great regularity. I would note that moore's law has beaten most attempts to be smart on performance for the last ~20+ years. If code is written correctly, with correct algorithms, in a reasonably performant language - no amount of micro-optimization will beat moore's law.

Sure. How does this connect with what I mentioned? Not being snarky, but I think I'm missing something because this remark seems to be completely random.

An abstraction that happens to harm performance is rarely a long-term problem. Eventually the hardware catches up. More software has been retired due to crappy abstractions than poor performance.

So how does this contradict what I said? The leak is still there. As I said... people just choose to ignore it.

IO has not been following moores law. So you can see high level abstractions around IO leak like crazy.

And also, there is no "retiring" going on. Retiring something because of a bad abstraction almost never happens, whether it's because of performance or not.

The bad abstractions stay forever. You just don't notice it. You just master the abstraction and keep on using it.

Learning about the leaks and learning how to manipulate the high level abstractions to get around the leaks becomes an intrinsic part of the culture. It becomes so intrinsic that most people can't even wrap their mind around the flaw in the abstraction. They get defensive if you talk about it. It's invisible, but it's still there.

SQL is the most leaky abstraction I have ever seen (in terms of performance). Learning to deal with it and all it's leaks are the norm now. People don't even refer to a query planner as a leak. The technical debt is so entrenched within our culture that it's actually considered to be a great abstraction. And who knows? Maybe mastering SQL with leaks and all is not a bad way to go. But mastering SQL involves dealing with a declarative language and manipulating it in ways that leak into the underlying implementation. No amount of optimization has fixed this.

SQL isn't the only thing with this issue. For SQL the crappy abstraction is just performance related. But you see other abstractions continued to be used because simply it's too hard to retire them. Javascript, CSS, HTML, C++. Python2 (finally this has been retired). Basically it's everywhere.

Abstractions are rarely retired. Abstractions with leaky performance problems are also never retired.

sorry IO is the wrong term. Memory has not been following moores law

I have found this to be very far from the truth, with most compute-bound or memory-bound tasks getting 10x speedups from minimal work by stuffing more work into each instruction and retiring more instructions per cycle without really changing the algorithm.

Yeah but why don’t we have both?

(Actually I think 99% of micro optimisations are harmful, it’s the higher level algorithmic optimisation that gives the best gains.)

Do you mean microopromizations done by programmers or the compiler? Because if the former, I agree (especially regarding old “wisdoms” like left shifting instead of multiplying by two, when that’s the first thing any compiler will do. But it is true of even more “complex” tricks, if anything they hurt performance).

Why is simdjson faster than json parsers that do not contain any simd intrinsics if using pshufb and movemask for string parsing hurts performance?

The former, absolutely. The compiler can do whatever the heck it wants as long as the result is the same.

Some abstractions leak more than others though, and I don’t think that has that much to do with the level at which the abstraction is. ORM-mappers are very leaky and you have to regularly read what SQL they generate and sometimes even read their source code. Most programmers will never have to read assembly generated by compilers and even less compiler source code.

Yeah you don't have to read assembly. But a higher level abstraction of a non-compiled language like python. The leakage occurs and is shown through performance.

Yes as said below, that implies you have leaky abstraction.

In that case, I'm with you, I'd rather see a pile of imperative code...

Every abstraction leakes and that is not a problem at all. Does IP leak under HTTP? Sure it does. But it is still a good abstraction used everywhere. Also, see the quote commented in a sibling comment.

It's bizarre to me how many people seem to be averse to any form of abstraction when literally anything you build is already atop an unbelievably large tower of abstraction.

We should as an industry be teaching people how to build good abstractions, not avoiding them at all costs.

Isn’t the venerable spreadsheet a popular and quite working example of a declarative functional environment?

the OP should preface the abstraction bulletpoint with a caveat that the abstraction must be well formed and logical.

spreadsheets are a great abstraction on calculating values based on tabular data.

ORM is not such a great abstraction on top of SQL.

> ORM is not such a great abstraction on top of SQL.

I agree wholeheartedly with that. I'm not sure that's the direction an ORM should be targeting the abstraction in the first place. It seems more productive to abstract the code away from the point of view of the database than to abstract the database away from the point of view of the code.

or ORM should be only abstracting the slightly different syntax of different database vendor's sql implementation, not the actual mapping of objects or fields to database values.

I think the database's query results should be exposed directly, so that the developer could control the DB properly and directly (such as get access to cursors, etc)

It wouldn't really be an ORM at that point, but there's definitely a point to be made that an ORM is the wrong level of abstraction altogether when dealing with data-heavy applications. I actually really hate seeing SQL embedded in a web application serving things to customers, spread out across a bunch of routes and sometimes different route servers accessing the same database. Having a data API service for your data, or one for each major kind of data (for a store let's say customers, products, orders, etc) that talks to the DB and another layer with the application logic that reads and writes through that layer works well for performance and maintenance. It puts your SQL and your grants in a more controlled realm. The user-facing application and the data API can more easily be maintained, especially if you need a new schema or to further shard your data.