The planner may still struggle but it’s not true that a CTE is an optimization fence.
https://www.depesz.com/2019/02/19/waiting-for-postgresql-12-...
https://oracle-base.com/articles/12c/with-clause-enhancement...
The escape hatch in MSSQL for the long tail is materialising an intermediate result to temp table.
PG re-optmizes every query over and over unless you manually do a prepared statement and that only last for the session its prepared on. Therefore it's very important the optimizer not take much time before execution.
But you really should using at least query builder that does the prepared statement thing under the hood, and also because writing dynamic queries with string concatenation sucks.
That the client must explicitly prepare the statement and the preparation is tied to the connection is pretty clunky and still means the statement must be replanned for each separate connection.
Also since it is assumed by the devs of PG that prepared statements are not the norm they seem to avoid making the optimizer take its time and shoot for a fast optimizer rather than the fastest query.
DB's like MSSQL and Oracle concern themselves much less with how long it take to optimize since repetitive queries will all use the same plan automatically it also allows interesting feature like saving and loading plans from different systems and freezing them so they don't all the sudden go bad in production, but those systems also supports hints unlike PG.
TBH, the solution is to improve the Postgres engine in this regard?
For example, in Bigquery I often use CTEs and see predictable speedups that are explained by predicate pushdown when slapping a WHERE clause on the user of a CTE etc. Bigquery has plenty of performance gotchas but I don't think CTEs are one of them; they seem to be transparent to the optimiser.
Something else that Bigquery supports is user-defined 'table' functions. These are like the 'functors' the article creates.
However, in practice they aren't very widely used, and I think this hints that they aren't as useful as the article is hoping for.
The thing I _really_ want is scoping. If only functions could be described like CTE WITH blocks, e.g.
Apart from the tidiness of this, this allows inlining of functions into the SQL statement body rather than as separate statements. Often various tooling e.g. DBT makes it really hard to with separate function definitions etc, and being able to inline logic would make things possible.Bigquery also supports user-defined aggregate functions. Although there is rarely the need to stray from the standard builtin aggregate functions, when you need them they are brilliant.
(Anyone working _on_ Bigquery reading? If so, please ping me, I have loads of feedback on what works well and where the user pain and toil is ;) )