News for hackers. Stuff that matters.
Re:How Postgres chooses which index to use for a query by gfody on Thursday April 21, 2022 @06:50PM (#31116148)
seems to be a matter of principle with the (some?) core devs position that it adds complexity and shouldn't be necessary. I remember the big debate long ago that resulted in postgres rejecting the standard sql merge command in favor of a proprietary syntax for upserting despite the common practice of using merge w/hints for doing upserts in most other engines - iirc it boiled down to postgres devs do not want hints, ever.
Re:How Postgres chooses which index to use for a query by petergeoghegan on Thursday April 21, 2022 @09:03PM (#31117079)
> a proprietary syntax for upserting despite the common practice of using merge w/hints for doing upserts in most other engines

I was the primary author of the feature. That decision wasn't related to hints at all. MERGE just doesn't address the problem that I wanted to address.

I think that you must be referring to Oracle's ignore_row_on_dupkey_index. Is that really in widespread use? While it is technically a hint, it is unlike any other hint it at least one important way: it changes the semantics of the query!

Re:How Postgres chooses which index to use for a query by gfody on Friday April 22, 2022 @01:08AM (#31118410)
> ignore_row_on_dupkey_index

I was thinking of the hints you used to have to use with a merge statement in order to get an atomic upsert - holdlock in sql server and lock table in oracle. My recollection of the terribly old newsgroup thread where it was (presumably you) said merge doesn't perform atomic upserts without these hints and since people were primarily wanting merge to perform atomic upserts and postgres doesn't support hints it made more sense to just make something for upserts hence the on conflict update stuff. Do I have it wrong?

Re:How Postgres chooses which index to use for a query by dspillett on Friday April 22, 2022 @03:05AM (#31119069)
> use with a merge statement in order to get an atomic upsert - holdlock in sql server

Because of all the potential edge cases that MERGE didn't handle properly when introduced to SQL Server (2008?), I've avoided its use. Despite several revisions over the years there are apparently still significant behaviours that I'd consider bugs¹, some of which have been closed as WONTFIX² so are a fact of life you just have to accept if you use MERGE in SQL Server.

[1] see https://sqlsunday.com/2021/05/04/how-merge-can-deadlock-you/ & https://michaeljswart.com/2021/08/what-to-avoid-if-you-want-... and many other similar articles

[2] https://www.mssqltips.com/sqlservertip/3074/use-caution-with...

Re:How Postgres chooses which index to use for a query by gfody on Friday April 22, 2022 @02:49PM (#31126317)
I'm in an agree-to-disagree state with various sql server consultants on this matter. I disagree that merge is harmful and should be avoided. Using merge w/hints for upserts has a decent performance advantage over the alternatives and comes with some theoretical risk of deadlocks that depend on implementation details of your use case and/or the shape of your data (I've yet to encounter it in practice, I tend to batch upserts and limit concurrency and suspect the folks having trouble do the opposite)
Re:How Postgres chooses which index to use for a query by petergeoghegan on Friday April 22, 2022 @01:44AM (#31118578)
The Oracle docs introduce MERGE as follows:

"Use the MERGE statement to select rows from one or more sources for update or insertion into a table or view. You can specify conditions to determine whether to update or insert into the target table or view.

This statement is a convenient way to combine multiple operations. It lets you avoid multiple INSERT, UPDATE, and DELETE DML statements."

MERGE doesn't claim to work as an atomic statement, and doesn't work as an atomic statement in practice. In other words you can get a unique violation error where a "true upsert" would update the row instead. You can work around this "limitation" in MERGE by retrying the statement until it succeeds, but at that point you might as well use a regular INSERT instead.

The syntax for INSERT .. ON CONFLICT constrains the problem in certain ways, which is quite deliberate. It's a pretty faithful representation of what's really going on. Hiding that creates a lot of subtle but real problems.

Re:How Postgres chooses which index to use for a query by gfody on Friday April 22, 2022 @01:00PM (#31124424)
the docs also say "every statement is atomic" and the ambiguity around whether merge is one statement or two (or three) ought to be settled by explicitly stating our atomicity expectations with a transaction. anyway my experience has been that adding the lock hints "fixes" the problem with pk violations during concurrent upserts and it seemed to be commonly known and widely used afaict. fwiw I agree with not adopting "merge w/hints" as the syntax for upsert I just thought the decision had more to do with the aversion to hints but I stand corrected (also from reading some of the links here I see that even merge w/hints has a potential deadlock if you get unlucky with the range locks it uses under the hood - I never hit that in practice myself, but it seems clear that sql should have a standard more constrained upsert so that further supports your original decision)
Re:How Postgres chooses which index to use for a query by petergeoghegan on Friday April 22, 2022 @02:48PM (#31126299)
> "every statement is atomic"

Statement-level atomicity doesn't mean that you cannot get race conditions like the one that I described - it just means that the statement succeeds or fails as a whole.

It's quite clear that you can get those race conditions in both Oracle and SQL Server's MERGE (same with the new Postgres implementation). Last I checked neither SQL Server nor Oracle address the fact that MERGE pretty much doesn't do what many of its users imagine it can do. (There is a narrow theoretical sense in which it does meet expectations provided serializable isolation level is used, but in practice that isn't worth much to users.)

> anyway my experience has been that adding the lock hints "fixes" the problem with pk violations during concurrent upserts and it seemed to be commonly known and widely used afaict

That's not my impression:

https://www.mssqltips.com/sqlservertip/3074/use-caution-with...

Re:How Postgres chooses which index to use for a query by gfody on Friday April 22, 2022 @03:04PM (#31126537)
there it is, if I had a nickel for every time that link has been thrown at me.. twice in this thread alone ;) we are now in agree-to-disagree territory because my experience has been different and I don't like to leave performance on the table no matter what the consultants say (I suspect consultants are overexposed to the kind of code written by teams who ultimately need a consultant to save them - that has never been my team)
Re:How Postgres chooses which index to use for a query by SigmundA on Thursday April 21, 2022 @07:59PM (#31116667)
Of course they are now adding merge anyway: https://www.depesz.com/2022/03/31/waiting-for-postgresql-15-...
Re:How Postgres chooses which index to use for a query by uhoh-itsmaciek on Thursday April 21, 2022 @08:39PM (#31116918)
Re:How Postgres chooses which index to use for a query by SigmundA on Thursday April 21, 2022 @08:58PM (#31117035)
Did you read the article I posted, its being added to PG 15: https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...
Re:How Postgres chooses which index to use for a query by petergeoghegan on Thursday April 21, 2022 @09:08PM (#31117109)
Did you read the text that uhoh-itsmaciek posted?
Re:How Postgres chooses which index to use for a query by nitrogen on Friday April 22, 2022 @06:27AM (#31120005)
From 2014. The MERGE discussion is from 2022.
Re:How Postgres chooses which index to use for a query by uhoh-itsmaciek on Friday April 22, 2022 @10:06AM (#31121689)
I don't see anything in the 2022 blog post that contradicts the 2014 mailing list post. These are different features with different trade-offs and somewhat overlapping use cases.
Re:How Postgres chooses which index to use for a query by SigmundA on Friday April 22, 2022 @11:03AM (#31122455)
Other systems like SQL server solve this with hints (HOLDLOCK) which is also discussed here that PG really needs hints.

So PG doesn't like hints and instead makes non standard syntax for upsert instead of standard merge syntax with a hint.

Now PG is adding merge anyway and probably could have just extended it with non standard syntax to get an atomic upsert instead it has two different approaches.

IMO it probably would have been better to go with the SQL standard MERGE with a non-standard hint syntax so its more readable between systems instead of adding non standard syntax then adding the standard much later.

Not sure if there is any movement on adding the upsert approach to the SQL standard instead.

Re:How Postgres chooses which index to use for a query by gfody on Friday April 22, 2022 @01:54PM (#31125447)
to be fair sql server's holdlock is pretty specific to sql server, there's this note on pg's merge project page (https://wiki.postgresql.org/wiki/SQL_MERGE)

> PostgreSQL doesn't have a good way to lock access to a key value that doesn't exist yet--what other databases call key range locking (SQL Server for example). Improvements to the index implementation are needed to allow this feature.

as of 2017 at least the plan was to implement merge using the underlying insert on conflict mechanism and I think everyone agrees that the only reasonable expected behavior from merge is atomic upserts (and the pk violations from concurrent merges on sql server and oracle really should be recognized as bugs!) so maybe postgres merge will be the first non-buggy implementation w/o hints. reading the related threads exposes a surprising amount of complexity though so who knows.

Re:How Postgres chooses which index to use for a query by petergeoghegan on Friday April 22, 2022 @02:56PM (#31126422)
> as of 2017 at least the plan was to implement merge using the underlying insert on conflict mechanism

That was something that was discussed at length at the time, but ultimately rejected. And so the implementation of MERGE in Postgres has essentially the same limitations as every other MERGE implementation that I'm familiar with.

Postgres clearly stresses the trade-off that MERGE makes in the docs, though. Which probably sets it apart.

Re:How Postgres chooses which index to use for a query by gfody on Friday April 22, 2022 @03:59PM (#31127292)
afaik postgres still doesn't have a merge command (just tested on 14.2)?
Re:How Postgres chooses which index to use for a query by petergeoghegan on Friday April 22, 2022 @04:15PM (#31127463)
It will be in Postgres 15, which will be out later in the year