Re:Creating a Document-Store Hybrid in Postgres 9.5
by anarazel on Friday February 05, 2016 @04:39PM
(#11044853)
Using NOT EXISTS() is a lot better than NOT IN(). The latter, as defined by the SQL standard, has confusing behaviour with NULLs. x NOT IN(y1, y2) is defined to expand to x <> y1 AND x <> y2. If any of these are NULL, the whole expression is NULL. Which means no rows match. Fun, ain't it?
Re:Creating a Document-Store Hybrid in Postgres 9.5
by empthought on Friday February 05, 2016 @04:28PM
(#11044776)
> Stick it in one big 'DELETE WHERE id NOT IN (...)' query.
I think that's a terrible idea, because it will scan the entire NOT IN (...) array for each row. Since it's NOT IN, it will even have to scan the whole thing every time. A temporary table is generally the way to go for unbounded lists of things; there's even support for it in PostgreSQL and the SQL standard.
Re:Creating a Document-Store Hybrid in Postgres 9.5
by orf on Friday February 05, 2016 @05:03PM
(#11044991)
I always assumed pg would be smart and use something with a constant lookup time. This[1] is a query plan 9.5 made with a table containing 10,000,000 sequential rows, executing "DELETE WHERE id IN (select * from generate_series(1, 1000000));". Maybe your right, it spends 5.8 seconds on the nested loop. I've only used this for smaller volumes where an extra table might be overkill, but I guess with loads more rows a temporary table is better.
Re:Creating a Document-Store Hybrid in Postgres 9.5
by empthought on Friday February 05, 2016 @05:35PM
(#11045189)
PG is doing the smart thing; it's optimizing for the common case of short IN/NOT IN literals in SQL queries. Simple arrays have much better constant factors.
If the clause is a subselect (like yours) and is also an actual table (unlike yours), it will be smart and rewrite it as a JOIN, which will generally have better complexity if the indexes are correct.
Or better yet just TRUNCATE beforehand.