News for hackers. Stuff that matters.
Re:Just use Postgres for everything by trifurcate on Saturday December 10, 2022 @01:27PM (#33935215)
I'm confused, how is matching against a hashed index faster than just matching against a string field? Surely postgres' indexing engine should treat these two things more or less the same, perhaps quietly performing the text -> hash conversion on the email field for quick lookups when it's used directly in an index (and perhaps performing even more optimizations than this basic transform)?
Re:Just use Postgres for everything by zffr on Saturday December 10, 2022 @01:48PM (#33935474)
I can confirm that indexing on a hashed value is definitely faster on SQLite at least.

The reason is that indexing on a hash produces a much smaller index so more of it can fit in memory. By minimizing disk seeks, you can speed up query time even if the Big O is the same. In both cases it should be O(log n) since SQLite uses btree indices.

Re:Just use Postgres for everything by ComputerGuru on Saturday December 10, 2022 @05:06PM (#33937396)
This is purely up to the schema designer or dbadmin. You can create a Postgres index specifying “using hash” to specify that the index will not contain the contents of the field, just its hash.

I’m pretty sure that still necessitates a hit to the db to prevent false hashes, but that’s going to be the case with your approach, too.

Re:Just use Postgres for everything by quickthrower2 on Saturday December 10, 2022 @03:33PM (#33936524)
Surely you give up LIKE queries benefiting from the index this way?
Re:Just use Postgres for everything by necovek on Sunday December 11, 2022 @08:46AM (#33943121)
Last time I needed this, LIKE queries still did sequential scans anyway, so the presence of an index or not did not matter.

Also note that you can have multiple indexes in Postgres (well, any RDBMS).

For emails in particular, you can even do partial indexes (eg. imagine filtering emails on @gmail.com to hit a separate index that's only half of your full table index). This requires some care with queries, but a wonderful feature regardless.

Re:Just use Postgres for everything by tpetry on Monday December 12, 2022 @05:54AM (#33953302)
With a trigram index in PostgreSQL you can even do index-supported leading wildcard searches.
Re:Just use Postgres for everything by CodeWriter23 on Saturday December 10, 2022 @01:43PM (#33935406)
I think GP may be referring to the use of GIN indexes https://www.postgresql.org/docs/current/textsearch-tables.ht...
Re:Just use Postgres for everything by jamal-kumar on Saturday December 10, 2022 @01:50PM (#33935507)
Yes that.

When you get over a million users...

I've yet to try the index of hashed values trick though, when I revisit this problem in the future I'll make sure to take note of this!

Re:Just use Postgres for everything by CodeWriter23 on Saturday December 10, 2022 @08:56PM (#33939217)
I've used it successfully at admittedly a small scale to enable full text search of notes-type records on a TEXT field.
Re:Just use Postgres for everything by jefftk on Saturday December 10, 2022 @03:11PM (#33936336)
I was worried they were talking about doing it manually, and was thinking "surely there's a built-in way to create a hashed index" -- Glad to learn there is!
Re:Just use Postgres for everything by necovek on Monday December 12, 2022 @01:51AM (#33951728)
A couple of obvious reasons why it's not the same: a hashed index is not useful for anything but direct equality comparison.

If you need any normalization like lowercasing, you need to do that yourself.

Next, you can't do collation/ordering using the index (and greater/less than comparisons), unless the hash function maintains ordering properties.

By having an option of an expression index customised to your data, you can use it to get the fastest query performance.