News for hackers. Stuff that matters.
Re:Do We Still Have to Denormalize? Have Your Schema and Sea... by jillesvangurp on Monday November 16, 2026 @05:50AM (#50516178)
Searching what you stored is nice. But usually, you want to put some ETL (extract, transform, and load) in between your data base and search indices that enables you to enrich, transform, and improve data. If you are using vector search, this is also where you calculate your embeddings and run your data to elaborate tokenization pipelines. And you might do things like calculate categorizations and other classifications that you might use for filtering.

Search quality tuning inevitably involves iterating on how you index your data. The value of separation of where your database and search index live is that you can try out new indexing strategies and try different strategies.

I've been consulting on how to improve search quality for some time and sorting out ETL is usually the first thing I need to do for clients. A good question is "when did you last reindex your data". If the answer is never, that usually means the clien is probably not able to do this easily and that they probably made a lot of mistakes when they set up their system first. I've helped quite a few clients like that.

ETL doesn't have to mean bringing in a lot of complexity. A few simple python or whatever scripts can do the job. And many people don't actually have a lot of data which means these scripts can run quickly. I've had clients where it only took a few seconds to rebuild their search indices.

ParadeDB seems premised on the notion of just skipping all this. IMHO this is usually a mistake and if you were to do this properly, you'd still end up with a separate db from which you index into ParadeDB. Of course, if you do that, you could question if you could not pick something else instead.

Re:Do We Still Have to Denormalize? Have Your Schema and Sea... by Veliladon on Monday November 16, 2026 @09:58AM (#50518283)
You'll spend what time you save and more trying to keep all the unnormalized data consistent.
Re:Do We Still Have to Denormalize? Have Your Schema and Sea... by philippemnoel on Monday November 16, 2026 @12:19PM (#50520060)
Agreed that people need to iterate on their search index. Our users do this a lot. Postgres has a clause called INDEX CONCURRENTLY, which allows you to build an index in the background while the existing index continues to serve queries.

As for embedding, etc., you can of course do this in Postgres as well. The "T" in ETL can still be done in a Postgres-based product like ParadeDB, but you can have it participate in transactions and avoid denormalizing (or denormalize much less), etc.

Re:Do We Still Have to Denormalize? Have Your Schema and Sea... by philippemnoel on Monday November 16, 2026 @12:33PM (#50520252)
And of course, you can also index expressions to achieve this: https://www.paradedb.com/docs/reference/indexing/indexing-ex...