News for hackers. Stuff that matters.
Re:SQLite Archive Files (2018) by noxer on Monday September 27, 2021 @11:12AM (#28671620)
>You could probably pass the data through a gzip filter before storage

That p much voids all the benefit of using SQLite.

>And to be honest, if you need to search through or index JSON blobs in a database you need to reconsider your design.

This is absolutely intentional the log contains information to manually re-play certain events in case of an fatal error mostly for debug purpose or performance benchmarks.

This means 99.99% of the data will never be needed and eventually will be discarded.

But when I need to re-play data I need to be able to query for exactly the data I need. Each JSON "blob" (which is actually store as TEXT in SQLite) has a few additional fields to quickly filter off 95% or more of unwanted data. The rest SQLite simply reads completely to separate wanted and unwanted entries further.

With a DB size around 100GB I get the data I need by reading 5GB or less. That's perfectly fine for my use case even if I would let the DB grow quite a lot larger it would be just fine.

There are no comments on this comment yet.