News for hackers. Stuff that matters.
Re:JSON and Virtual Columns in SQLite by remram on Monday May 16, 2022 @01:17PM (#31400508)
If it is a cache, is it really beneficial to store that data (wasting space) rather than query it again after migration (wasting bandwidth)?

If a lot of my cached data is never used again and waiting to be replaced by e.g. LRU strategy, storing all those fields that can't even be used in the current version seems like a huge waste of space.

It seems that it would only be a good idea if most of the fields of the JSON objects are extracted and used, and most of the cached objects are used again. Which might be the case at Notion (where the cached data is probably pages) but probably not universally.

Re:JSON and Virtual Columns in SQLite by jitl on Monday May 16, 2022 @02:34PM (#31401543)
There are some object properties the clients need to query, and other object properties the clients don’t care to query, but are needed to render the data in the UI, so it still needs to be stored locally.

For example, today we run queries on "parent page ID", so that is a normalized parent_id column in SQLite, instead of being stored in the JSON column.

We don't currently run queries on "text color" or "image width" object properties, but we still need to store that data in the cache to render a block. So, it's fine to leave those things in a JSON column. If we need to run a query like "find all images that are full-width" in the future, we can add a new image_width column/virtual column/index at that time.

Re:JSON and Virtual Columns in SQLite by remram on Monday May 16, 2022 @10:25PM (#31405410)
Ah gotcha -- thanks!
Re:JSON and Virtual Columns in SQLite by conradev on Monday May 16, 2022 @02:12PM (#31401262)
You can choose whether to store the values or to compute them every time in SQLite

But in terms of the overall performance budget, for a mobile app, I don't think the choice matters much. JSON in SQLite is fast and even in the case of "storing every generated column with a lot of user data", I imagine the app binary is going to be larger than the database it produces.

Re:JSON and Virtual Columns in SQLite by remram on Monday May 16, 2022 @10:24PM (#31405405)
I'm talking about a storage space/bandwidth tradeoff here, I don't know what you are saying about compute.